> ## Content Index
> Fetch the complete content index at: https://blog.sudowl.com/llms.txt
> Use this file to discover other available public pages before exploring further.

# Linux Disk Full: Finding What's Eating Your Space
- URL: https://blog.sudowl.com/linux-disk-full-forensics/
- Published: 2026-04-06T10:00:00.000Z
- Updated: 2026-10-11T08:47:37.000Z
- Description: Disk full on Linux: df, du, ncdu, deleted-but-open files, inodes, journald, Docker and snap leftovers, and how to free space safely.
- Author: Lukas
- Tags: Linux, Debian, Ubuntu

"No space left on device" usually shows up as something else first: a database refusing writes, a service that won't start, `apt` dying halfway, a game server that stopped saving. When it happens I don't start deleting things. I go through the same short list, in the same order, because the culprit is almost always one of maybe six things.

**TL;DR:** `df -h` and `df -i` to see *what* is full, `du -xh -d1 / | sort -h` (or `ncdu -x /`) to find *where*, `lsof +L1` for deleted files still held open, then check the usual suspects: journald, Docker, old snaps, apt cache, runaway logs. Truncate live log files, don't `rm` them.

```bash
df -hT                                   # which filesystem is full
df -i                                    # or is it inodes?
sudo du -xh -d1 / 2>/dev/null | sort -h  # biggest top-level dirs, this fs only
sudo lsof +L1                            # deleted but still open
journalctl --disk-usage
docker system df                         # if Docker is installed

```

## Step 1: what exactly is full

```bash
df -hT

```

`-T` shows the filesystem type, which saves a wrong turn later. Look at `Use%` for each mount. It's not always `/`. A full `/boot` breaks kernel updates. A full `/var` on a separate partition kills logging and databases while `/` looks fine. A full `/tmp` as tmpfs is RAM, not disk.

Then check inodes:

```bash
df -i

```

If `IUse%` is at 100% but `df -h` shows free space, you're out of inodes, not bytes — millions of tiny files somewhere. Different hunt, see below.

## Step 2: where is it

```bash
sudo du -xh -d1 / 2>/dev/null | sort -h

```

- `-x` stays on one filesystem, so it doesn't wander into `/proc`, other mounts or network shares.
- `-d1` is one level deep. Then drill down: `sudo du -xh -d1 /var | sort -h`, then `/var/lib`, and so on.

That's fine on any box, but if I can install something, I use `ncdu`:

```bash
sudo apt install ncdu
sudo ncdu -x /

```

It scans once and lets you browse with arrow keys, sorted by size. `d` deletes — careful, that's real. It's simply faster than re-running `du` five levels deep.

For "just show me big files":

```bash
sudo find / -xdev -type f -size +500M -exec ls -lh {} + 2>/dev/null

```

## Step 3: df and du disagree — deleted files still open

The most confusing case: `df` says 100% full, `du` adds up to way less. Almost always a file was deleted while a process still has it open. The directory entry is gone, so `du` doesn't count it, but the blocks aren't freed until the last file handle closes.

```bash
sudo lsof +L1

```

`+L1` lists open files with a link count below 1, i.e. deleted. You'll see something like a log file with `(deleted)` behind it and a `SIZE/OFF` in gigabytes. Typical cause: someone `rm`'d a huge log while the service was writing to it.

Fix either by restarting the process holding it, or — when you can't restart right now — truncate it through `/proc`:

```bash
# PID and FD from the lsof output (FD column "4w" means fd 4)
sudo truncate -s 0 /proc/1234/fd/4

```

Space comes back immediately, process keeps running.

## Gotcha: never `rm` a log that's being written to

That's how you *get* the situation above. If a live log is huge, empty it in place:

```bash
sudo truncate -s 0 /var/log/myapp/app.log
# or
sudo sh -c ': > /var/log/myapp/app.log'

```

The file stays, the inode stays, the process keeps writing to it, and the space is freed. Then go fix the actual problem, which is missing rotation — see [logrotate](/logrotate-linux-log-rotation/).

## The usual suspects

### journald

```bash
journalctl --disk-usage
sudo journalctl --vacuum-size=500M
# or by age
sudo journalctl --vacuum-time=14d

```

Vacuuming only removes archived journal files, not the active one, so it may free less than you expect. To make the limit permanent:

```ini
# /etc/systemd/journald.conf.d/size.conf
[Journal]
SystemMaxUse=500M

```

```bash
sudo systemctl restart systemd-journald

```

### Docker

```bash
docker system df          # images, containers, volumes, build cache
docker system df -v       # per item
sudo du -sh /var/lib/docker/containers/*/*-json.log 2>/dev/null | sort -h | tail

```

The last line is the one people miss: container logs with the default `json-file` driver and no size limit. Fix that in `daemon.json` as shown in [installing Docker](/docker-install-ubuntu/). Cleanup:

```bash
docker image prune -a      # unused images
docker builder prune       # build cache — often the biggest chunk on build hosts
docker container prune     # stopped containers

```

I deliberately don't run `docker system prune --volumes` on reflex. `docker volume prune` removes volumes that no container references right now — and a stopped-and-removed compose stack's database volume is exactly that. Check `docker volume ls` first.

### Old snap revisions (Ubuntu)

Snap keeps old revisions of every snap around, disabled:

```bash
snap list --all | awk '/disabled/{print $1, $3}' |
  while read -r name rev; do sudo snap remove "$name" --revision="$rev"; done

sudo snap set system refresh.retain=2

```

### apt

```bash
sudo apt clean          # downloaded .deb files in /var/cache/apt/archives
sudo apt autoremove     # old kernels and orphaned deps

```

On a small `/boot`, `autoremove` is usually what fixes "no space" during kernel upgrades.

### Logs that never rotated

```bash
sudo du -xh -d1 /var/log | sort -h
sudo find /var/log -type f -size +100M -exec ls -lh {} +

```

A single multi-GB `.log` file means rotation is missing or broken for that app. Truncate it (see above), then write a logrotate config for it.

### Backups and dumps pointing at the wrong place

Database dumps, panel backups, tarballs from "I'll just quickly make a copy". Search for archives:

```bash
sudo find / -xdev -type f \( -name '*.tar*' -o -name '*.sql*' -o -name '*.zip' -o -name '*.dump' \) -size +200M -exec ls -lh {} + 2>/dev/null

```

Backups that live on the same disk as the data they back up aren't backups anyway. Move them off the box — [rsync backups](/rsync-backup-linux/) for the simple version.

## Out of inodes

Bytes free, inodes gone. Find which directory has the most files:

```bash
sudo du --inodes -x -d1 / 2>/dev/null | sort -n | tail

```

Drill down the same way as with `du -h`. Typical results: a PHP session directory nobody cleans, a mail queue full of undeliverable bounces, a cache directory with millions of small files, or an app writing one file per request. Deleting millions of files with `rm *` fails with "argument list too long"; use find:

```bash
sudo find /var/lib/php/sessions -type f -mmin +1440 -delete

```

## Gotcha: files hidden under a mount point

Rare, but nasty: something wrote to `/data` *before* the actual disk was mounted there (or while it was unmounted). Those files sit on the root filesystem, underneath the mount, invisible to `du`. Symptom: `/` is fuller than everything `du` can find. Bind-mount root somewhere to look underneath:

```bash
sudo mkdir -p /mnt/rootfs
sudo mount --bind / /mnt/rootfs
sudo du -xh -d1 /mnt/rootfs/data
sudo umount /mnt/rootfs

```

## Gotcha: the 5% root reserve

ext4 reserves 5% of blocks for root by default. That's why a non-root process sees "disk full" while `df` shows some space still available, and why root can still log in and clean up. On a big data-only volume, 5% is a lot of wasted space and you can lower it:

```bash
sudo tune2fs -l /dev/sdb1 | grep -i 'reserved block count'
sudo tune2fs -m 1 /dev/sdb1

```

I leave it alone on the root filesystem. That reserve is what keeps the system usable enough to fix itself.

## Afterwards

Freeing space is the easy part. The real fix is whatever let it fill: missing log rotation, no Docker log limits, backups on the wrong disk. And next time you'd like to hear about it at 85%, not at 100% — any monitoring with a disk alert does that; I've compared a few options in [Grafana vs Netdata vs Uptime Kuma](/grafana-vs-netdata-vs-uptime-kuma/).

---

### Related posts

- [Run Your Own Daemon with systemd — Writing Service Unit Files](https://blog.diengdoh.com/systemd-service-unit-file/?ref=blog.sudowl.com)
- [UFW Firewall Rules That Don't Lock You Out](https://blog.diengdoh.com/ufw-firewall-rules-ubuntu/?ref=blog.sudowl.com)
- [Linux Log Rotation with logrotate — Before Your Disk Fills Up](https://blog.diengdoh.com/logrotate-linux-log-rotation/?ref=blog.sudowl.com)