Linux Disk Full: Finding What's Eating Your Space
Disk full on Linux: df, du, ncdu, deleted-but-open files, inodes, journald, Docker and snap leftovers, and how to free space safely.
"No space left on device" usually shows up as something else first: a database refusing writes, a service that won't start, apt dying halfway, a game server that stopped saving. When it happens I don't start deleting things. I go through the same short list, in the same order, because the culprit is almost always one of maybe six things.
TL;DR: df -h and df -i to see what is full, du -xh -d1 / | sort -h (or ncdu -x /) to find where, lsof +L1 for deleted files still held open, then check the usual suspects: journald, Docker, old snaps, apt cache, runaway logs. Truncate live log files, don't rm them.
df -hT # which filesystem is full
df -i # or is it inodes?
sudo du -xh -d1 / 2>/dev/null | sort -h # biggest top-level dirs, this fs only
sudo lsof +L1 # deleted but still open
journalctl --disk-usage
docker system df # if Docker is installed
Step 1: what exactly is full
df -hT
-T shows the filesystem type, which saves a wrong turn later. Look at Use% for each mount. It's not always /. A full /boot breaks kernel updates. A full /var on a separate partition kills logging and databases while / looks fine. A full /tmp as tmpfs is RAM, not disk.
Then check inodes:
df -i
If IUse% is at 100% but df -h shows free space, you're out of inodes, not bytes — millions of tiny files somewhere. Different hunt, see below.
Step 2: where is it
sudo du -xh -d1 / 2>/dev/null | sort -h
-xstays on one filesystem, so it doesn't wander into/proc, other mounts or network shares.-d1is one level deep. Then drill down:sudo du -xh -d1 /var | sort -h, then/var/lib, and so on.
That's fine on any box, but if I can install something, I use ncdu:
sudo apt install ncdu
sudo ncdu -x /
It scans once and lets you browse with arrow keys, sorted by size. d deletes — careful, that's real. It's simply faster than re-running du five levels deep.
For "just show me big files":
sudo find / -xdev -type f -size +500M -exec ls -lh {} + 2>/dev/null
Step 3: df and du disagree — deleted files still open
The most confusing case: df says 100% full, du adds up to way less. Almost always a file was deleted while a process still has it open. The directory entry is gone, so du doesn't count it, but the blocks aren't freed until the last file handle closes.
sudo lsof +L1
+L1 lists open files with a link count below 1, i.e. deleted. You'll see something like a log file with (deleted) behind it and a SIZE/OFF in gigabytes. Typical cause: someone rm'd a huge log while the service was writing to it.
Fix either by restarting the process holding it, or — when you can't restart right now — truncate it through /proc:
# PID and FD from the lsof output (FD column "4w" means fd 4)
sudo truncate -s 0 /proc/1234/fd/4
Space comes back immediately, process keeps running.
Gotcha: never rm a log that's being written to
That's how you get the situation above. If a live log is huge, empty it in place:
sudo truncate -s 0 /var/log/myapp/app.log
# or
sudo sh -c ': > /var/log/myapp/app.log'
The file stays, the inode stays, the process keeps writing to it, and the space is freed. Then go fix the actual problem, which is missing rotation — see logrotate.
The usual suspects
journald
journalctl --disk-usage
sudo journalctl --vacuum-size=500M
# or by age
sudo journalctl --vacuum-time=14d
Vacuuming only removes archived journal files, not the active one, so it may free less than you expect. To make the limit permanent:
# /etc/systemd/journald.conf.d/size.conf
[Journal]
SystemMaxUse=500M
sudo systemctl restart systemd-journald
Docker
docker system df # images, containers, volumes, build cache
docker system df -v # per item
sudo du -sh /var/lib/docker/containers/*/*-json.log 2>/dev/null | sort -h | tail
The last line is the one people miss: container logs with the default json-file driver and no size limit. Fix that in daemon.json as shown in installing Docker. Cleanup:
docker image prune -a # unused images
docker builder prune # build cache — often the biggest chunk on build hosts
docker container prune # stopped containers
I deliberately don't run docker system prune --volumes on reflex. docker volume prune removes volumes that no container references right now — and a stopped-and-removed compose stack's database volume is exactly that. Check docker volume ls first.
Old snap revisions (Ubuntu)
Snap keeps old revisions of every snap around, disabled:
snap list --all | awk '/disabled/{print $1, $3}' |
while read -r name rev; do sudo snap remove "$name" --revision="$rev"; done
sudo snap set system refresh.retain=2
apt
sudo apt clean # downloaded .deb files in /var/cache/apt/archives
sudo apt autoremove # old kernels and orphaned deps
On a small /boot, autoremove is usually what fixes "no space" during kernel upgrades.
Logs that never rotated
sudo du -xh -d1 /var/log | sort -h
sudo find /var/log -type f -size +100M -exec ls -lh {} +
A single multi-GB .log file means rotation is missing or broken for that app. Truncate it (see above), then write a logrotate config for it.
Backups and dumps pointing at the wrong place
Database dumps, panel backups, tarballs from "I'll just quickly make a copy". Search for archives:
sudo find / -xdev -type f \( -name '*.tar*' -o -name '*.sql*' -o -name '*.zip' -o -name '*.dump' \) -size +200M -exec ls -lh {} + 2>/dev/null
Backups that live on the same disk as the data they back up aren't backups anyway. Move them off the box — rsync backups for the simple version.
Out of inodes
Bytes free, inodes gone. Find which directory has the most files:
sudo du --inodes -x -d1 / 2>/dev/null | sort -n | tail
Drill down the same way as with du -h. Typical results: a PHP session directory nobody cleans, a mail queue full of undeliverable bounces, a cache directory with millions of small files, or an app writing one file per request. Deleting millions of files with rm * fails with "argument list too long"; use find:
sudo find /var/lib/php/sessions -type f -mmin +1440 -delete
Gotcha: files hidden under a mount point
Rare, but nasty: something wrote to /data before the actual disk was mounted there (or while it was unmounted). Those files sit on the root filesystem, underneath the mount, invisible to du. Symptom: / is fuller than everything du can find. Bind-mount root somewhere to look underneath:
sudo mkdir -p /mnt/rootfs
sudo mount --bind / /mnt/rootfs
sudo du -xh -d1 /mnt/rootfs/data
sudo umount /mnt/rootfs
Gotcha: the 5% root reserve
ext4 reserves 5% of blocks for root by default. That's why a non-root process sees "disk full" while df shows some space still available, and why root can still log in and clean up. On a big data-only volume, 5% is a lot of wasted space and you can lower it:
sudo tune2fs -l /dev/sdb1 | grep -i 'reserved block count'
sudo tune2fs -m 1 /dev/sdb1
I leave it alone on the root filesystem. That reserve is what keeps the system usable enough to fix itself.
Afterwards
Freeing space is the easy part. The real fix is whatever let it fill: missing log rotation, no Docker log limits, backups on the wrong disk. And next time you'd like to hear about it at 85%, not at 100% — any monitoring with a disk alert does that; I've compared a few options in Grafana vs Netdata vs Uptime Kuma.
