No space left on device but df shows free space
Fix "No space left on device" with free space: run df -i for inodes and lsof +L1 for deleted open files. Covers wrong filesystems and finding the space with du.
cp: cannot create regular file '/var/log/app.log': No space left on device
Incident Overview
Fix "No space left on device" with free space: run df -i for inodes and lsof +L1 for deleted open files. Covers wrong filesystems and finding the space with du.
medium fix·6 min read
By Vishvesh Patel·DevOps Engineer·Updated 21 September 2026
cp: cannot create regular file '/var/log/app.log': No space left on device OSError: [Errno 28] No space left on device bash: cannot create temp file for here-document: No space left on device
Do this first3 steps
Run these in order. Each one tells you what its output means before you change anything.
• Check inodes, not just blocks
A filesystem at 100% in the IUse% column of df -i is out of inodes and cannot create another file regardless of free space. Millions of tiny files, typically sessions or cache entries, are the usual cause.
• Find deleted files that a process is still holding open
A log deleted with rm while the writer still has it open frees no space until that process closes the descriptor or restarts. The seventh column is the size, so the biggest offenders sort to the top.
• Confirm you are looking at the right filesystem
A path can sit on its own mount. A full /var while / has plenty free is invisible if you only run df -h and read the first line.
ENOSPC when df -h shows free space means the free space is not the thing that ran out. Three candidates, in order of frequency.
df -h && df -i
1. Inodes
Every file consumes an inode, and the count is fixed when the filesystem is created. Run out and you cannot create another file, however many gigabytes are free.
IUse% 100 is the answer. Find where they went:
-xdev keeps it on one filesystem, which stops it wandering into /proc and network mounts.
The usual sources are PHP sessions in /var/lib/php/sessions, a mail spool, a cache directory with millions of tiny entries, or a container runtime's layer store.
Deleting many files at once can be slow and I/O heavy. find -delete is better than rm with a huge argument list, which fails with Argument list too long.
You cannot add inodes to an ext4 filesystem after creation. If this recurs, either fix what is creating the files or recreate the filesystem with mkfs.ext4 -i 8192 for a higher density. XFS allocates inodes dynamically and does not have this limit, which is one reason it is a better default for anything file-heavy.
df -i
2. A deleted file still held open
rm on a file removes the directory entry. The space is only reclaimed when the last file descriptor closes. A log rotated by deletion while the writer still holds it open frees nothing.
NLINK 0 with a live process is exactly this: 40GB held by a file nobody can see.
Three ways out, in increasing order of disruption:
The /proc trick works because the descriptor is still a path into the file. It frees the space without stopping anything, and the process carries on writing at its previous offset, which leaves a sparse file that is harmless in practice.
The real fix is logrotate with copytruncate or a proper reopen signal, rather than anything deleting an open log:
sudo lsof +L1
3. You are looking at the wrong filesystem
A separate /var can be full while / has plenty. df -h alone shows every mount, and it is easy to read the first line and stop.
Docker is the usual culprit on a build host:
--volumes deletes named volumes not attached to a container, which can be real data. Run it without the flag first and read the reclaim estimate.
df -h /var/lib/docker /var/log /tmp /
Two things that look like this and are not
Reserved blocks. ext4 reserves 5% for root by default. A filesystem at "100%" for an ordinary user still has that reserve, so root can write and nobody else can.
On a large data volume, 5% is a lot of wasted space, and 1% is ample.
A full /tmp on tmpfs. /tmp is often RAM-backed and sized as a fraction of memory, so it can fill while the disk is empty:
sudo tune2fs -l /dev/nvme0n1p1 | grep -i 'reserved block count' sudo tune2fs -m 1 /dev/nvme0n1p1
Finding the space
-x stays on one filesystem. ncdu -x / is much nicer if you can install it.
Note that du reports what files use and df reports what the filesystem has allocated. A large gap between them is the deleted-file case above.
sudo du -xh --max-depth=1 / 2>/dev/null | sort -rh | head -15
A checklist
Frequently Asked Questions
Why does df show free space when I cannot create a file?
Because blocks are not the only finite resource. The filesystem may be out of inodes, which df -i reports separately and which cap the number of files regardless of their size. Or a process is holding a deleted file open, so the space is allocated but invisible to du and unreclaimable until the descriptor closes. Or the path you are writing to is on a different mount from the one you checked. Run df -i and lsof +L1 before anything else.
What are inodes and why do they run out?
An inode stores a file's metadata, and every file and directory needs exactly one. On ext4 the total is fixed when the filesystem is created, derived from its size on the assumption of average-sized files. Millions of tiny files break that assumption, so a filesystem with plenty of free bytes can have no inodes left. Typical causes are PHP session files, mail spools and cache directories. You cannot add inodes to an existing ext4 filesystem; XFS allocates them dynamically and does not have the limit.
How do I free space from a deleted file that is still open?
Get the process and descriptor from sudo lsof +L1, then either ask the process to reopen its files, with systemctl reload or a signal such as nginx's SIGUSR1, or truncate it in place with sudo truncate -s 0 /proc/<pid>/fd/<fd>. The truncate reclaims the space immediately without stopping the service; the process continues writing at its old offset, producing a sparse file that is harmless. Restarting the service also works and is the most disruptive option.
Why can root write when my user cannot?
ext4 reserves a percentage of the filesystem for the superuser, 5% by default. The reservation exists so that root can still log in and system daemons can still write when a filesystem fills, rather than the machine becoming unrecoverable. It means unprivileged writes fail at what looks like 100% while root has headroom. On a large data volume 5% is a substantial amount to hold back, and lowering it with tune2fs -m 1 is reasonable.
Why do du and df disagree about disk usage?
du walks the directory tree and adds up the files it can see, while df reports what the filesystem has allocated. Files that have been deleted but are still held open by a process appear in df and not in du, because they no longer have a directory entry. A large discrepancy between the two is therefore a strong signal to run lsof +L1. Sparse files and filesystem overhead produce smaller differences that are normal.
Reference and practice
Learn the underlying concept
Other Linux errors
Free, comprehensive DevOps tutorials and learning roadmaps. Master Docker, Kubernetes, CI/CD, and more.
Get notified about new tutorials and features.
• Roadmaps
• Interview Prep
• Certifications
• Company
© 2026 DevOpsLesson. All rights reserved.
Need to practice this scenario?
Simulate this error condition and practice recovery commands safely in our in-browser terminal.