I've seen more production outages from full disks than from almost any other infrastructure failure. A web server runs out of space, logs stop writing, databases lock up, and suddenly you're explaining to a client why their checkout page returned a 500 for two hours.
The fix is never just running df -h when things break. You need automated monitoring, the right tools for different scenarios, and a baseline understanding of what normal looks like for your workload.
Why df alone isn't enough
Most engineers reach for df -h first. It's fast, it's built-in, and it shows partition usage at a glance:
df -h
But df only tells you how full each filesystem is right now. It won't show you what is filling the disk, which directory tree is growing, or whether inodes are exhausted even though you still have free space. I've debugged servers where df reported 40% usage but the partition was functionally full because every inode was allocated.
Check inode usage separately:
df -i
If you see 100% inode usage, you have too many small files—often session data, cache fragments, or mail queue files. The solution is different from a space problem.
What you should actually use
du for directory-level investigation
When a partition fills up, du locates the culprit. Start at the root of the problem filesystem:
du -sh /* 2>/dev/null | sort -rh | head -n 10
This shows the ten largest top-level directories in human-readable sizes, sorted descending. The 2>/dev/null hides permission errors you'll hit scanning /proc or /sys.
Once you identify the big directory, drill down:
du -sh /var/* 2>/dev/null | sort -rh | head -n 10
In my experience, /var/log and /var/lib/mysql account for most unexpected growth on hosting servers. Sometimes it's a single 80GB error log from a misconfigured PHP app.
ncdu for interactive exploration
When you need to navigate the filesystem visually, ncdu beats du by miles. Install it first:
# Debian/Ubuntu
apt install ncdu
# RHEL/Rocky/AlmaLinux
dnf install ncdu
Then run it on the partition:
ncdu /var
You get a navigable tree view sorted by size. Arrow keys move around, d deletes files directly (careful), and q quits. It's especially useful when you're hunting through nested subdirectories and need to see context.
Patterns that catch you off guard
Deleted files still holding space
A process can keep a file descriptor open even after the file is deleted. The space won't free until the process closes the handle or restarts. This trips up people constantly.
Find them with lsof:
lsof +L1
The +L1 flag shows files with a link count less than 1—deleted but still open. You'll see the process ID and the file path (marked as (deleted)).
Restart the offending service, or kill the process if it's safe. The space returns immediately.
Log rotation misconfiguration
Logrotate is supposed to keep /var/log under control. But if a service holds its log file open and logrotate only renames the file without signaling the process to reopen, logs continue writing to the renamed (often compressed) file and break everything.
Check your logrotate config for the relevant service:
cat /etc/logrotate.d/nginx
You want a postrotate block that reloads the daemon:
/var/log/nginx/*.log {
daily
missingok
rotate 14
compress
delaycompress
notifempty
postrotate
systemctl reload nginx > /dev/null 2>&1 || true
endscript
}
The delaycompress is key—it waits one rotation cycle before compressing, giving the service time to finish writing to the old file.
Temporary files in /tmp or /var/tmp
Applications create temporary files and crash before cleanup. Over months, /tmp accumulates gigabytes. Modern distros usually mount /tmp as tmpfs (RAM-backed), but /var/tmp persists across reboots and follows the same filesystem as /var.
Set up a daily cron job to purge old temp files:
find /var/tmp -type f -atime +7 -delete
This deletes files not accessed in seven days. Adjust the threshold to match your workload.
How to automate monitoring
Manual checks don't scale. What works is threshold-based alerting tied to your existing monitoring stack.
Disk usage alerts in Prometheus
If you run Prometheus and node_exporter, add an alert rule for filesystem usage:
groups:
- name: disk
interval: 60s
rules:
- alert: DiskSpaceWarning
expr: (
node_filesystem_avail_bytes{mountpoint="/"}
/ node_filesystem_size_bytes{mountpoint="/"}
) < 0.2
for: 5m
labels:
severity: warning
annotations:
summary: "Disk space below 20% on {{ $labels.instance }}"
This triggers when the root filesystem drops below 20% free for five consecutive minutes. Adjust the mountpoint and threshold for each critical partition.
Simple shell script for smaller setups
No Prometheus? A cron job with a bash script does the job:
#!/bin/bash
THRESHOLD=85
EMAIL="[email protected]"
df -H | grep -vE '^Filesystem|tmpfs|cdrom' | awk '{ print $5 " " $1 }' | while read output;
do
usage=$(echo $output | awk '{ print $1}' | sed 's/%//g')
partition=$(echo $output | awk '{ print $2 }')
if [ $usage -ge $THRESHOLD ]; then
echo "ALERT: $partition is ${usage}% full" | mail -s "Disk Alert: $(hostname)" $EMAIL
fi
done
Drop it in /etc/cron.hourly/disk-check and make it executable. You'll get an email when any partition crosses 85%.
Anti-patterns that make things worse
Running du on the entire root filesystem in production
Running du -sh / during peak hours hammers the disk I/O and slows everything else down. The filesystem cache gets evicted, database queries slow, and users notice.
Always scope du to a specific subtree. If you must scan everything, use ionice to lower the priority:
ionice -c3 du -sh /var
Class 3 is idle priority—it only runs when no other I/O is waiting.
Ignoring inode exhaustion
You can have 50GB free and still be unable to create files. Check inodes first when df looks fine but writes fail with "No space left on device."
Maildir setups and session storage are common causes. Each email or session is a separate file. Thousands of tiny files eat inodes fast.
Not monitoring growth rate
Knowing a partition is 60% full today means nothing without context. Is it growing 1% per day or 10%?
Track usage over time. Most monitoring systems graph this automatically, but even a weekly cron job logging df output to a CSV gives you trend data:
date +"%Y-%m-%d %H:%M" >> /var/log/disk-usage.log
df -h >> /var/log/disk-usage.log
Plot it monthly. You'll see seasonal patterns (backup schedules, log volume spikes) and catch runaway growth before it becomes an emergency.
What happens when you're already at 100%
Panic doesn't help. Here's the checklist.
-
Identify the largest files immediately:
bash find / -type f -size +100M -exec ls -lh {} \; 2>/dev/nullThis lists every file over 100MB. You'll spot the log file or database dump that exploded overnight. -
Check for deleted-but-open files:
bash lsof +L1 | grep deletedRestart the services holding those descriptors. -
Clear old logs manually: ```bash
/var/log/huge-logfile.log
`` The>` truncates the file in place without unlinking it, so open file descriptors still work. -
Remove old package caches: ```bash # Debian/Ubuntu apt-get clean
# RHEL/Rocky dnf clean all ```
- Purge old kernels (if you're on a system that doesn't auto-clean):
bash # Debian/Ubuntu apt autoremove --purge
Once you have breathing room, figure out what caused the spike and fix the root cause—usually log rotation, backup retention, or a runaway process.
What to check first
When disk issues hit production, speed matters. Check df -h and df -i together—you need both space and inode counts. Run lsof +L1 to catch deleted files still holding space, then use du or ncdu to locate the largest directories.
Automate this before the next outage. A five-minute monitoring setup beats an hour of emergency firefighting at 2 AM. Set alerts at 80%, watch growth trends weekly, and keep log rotation configs tight.
The tools are simple. What separates working systems from broken ones is using them before you have to.