Inode Exhaustion: Disk Full with Space Available

What Are Inodes?

An inode is a data structure that stores metadata about every file and directory on your filesystem (name, size, owner, permissions, etc.). Even if you have gigabytes of free disk space, if you run out of inodes, you cannot create new files.

This is different from disk space, you can have 50% disk usage but 100% inode usage, and still get "No space left on device" errors.

Identify the Problem

The Symptom

touch /tmp/testfile
# touch: cannot touch '/tmp/testfile': No space left on device

But checking disk space shows plenty available:

df -h
# Filesystem      Size  Used Avail Use% Mounted on
# /dev/vda1        50G  20G   25G  44% /

Check Inode Usage

df -i
# Filesystem     Inodes  IUsed IFree IUse% Mounted on
# /dev/vda1     3200000 3197432 2568  100% /

If IUse% is 100% (or very high like 99%), you've exhausted inodes.

Find What's Consuming Inodes

The most reliable method to find directories with the most files:

find / -xdev -printf '%h\n' | sort | uniq -c | sort -k 1 -rn | head -20

This command:

• Searches from root without crossing filesystem boundaries (-xdev)
• Counts files by parent directory
• Shows top 20 directories

Common Inode Culprits

1. PHP Session Files

Location: /var/lib/php/sessions

Check:

ls -la /var/lib/php/sessions | wc -l
find /var/lib/php/sessions -type f | wc -l

Clean old sessions (older than 7 days):

find /var/lib/php/sessions -type f -mtime +7 -delete

Enable automatic cleanup in PHP config:

Edit /etc/php/8.1/fpm/pool.d/www.conf (or your pool):

php_admin_value[session.save_path] = "/var/lib/php/sessions"

Or set cron job to clean daily:

0 2 * * * find /var/lib/php/sessions -type f -mtime +7 -delete

2. Mail Spool

Locations: /var/mail, /var/spool/mail, /var/spool/postfix

Check Postfix queue:

postqueue -p

Clear old messages:

postsuper -d ALL

Check mail spool:

du -sh /var/spool/postfix/*
ls -lh /var/mail/

3. Log Files and Rotations Gone Wrong

du -sh /var/log
ls -lhS /var/log | head -20

If log rotation is broken, old logs accumulate. Compress or delete:

gzip /var/log/apache2/access.log
# or delete entirely
rm /var/log/apache2/access.log.1 /var/log/apache2/access.log.2 /var/log/apache2/access.log.3

4. Temp Files and Cache

# /tmp cleanup
find /tmp -type f -atime +30 -delete

# /var/tmp cleanup
find /var/tmp -type f -atime +30 -delete

# Application cache (if applicable)
rm -rf /var/www/myapp/cache/*

5. Uploaded Files or User Content

If you host file upload services, check:

find /var/www -type f | wc -l
du -sh /var/www

Look for:

• Unused or orphaned files
• Failed upload directories
• Temporary upload directories

Advanced: Truncate Large Open Files

If a log file is still open (being written to), deleting it won't free inodes immediately. You need to truncate it:

# Find large files in use
lsof +L1 2>/dev/null | head -20

# Truncate without closing the file
truncate -s 0 /var/log/largefile.log

The difference:

• rm deletes the inode, but the space isn't freed if the file is still open
• truncate -s 0 empties the file while keeping it open, freeing space immediately

Permanent Solutions

1. Increase Inode Count (For New Filesystems)

When formatting a new partition:

mkfs.ext4 -i 4096 /dev/vda1
# or
mkfs.ext4 -N 4000000 /dev/vda1

The -i flag sets the ratio of bytes per inode. Lower value = more inodes.

2. Implement Automatic Cleanup

Create a cron job to regularly clean old files:

# /etc/cron.d/cleanup-inodes
0 2 * * * root find /var/lib/php/sessions -type f -mtime +7 -delete
0 3 * * * root find /tmp -type f -atime +30 -delete
0 4 * * * root find /var/tmp -type f -atime +30 -delete

3. Monitor Inode Usage

Add to your monitoring/alerting:

# Alert if inode usage > 85%
df -i | awk 'NR==2 {if ($5+0 > 85) print "Warning: Inode usage at " $5}'

Quick Diagnostic Checklist

• Confirm the problem: df -i shows 100% usage
• Find culprits: find / -xdev -printf '%h\n' | sort | uniq -c | sort -rn | head -20
• Check PHP sessions: ls /var/lib/php/sessions | wc -l
• Check mail spool: postqueue -p, du -sh /var/mail
• Clean old sessions: find /var/lib/php/sessions -type f -mtime +7 -delete
• Clean /tmp: find /tmp -type f -atime +30 -delete
• Verify recovery: df -i (should show lower usage)
• Set up automatic cleanup via cron