OOM Killer - Out of Memory Process Kills

Understand and prevent the Linux OOM Killer from terminating critical processes when RAM is exhausted.

The OOM (Out of Memory) Killer is a Linux kernel mechanism that kills processes when the system runs out of RAM and swap. It selects the process with the highest oom_score to terminate.

Detect OOM Events

# Check kernel ring buffer
dmesg | grep -i "oom\|killed\|out of memory"

# Systemd journal (persistent)
journalctl -k | grep -i "oom\|killed process"

# Search syslog
grep -i "oom\|killed" /var/log/syslog | tail -50

Typical OOM message:

kernel: Out of memory: Kill process 12345 (php-fpm) score 892 or sacrifice child
kernel: Killed process 12345 (php-fpm) total-vm:512MB, anon-rss:256MB

Understand oom_score

Each process has a score from 0 to 1000. Higher score = more likely to be killed.

# Check score for a process
cat /proc/$(pgrep nginx)/oom_score

# Check score_adj (manual tuning offset, -1000 to +1000)
cat /proc/$(pgrep nginx)/oom_score_adj

# List all processes sorted by oom_score
for pid in $(ls /proc | grep -E '^[0-9]+$'); do
  score=$(cat /proc/$pid/oom_score 2>/dev/null)
  comm=$(cat /proc/$pid/comm 2>/dev/null)
  echo "$score $pid $comm"
done | sort -rn | head -20

Protect a Critical Process

Lower the oom_score_adj to make a process less likely to be killed:

# Protect a running process (temporary)
echo -500 > /proc/$(pgrep mysqld)/oom_score_adj

# Never kill this process (-1000 = completely immune)
echo -1000 > /proc/$(pgrep sshd)/oom_score_adj

Persistent via systemd unit (/etc/systemd/system/mysql.service.d/oom.conf):

[Service]
OOMScoreAdjust=-500

systemctl daemon-reload
systemctl restart mysql

Common Culprits

PHP-FPM spawning too many workers:

; /etc/php/8.x/fpm/pool.d/www.conf
pm = dynamic
pm.max_children = 20        ; reduce this
pm.start_servers = 5
pm.min_spare_servers = 2
pm.max_spare_servers = 10
pm.max_requests = 500       ; recycle workers to prevent leaks

Java heap unbounded:

# Always set explicit heap limits
java -Xms512m -Xmx2g -jar app.jar

Redis without maxmemory:

# /etc/redis/redis.conf
maxmemory 512mb
maxmemory-policy allkeys-lru

MySQL innodb_buffer_pool_size too large:

# /etc/mysql/mysql.conf.d/mysqld.cnf
innodb_buffer_pool_size = 1G   # max ~70% of available RAM

Immediate Fixes

# 1. Add swap space (emergency relief)
fallocate -l 2G /swapfile
chmod 600 /swapfile
mkswap /swapfile
swapon /swapfile
echo '/swapfile none swap sw 0 0' >> /etc/fstab

# 2. Find and kill the memory hog
ps aux --sort=-%mem | head -10
kill -9 <PID>

# 3. Free page cache (safe to run on production)
sync && echo 3 > /proc/sys/vm/drop_caches

Memory Limits via cgroups (systemd)

Limit memory per service without OOM risk:

# /etc/systemd/system/myapp.service.d/limits.conf
[Service]
MemoryMax=1G
MemorySwapMax=0    # disable swap for this service

systemctl daemon-reload
systemctl restart myapp

Tune swappiness

# Check current (default: 60)
cat /proc/sys/vm/swappiness

# Reduce to prefer RAM over swap (better for servers)
sysctl vm.swappiness=10

# Persistent
echo "vm.swappiness=10" >> /etc/sysctl.d/99-swappiness.conf
sysctl -p /etc/sysctl.d/99-swappiness.conf

Monitor Memory Proactively

# Watch memory in real time
watch -n 2 free -h

# Alert when free memory < 10% (add to monitoring)
FREE=$(free | awk '/^Mem:/ {printf "%.0f", $4/$2 * 100}')
[ "$FREE" -lt 10 ] && echo "WARNING: only ${FREE}% RAM free"

Adding swap is a short-term fix, not a solution. If OOM events are frequent, find and fix the memory leak or increase RAM.

OOM Killer - Out of Memory Process Kills

On this page