System Logs

How to read, search and manage server logs with journalctl and traditional log files

Logs are the first thing to check when something doesn't work. On modern Linux there are two systems: journald (journalctl) for service logs, and text files in /var/log/ for application logs.

journalctl: system and service logs

Most used commands

# All logs (from oldest)
journalctl

# Logs in real time (like tail -f)
journalctl -f

# Last N messages
journalctl -n 50

# Log of a specific service
journalctl -u nginx
journalctl -u mariadb
journalctl -u sshd
journalctl -u php8.2-fpm

# Log of a service in real time
journalctl -u nginx -f

# Log of current boot
journalctl -b

# Log of previous boot (useful to understand why the server restarted)
journalctl -b -1

# Only errors and critical messages
journalctl -p err
journalctl -p err -b    # Only errors from current boot

# Logs in a time interval
journalctl --since "2024-03-01" --until "2024-03-28"
journalctl --since "1 hour ago"
journalctl --since "today"

Filter by keyword

journalctl -u nginx | grep "error"
journalctl -u sshd | grep "Failed"
journalctl | grep -i "oom\|killed"

Traditional log files in /var/log/

File	Content
`/var/log/syslog`	General system log (Debian/Ubuntu)
`/var/log/messages`	General log (CentOS/AlmaLinux)
`/var/log/auth.log`	Login, sudo, SSH (Debian/Ubuntu)
`/var/log/secure`	Login, SSH (CentOS)
`/var/log/nginx/access.log`	HTTP requests received by Nginx
`/var/log/nginx/error.log`	Nginx errors
`/var/log/apache2/error.log`	Apache errors
`/var/log/mysql/error.log`	MySQL/MariaDB errors
`/var/log/fail2ban.log`	Fail2ban blocks
`/var/log/kern.log`	Kernel log
`/var/log/dpkg.log`	Package installations (Debian/Ubuntu)

Useful commands for log files

# Read a file in real time
tail -f /var/log/nginx/error.log

# Last 100 lines
tail -100 /var/log/nginx/access.log

# Search for a word in logs
grep "error" /var/log/nginx/error.log
grep -i "refused\|denied" /var/log/auth.log

# How many times an IP appears in the access log
grep "1.2.3.4" /var/log/nginx/access.log | wc -l

# IPs making the most requests (possible attack)
awk '{print $1}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head -20

Understand why the server restarted

# Last reboot
last reboot | head -5

# OOM killer (process terminated due to lack of RAM)
journalctl -b -1 | grep -i "oom\|killed process\|out of memory"
dmesg | grep -i "oom"

# Kernel panic or crash
journalctl -b -1 -p 0..3

# Current uptime
uptime

Find suspicious SSH access

# Failed login attempts
grep "Failed password" /var/log/auth.log | tail -20

# IPs attempting brute force
grep "Failed password" /var/log/auth.log | awk '{print $11}' | sort | uniq -c | sort -rn | head -10

# Successful logins
grep "Accepted" /var/log/auth.log | tail -20

# Last accesses
last | head -20

Manage log size

# How much space journald logs use
journalctl --disk-usage

# Delete logs older than 30 days
journalctl --vacuum-time=30d

# Limit logs to 500MB total
journalctl --vacuum-size=500M

# Configure limits permanently
nano /etc/systemd/journald.conf

[Journal]
SystemMaxUse=500M
MaxRetentionSec=1month

systemctl restart systemd-journald

Automatic log rotation (logrotate)

Log rotation prevents files from becoming enormous. Verify it is configured:

# Test the configuration
logrotate --debug /etc/logrotate.conf

# Force manual rotation
logrotate -f /etc/logrotate.conf

# Nginx configuration (example)
cat /etc/logrotate.d/nginx

On this page