Query the journal
journalctl -u <unit>
Show the logs of one systemd unit, including stdout and stderr of the service.
journalctl -u webapp -e
Sep 23 09:41:02 web-01 server[2211]: config: open /etc/webapp/app.env: permission denied
Sep 23 09:41:02 web-01 systemd[1]: webapp.service: Main process exited, code=exited, status=1/FAILURE
-u UNIT- Filter by unit; repeat it to merge several units.
-e- Jump to the end of the output in the pager.
-n N- Show only the last N lines.
--no-pager- Print directly, for pasting or piping.
Gotcha The first error after a restart is usually the real cause. Everything after it is fallout.
journalctl -f
Follow new journal entries live, like tail -f for the whole system or one unit.
journalctl -u webapp -f
-f- Keep printing new entries as they arrive.
-n 50 -f- Start with the last 50 lines, then follow.
journalctl --since <time>
Limit logs to a time window, such as the minutes around when the alert fired.
journalctl -u nginx --since "10 min ago"
--since- Relative ("1 hour ago", yesterday) or absolute ("2026-09-23 09:00").
--until- End of the window.
-S / -U- Short forms of --since and --until.
Gotcha Timestamps are in the host's local time zone unless you pass --utc. Match that to the alert's time zone before comparing.
journalctl -p <priority>
Show only entries at or above a syslog priority.
journalctl -p err -b
0-7- emerg, alert, crit, err, warning, notice, info, debug.
-p err- err and everything more severe.
-p warning..err- A range of priorities.
journalctl -b
Restrict output to one boot. -b -1 is the previous boot, which is where a crash before a reboot lives.
journalctl -b -1 -e
-b- The current boot.
-b -1- The boot before this one.
--list-boots- Every recorded boot with its ID and time range.
Gotcha Previous boots exist only with persistent storage. If /var/log/journal does not exist, the journal lives in /run and disappears on reboot.
journalctl -k
Kernel messages for this boot: OOM kills, disk errors, network link flaps and segfaults.
journalctl -k -p warning
-k- Kernel ring buffer entries only (like dmesg).
-g PATTERN- Grep inside the journal, for example -g oom.
journalctl -g <regex>
Filter messages by a regular expression inside journalctl itself.
journalctl -u webapp -g "timeout|refused"
-g REGEX- Match the MESSAGE field (PCRE2).
--case-sensitive=false- Force case-insensitive matching.
journalctl _COMM=<name>
Filter by structured journal fields such as process name, PID, UID or unit.
journalctl _COMM=sshd --since today
_COMM=- Executable name.
_PID=- Process ID.
_UID=- User ID of the logging process.
-F FIELD- List every value a field has taken.
journalctl -o <format>
Change the output format: JSON for tooling, bare messages for grepping, ISO timestamps for tickets.
journalctl -u webapp -n 5 -o short-iso
short-iso- ISO 8601 timestamps.
cat- Message text only.
json / json-pretty- Every field, for jq.
verbose- All fields, human readable.
journalctl --vacuum-size=<size>
Check and shrink journal disk usage by deleting the oldest archived journal files.
journalctl --disk-usage
Archived and active journals take up 3.8G in the file system.
--disk-usage- Current journal size.
--vacuum-size=500M- Delete archives until the total is under 500 MB.
--vacuum-time=7d- Delete archives older than seven days.
Gotcha Vacuuming frees space once. Set SystemMaxUse= in journald.conf so it does not refill.
Log files
ls -lt /var/log
Find the traditional log files and see which ones are still being written.
syslog / messages- General system log (Debian / RHEL naming).
auth.log / secure- Logins, sudo and SSH authentication.
kern.log- Kernel messages on Debian-based systems.
nginx/, apache2/- Web server access and error logs.
grep -i <pattern> <log> | tail
Pull the most recent matching lines out of a log file.
grep -i "error" /var/log/syslog | tail -n 20
-i- Ignore case.
-C N- Show N lines of context around each match.
-c- Count matches instead of printing them.
grep "Failed password" /var/log/auth.log
Spot SSH brute-force attempts or a user fighting a wrong password.
grep "Failed password" /var/log/auth.log | tail -n 10
Accepted publickey- Successful key-based logins.
Invalid user- Attempts for accounts that do not exist.
zgrep <pattern> <log>*
Search current and gzip-compressed rotated logs in one pass.
zgrep -h "Out of memory" /var/log/syslog*
-h- Do not prefix matches with file names.
zcat / zless- Read compressed logs directly.
sudo dmesg -T
Read the kernel ring buffer with human timestamps. Where OOM kills, failed mounts and hardware errors show up first.
sudo dmesg -T | tail -3
[Tue Sep 23 09:40:57 2026] Out of memory: Killed process 1432 (server) total-vm:912340kB, anon-rss:786120kB
-T- Human-readable timestamps.
--level=err,warn- Only those priorities.
-w- Follow new messages.
Gotcha A service that dies with no application error often shows up here as 'Killed process' from the OOM killer. On Ubuntu only root may read the buffer.
logger -t <tag> <message>
Write your own entry to syslog and the journal, for example to mark when a deploy or manual fix started.
logger -t deploy "release 42 started by alice"
-t TAG- Identifier shown instead of the program name.
-p facility.level- Priority, for example user.warning.
journalctl -t TAG- Read the entries back.
last [-n <N>] [<user>]
Login history from /var/log/wtmp, newest first: who logged in, from which address and for how long, and when the host was booted.
last -n 5
user pts/0 10.0.0.10 Thu Jan 15 09:28 still logged in
reboot system boot 6.8.0-49-generic Wed Jan 14 07:12 still running
wtmp begins Wed Jan 14 07:12:18 2026
-n N- Only the N latest entries.
reboot- The pseudo-user that marks each boot.
sudo lastb- Failed logins instead, from /var/log/btmp.
lastlog- The latest login of every account.
Gotcha wtmp is rotated: 'wtmp begins' is how far back the history goes, not when the host was installed.
Rotation & retention
logrotate policy
A drop-in for /etc/logrotate.d/ that rotates daily, keeps a week and compresses old files.
/var/log/webapp/*.log {
daily
rotate 7
compress
delaycompress
missingok
notifempty
create 0640 webapp adm
postrotate
systemctl reload webapp > /dev/null 2>&1 || true
endscript
}
rotate N- Keep N old files.
delaycompress- Leave the newest rotated file uncompressed for slow writers.
create MODE USER GROUP- Recreate the log with these permissions.
copytruncate- For apps that cannot reopen logs; can lose lines during the copy.
postrotate- Tell the app to reopen its log file.
Gotcha Without postrotate (or copytruncate) the app keeps writing to the rotated, deleted file and disk space is never freed.
logrotate -d <config>
Dry-run a rotation config and print exactly what would happen.
sudo logrotate -d /etc/logrotate.d/webapp
-d- Debug and dry run; changes nothing.
-f- Force a rotation now.
-v- Verbose output.
truncate -s 0 <file>
Empty a huge log in place without deleting it, so the writing process keeps a valid file handle.
sudo truncate -s 0 /var/log/webapp/debug.log
-s 0- Set the size to zero bytes.
: > file- Shell equivalent that truncates via redirection.
Gotcha Copy the tail somewhere first if you might need it for the postmortem. Truncation is permanent.
journald retention
Cap the journal and make it survive reboots with a drop-in under /etc/systemd/journald.conf.d/.
[Journal]
Storage=persistent
SystemMaxUse=1G
SystemKeepFree=2G
MaxRetentionSec=14day
Storage=persistent- Keep logs in /var/log/journal across reboots.
SystemMaxUse- Maximum disk space for persistent journals.
SystemKeepFree- Always leave this much free on the filesystem.
MaxRetentionSec- Drop entries older than this.
Gotcha Apply with systemctl restart systemd-journald.
journalctl -o cat | jq <filter>
Slice structured JSON application logs by field instead of by fragile text matching.
journalctl -u api -o cat | jq -r 'select(.level=="error") | .msg'
-o cat- Emit only the message, which is the JSON document.
jq -r- Raw strings without quotes.
select(...)- Keep only the objects that match.