Linux systems 30 entries 24 run in the sandbox

systemctl and Linux processes cheat sheet

Find what is running, control services and read unit state. Each entry has a worked example and the flags worth knowing, and where one is easy to get wrong, the gotcha.

Find processes

ps aux

Snapshot every process with its owner, CPU and memory share, state and full command line.

ps aux --sort=-%mem | head
USER       PID %CPU %MEM    VSZ   RSS TTY  STAT START   TIME COMMAND
webapp    1432 12.4 18.9 912340 386120 ?   Ssl  09:12  4:21 /opt/webapp/bin/server
root         1  0.0  0.5 169356 11020 ?    Ss   Sep22  0:09 /sbin/init
a
Processes of all users with a terminal.
u
User-oriented columns (%CPU, %MEM, RSS, START).
x
Include processes without a terminal, such as daemons.
--sort=-%mem
Sort descending by memory (GNU ps).

Gotcha STAT D means uninterruptible sleep, usually stuck on disk or NFS I/O. kill -9 will not remove it; fix the I/O.

ps -eo <columns>

Choose exactly the columns you need, including parent PID and elapsed time.

ps -eo pid,ppid,user,%cpu,%mem,etime,cmd --sort=-%cpu
-e
Every process.
-o LIST
Output columns: pid, ppid, user, stat, etime, rss, cmd.
-p PID
Only the listed PIDs.
--forest
Draw the parent/child tree.

pgrep -a <pattern>

Find PIDs by process name or command line without the grep-matches-itself problem.

pgrep -a nginx
812 nginx: master process /usr/sbin/nginx
813 nginx: worker process
-a
Print the full command line next to each PID.
-f
Match against the full command line, not just the name.
-u USER
Only processes owned by USER.
-x
Require an exact name match.

pstree -p [<pid>|<user>]

Draw the process table as a tree: which service or shell started a process, and what it started in turn.

pstree -p $$
bash(4242)───pstree(4301)
-p
Show PIDs; without it identical children fold into N*[name].
-a
Show command-line arguments.
-u
Mark where the user changes.
$$
The PID of your own shell.

Gotcha A process whose parent is systemd (PID 1) is either a service or an orphan: its original parent has exited.

top

Live view of load, CPU, memory and the busiest processes.

P / M
Sort by CPU or by memory.
1
Toggle per-CPU rows, which exposes one pegged core.
-b -n 1
Batch mode, one iteration: safe to pipe or paste into a ticket.
-p PID
Watch specific processes only.

Gotcha High %wa (iowait) with idle CPUs means the bottleneck is storage, not compute.

uptime

Time since boot plus the 1, 5 and 15 minute load averages.

10:47:40 up 1 day,  3 users,  load average: 0.12, 0.08, 0.05
-p
Pretty uptime only.
-s
The time the system booted.

Gotcha Compare load to the core count (nproc). Load 8 on 16 cores is fine; on 2 cores it is a queue. Linux load also counts processes blocked on I/O.

free -h

Memory and swap usage in human-readable units.

total        used        free      shared  buff/cache   available
Mem:           3.8Gi       382Mi       2.2Gi       1.2Mi       1.3Gi       3.5Gi
Swap:          2.0Gi          0B       2.0Gi
-h
Human-readable units.
-s N
Repeat every N seconds.

Gotcha Low 'free' is normal; the kernel uses spare memory as cache. Watch 'available' and whether swap is in use.

cat /proc/<pid>/<file>

Read live kernel data about a process: limits, status, environment and open file descriptors.

cat /proc/loadavg
0.12 0.08 0.05 1/412 20931
/proc/PID/status
State, threads, VmRSS and capability sets.
/proc/PID/limits
Effective ulimits, such as Max open files.
/proc/PID/fd/
One symlink per open file descriptor.
/proc/PID/environ
Environment, NUL-separated: tr '\0' '\n' < file.
/proc/loadavg, /proc/meminfo
The whole machine: what uptime and free read.

sysctl <key>[=<value>]

Read or change a kernel parameter. A change with -w lasts until the next reboot; a file in /etc/sysctl.d makes it permanent.

sysctl vm.swappiness net.ipv4.ip_forward
sudo sysctl -w KEY=VALUE
Change it now; the same as writing /proc/sys/...
/etc/sysctl.d/60-local.conf
KEY = VALUE lines applied at every boot.
sudo sysctl --system
Apply every settings file now, in boot order.
sysctl -a | grep KEY
Find a parameter by part of its name.

Gotcha sudo echo 1 > /proc/sys/net/ipv4/ip_forward fails: the redirect runs as you. Use sysctl -w or tee.

timedatectl [set-timezone <zone>]

Show the clock, the time zone and whether a time service keeps the clock right, or change the time zone.

timedatectl
sudo timedatectl set-timezone Europe/Berlin
Point /etc/localtime at that zone; date follows at once.
timedatectl list-timezones
The zone names; pipe them into grep.
sudo timedatectl set-ntp true
Start and enable the installed time service.
chronyc sources
With chrony installed: the time servers, and ^* marks the one the clock follows.

Gotcha NTP service: n/a means no time service is installed. Install chrony, then its servers are in /etc/chrony/chrony.conf.

lsof -p <pid>

List open files and sockets per process. Finds deleted-but-open files that still hold disk space.

sudo lsof +L1
-p PID
Files opened by one process.
-i :PORT
Processes using a TCP or UDP port.
+L1
Open files with zero links: deleted but still using space.
-u USER
Files opened by a user.

Gotcha Deleting a log a process still writes to does not free space until the process closes it. Truncate it (: > file) or restart the process.

strace -f -p <pid>

Trace the system calls of a running process to see what it is waiting on or failing to open.

sudo strace -f -e trace=openat,connect -p 1432
-f
Follow child processes and threads.
-e trace=SET
Limit to syscalls, such as network, file or openat.
-T
Show time spent in each syscall.
-o FILE
Write the trace to a file.

Gotcha strace slows the traced process a lot. Attach briefly on production and detach with Ctrl-C.

Control processes

kill -<SIGNAL> <pid>

Send a signal to a process. TERM asks it to shut down cleanly; KILL cannot be caught.

kill -TERM 1432
-TERM (15)
Default. Graceful shutdown; the process can clean up.
-HUP (1)
Many daemons reload their configuration.
-KILL (9)
Immediate termination. No cleanup, no flushed buffers.
-l
List signal names and numbers.

Gotcha Reach for -9 last. It skips shutdown hooks, can leave lock files behind and corrupt half-written data.

killall <name>

Signal every process with a given name.

killall -HUP rsyslogd
-SIGNAL
Signal to send (TERM by default).
-u USER
Only processes owned by USER.
-w
Wait until the processes have exited.

Gotcha On some Unix systems (Solaris) killall kills every process. Prefer pkill with an exact pattern in shared scripts.

nice -n <N> <cmd>

Start a command with lower CPU priority so batch work does not starve the service.

nice -n 10 tar -czf backup.tgz projects
-n N
Niceness from -20 (highest priority) to 19 (lowest).
renice -n N -p PID
Change the niceness of a running process.
ionice -c3
Idle I/O class, for disk-heavy jobs.

Gotcha Anyone may raise niceness; lowering it, even back to 0, needs root.

nohup <cmd> > <log> 2>&1 &

Run a long task that keeps going after you log out, with its output in a file.

nohup ./reindex.sh > reindex.log 2>&1 &
&
Run in the background of the current shell.
jobs / fg / bg
List, foreground or resume background jobs.
disown
Detach an already running job from the shell.

Gotcha For anything that matters, use tmux or a systemd-run transient unit so you can reattach and see its logs.

jobs

List what this shell runs in the background or has stopped. fg and bg move a job between the foreground and the background.

sleep 300 & jobs -l
[1] 4301
[1]+  4301 Running                 sleep 300 &
%1, %+, %-
Job 1, the current job and the previous one; kill, fg, bg and wait take them.
fg %1
Bring job 1 to the foreground and wait for it.
Ctrl+Z, then bg
Stop the foreground command, then let it continue in the background.
wait
Block until background jobs finish; wait PID returns that job's exit status.
$!
PID of the last background command.

Gotcha Jobs belong to the shell that started them. From another terminal the job table is empty: find the process with pgrep or ps instead.

systemd services

systemctl status <unit>

Show whether a unit is active, its main PID, memory, cgroup tree and the last journal lines.

systemctl status webapp
● webapp.service - Web application
     Loaded: loaded (/etc/systemd/system/webapp.service; enabled)
     Active: failed (Result: exit-code) since Tue 09:41:02 UTC
    Process: 2211 ExecStart=/opt/webapp/bin/server (code=exited, status=1/FAILURE)
-l
Do not truncate long lines.
-n N
Show N journal lines (default 10).
--no-pager
Print straight to the terminal.

Gotcha The last lines are a teaser. For the real error, run journalctl -u <unit> -e.

systemctl restart <unit>

Stop and start a service. Use start and stop for one direction only.

sudo systemctl restart webapp
start / stop
Start or stop the unit now.
restart
Stop then start, even if it was not running.
try-restart
Restart only if it is already running.

Gotcha Restarting clears the symptom and the evidence. Capture status and logs before restarting a broken service.

systemctl reload <unit>

Ask a service to re-read its configuration without dropping connections, if the unit defines ExecReload.

sudo systemctl reload nginx
reload
Fails if the unit has no reload action.
reload-or-restart
Reload if supported, otherwise restart.

Gotcha Validate first (nginx -t, sshd -t). A reload with a broken config can take the service down.

systemctl enable --now <unit>

Start a unit at boot and start it right now in one command.

sudo systemctl enable --now webapp
enable / disable
Add or remove the boot-time symlinks.
--now
Also start (or stop) the unit immediately.
mask
Link the unit to /dev/null so nothing can start it.

Gotcha enable alone does not start anything. A service that is fine until the next reboot is usually enabled but was never started, or the reverse.

systemctl is-active <unit>

Print one word and set the exit code. Built for scripts and health checks.

systemctl is-active webapp
active
is-active
active, inactive, failed, activating.
is-enabled
enabled, disabled, masked, static.
is-failed
Exit 0 only when the unit is failed.
--quiet
No output, exit code only.

systemctl --failed

List every unit currently in the failed state.

systemctl list-units --failed
list-units --type=service
All loaded services.
--state=running
Filter by state.
reset-failed
Clear the failed state after you fix the cause.

systemctl cat <unit>

Print the unit file and every drop-in override, in the order systemd applies them.

systemctl cat webapp
cat
Unit file plus drop-ins with their paths.
show -p KEY
One effective property, for example -p ExecStart.

systemctl show <unit> -p <props>

Read effective properties: main PID, restart counter, memory, the resolved user and more.

systemctl show webapp -p MainPID,ActiveState,NRestarts
MainPID=1432
ActiveState=active
NRestarts=3
-p A,B
Only the named properties.
--value
Print values without the KEY= prefix.

Gotcha A rising NRestarts with Restart=always is a crash loop hiding behind an 'active' status.

systemctl edit <unit>

Create a drop-in override instead of editing the packaged unit file, which upgrades overwrite.

sudo systemctl edit webapp
edit
Opens /etc/systemd/system/<unit>.d/override.conf.
--full
Edit a full copy of the unit instead.
revert
Drop every local override for the unit.

Gotcha To replace ExecStart in a drop-in, first clear it with an empty ExecStart= line, then set the new value.

systemctl daemon-reload

Reload unit files from disk after you create or change them.

sudo systemctl daemon-reload

Gotcha Editing a unit and restarting without daemon-reload runs the old definition. systemctl status warns 'changed on disk'.

systemctl list-timers

List systemd timers with their last and next run times, the modern alternative to cron.

systemctl list-timers --all
--all
Include inactive timers.
OnCalendar=
Schedule syntax in the .timer unit, such as daily or *-*-* 02:00.

Service unit file

A minimal, production-ready service unit with a dedicated user, restart policy and environment file.

[Unit]
Description=Web application
After=network-online.target
Wants=network-online.target

[Service]
User=webapp
Group=webapp
WorkingDirectory=/opt/webapp
EnvironmentFile=/etc/webapp/app.env
ExecStart=/opt/webapp/bin/server --port 8080
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target
After / Wants
Ordering and soft dependency on the network being up.
User / Group
Run unprivileged; the user must own what it writes.
EnvironmentFile
KEY=value file; keep it mode 640 root:webapp.
Restart=on-failure
Restart after crashes, not after a clean stop.
WantedBy
The target that enable links the unit into.

Gotcha ExecStart needs an absolute path. systemd does not search PATH the way your shell does.

crontab -e

Edit the current user's cron table. Five time fields, then the command.

*/5 * * * * /usr/local/bin/sync.sh >> /var/log/sync.log 2>&1
m h dom mon dow
Minute, hour, day of month, month, day of week.
*/5
Every 5 units.
-l
List the current table.
-u USER
Edit another user's table (as root).

Gotcha Cron runs with a tiny PATH and no profile. Use absolute paths and redirect output, or failures vanish silently.

Your pager is ready.

Free, instant, and it works on your phone. No signup: start as a guest and save your progress later.