grep, sed, awk and jq cheat sheet

grep -rn <pattern> <dir>

Search every file under a directory and print file names with line numbers.

grep -rn "listen" /etc/nginx/

/etc/nginx/sites-enabled/default:2:    listen 80 default_server;
/etc/nginx/sites-enabled/default:3:    listen [::]:80 default_server;

-r

Recurse into directories.

-n

Prefix each match with its line number.

-i

Ignore case.

-l

Print only the names of matching files.

--include='*.conf'

Only search files matching a glob.

grep -E <regex> <file>

Use extended regular expressions: alternation, +, ? and grouping without backslashes.

grep -E "error|timeout|refused" app.log

-E

Extended regex (egrep).

-w

Match whole words only.

-x

The pattern must match the whole line.

grep -v <pattern>

Invert the match to drop noise such as health checks before you read a log.

grep -v "GET /healthz" access.log

-v

Print lines that do not match.

-e P1 -e P2

Several patterns, for example to exclude two kinds of noise.

grep -C <N> <pattern> <file>

Show lines around each match, which is where the cause of a stack trace usually is.

grep -C 3 "Traceback" app.log

-C N

N lines before and after.

-B N

N lines before only.

-A N

N lines after only.

grep -c <pattern> <file>

Count matching lines instead of printing them.

grep -c "ERROR" app.log

137

-c

Count matching lines per file.

-o

Print only the matched part, one match per line.

Gotcha -c counts lines, not occurrences. For every occurrence use grep -o PATTERN | wc -l.

grep -F <literal>

Match a fixed string, so characters like . [ ] * are not treated as regex syntax.

grep -F "[error]" error.log

-F

Fixed-string matching (fgrep).

-f FILE

Read the patterns from FILE, one per line.

grep -oE <regex> | sort | uniq -c

Extract a value from each line and count how often each distinct value occurs.

grep -oE "status=[0-9]+" app.log | sort | uniq -c | sort -rn

9412 status=200
    318 status=502
     41 status=404

-o

Only the matching part.

uniq -c

Collapse adjacent duplicates with a count; needs sorted input.

sort -rn

Numeric, descending.

sed -n '<start>,<end>p' <file>

Print a range of lines, for example around the line number from an error message.

sed -n '120,140p' app.log

-n

Do not print every line automatically.

p

Print the selected lines.

'/start/,/end/p'

Select by pattern instead of line numbers.

sed 's/<old>/<new>/g' <file>

Substitute text and print the result without touching the file. Preview before editing in place.

sed 's/8080/9090/g' app.conf

s/a/b/

Replace the first match on each line.

g

Replace every match on the line.

I

Case-insensitive match (GNU).

&

In the replacement, the whole matched text.

sed -i.bak 's/<old>/<new>/' <file>

Edit a file in place and keep a backup copy with the given suffix.

sed -i.bak 's/^debug=true/debug=false/' app.conf

-i.bak

Edit in place, saving the original as FILE.bak.

^ and $

Anchor to the start or end of the line to avoid partial matches.

Gotcha BSD and macOS sed need an explicit suffix argument: sed -i '' 's/a/b/' file. The GNU form fails there.

sed '/<pattern>/d' <file>

Delete matching lines. Stripping comments and blanks shows the configuration that is actually in effect.

sed '/^\s*#/d;/^\s*$/d' nginx.conf

/re/d

Delete lines matching re.

;

Separate several commands.

Nd

Delete line N, for example 1d drops a CSV header.

sed 's|<old>|<new>|g'

Use a different delimiter when the pattern contains slashes, such as file paths or URLs.

sed 's|/var/www/old|/var/www/new|g' site.conf

s|a|b|

Any character after s becomes the delimiter.

awk '{print $N}' <file>

Print chosen whitespace-separated columns from each line.

awk '{print $1, $9}' access.log

$1..$NF

Fields; $0 is the whole line and $NF the last field.

NR

Current line number.

OFS

Output field separator, for example -v OFS=,

awk -F<sep> '<cond> {print}'

Split on a custom separator and filter rows with a condition.

awk -F: '$3 >= 1000 {print $1}' /etc/passwd

-F:

Use : as the field separator; -F, for CSV without quoted commas.

cond {action}

Run the action only on lines where cond is true.

awk '$9 >= 500' access.log

Keep only the lines where a column matches a numeric test, such as HTTP 5xx in a combined access log.

awk '$9 >= 500 {print $4, $7, $9}' access.log | tail

$9

Status code field in nginx and Apache combined log format.

$7

Request path.

Gotcha Field numbers shift if the format changes. Check one raw line first, or switch the server to JSON logs.

awk '{s += $N} END {print s}'

Sum or average a numeric column in one pass.

awk '{s += $10} END {printf "%.1f MiB\n", s/1048576}' access.log

END { }

Runs once after the last line.

printf

Formatted output, as in C.

awk '{c[$N]++} END {for (k in c) print c[k], k}'

Count occurrences per key with an associative array: requests per client, errors per endpoint.

awk '{c[$1]++} END {for (ip in c) print c[ip], ip}' access.log | sort -rn | head

c[key]++

Increment the counter for key.

for (k in c)

Iterate keys in unspecified order; sort afterwards.

sort | uniq -c | sort -rn | head

The classic top-N pipeline: count distinct values and rank them.

cut -d' ' -f1 access.log | sort | uniq -c | sort -rn | head

18231 203.0.113.9
   2210 198.51.100.24
    914 192.0.2.61

sort

Groups identical lines so uniq can count them.

uniq -c

Prefix each distinct line with its count.

sort -rn

Largest count first.

Gotcha uniq only collapses adjacent duplicates. Without the first sort the counts are wrong.

sort -t<sep> -k<col>

Sort by a specific column, numerically or by human-readable sizes.

sort -t, -k3,3nr latency.csv | head

-t SEP

Field separator.

-k3,3

Sort on field 3 only.

-n / -h

Numeric, or human sizes like 2K 1G.

-r

Reverse.

-u

Output unique lines only.

cut -d<sep> -f<fields>

Extract fields separated by a single character.

cut -d: -f1,7 /etc/passwd

root:/bin/bash
daemon:/usr/sbin/nologin

-d C

Delimiter character (tab by default).

-f 1,3-5

Fields to keep.

-c 1-8

Character positions instead of fields.

Gotcha cut treats every delimiter as a field boundary, so runs of spaces create empty fields. Use awk for space-aligned output.

tr <set1> <set2>

Translate or delete characters: fix Windows line endings, change case, squeeze spaces.

tr -d '\r' < windows.conf > unix.conf

-d SET

Delete characters in SET.

-s SET

Squeeze repeats into one.

'[:upper:]' '[:lower:]'

Lowercase everything.

column -t

Align whitespace-separated output into a readable table.

mount | column -t

-t

Create a table.

-s SEP

Input separator, for example -s, for CSV.

jq .

Pretty-print and validate JSON. A parse error tells you the payload is not JSON.

curl -s localhost:8080/health | jq .

{
  "status": "degraded",
  "checks": {
    "db": "ok",
    "cache": "timeout"
  }
}

.

The identity filter: the whole input.

-c

Compact, one document per line.

-S

Sort object keys.

jq -r '<path>'

Extract fields as raw text for use in shell loops.

jq -r '.items[].metadata.name' pods.json

-r

Raw output: strings without quotes.

.a.b

Nested field access.

.[] / .items[]

Iterate an array.

jq '.[] | select(<cond>)'

Keep only the elements that match a condition.

jq '.[] | select(.status != "ok")' checks.json

select(expr)

Pass the value through only when expr is true.

|

Feed the output of one filter into the next.

test("re")

Regex match on a string.

jq '{a: .x, b: .y}'

Build new objects with only the fields you care about.

jq -c '.[] | {name, restarts: .restartCount}' containers.json

{name}

Shorthand for {name: .name}.

length

Array length or number of keys.

map(f)

Apply f to every array element.

jq --arg <name> <value> '<filter>'

Pass shell values into a filter safely instead of splicing them into the quoted program.

jq --arg env prod '.[] | select(.env == $env) | .name' services.json

--arg n v

Bind $n to the string v.

--argjson n v

Bind $n to parsed JSON, such as a number.

kubectl get -o json | jq

Answer questions kubectl's columns cannot, such as which pods are not Running and why.

kubectl get pods -o json | jq -r '.items[] | select(.status.phase != "Running") | .metadata.name'

.status.phase

Pending, Running, Succeeded, Failed or Unknown.

.status.containerStatuses[]

Per-container state, restarts and last termination reason.

yq '<path>' <file>

Read a value out of a YAML file with the paths jq uses. Scalars print bare, so the output goes straight into a variable.

yq '.spec.template.spec.containers[].image' deploy.yaml

.a.b

A key below a key.

.items[0], .items[]

One item, or every item.

-o json

Print JSON instead of YAML.

-e

Exit 1 when the result is null or false.

Gotcha This is the Go yq (mikefarah). The Python yq of some distributions wraps jq and takes other options.

yq -i '<path> = <value>' <file>

Change a YAML file in place: set a value, add a key, delete one. Comments in the file are kept.

yq -i '.spec.replicas = 5' deploy.yaml && yq .spec.replicas deploy.yaml

.a.b = "text"

Set a value; missing keys are created.

.list += ["x"]

Append to a list.

del(.a.b)

Remove a key or an item.

(.[] | select(.name == "x") | .value) = 1

Change the item that matches.

Gotcha yq writes the whole file again: blank lines go and lists are indented under their key, so the first edit of a hand-written file shows up as a large diff. Without -i the result is only printed.