Incident response runbook

First five minutes

What to do between the page and the first update. Stabilise first, understand later.

1. Acknowledge the page so the escalation stops and others know someone is on it. 2. Confirm impact from the user's side: dashboards, synthetic checks, a real request. 3. Declare an incident and a severity if users are affected. Open the incident channel. 4. Name an incident commander. Until someone else takes it, that is you. 5. Ask 'what changed?' before deep debugging: deploys, config, flags, traffic, dependencies. 6. Post the first update within 15 minutes, even if it only says 'investigating'.

Acknowledge

Stops escalation; does not mean you have a fix.

Impact first

Who is affected, how badly, since when.

Mitigate before root cause

A rollback now beats a perfect diagnosis in an hour.

Severity levels

A common SEV scale. Agree on yours before you need it; when in doubt, declare higher and downgrade.

Severity sets who is paged, how often you update, and whether a postmortem is required.

SEV1

Critical: core service down or data loss for many users. All hands, updates every 15-30 minutes, postmortem required.

SEV2

Major: significant degradation or a key feature broken for a subset of users. Postmortem expected.

SEV3

Minor: limited impact or a workaround exists. Fix in business hours.

SEV4

Low: cosmetic or internal-only issue, no customer impact.

Incident roles

Separate coordinating from fixing. The commander does not type commands.

Small incident: one person may hold several roles. Large incident: one person per role, and say it out loud when a role changes hands.

Incident commander

Owns decisions, priorities and delegation; keeps the big picture.

Ops / tech lead

Drives investigation and mitigation with responders.

Comms lead

Writes internal and external updates on a fixed cadence.

Scribe

Keeps a timestamped timeline of actions and findings.

What changed?

Most incidents follow a change. Walk this list before guessing.

Deploys in the last few hours (git log, kubectl rollout history, helm history) Config or feature-flag changes Infrastructure changes (Terraform applies, scaling events, certificate renewals) Traffic shape (a marketing push, a crawler, a retry storm) Dependencies (cloud provider status, third-party APIs, DNS)

Deploy correlation

Line up the error graph with deploy markers.

Blast radius

One host, one zone, one region or everything? It narrows the cause.

find /etc -type f -mmin -120

Configuration files changed on this host in the last two hours.

-mmin -120

Modified within 120 minutes.

uptime

Load averages and whether the host recently rebooted.

1 / 5 / 15 minute load

Rising 1-minute versus 15-minute load means the problem is getting worse.

systemctl list-units --failed

Services that systemd already knows are broken.

journalctl -p err -b

Error-level log entries since boot across every service.

journalctl -p err -b --no-pager | tail -n 40

-p err

err and more severe.

-S -15min

Or narrow to the last 15 minutes.

df -h

Full disks break databases, logging and deploys in confusing ways. Rule it out early.

df -i

Inodes too.

free -h

Available memory and swap activity.

available

The number that matters, not free.

ps aux --sort=-%cpu | head

The processes using the most CPU right now.

--sort=-%mem

Or the largest memory users.

dmesg -T | tail

OOM kills, disk errors and dropped packets reported by the kernel.

dmesg -T | tail -n 20

vmstat 1 5

Run queue, swapping, I/O wait and CPU split, once per second.

procs -----------memory---------- ---swap-- -----io---- -system-- -------cpu-------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st gu
 9  2      0 412332  40112 2210440    0    0  1204   880 2210 4102 71 18  2  9  0  0

r

Runnable processes; above the core count means CPU saturation.

b

Blocked on I/O.

si / so

Swap in and out; non-zero under load means memory pressure.

wa / st

I/O wait and hypervisor steal.

Gotcha The first line is the average since boot. Judge the present by the lines after it.

USE and RED methods

Two checklists that stop you from staring at random graphs.

USE, for every resource (CPU, memory, disk, network): Utilisation, Saturation, Errors. RED, for every service: Rate of requests, Errors, Duration (latency).

Utilisation

How busy the resource is.

Saturation

Queued work it cannot serve yet.

Errors

Error events or failed requests.

Duration

Latency percentiles, not averages.

Mitigation options

Reversible actions that reduce impact while you find the cause, roughly from safest to riskiest.

Roll back the last deploy or config change. Turn off the feature flag. Scale out, or move traffic away from the bad zone or host. Fail over to a replica or secondary region. Shed load: rate-limit, disable non-critical features, serve cached pages. Block abusive traffic at the edge.

Reversible

Prefer actions you can undo in one command.

Announce it

Say in the channel what you are about to change, then do it.

One change at a time

Otherwise you cannot tell what helped.

Verify recovery

An incident is over when users are fine, not when the process restarted.

The user-facing symptom is gone (synthetic check, real request, error graph back to baseline). Alerts have resolved on their own. No new errors in the logs of the fixed component and its neighbours. The fix survives a restart or a new pod (it is in code or config, not only in memory). Watch for a full cycle of the traffic pattern or batch job that triggered it.

Baseline

Compare with the same hour yesterday or last week.

Initial internal update

The first post in the incident channel. Short, factual, with the time of the next update.

[SEV2] Checkout API returning errors
Impact: ~20% of checkout requests fail with 502 since 09:40 UTC.
Status: Investigating. Suspected link to the 09:35 release of api 1.4.2.
IC: @alice  Ops: @bob  Comms: @carol
Next update: 10:15 UTC or sooner if anything changes.

Impact

Who, what and since when, in user terms.

Status

Investigating, Identified, Mitigating, Monitoring or Resolved.

Next update

A time, and keep that promise even with no news.

Public status page update

Customer-facing wording: empathy and facts, no internal jargon or blame, no guessed ETAs.

Investigating: Some customers are unable to complete checkout. We are investigating and will post an update within 30 minutes.

Identified: We have identified the cause of the checkout failures and are rolling out a fix.

Resolved: Between 09:40 and 10:22 UTC some customers could not complete checkout. The issue is resolved and we are monitoring. We are sorry for the disruption.

No internals

Say 'a configuration change', not host names or people.

Time window in UTC

Customers across time zones read the same page.

Shift handoff note

What the next responder needs to continue without re-investigating.

Current state: Mitigated by rollback to api 1.4.1 at 10:22 UTC. Error rate back to baseline.
Still open: Root cause of the 1.4.2 regression; release pipeline paused.
Watch: checkout 5xx panel and the ApiHighErrorRate alert.
Do not: re-enable the pipeline before the fix is reviewed.
Timeline and notes: incident doc link.

Do not

Traps the next person would otherwise walk into.

Resolution message

Close the incident clearly and point to the follow-up.

[RESOLVED] Checkout API errors
Duration: 09:40 to 10:22 UTC (42 min).
Cause: Connection pool size reduced in api 1.4.2 config.
Fix: Rolled back to 1.4.1; corrected config under review.
Postmortem: scheduled for Thursday, owner @alice.

Duration

Start of user impact to confirmed recovery, not to the fix commit.

Cause

One line in plain words; the detail belongs in the postmortem.

Postmortem

A date and an owner, so the follow-up actually happens.

Blameless postmortem

A structure that produces learning and fixes instead of blame.

Title, date, severity, authors

Summary: two or three sentences a stakeholder can read in 30 seconds.

Impact: users affected, duration, error budget consumed, revenue or data impact.

Timeline (UTC): detection, key decisions, mitigation, resolution.

Root cause and contributing factors: why it happened and why it was not caught earlier.

Detection: how we found out, and how we could have found out sooner.

What went well / what was hard.

Action items: each with an owner, a due date and a ticket. Prefer fixes that make the failure impossible over fixes that rely on people being careful.

Blameless

Describe what people knew and why their actions made sense at the time.

Contributing factors

Usually several; a single 'root cause' is rarely the whole story.

Action items

Few, specific and owned, rather than many vague ones.

Gotcha 'Be more careful' is not an action item. Change the system: a guard rail, a test, an alert or automation.

Five whys, done well

Keep asking why to get from the symptom to the systemic causes. Branch whenever there is more than one answer.

Checkout failed -> the API returned 502. Why? -> API pods ran out of database connections. Why? -> The 1.4.2 config cut the pool from 50 to 5. Why? -> A staging value was copied into the production overlay. Why was it not caught? -> Config diffs are not shown in review and there is no pool-exhaustion alert.

Stop at systems

Stop when you reach something you can change, not at a person.