Observability Advanced About 30 min +250 XP

Critical Alerts Arrive Without Pages

Critical alerts reach the alert router and are accepted, but no on-call page has gone out for over an hour.

Stage 3 of 3 in Observability Blackout.

Briefing

The last page of the observability blackout is the one that nearly stayed silent: critical alerts have been firing in Prometheus, the alert router accepts them, and not one on-call phone has buzzed for over an hour.

Sam happened to be watching the alert list by hand, which is the only reason anyone knows.

The routing file was restructured this afternoon to add a quiet receiver for noisy synthetic test alerts.

Dev suspects the pager integration key expired.

The fix is yours to find: hints, the recovery checks and the debrief unlock inside the incident.

How it plays

  1. 01

    Get paged

    The alert fires and the clock starts. Read the page and the briefing.

  2. 02

    Investigate

    Work in a simulated shell with realistic output: logs, configs, services.

  3. 03

    Fix it

    Change the system the way you would in production. Hints are there if you get stuck.

  4. 04

    Prove it

    Automated checks verify the recovery, then the debrief explains what happened.

Your pager is ready.

Free, instant, and it works on your phone. No signup: start as a guest and save your progress later.