Linux
Beginner
Storefront Stuck in Maintenance Mode
Every Northstar Shops storefront page answers 503 with the maintenance banner, and checkout has taken no orders for 20 minutes.
Incident catalog
Pick an outage.
61 simulated incidents across Linux, Networking, Docker, Kubernetes, Security, Git, CI/CD, Web, Observability and Cloud. Each one is a broken system you investigate in a real-feeling shell, fix, and prove recovered. Nothing to install.
Linux
17 incidents
Linux
Intermediate
Backup Pager Fires After Every Run
The nightly backup logs a finished archive every run, yet BackupFailure has paged the night shift three nights running.
Linux
Beginner
Cache Logs Approach the Disk Limit
The capacity forecast projects the rebuilt cache host's disk full within a week at the cache log's usual growth rate.
Linux
Beginner
Overnight Sync Has No New Run
The 02:00 content sync never published, and product pages are still showing yesterday's prices.
Linux
Beginner
Developer Shell Shortcuts Disappear
Three operators report that ll output on the ops host differs from what the handoff runbooks show.
Linux
Intermediate
Incident Notes Service Fails to Start
The internal incident notes service is down, leaving responders to coordinate the outage from chat scrollback.
Linux
Beginner
Builds Stall Before Compilation
Eight jobs are stuck in the build queue, none has reached the compile step in 40 minutes, and deploys are frozen.
Linux
Intermediate
Queue Error Pager Keeps Firing
QueueWorkerCritical has paged 42 times in the last hour, although the queue workers report zero lag.
Linux
Intermediate
Database Backup Archive Is Missing
The nightly orders database backup exits successfully but leaves nothing in /backups.
Linux
Intermediate
Report API Fails To Start
The report API does not answer, and the report dashboard shows errors ahead of Finance's 10:00 run.
Linux
Intermediate
Batch Host Cannot Write New Exports
Batch exports fail with 'no space left on device', and the batch host's root filesystem is at 100%.
Linux
Intermediate
Metrics API Refuses to Bind
The metrics API will not stay up after its 09:18 restart, and the dashboards that query it show no data.
Linux
Intermediate
Batch Worker Saturates the Host
Host CPU has sat above 95% for ten minutes, and storefront response times are climbing with it.
Linux
Advanced
Forgeline Returns 500 on Every Page
forgeline.dev answers 500 on the web UI and API, and its readiness probe fails from every region.
Linux
Advanced
Exchange Gateway Throttling a Flood of Child Orders
The process list shows one order-router process near 98% CPU while the other seven sit between 2% and 5%.
Linux
Intermediate
Scan Host Still 98% Full After the Big Log Was Deleted
Depot scan uploads are being refused.
Linux
Intermediate
Larder Marketplace Shows Our Stock Frozen Since 08:08
The Larder delivery marketplace has accepted no stock feed from Greengage.
Networking
5 incidents
Networking
Intermediate
Storefront API Calls Fail
Storefront pages load again, but cart, search and checkout calls through /api fail for every customer.
Networking
Beginner
Outbound CLI Requests Fail
Package installs and API calls from the reimaged host time out whenever they leave the network.
Networking
Intermediate
External Hostnames Stop Resolving
billing-sync fails every export to the payments API, and no external hostname resolves on the host.
Networking
Intermediate
Public HTTP Requests Time Out
Customers outside the network cannot load the site: every external request times out while nginx reports healthy.
Networking
Beginner
Internal API Hostname Does Not Resolve
The migrated host cannot reach the internal API by name; every call fails before it leaves the machine.
Docker
8 incidents
Docker
Intermediate
Web Container Is Unreachable on Port 80
Browser customers get connection refused on port 80 while the storefront's web container reports Up.
Docker
Beginner
Web Container Exits After Startup
The launch API's container exits with status 0 a moment after every start, so nothing is serving.
Docker
Intermediate
Database Rows Vanish After Restart
Each time the postgres container is recreated, the orders database comes back empty and earlier rows are gone.
Docker
Intermediate
Finance Database Exits After Startup
The finance database container exited ten minutes ago, and every finance API write has failed from that moment on.
Docker
Intermediate
Worker Cannot Resolve Web Backend
The shop worker gets 'bad address' for the web backend on every job, so nothing it processes completes.
Docker
Intermediate
API Image Cannot Be Built
The release-api image build fails at a COPY step, so there is no image to run and nothing answers on the release port 3001.
Docker
Intermediate
Published Frontend Port Refuses Connections
Requests to the frontend on host port 8089 connect and then get an empty reply, although the container is Up and the port is published.
Docker
Intermediate
Web Container Reports Unhealthy
Compose marks the edge web container unhealthy even though its page answers 200, and the release gate that waits on health is blocked.
Kubernetes
11 incidents
Kubernetes
Intermediate
API Pods Restart Across the Rollout
Every api replica is in CrashLoopBackOff and the Deployment has shown 0 of 3 available for ten minutes.
Kubernetes
Intermediate
Webapp Pods Never Become Available
The migrated webapp has shown 0 of 1 available replicas from the moment it was applied to the new cluster four minutes ago.
Kubernetes
Beginner
Order Scoring Pods Keep Restarting
Both order-scoring worker pods die within seconds of every start, and the orders.score backlog keeps climbing.
Kubernetes
Intermediate
Checkout Pods Never Start
Ten minutes after tonight's release, the checkout Deployment has 0 of 2 replicas available and neither pod has ever started.
Kubernetes
Intermediate
Catalog Service Has No Backends
Calls to the catalog Service are refused even though both catalog pods are Running and Ready.
Kubernetes
Intermediate
Payment Pods Stuck Before Startup
The payment pod has been stuck before startup for ten minutes, leaving the payment Deployment with zero ready replicas.
Kubernetes
Intermediate
Report Workers Remain Pending
The report Deployment has run zero workers for ten minutes; its only pod is still waiting to be scheduled.
Kubernetes
Intermediate
Gateway Pods Run but Receive No Traffic
Both gateway pods have been Running for ten minutes and neither has ever become Ready, so no gateway replica is eligible for traffic.
Kubernetes
Advanced
Ledger Rollout Leaves Every Pod Crashing
Both pods from the new ledger revision crash on startup, leaving one old pod to carry every ledger write.
Kubernetes
Advanced
Ledger Deployer Is Forbidden to Patch
The payments deploy job's service account gets Forbidden on every Deployment patch, so the ledger change it carries cannot roll out.
Kubernetes
Advanced
Sensor Agent Crash-Looping on Every Node
The endpoint sensor on all five fleet-east nodes has been crash-looping.
Security
4 incidents
Security
Intermediate
Developer SSH Login Is Rejected
The dev automation account's SSH logins from the deploy runner are refused, and scheduled deploy jobs cannot reach the host.
Security
Intermediate
Hotfix Deploy Blocked
The stabilization hotfix is staged, but deploy.service cannot be restarted to ship it.
Security
Intermediate
Deploy Key Exposed to Local Users
A security scan flagged the deploy account's private SSH key as readable beyond its owner on the shared deploy host.
Security
Advanced
Players Cannot Log In Across Every Game and the Store
Login, profiles and matchmaking fail for every player through the public gateway while its health checks stay green.
Git
8 incidents
Git
Intermediate
Checkout Errors Rise After Release
Checkout's error rate jumped from about 0.2% to 23% within minutes of the latest configuration release.
Git
Intermediate
Checkout Hotfix Is Missing From Main
The retry change that passed checkout testing is absent from main, so the release build would ship the old behaviour.
Git
Intermediate
Tested Hotfix Vanishes After Checkout
main is clean, the tested retry change is absent from it, and no branch anywhere contains that commit.
Git
Intermediate
Release Config Merge Stops in Conflict
Merging release/config into main stops on a conflict in deploy/config.yaml, so the checkout retry change cannot ship.
Git
Advanced
Release Token Appears in Commit History
A release token file is tracked at the tip of main, and the credential has to be treated as exposed until it is revoked.
Git
Beginner
Status Page Timeout Fix Is Not on Main
main still carries upstream_timeout_seconds: 2 in its committed deploy/status.yaml.
Git
Beginner
Rate-Limit Hotfix Push Rejected by Origin
A third of partner API calls are answered with HTTP 429 while the agreed rate-limit hotfix waits to deploy.
Git
Beginner
Invoice Uploads Over 2 MB Fail With 413
Since this morning's 2.4.0 deploy, invoice uploads larger than 2 MB are rejected with HTTP 413.
CI/CD
1 incident
CI/CD
Intermediate
Deploy Workflow Stops Before Any Job Starts
The deploy workflow is rejected before any job is scheduled, so none of tonight's changes can reach production.
Web
4 incidents
Web
Intermediate
Order API Returns 502 Through Nginx
Every order API request through the edge returns 502 Bad Gateway, so checkout cannot submit orders.
Web
Intermediate
Web Tier Has No Listener After Reload
The web host refuses every connection on port 80, taking the site down for visitors routed to it.
Web
Advanced
Every Site Behind the Edge Returns 502
Customer sites proxied through edge-fra-07 answer 502 Bad Gateway for almost every request while the edge host's CPU sits at 100%.
Web
Intermediate
Shoppers Get a Browser Security Warning at Checkout
Every HTTPS request to the storefront fails certificate validation, so browsers and the mobile app refuse to connect.
Observability
2 incidents
Observability
Intermediate
API Metrics Disappear from Prometheus
API dashboards have gone flat: Prometheus reports its api scrape target down while the exporter answers from the host.
Observability
Advanced
Critical Alerts Arrive Without Pages
Critical alerts reach the alert router and are accepted, but no on-call page has gone out for over an hour.
Cloud
1 incident
Cloud
Advanced
Object Requests Failing Across nbx-east-1
GET, LIST and PUT requests fail for every customer in nbx-east-1, and new uploads cannot be placed.
Campaigns
One outage, several escalating stages
5 stages
Linux Outage Ladder
One production incident, five escalating Linux recoveries across config, routing, service control, access, and stabilization.
4 stages
Platform Recovery Ladder
Recover a broken delivery path from container build, to runtime networking, to persistence, to cluster health.
4 stages
Network Response Ladder
Work through outbound access, local name resolution, DNS repair, and firewall recovery as one incident chain.
5 stages
Kubernetes Night Shift
Bring a troubled cluster release back from image pulls, configuration errors, memory kills, empty Services, and unscheduled workers.
7 stages
Friday Deploy Meltdown
Contain a committed token, then recover a bad deploy, lost Git work, a config conflict, and a broken image build.
3 stages
Observability Blackout
Restore a misleading container health signal, a missing scrape target, and a paging route that hides alerts.
4 stages
Breach Response
Contain an exposed key and restore only the access responders need.
Your pager is ready.
Free, instant, and it works on your phone. No signup: start as a guest and save your progress later.