FixOps
Play now
Cloud Advanced About 30 min +260 XP

Object Requests Failing Across nbx-east-1

GET, LIST and PUT requests fail for every customer in nbx-east-1, and new uploads cannot be placed.

Take this incident no signup All incidents

Briefing

Every GET, LIST and PUT against Nimbic Object Storage in nbx-east-1 has failed.

Nothing is crash-looping and the cluster itself looks calm, yet the metadata tier answers nothing and the status page cannot even load its own images.

Billing alerts are the loudest thing on the board, and the engineer who ran the change wants to finish the retirement properly before anyone touches anything else.

Inspired by a real outage: Summary of the Amazon S3 Service Disruption in the Northern Virginia (US-EAST-1) Region (Amazon Web Services, 2017). Names and details are fictionalized.

The fix is yours to find: hints, the recovery checks and the debrief unlock inside the incident.

How it plays

  1. 01

    Get paged

    The alert fires and the clock starts. Read the page and the briefing.

  2. 02

    Investigate

    Work in a simulated shell with realistic output: logs, configs, services.

  3. 03

    Fix it

    Change the system the way you would in production. Hints are there if you get stuck.

  4. 04

    Prove it

    Automated checks verify the recovery, then the debrief explains what happened.

All 61 incidents
Kubernetes Intermediate

API Pods Restart Across the Rollout

Every api replica is in CrashLoopBackOff and the Deployment has shown 0 of 3 available for ten minutes.

+220 XP35 minStage 4/4

Kubernetes Intermediate

Webapp Pods Never Become Available

The migrated webapp has shown 0 of 1 available replicas from the moment it was applied to the new cluster four minutes ago.

+190 XP30 min

Kubernetes Beginner

Order Scoring Pods Keep Restarting

Both order-scoring worker pods die within seconds of every start, and the orders.score backlog keeps climbing.

+160 XP20 minStage 3/5

Your pager is ready.

Free, instant, and it works on your phone. No signup: start as a guest and save your progress later.

Take this incident no signup Get it on Google Play