Challenge 7 · Final Incident: Restore Global Check-In Before Peak Departure

Challenge 07 of 08 · Act V: Major Incident

Run mode: Review + bounded automation · Governance: you decide what is approved and what is automated · Time: ~60 min

Stage: Foundation → Operations → Engineering → Autonomous → Major Incident

Before you start: is your Sev1 plan armed?

This challenge depends on the response plan you built in Challenge 6. Open Incidents → Triggers + response plans and confirm one plan shows Status: On and Severity: Sev1. If it doesn't, fix that first — the agent will never pick the incident up, and Task 1 has nothing to hand over.

Situation. Minutes before peak departures, several services fail together under a passenger surge: the flight board is dark, crew scheduling is timing out, check-in has slowed to a crawl, and the API front door is failing legitimate partner and mobile traffic. The risk gauge is pinned. Everything you built today is now in play.

You are not first on scene. The Sev1 plan you armed in Challenge 6 matched the alert before anyone paged you, and the agent has been working the incident unattended. Your first job is not to fix something. It is to find out what has already been changed in your absence.

Incident timeline

Time	Event
18:07	A change rolls out to flight-ops
18:12	Check-in / booking latency begins to climb
18:14	Crew scheduling starts timing out under load
18:17	The live flight board goes dark for all stations
18:21	The API front door starts failing partner traffic
18:23	aetherion-major-incident fires at Sev1 and auto-triggers the agent
18:25	Major incident declared. You are incident commander
18:26	The agent is already acting on its own in a thread it opened itself

Mission. Take handover from the agent, then bring Aetherion AirOps back to full health before peak departure: verify what it changed without you, triage the rest by business tier, remediate within the runbook guardrails, and verify recovery service by service.

Start this challenge

./scripts/start-challenge.ps1 7   # open the incident

Four faults land at once and the board takes two or three minutes to reflect all of them. Don't start triaging from a half-drawn picture.

Tasks

1. Take handover from the agent. Find the thread the Sev1 alert opened by itself, read what the agent has already changed unattended, and verify it independently before you trust it.
2. Read the whole board. Order the remaining work by business impact (situational awareness and legal-to-fly first), not the loudest alert.
3. Check the agent's memory. Ask whether a similar incident has happened and let session insights surface the earlier RCA.
4. Delegate and recover. Hand AKS triage to your specialist subagent and reuse your crew recovery skill, keeping every action governed.
5. Localize the front door. Use the direct-vs-APIM comparison and treat the policy change as customer-facing.
6. Verify every service. Confirm each fix from telemetry, then the whole platform back to green.

Why Sev1?

The environment pre-provisions a fixed Sev1 Azure Monitor alert (aetherion-major-incident, on Application Insights failed requests). A response plan matches incoming alerts by severity, so your filter must be Sev1 to catch it; a broader filter fires on everything, a mismatched one catches nothing.

This one is graded. check-challenge.ps1 7 marks the platform's end state and confirms a Sev1 major-incident alert actually fired since you started the challenge. Driving the whole incident by hand is not enough on its own.

Suggested Azure SRE Agent prompt

Paste into the [Sev1] aetherion-major-incident thread

You picked this incident up before I did. List every action you have already taken without my approval, and how I can verify each one. Then tell me what is still broken, and what you recommend I deal with first.

Paste it into the right thread

This prompt only works inside the thread the response plan opened by itself. Started in a fresh chat it has no incident to refer to, so it answers about whatever it worked on last — validated: pasted into a new thread it reported on the previous challenge's baggage canary and declared traffic healthy while the flight board was 100% down. Open the [Sev1] aetherion-major-incident thread first (Task 1), then ask.

The second half is the part you own. It will give you an order; your job is to decide whether it's the right one for an airline at peak departure.

Success criteria

• You can state exactly what the agent changed without your approval, and you verified it from the cluster rather than from its summary.
• All remaining failing services are triaged by priority and restored with sanctioned, reversible actions.
• The API front door serves legitimate traffic again, and every fix is verified from telemetry/health before closing.
• The platform is healthy and check-challenge.ps1 7 passes.

Verify your work

Run this when you're done. It grades the real end state:

./scripts/check-challenge.ps1 7

Hints

Triage, don't firefight. Don't fix the first red tile you see; read the whole board and order by business tier. You've solved every one of these failure classes already: delegate AKS triage to your specialist and apply your crew recovery skill.

Is it the service, or the front door? If the backend is healthy when you hit it directly but clients still fail, the problem is at the API front door, not the services. Treat the policy change as customer-facing.

Expect the board to get worse once the front door is fixed: with traffic flowing again, the degradations further back become visible. That is normal in a layered incident. Restoring the edge doesn't create new faults, it reveals the ones the outage was masking.

How an experienced incident commander would have scoped it. If the agent's account is scattered, this is how someone who has run a major incident would ask for it:

Read the whole Aetherion board and give me an incident-command triage: list every service still failing, order remediation by business tier (situational awareness and legal-to-fly first), and for each name the sanctioned, reversible fix. Have we seen a similar crew-scheduling / check-in incident before? Pull the earlier RCA from session memory.

Two things in there are doing the work: order by business tier, which stops you fixing whatever is loudest, and pull the earlier RCA from memory, which is the difference between solving crew scheduling for the first time and the second.

Stuck? Step-by-step for each task

Give each task a genuine attempt first, and skim the hints above. When you want the exact clicks, open the matching task below.

Task 1 · Take handover from the agent

The agent started before you did. Treat this exactly like taking handover from a colleague who has been on the incident for ten minutes.

Find the thread it opened by itself. In the agent's chat list, look for a thread whose title starts with [Sev1] aetherion-major-incident. You did not create it. The response plan did, off the alert.

Read what it already changed. Ask it directly, in that thread:

You picked this incident up before I did. List every action you have already taken without my approval, what each one changed, and how I can verify it. Then list what you deliberately did not act on, and why.

Verify it yourself. Do not take its word for it. The whole point of autonomy is that you audit the outcome, not the intention:

kubectl -n aetherion rollout history deployment/flight-ops
kubectl -n aetherion get deploy flight-ops -o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}'

Expect it to have acted on some tiers and not others. It moves where the fix is unambiguous and reversible, and holds where the call is a judgement. Those held tiers are your work.

If no thread was auto-created

The plan did not match. Open Incidents → Triggers + response plans and confirm your Sev1 plan shows Status: On and Severity: Sev1, that Subagent name reads Set up (none bound), and that no leftover quickstart plan is competing with it. Fix it, then re-run the start command so the alert fires again. You can still command the incident by hand, but the alert has to fire for this challenge to pass.

Task 2 · Read the whole board

• Open the Operations Center and list every failing service. Order remediation by business tier: situational awareness (flight board) and legal-to-fly (crew) first, not by whichever alert is loudest.

The board cannot see your front door

The Operations Center polls each service directly, inside the cluster. It has no view of API Management. So the board can read fully green while every partner and mobile customer is getting a 500 at the edge, which is exactly what happens in this incident.

Validated: an agent told to "read the whole board" recovered all four in-cluster faults, declared the incident stable, and never mentioned the front door once. It was not wrong about the board. The board was incomplete.

Treat the internal board and the customer-facing path as two separate questions, and always ask the second one. That is Task 5.

Task 3 · Check the agent's memory

Ask it directly, in the same thread as the investigation:

Have you investigated a crew-scheduling or check-in slowdown on this platform before? Recall what the root cause was and which fix worked.

If you saved a hypothesis in Challenge 2 or grounded the runbooks in Challenge 4, session insights should surface the earlier RCA and the fix. That is the point of the exercise: the second incident should cost less than the first.

Task 4 · Delegate and recover

Name both the subagent and the skill explicitly. Neither fires on its own.

For the AKS tier:

Delegate this to your aks-triage subagent: which workloads in the aetherion namespace are unhealthy, and why?

Expect it to investigate and then hand back rather than fix. You scoped it without RunKubectlWriteCommand in Challenge 5, so it physically cannot change the cluster. It reports; you and the main agent act on it. That split is deliberate, and under incident pressure it is worth noticing that you do not have to trust a prompt to keep it in its lane.

For the crew tier:

Use the crew-query-path-recovery skill to restore crew scheduling.

Crew is failing the same way it did in Challenge 4, so the skill carries. Asking the agent to "act as" a specialist does not invoke the subagent, it only adopts the persona.

• Keep actions governed and follow the runbooks: fix the layer that is actually saturated, never delete the database.

Time the crew tier specifically

Note the minutes from starting on crew scheduling to the point it is healthy again. This is the same failure class you worked cold in Challenge 2 and diagnosed in Challenge 4 — but now the agent has your runbooks, a skill written for exactly this, and a memory of the earlier incident.

Compare it with the number you wrote down in Challenge 2. Whatever the gap is, that is the finding you take to Challenge 8. If it is smaller than you expected, that is worth reporting honestly too.

Task 5 · Localize the front door

• Compare backend health directly (http://<gateway-ip>/api/status) against the same call through APIM. If direct returns 200 but APIM fails, the fault is the edge policy, not the service; treat the change as customer-facing.

Run the comparison from your machine, not the agent

The agent's sandbox egress can block the public gateway IP and *.azure-api.net, so an external probe it runs itself comes back as a proxy-generated 403 before the request ever reaches Azure. Validated: the agent repaired the policy correctly and then reported it could not confirm the fix externally.

Do the direct-vs-APIM comparison yourself, and treat the agent's "external check failed" as a tooling limit rather than evidence the fix did not land. Confirm from the policy XML plus your own call:

$st = Get-Content ./scripts/.env.aetherion.json -Raw | ConvertFrom-Json
(Invoke-WebRequest "http://$($st.gatewayIp)/api/status").StatusCode
(Invoke-WebRequest "$($st.apimGatewayUrl)/aetherion/api/status" `
   -Headers @{ 'Ocp-Apim-Subscription-Key' = $st.apimSubscriptionKey }).StatusCode

Why it can fix this at all

Everywhere else today the agent has held write access to Kubernetes and nothing else. It can repair the front door because its identity also holds API Management Service Contributor, scoped to the API Management service alone rather than the resource group.

Narrow enough to fix this, too narrow to touch anything else. That is what least privilege looks like when it is actually load-bearing: had it been scoped to the resource group, the same repair would have carried the authority to change every resource in the environment.

Task 6 · Verify every service

• Confirm each fix from telemetry / health before closing, then check the whole platform is green and /api/flights responds.

Reference

• Automate incident response
• Incident response plans
• Review and approve mitigations

Up next: brief airline leadership

The incident is technically closed; now leadership needs it closed formally.

Proceed to Challenge 8 · Leadership Briefing →

---