Challenge 6 · Autonomous Recovery and Cost-Aware Governance

Challenge 06 of 08 · Act IV: Autonomous & Cost

Run mode: Autonomous (bounded) · Governance: bounded blast radius, no per-action approval · Time: ~50 min

Stage: Foundation → Operations → Engineering → Autonomous → Major Incident

Situation. Every write so far has waited for your approval. Now a minor issue appears: the baggage service returns intermittent errors to a slice of traffic. Bags still move, no flight is grounded, and the fix is one you've already seen. That's exactly when it's safe to let the agent run autonomously while you supervise. Meanwhile Priya Nair from FinOps wants to know what autonomy costs before you make more of it.

Mission. Let the agent recover the baggage service autonomously within guardrails, design a cost-aware operating model for its Azure Agent Unit (AAU) usage, and arm a Sev1 response plan so the next major incident auto-triggers an investigation, before you reach Challenge 7.

Start this challenge

./scripts/start-challenge.ps1 6   # open the incident

Give it a couple of minutes. The errors only show up once enough traffic has hit the affected slice for the board to average it.

Tasks

1. Confirm the symptom. Verify the baggage errors in the Ops Center and telemetry so you know what "recovered" looks like.
2. Decide the bounds, then hand over. Judge whether this incident is safe to automate, switch to Autonomous mode, let the agent fix it end-to-end, then review the action log.
3. See where AAUs go. Inspect the agent's consumption by thread type and purpose.
4. Design a cost-aware model. Name concrete reductions without losing reliability or investigation quality.
5. Arm the Sev1 response plan. Connect Azure Monitor as an incident platform, then bind a Sev1 plan to the pre-provisioned aetherion-major-incident alert so the next major incident auto-triggers. Challenge 7 does not work without this — do it before you move on.

Suggested Azure SRE Agent prompt

Paste into the agent chat

The baggage service is returning intermittent errors to a slice of traffic while other requests succeed. Detect the fault, decide the sanctioned reversible fix, apply it autonomously, and report exactly what you changed and why it was safe to automate.

Success criteria

• The agent detects the baggage errors, executes a sanctioned fix autonomously, the service returns to healthy, and you can explain why it was safe to automate: small blast radius, a reversible and well-understood remedy, a service that is degraded rather than down, and a run mode you chose deliberately.
• You've reviewed real consumption data and produced a written, defensibly balanced operating model: the two largest consumers with their actual figures, at least three named levers each with a trade-off, and an explicit reliability floor. Cost-aware, not merely cheapest.
• A Sev1 major-incident response plan is active on the agent (bound to the pre-provisioned aetherion-major-incident alert), ready to auto-trigger the agent when the next major incident fires.
• check-challenge.ps1 6 passes.

Ground truth only

Don't invent AAU rates, prices, or savings figures. Use only values present in the official docs or your environment. Don't claim that stopping an agent ends all billing.

Verify your work

Run this when you're done. It grades the real end state, then walks your response plan setting by setting. Answer those honestly: a plan that looks right and never fires is the one failure that doesn't surface until Challenge 7 is already open.

./scripts/check-challenge.ps1 6

Hints

Make autonomy safe. Confirm the symptom first; you can't judge an autonomous recovery without knowing what healthy looks like.

What makes this one safe to automate isn't a permission setting. It's the shape of the incident: one degraded (not down) service, a remedy you can undo in seconds, and a blast radius you can describe in a sentence. Decide that before you switch mode, then switch to Autonomous and resist helping.

Cost model and the Sev1 plan. Optimize what you can measure: look at real consumption by thread type and purpose — consolidation, deleting unused pilots, model fit, and plan/schedule noise are the biggest levers. Aim for a balanced model, not the cheapest. Then wire the Sev1 response plan now, matching the pre-provisioned aetherion-major-incident alert's severity exactly, or you'll be wiring it by hand while the board is red in Challenge 7.

Stuck? Step-by-step for each task

Give each task a genuine attempt first, and skim the hints above. When you want the exact clicks, open the matching task below.

Task 1 · Confirm the symptom

• Check the baggage tile and telemetry (Application Insights failed requests) so you know the current error rate and what "recovered" means.
• Some requests succeed and some fail. Ask the agent what is different about the ones that fail — the answer is in what sits behind the Service, not in the Service itself.
• kubectl get pods -n aetherion -l app=baggage --show-labels shows the same thing directly, if you want to confirm the agent's account.

Task 2 · Decide the bounds, then hand over

• Write down the bounds first: what may the agent change, what must it never touch, and how would you undo it? That judgement is the governance here.
• Set the run mode to Autonomous (on the response plan / task), then let the agent detect → decide → fix on its own. Review the action log after.

\"Never delete\" applies to data, not to a bad revision

Aetherion's guardrails forbid deleting data resources: databases, caches, storage. Removing a workload revision that shouldn't be serving is an ordinary, reversible rollback, and it is the sanctioned fix here.

Task 3 · See where AAUs go

• In the agent, open Settings → Agent Consumption and read the breakdown by thread type (Chats, Incidents, Scheduled tasks, Triggers) and by thread.
• Write down the two largest consumers and their actual figures. Task 4 is only worth doing against real numbers, and "incidents cost more than chats" is a guess until you have looked.

Task 4 · Design a cost-aware model

Produce something a finance partner could actually review, not an essay. Ask the agent for it, then correct it against what you saw in Task 3:

Using my agent consumption figures, draft a cost-aware operating model as a table: each row a named lever, what it would save, and what it costs us in reliability, ownership, isolation or investigation quality. Add one line stating the floor we will not go below. Use only the figures I give you and do not invent numbers.

It has to contain, at minimum:

• Three named levers with a real figure attached to each — for example consolidating workloads under one agent, deleting unused pilot agents, matching the model to the task, or trimming noisy response plans and schedules.
• A reliability floor stated explicitly: the thing you will not trade away to save AAUs. Cheapest is not the target.

Keep it. Challenge 8 asks you to defend it to a director, and check-challenge.ps1 6 asks you about both parts.

Task 5 · Arm the Sev1 response plan

Connect an incident platform first. Response plans do not exist until you do, and this catches people out.

1. Incidents in the left-hand menu → Triggers + response plans.
2. Connect an incident platform → Azure Monitor → Save. It configures itself from your agent's scope, so there is nothing to fill in. You'll see Azure Monitor is connected in the top right when it's done.
3. Delete the quickstart plan. Connecting a platform automatically creates a default quickstart response plan. Leave it there and it runs alongside yours, which can route the incident to the wrong agent or process it twice. Select it and delete it before you continue.
4. + Create a response plan.

Fill it in as follows:

Field	Value
Incident response plan name	aetherion-major-incident-sev1
Severity	Sev1
Title contains / does not contain	leave empty
Response subagent	leave empty
Agent autonomy level	Autonomous
Alert reinvestigation cooldown	disable it

Then Next to see Incidents preview, and create.

Severity is the whole match. The environment pre-provisions aetherion-major-incident at Sev1, so a plan filtered to anything else catches nothing and a broader plan fires on everything. Leave the title filters empty. They are an extra way to miss the alert, not an extra safeguard.

Why not route it to your AKS specialist?

The Response subagent dropdown is populated by the subagent you built in Challenge 5, so it is tempting to select aks-triage here. Don't.

Challenge 7's incident spans AKS, API Management and PostgreSQL. Handing the auto-triggered investigation to an AKS-only specialist scopes it to one tier of a four-tier incident. Leave it empty so the main agent commands the whole board, and delegate to the specialist within the incident instead.

Deciding not to use a tool you just built is a real operational judgement, and this is the moment to make it.

Turn the cooldown off for this workshop

Alert reinvestigation cooldown defaults to enabled at 3 hours: the plan skips reinvestigation if the same plan already started one inside that window. Sensible in production, wrong here. Re-run Challenge 7 and the plan silently will not fire the second time.

Choose Autonomous, and understand what you are choosing

Selecting Autonomous shows an information icon. Open it and read the Autonomous mode acknowledgment before you accept: it covers the agent's boundaries, the model's limitations, and the fact that scoping its access and reviewing its outcomes remain your responsibility.

In production you would start at Review and only promote a plan to Autonomous once you trust its tool selection. You are promoting it after one supervised autonomous recovery in this challenge, which is faster than you should move on a real platform. That is a deliberate trade so Challenge 7 can show you what unattended response looks like.

Autonomy level is per plan, separate from the agent's own run mode. At Autonomous, the auto-triggered investigation can mitigate without waiting for you. It will not fix everything: it acts where the fix is unambiguous and reversible, and leaves the judgement calls. Challenge 7 is built around that split, and your first job there is to audit what it decided to do alone.

Do this now. Challenge 7 depends on it auto-triggering, and check-challenge.ps1 6 will ask you about each of the six settings above one at a time. Every one of them is a way to end up with a plan that is listed, looks correct, and quietly matches nothing.

Reference

• Run modes
• Pricing and billing
• Incident response plans

Up next: the final major incident

Your Sev1 plan is armed at Autonomous. When everything breaks at once, the agent triggers itself and starts working the incident before anyone pages you. You arrive second, and your first job is to audit what it already did.

Proceed to Challenge 7 · Final Major Incident →

---