An observability agent may correctly identify the root cause of an outage and still be the wrong component to change production. AWS’s reference workflow keeps AWS DevOps Agent in observe-and-report mode, then sends its investigation into a separate orchestrator that can inspect approved configurations, propose a remediation and wait for a person before any mutating tool runs.[1]
That architecture treats diagnosis and authority as different capabilities. A model may be permitted to read logs, correlate symptoms and recommend a change without receiving a general credential that can modify infrastructure. The boundary reduces the damage from a mistaken root cause, a compromised prompt or an overly broad tool call.
The reference workflow
| Measure | Value | Context |
|---|---|---|
| Investigation input | Symptoms, findings and root-cause analysis | AWS DevOps Agent emits an event when the investigation completes.Source 1 |
| Tool surface | Curated Lambda allowlist | The model can select only purpose-built approved operations.Source 1 |
| Read-only actions | Automatic | Configuration retrieval can run without interrupting the on-call engineer.Source 1 |
| Mutating actions | Human approval required | The durable function suspends until an explicit callback is received.Source 1 |
| Approval wait | Minutes, hours or days | Execution checkpoints and pauses without consuming compute while waiting.Source 1 |
A proposed fix is an executable claim
In the demonstration, a Lambda function has a three-second timeout. The agent retrieves the current configuration, proposes changing the timeout to 30 seconds and produces the exact function name and parameter. Increasing the limit by a factor of ten may remove the observed timeout, but it does not prove that the underlying workload is healthy. The extra time could hide a performance regression or increase cost.[1]
The approver therefore needs more than a sentence saying ‘increase the timeout’. They need the exact resource, current value, proposed value, reason, likely side effects and the evidence that will be checked after execution. Approval should be closer to reviewing a small change set than clicking yes on an explanation.
Four levels of remediation authority
| Level | Agent may do | Control needed |
|---|---|---|
| Diagnose | Read telemetry and propose a root cause | Source-linked investigation recordSource 1 |
| Inspect | Run approved read-only tools | Least-privilege read access and complete logsSource 1 |
| Prepare | Select a bounded mutating tool and populate parameters | Schema validation, policy checks and impact previewSource 1 |
| Execute | Change production after approval | Named approver, rollback, verification and blast-radius limitsSource 1 |
The allowlist is more important than the prompt
The orchestrator exposes purpose-built tools such as reading a function configuration or updating a specific policy statement. This is safer than handing the model an unrestricted shell or a cloud administrator credential. A prompt can ask the model to be careful; an allowlist removes entire classes of action from its reach.[1]
The tool itself should also validate parameters. A timeout update can enforce an approved range. A policy editor can reject wildcard resources. A deployment tool can require a known artifact and environment. Security should narrow what the function is capable of doing even if the model selects the wrong operation.
Every automated fix needs a second measurement
After the change, the system should repeat the failing transaction, inspect the relevant health indicators and confirm that no neighbouring metric deteriorated. ‘API call succeeded’ only proves the configuration was written. It does not prove the customer-visible incident ended.
- Capture the incident evidence and the agent’s confidence before proposing a fix.
- Present exact resource identifiers and parameters to the approver.
- State the rollback operation before the forward change executes.
- Run a bounded verification test and compare the original symptoms.
- Escalate when the result is inconclusive instead of chaining more changes automatically.
Sources & further reading
- Automate remediation post AWS DevOps Agent investigation AWS Artificial Intelligence Blog · 7 Oct 2026 · Accessed 9 Oct 2026
- AWS sample: automate remediation after a DevOps Agent investigation AWS Samples on GitHub · Accessed 9 Oct 2026