CloudOpsBench opens 45 AWS tasks that judge what an AI agent actually changes

Telling an AI agent to delete an abandoned cloud project sounds easy until the project shares a storage bucket, notification service or access role with a live business. The question is not simply whether the agent can execute the deletion command. It is whether it can identify everything that belongs to the retired environment, remove only those resources, preserve dependent services and leave behind a verifiable final state. CloudOpsBench, a public task suite hosted on GitHub, turns these ambiguities into a set of 45 AWS operational challenges.[1]

The repository covers decommissions, data migrations, access-policy repairs, alarm routing, container security and broken event workflows. One task asks an agent to retire a customer portal without damaging a billing workload. Another demands a repair to a compliance evidence pipeline that has silently stopped recording events. Other scenarios require restricting an external vendor's access without disrupting legitimate activity. The interest lies in the constraints: the quickest command can be the wrong command, and a reassuring natural-language summary cannot compensate for an unchanged or damaged cloud account.[1]

Checking the resulting cloud state

Each scenario includes an instruction, environment setup, executable checks and a reference solution. The documented files include instruction.md, task.toml, setup.sh, check.py and solve.sh. A runner prepares the state, gives the agent its objective, then invokes the unchanged grader. Under the evaluation contract, grader exit code zero gives a pass and code one a failure; setup errors or other invalid grading outcomes are reported separately. One subtlety matters: if a grader catches an infrastructure error and explicitly returns a failure, the wrapper cannot infer that the result should have been excluded. The grader's own behaviour is part of the evaluation.[1][2]

A useful example is cleaning up a temporary analytics workload. A superficial test might check that one named bucket disappeared. A stronger task checks every resource the workload created, including query results in shared storage, and confirms that other teams' data remains. The benchmark's instruction that tests should not be rewritten for the emulator is an attempt to prevent test-specific success: an agent should satisfy the same operational requirements regardless of where the test runs.[1]

The source design is more interesting than a hypothetical leaderboard. By releasing reference solutions and graders alongside tasks, the maintainers give independent auditors a way to investigate what success means. A reviewer can check whether the reference solution really preserves the protected workload, whether a task uses leaked hidden identifiers or whether the grading conditions reward shortcuts. For complex cloud work, this kind of transparent contract is itself an important contribution. It is also a potential attack surface: a benchmark that exposes its answer and assertions cannot assume a model has not memorized them. Evaluation teams should therefore maintain separate, genuinely unseen operational cases.[1]

A public task library and an execution contract

The repository defines tasks that can run against the Vera AWS emulator or an isolated real AWS account while preserving the same task and grader semantics. Users supply the execution harness and backend configuration. Emulator examples use dummy credentials; a real AWS environment requires its own scoped credentials and isolation. The release distinguishes packaging checks on ten curated exports from full Docker/Harbor lifecycle execution and adversarial isolation checks, which remain release gates. It has not published a comparative 45-task scoreboard for named models.[1][2]

The evaluation documentation specifies how such a scoreboard should be assembled. A pass-at-one result counts successful valid attempts, while excluded infrastructure errors are disclosed separately. Repeated trials should be averaged rather than reduced to the best run. That makes the denominator as important as the headline percentage: excluding difficult but valid failures would inflate the apparent success rate, while counting a failed test setup as an agent mistake would conflate two different problems.[2]

For cloud teams, the suite suggests a test plan more useful than asking whether an agent can write infrastructure code. Seed a realistic failure; give the agent only scoped permissions; verify the postcondition independently; inspect the sequence of tool calls; and test how it handles partial completion. Include tasks in which the correct outcome is to stop, seek approval or refuse to remove a shared resource. Track time and operational cost as secondary metrics, but never let an inexpensive incorrect deletion count as efficiency.[1]

CloudOpsBench makes a difficult operational question concrete: after an agent has finished, what exactly must be true of the account? Publishing the setup, grader and reference solution lets others inspect that answer, challenge weak assertions and reproduce a task under their own runner. Comparative scores will add another layer of evidence. The immediate contribution is a collection of explicit, testable operational requirements from which those comparisons can be built.[1]

Report an error or suggest a correction