Top AI engineering agent cleared every hard check in 41.3% of deliverables

The strongest configuration in a new physical-engineering benchmark met 91% of individual constraints on its co-authored tasks, yet only 41.3% of deliverables cleared every hard safety and feasibility check. Google's AI and Infrastructure researchers and Mercor released Physical-Engineering's Last Exam, or PLE, alongside an open competition on November 3. The contrast captures a practical problem: a design can satisfy most of a brief while one failed requirement makes the completed work unacceptable.[1][2]

The benchmark extends beyond question-answer tests. It asks agents to inspect project folders containing emails, calculations, drawings and changing documentation, then produce work a licensed engineering reviewer could examine. There are 30 mechanical tasks, 27 electrical, 18 civil and 10 structural. The complete set contains 5,170 files, an average of roughly 61 per task. Typical materials include PDFs, spreadsheets, two-dimensional CAD, building-information models and earlier design revisions. In practice, the challenge is often knowing which specification controls the final design, not simply calculating a number correctly.[1]

A wrong part can invalidate an otherwise good package

Engineering is full of non-negotiable constraints. A design may be cheap, elegant and mostly accurate, yet fail because its load rating is too low or an electrical installation violates a code requirement. PLE therefore applies hard gates before awarding objective credit for cost or performance. Its headline 41.3% complete-pass figure refers to deliverable items in co-authored tasks, not to every piece of engineering made by AI worldwide. A score for satisfying 91% of checks cannot be converted into a 91% probability that a building or circuit is safe.[1]

The tasks have two origins. Practising engineers wrote 44 of them directly. Another 41 began as expert briefs, with agents helping assemble project environments and grading procedures before engineering specialists validated the results and increased difficulty. These distinct creation methods matter because benchmark design can make tests unrealistically easy, overly hard or narrowly tailored to a particular tool. The authors made 29 tasks public and retained 56 for private scoring to limit the risk that an agent memorises published solutions.[1][2]

Fourteen frontier models were tested across seven agent harnesses, including Codex and Claude Code. In the private-set native-harness comparison, all models scored below 50%; the top configuration reached a mean of 48.3%. This is not the same measurement as the percentage of individual technical constraints satisfied. Mercor identifies the leading configuration as Claude Opus 5.5 running with Claude Code, at an average of 27 million tokens and 63 minutes per task. Those figures indicate how resource-intensive the strongest runs were under the organisers' conditions; they are not a forecast for a customer's typical invoice.[1][2]

What the comparison actually tells a buyer

Google says model selection influenced benchmark performance substantially more than the choice of agent harness in its runs: the spread across models reached about 50 percentage points, compared with a median three-point difference between harnesses for the same model. Harnesses still affected cost and workflow. That result is specific to the tested models, configurations and tasks, and does not imply that orchestration is unimportant in production. A company evaluating an engineering assistant should examine its actual project files, regulatory requirements, review burden and output formats.[1]

The public release allows developers to test their systems on 29 available tasks and their graders, while submissions to the competition face hidden evaluation. The private set matters because an agent can be finely adjusted to public examples without becoming more competent on unfamiliar work. The benchmark also tests finished artefacts, rather than awarding high marks solely for a plausible explanation. Someone reviewing an agent's output can therefore ask whether the wiring diagram, bill of materials and accompanying calculations agree with one another.[1][2]

For engineering teams, these results make the review burden central to an assistant's value. Drafting drawings and calculations faster saves little if a qualified reviewer must spend longer finding an incompatible component or an overlooked constraint. PLE's finished-artefact scoring lets evaluators ask a more useful question: how often does a complete design survive every necessary check, and how much work is needed to correct the rest?[1][2]

Report an error or suggest a correction