How to compare AI models on ten pieces of your own work

The most useful comparison between two AI systems is usually not a public leaderboard. It is a small, repeatable set of work that matters to the person making the choice: the files they actually upload, the sources they need checked, the artifact they need returned and the errors they cannot afford. Ten well-designed tasks can reveal more about practical fit than a hundred vague impressions.

This does not require a research laboratory. Anthropic defines an evaluation as a task with explicit inputs and success criteria, run through a harness and graded from both the transcript and the final outcome. OpenAI’s evaluation tools likewise organise test data and grading criteria, and can run external models. The same structure can be reproduced manually in a spreadsheet when the goal is choosing a subscription rather than shipping a platform.[1][2][3]

A ten-task evaluation can remain small and rigorous

• Choose ten representative jobs, including routine work, demanding work and known failure cases.
• Run multiple trials when output varies, rather than treating one success or failure as definitive.
• Attach an external standard: a known answer, source set, test, schema or expert judgement.
• Record the outcome as accepted, repairable or failed, and grade the real artifact or external state.
• Measure elapsed time, retries, correction time, allowance and cost.
• Read several complete traces to catch unfair grading, shortcuts or valid alternative paths.

Choose tasks that expose the whole product

A useful set might include a current-news article with citations, a long-file synthesis, a structured JSON refresh that must preserve IDs, a coding change that must pass tests, a spreadsheet with a known total, a presentation or document export, an image or PDF ingestion task, a multi-step web research question, an instruction-following stress test and a handoff that must preserve earlier decisions. The exact list should mirror the work the user expects to do.

This is where file support and interface design enter the evaluation. A powerful model that cannot ingest the supplied ZIP, preserve a spreadsheet formula or return an importable file may be the wrong product for that job. Conversely, a polished interface that produces an attractive but incorrect artifact should not win because it was easier to start.

Grade the outcome, then inspect the path

Anthropic recommends grading what the agent produced rather than forcing one exact tool sequence, because a capable system may find a valid approach the evaluator did not anticipate. It also recommends reading transcripts to confirm that graders are fair and the environment is not responsible for the failure. A model that says “published successfully” should not pass unless the expected record or artifact actually exists.[1]

The environment should remain stable. Use the same source bundle, task wording, tool permissions, time limit and acceptance criteria. Clear leftover state between trials. When comparing consumer products, record the selected model and effort because defaults can differ. If one product has web access or file tools and the other does not, that is a valid product comparison—but it should not be mislabeled as a pure comparison of model weights.

A worked example shows why correction time matters

Consider an illustrative ten-task comparison. Model A produces eight first-pass acceptances and needs 30 minutes of correction across the remaining work. Model B produces six first-pass acceptances and needs 100 minutes of correction. At an assumed review value of £30 an hour, A creates £15 of correction work and B creates £50. These invented figures demonstrate the calculation; they are not measurements of named systems.

Do not misuse benchmark “time horizons”

METR’s task-completion time horizon is the human difficulty of tasks that a model can complete with a stated success probability, not the number of minutes the AI is allowed to think. A four-hour horizon means performance on tasks that take skilled humans around four hours under the benchmark’s conditions; it does not predict that the AI will run for four hours or finish the user’s particular four-hour job.[5]

NIST’s ARIA programme similarly combines model testing, red teaming and field testing because capability in a controlled test does not settle real-world impact. For a personal evaluation, the analogue is simple: use a known test, inspect failure modes, then try the workflow in its actual environment before making the model your default.[4]

A ten-task suite will not establish which model is universally best. It can establish something more useful: which current product, plan and configuration fits the work being purchased. Keep the task bank and rerun it after major model or interface changes. That turns model choice from an anecdote into a small operating system for evidence.

Report an error or suggest a correction