A 13-million-token AI research run needs more than a success score. Transect lets evaluators inspect the path

A language-model agent can finish a long research assignment, hand over a manuscript and receive a favourable score. That result says little about how the work unfolded. Did it examine alternatives? Did it generate a testable hypothesis? Which tools failed before it changed direction? Which parts were delegated to other agents? As evaluations involve longer runs and larger teams of AI systems, those questions become harder to answer from a single final output. The UK AI Security Institute has released Transect, an open-source package intended to make the recorded path of an agent's work inspectable.[1]

Transect is built on the Inspect Scout evaluation ecosystem. Its input is an existing transcript of the agent's interaction with tools, people and other agents. Evaluators supply task context and behavioural categories appropriate to that family of tests, while configuring models used to judge sections of the transcript. The software produces a navigable report that aligns model-generated activity labels, token use, recorded events and sub-agent activity with the turns where they occurred. The premise is not that a language model judge automatically knows what another agent intended; it is that a useful interpretation should remain connected to the original evidence.[1]

A timeline with receipts, not just a summary

When an evaluator sees a long interval labelled as coding, experiment design or manuscript writing, they can inspect the underlying passages. They can also examine recorded events, such as a human intervention or the launch of a sub-agent, near the time the label was assigned. Results can be exported as structured tables. Those capabilities may sound mundane, but they address a real problem: a fluent narrative written by a summarising model can accidentally conceal transitions, merge distinct activities or imply causal links that are absent from the transcript.[2]

The researchers illustrate the method using an AI research-and-development evaluation that generated almost 13 million tokens. Their interpretation identified phases of operational work and manuscript production, with little sustained evidence of hypothesis generation in the recorded run. That observation is not proof that the system is scientifically incapable. It is a finding about the trace available from this particular case under specified behavioural categories. It illustrates why the quality of a research process cannot be established simply by measuring the length or apparent polish of the final manuscript.[1]

Multi-agent collaboration creates a second audit problem

The Institute's case study also examines shared documents and the interactions of delegated agents. It follows writes and reads of work products such as plans, code and experiment drafts, allowing reviewers to investigate whether research tasks were handed between agents. But the authors caution that reading a file does not demonstrate that the contents affected a later decision. A timeline based on turns likewise does not establish that every activity happened continuously or simultaneously in clock time. Such distinctions prevent activity visualisations from becoming new sources of false certainty.[2]

There is an important line between observability and surveillance of an inner mind. Transect can organise recorded actions, visible tool calls and model-generated interpretations of them. It cannot reveal hidden reasoning that was never logged, authenticate an event that was not recorded or infer intent beyond the evidence. Similarly, a label such as experiment design depends on category definitions and the behaviour of the judging model. Repeated model judgments may disagree; agreement does not guarantee a correct classification. The report's value increases when evaluators can inspect those uncertainties rather than treating its labels as facts.[1]

Why this matters for consequential deployments

For a company testing an AI software-maintenance agent, the same discipline could separate a successful code change from an unsafe route to that result. A system might pass unit tests while installing an unexpected dependency or bypassing a required review. The final test score alone would miss how the outcome was achieved. In an evaluation environment with reliable logging, linked transcripts would let a reviewer investigate the sequence of actions and the point where expectations diverged. That is an illustrative use case, not a claim that the Institute independently tested a company's production system.[1]

Reproducibility also requires more than saving a dashboard screenshot. The Institute says reviewers should preserve the underlying transcript, chosen behavioural vocabulary, analysis configuration, software version and model-judgment records. Some model-generated labels can vary when an evaluation is rerun, even with similar settings. Retaining the earlier outputs makes it possible to compare interpretations and identify discrepancies. That is particularly important when a safety finding is contested or used to support a release decision affecting other people.[2]

The challenge will intensify as agents use greater compute budgets and long-running workflows. Human specialists cannot read every token of a massive evaluation, yet they should not be forced to trust a second opaque system that condenses the evidence. Transect addresses that design tension by using machine assistance to organise a transcript while preserving routes back to the original record. The reported contribution is an auditing method, not a blanket guarantee that autonomous agents are now safe to deploy. The strongest principle is procedural: a claim about what an AI did should be reproducible from what it actually recorded.[1]

Report an error or suggest a correction