Defra’s AI assistant reads the whole toolkit with each question

Defra opened Ask the toolkit to the public on 27 October. It answers questions about the department’s AI guidance by sending all roughly 40 pages with each question to Claude Sonnet 4.6, running through Amazon Bedrock in London.[1]

That differs from the retrieval route described in AWS’s Knowledge Bases documentation. There, a search selects the source passages most relevant to a query, then a model generates an answer with citations to those passages. Selection determines which material reaches the answer-writing step.[2]

Imagine a staff guide in which a permission appears in one chapter and an exception in another. A system that selects only the passage closest to the question could leave the exception outside the answer’s evidence. Giving the model both chapters removes that particular omission. It still has to recognise the relationship between them.

For a compact handbook, the design question becomes whether the model can use a bounded collection well. As the collection grows, more input must be processed for every request; retrieval instead attempts to narrow that input before generation. There is no universal page-count threshold in this comparison: documents differ in length, and a question may need one paragraph or connections across several chapters.[2]

From the rulebook to an answer

Defra makes one model call, retrying only if the reply has the wrong structure. Answers carry links and quoted rules; the backend removes a quotation absent from its named page. The service has no browser or action-taking tools.[1]

Removing passage selection also removes a separate implementation choice. Builders do not have to decide how much of each page to put in a searchable chunk, how many chunks to return, or how to rank close matches. Those choices can be worthwhile for a large collection. In this small one, they would add machinery before the model reaches material that already fits.[2][3]

An AI-judged test on 19 October met four of six pass marks, missing its perfect-score targets for rule selection and grounding. Human checks also found uneven agreement with the judge.[1]

Anyone can ask without signing in; project-specific questions go to the team. The service cannot approve a project.[1]

For someone building a similar handbook assistant, the useful starting point is the collection itself: how much there is, how often rules depend on one another, and what an answer should help its reader do. Those choices determine whether selecting passages is a helpful economy or an extra place to lose the relevant context.

Report an error or suggest a correction