An AI can read a document without being allowed to obey it

A supplier’s document can tell an assistant the delivery date. It should not be able to tell that assistant to send the customer’s private files somewhere else. Both instructions may be expressed in ordinary language. Only one belongs to the task the user authorised. This is the central distinction a tool-using AI system must preserve.

OWASP describes indirect prompt injection as external content influencing an application’s model in unintended ways. The consequences depend on the powers surrounding that model. Retrieval and fine-tuning can improve answers without eliminating this class of risk. A sourced answer is not automatically a safe action.[2]

The dangerous transition is from reading to authority

Consider a benign test document containing delivery details and an irrelevant instruction to change an unrelated setting. A secure workflow can summarise the delivery details without treating the other sentence as a legitimate command. The document is a source of claims about the world. It is not the owner of the assistant.

This distinction is easy to describe and harder to maintain across a long workflow. A retrieved passage may become a summary, the summary may become part of a plan, and the plan may trigger a tool. At each transformation, the application must avoid laundering untrusted content into authority. Labelling an input helps, but the downstream execution layer must still check what the plan is allowed to do.

Permission is also about where information goes

Google Research’s 5 October workshop report uses contextual integrity to examine these problems: privacy depends on appropriate information flows in a particular setting, not merely whether data is secret. It proposes research across system, model and user layers, including contextual policy mechanisms. The report is a research agenda, not an announced universally effective security product.[1]

The contextual idea becomes concrete in a hypothetical expense workflow. The assistant may need to read a receipt, extract the total and submit that total to an internal system. It does not follow that the full receipt, the traveller’s other documents or the conversation history should be sent to every tool used along the way. Authority to perform the task is narrower than authority to move all available information.

The earlier Contextual Agent Security work proposes generating purpose-specific policies that can be inspected and used to govern an agent. Its useful question is not simply whether a tool is allowed in general, but whether a particular use of it fits the user’s task and context. Such policies still require trustworthy inputs and enforcement; a policy-generating model is not automatically immune to the problems of the model it supervises.[4]

A compromised plan should meet a smaller execution boundary

Anthropic’s sandboxing design illustrates a complementary engineering approach: enforce filesystem and network boundaries around execution. These controls can reduce what a manipulated process is able to read, change or contact. They do not establish that every statement produced inside the boundary is true, nor that every allowed destination is appropriate for every document.[3]

That is the difference between prevention and containment. A model may still form a bad plan. A separate mechanism can refuse to carry it out because the file, recipient or action lies outside the authorised scope. The strength of this design depends on the scope actually being narrow; a sandbox with broad sensitive-data access and unrestricted outbound communication would provide a much weaker containment story.

OWASP recommends least privilege, deterministic output validation, separation of untrusted content and human approval for higher-risk operations. None should be reduced to a ceremonial checklist. The operational test is whether the restricted action remains unavailable even when the model confidently proposes it.[2]

Why a confirmation button can be insufficient

A user cannot meaningfully approve “continue” without knowing what will continue. For a consequential transfer, our suggested confirmation should name the destination, describe the information being sent and show the action’s effect. The approval should be tied to those parameters, not treated as blanket permission for whatever the agent chooses next.

Too many prompts create another problem. If every routine step looks equally urgent, the important decision can disappear among trivial requests. Anthropic’s engineering discussion explicitly identifies approval fatigue as a motivation for bounded execution. A carefully scoped environment can allow ordinary work to proceed while reserving interruption for a genuine boundary crossing.[3]

There is an equally important usability test: can a person recover after a refusal? The system should explain the blocked action in ordinary terms and offer a bounded way to complete the legitimate task. Otherwise users may respond by granting broad permissions that erase the intended protection.

Trust comes from constrained authority, not confidence

For publishers and knowledge workers, the practical implication is reassuringly specific. An assistant does not need to distrust every document to remain useful. It needs to know that documents supply content, users supply task authority and the application enforces the permitted actions. Those roles should survive summarisation, tool use and delegation.

Provenance adds useful evidence, but not new authority. Just as a signed image is not a verified story, a document with a verified origin does not gain permission to redirect an assistant outside the user’s task.

A strong agent is therefore not one that never encounters hostile or confusing text. It is one whose useful capabilities remain bounded when that text affects its reasoning. The objective is to keep reading broad enough to be useful and acting narrow enough to be accountable—and to prove both through observable outcomes.

Report an error or suggest a correction