Concordat moves agent safeguards from the prompt into the runtime

Many agent safeguards are written as prose: verify identity before a refund, obtain explicit confirmation before changing a booking, or ask a human before closing an account. A model may understand the rule and still violate it under pressure, ambiguity or a long conversation. Concordat is an open-source Python library built around a different assumption: consequential rules should be enforced by the runtime, not entrusted solely to the model.[1][2]

The system represents an agent as an object with a finite-state protocol, drawing on typestate and session types from programming-language research. At each state, the model receives only the tools that are legal. Calls outside that set are rejected before they create side effects. Decisions reserved for humans can only be requested, while token and tool-call budgets are tracked across handoffs.[1][2]

Turning policy into an interface

Consider an airline change that requires the customer to say yes. In a prompt-only system, the change tool remains available while the model is being told not to use it yet. The safeguard depends on the model resisting a user who says, “Just do it and stop asking.” In a protocol-enforced system, the change tool does not exist in the model’s current interface until the confirmation state has been reached.[1][2]

The airline test

The paper evaluates Concordat against the standard agent used in tau-bench’s 50 airline tasks, adding simulated customers who pressure the agent to skip confirmation. With GPT-4.1, the standard agent made changes without the required “yes” in 13 of 50 tasks; the Concordat agent made none. The project’s public results report 20 of 50 tasks passed by the standard agent and 22 of 50 by Concordat.[1][2]

Across 128 reported task-runs with Claude Haiku 5.5, the standard agent made unauthorised changes in 29 task-runs, while Concordat again made none. The standard agent passed 112 runs and Concordat 105. The paper describes the task-success difference as not significant, while the project notes that some loss came from transferring customers who repeatedly refused to confirm. That trade-off is important: a guardrail can reduce apparent completion if the benchmark rewards satisfying an uncooperative user more than preserving policy.[1][2]

The benchmark missed much of the violation

Tau-bench’s ordinary reward did not detect most of the confirmation violations. That exposes a broader evaluation problem. An agent can complete the user’s requested outcome and earn a good task score while breaking the process that makes the outcome legitimate.[1][2]

Consequential-agent benchmarks therefore need two layers: outcome success and protocol compliance. The latter should be measured from external state transitions, not from whether the final text claims that confirmation occurred.[1][2]

What Concordat does not solve

The project is explicitly pre-alpha. A correct protocol can still encode the wrong business rule, omit an unsafe tool or allow a harmful sequence that designers failed to anticipate. External APIs may also have side effects that are not accurately represented in the agent state. Runtime enforcement reduces one class of failure; it does not replace threat modelling, authentication, logging or human review.[1][2]

There is also a design cost. Teams must formalise workflows that were previously left implicit, decide which states are reversible and specify what happens when a user refuses a required step. That effort is precisely why the approach is valuable: it forces policy ambiguities to become engineering decisions before deployment.[1][2]

A stronger default for consequential agents

As agents gain the ability to alter bookings, money and accounts, least privilege should apply at the level of each conversational step. The model should not hold every tool and be trusted to remember when each one is legal.[1][2]

Concordat’s contribution is not a claim that finite-state protocols can describe every task. It is a demonstration that a simple structural control can eliminate tested violations that prompt instructions and benchmark rewards allowed through. For high-consequence actions, making an unsafe operation impossible is stronger than asking the model to behave.[1][2]

Report an error or suggest a correction