Ordinary task history lifts AI-agent safety scores by 9–52 percentage points

An AI assistant may behave differently when a risky instruction appears as the first thing in a session than when it arrives after a sequence of ordinary jobs. This is not a subtle question for developers of tool-using agents. A system might be asked to read documents, run searches or change account settings, and an attacker may try to slip an unsafe instruction into that workflow at any point. Researchers studying the timing of such attacks found a surprising pattern: their tested agents were especially vulnerable at the start.[2]

The paper, published at EMNLP in October 2026, calls the pattern a cold-start safety gap. Its SODA benchmark varies how many legitimate tasks precede an unsafe request, from none to 20, across seven models from four families. Table 1 reports increases of 8.9–52.1 percentage points between those endpoints. These are absolute changes in the benchmark’s safety rates, rather than relative percentage gains. Intermediate steps do not improve monotonically for every model.[2]

The order of work becomes a variable

The largest endpoint change was for Llama 3.1-8B, whose safety rate rose from 5.7% with no preceding tasks to 57.8% after 20, a gain of 52.1 percentage points. Gemma 4-26B-A4B began at a much higher 82.9% and reached 91.8%, an 8.9-point gain. The starting rate therefore matters as much as the improvement: a large change can still leave a model below another model’s cold-start score.[2]

SODA covers 16 tool-use environments and 80 scenarios, each with five threats, for 400 threat cases. The authors test eight conversation depths and average three runs with different random seeds. Claude Opus 4.6 grades the resulting trajectories for harmful actions or intentions. This structured design isolates history as an experimental variable, while the safety scores still depend on the chosen threats and model-based judging procedure.[2]

The study also examines which part of the earlier conversation matters. Its analysis suggests the preceding legitimate tasks themselves were a major contributor to the safety change, whereas the assistant's earlier replies played a smaller role in safety and a larger role in preserving later task utility. The researchers also report shifts in model representations, providing a possible mechanism for the observed pattern. Those findings arise from controlled tests and analysis; they are not a complete causal account of all model safety behaviour.[2]

The mechanism is not a substitute for access control. Earlier legitimate work may influence how a model interprets the next request, while tool permissions still determine what an erroneous action can do. A deployment using a warm-up strategy would need to measure safety and task completion together, under the actual kinds of content and tools the agent encounters.[2]

Testing the agent at different conversation depths

For teams building persistent agents, the practical contribution is a new testing axis. Instead of assessing only a fresh-session prompt, repeat risky and ordinary tasks at different conversation depths. Keep the tools, permissions and evaluation criteria consistent, and record whether safety outcomes change after five, ten or twenty legitimate operations. A result that depends strongly on session history deserves closer examination before the system is entrusted with important actions.[2]

The paper examines useful work as well as safety. Its analysis indicates that previous assistant responses play a smaller part in the safety gain but help preserve performance on later tasks. Omitting those replies can therefore change the trade-off. A system that refuses more unsafe requests is valuable only if it can still complete the legitimate work for which it was deployed.[2]

There is another implication for research comparability. Two evaluations can use the same model and the same final request yet report different success rates because one silently includes previous tool calls. Without documenting conversation depth, an apparent model improvement may actually be a context effect. The same principle applies to user trust: results from a one-turn test should not be sold as a comprehensive forecast of behaviour in a long business process.[2]

Conversation depth is a reproducibility detail with operational consequences. Two teams can use the same model and final request yet measure different behaviour if the preceding work differs. Recording that history makes the result easier to compare and helps reveal whether an agent’s weakest point is the moment it starts.[2]

Report an error or suggest a correction