More memory made some LLM group decisions worse

Multi-agent language-model systems are often built on an intuitive idea: let several agents discuss a problem, compress the discussion into memory and give the summary to a final decision-maker. A new study shows why that pipeline can fail even when the memory system follows a seemingly sensible rule.[1]

“Rare, Relevant, and Still Wrong” introduces CrossProfile, a diagnostic benchmark with 60 templates and 480 tasks. It separates three properties that ordinary group-decision problems often confound: whether a fact supports the correct answer, whether only one agent knows it, and whether it is relevant to the stated decision criteria. By holding the text fixed while changing ownership and role, the benchmark can test what the memory policy is actually rewarding.[1]

Relevance can preserve the wrong evidence

A relevance filter sounds safe: keep the statements most connected to the decision. The problem is that a decoy can be highly relevant. It may discuss exactly the right criterion while pointing to the wrong choice. In the classic hidden-profile configuration, even a perfect relevance filter lost 23 to 31 accuracy points to rarest-first selection across the tested models because it readmitted these on-topic decoys.[1]

Rarest-first selection was not a general solution. It performed no better than random selection on average and won only when the benchmark happened to place the useful information among the rare facts. The result is a warning against treating minority status as a proxy for value.[1]

Attribution created artificial social proof

The final decision model did not only respond to content. It gave extra weight to facts attributed to multiple agents. When the researchers removed attribution tags, or reassigned them randomly while leaving line order unchanged, accuracy in the classic configuration rose by 13 to 24 points.[1]

That effect is especially important because multi-agent systems often create correlated agents. They may share a model, prompt, training data or retrieval source. Three agents repeating a claim can look like corroboration even when all three derived it from the same mistaken pattern. Attribution labels turn correlation into apparent social evidence.[1]

Why the full transcript did not rescue the system

In natural discussions, the study reports that the complete transcript produced lower accuracy than giving the final model no notes at all. The evidence had already been distorted during conversation: agents repeated some claims, failed to surface others and influenced one another before any memory policy acted.[1]

This means memory quality cannot be solved only at the summarisation stage. The discussion protocol must preserve provenance, independence and the balance of evidence. A perfect compressor cannot recover a fact that no agent raised or correct a consensus that formed through imitation.[1]

What system builders should change

Multi-agent applications should distinguish evidence from endorsement. A claim needs a source, confidence and dependency record, not merely a count of how many agents repeated it. Systems can also separate independent first-pass judgements from later discussion, so the final decision-maker can see whether apparent consensus existed before agents exchanged messages.[1]

Memory selection should be evaluated against adversarial configurations in which relevance, rarity and truth are deliberately separated. A policy that performs well only when useful facts are rare has learned the benchmark’s social structure rather than a reliable principle of evidence selection.[1]

Limits of the result

CrossProfile is a synthetic diagnostic benchmark, and the study evaluates Qwen2.5-7B, Llama-3.1-8B and two DeepSeek models. The exact magnitudes should not be assumed for every frontier model or real organisation. Natural groups also have richer roles, tools and incentives than the benchmark represents.[1]

The mechanism is still broadly relevant. More agents and more memory do not automatically create more independent evidence. Without provenance and controls for correlated reasoning, a multi-agent system can become a machine for manufacturing confidence. The study’s practical lesson is that collective intelligence depends on how evidence is represented—not on how many voices appear to agree.[1]

Report an error or suggest a correction