Search study finds 53% of sampled no-click queries were judged successful

A search can answer a question before the user clicks a link. A November 1 preprint examines how to recognise that outcome: trained judges rated 1,008 of 1,895 no-click queries successful, or 53.2%, while 325 of 5,009 clicked queries were judged unsuccessful, or 6.5%. The labels were inferred from observable session evidence, rather than obtained by asking users whether they were satisfied. Although the paper addresses measurement in the generative-AI search era, its evaluation does not isolate a generative-only search population.[2]

The researchers call their proposed metric Session Success Rate, or SSR. Human assessments of complete search sessions, including privacy-protected interaction evidence and recordings, provide labels for predictive models. The models then estimate success from signals available in online experiments, such as query patterns, page views and reformulations. The recordings support offline labelling; they are not inputs required by the online metric. This lets the researchers evaluate a potential success measure at a scale that human grading alone could not cover.[2]

A deliberately challenging sample

The out-of-sample evaluation covered 6,904 queries from 3,216 sessions. Its sampling design oversampled difficult multi-query sessions and mobile use, so the observed success shares are not prevalence estimates for all search traffic. SSR achieved 0.75 balanced accuracy overall, 0.68 within no-click queries and about 0.68 at session level. Balanced accuracy averages performance on the successful and unsuccessful classes; an always-majority prediction receives 0.5, even when its ordinary accuracy looks high.[2]

The headline 53.2% therefore answers a bounded question: how many no-click queries in this labelled sample were judged to have met the information need? It does not establish the success rate of every AI answer or show that withholding a click caused a better experience. The distinction still matters for experimentation. A click-only metric cannot tell a user who found the answer on the results page from one who gave up.[2]

Why an apparently successful answer may still be difficult to score

Consider three searches. In the first, a user asks for a simple conversion and reads a correct answer without clicking. In the second, a user seeks a complicated government form, clicks several pages and eventually finds the instructions. In the third, an AI search system offers a fluent but incorrect answer; the user accepts it without clicking or reformulating. The first is a good abandonment; the second may be successful despite many interactions; the third is a false sense of completion. These are illustrative cases, not observed study participants. They show why an evaluation must consider what users tried to accomplish and, where possible, whether the content supplied was accurate—not merely whether they stayed on the page.[2]

SSR remains a prediction of judge-labelled success, rather than a direct observation of each user’s understanding. A person might accept an incorrect answer or complete a task outside the recorded session. Human labelling can reduce ambiguity but cannot eliminate those blind spots. The paper accordingly treats SSR as an additional experimental signal, whose disagreements with established metrics need investigation.[2]

The new metric was tested against established experiments

In 12 validation experiments, SSR agreed with Sessions per User, the existing behavioural metric, in seven cases, was inconclusive in four and disagreed in one under the authors’ significance threshold. Across about 4,000 production experiments, the authors report roughly 2.5 times the binary sensitivity: a larger share of experiments produced a statistically significant change in the metric. That is a claim about detecting experimental movement, rather than a 2.5-fold improvement in search quality; the paper does not publish the underlying production counts.[2]

The research has particular significance for the relationship between search platforms and publishers. A platform that shows complete generated answers can deliver value without sending the reader to a publisher's site. That may be a good search experience while being a poor outcome for a publisher depending on referral traffic. SSR measures the former, not whether the publisher was credited adequately, licensed, compensated or visited. Measuring user satisfaction and measuring the economics of originating the underlying information are separate problems. A serious public account of AI search should not pretend one statistic resolves both.[2]

For search teams, the useful next step is to investigate experiments where SSR, clicks and repeat use point in different directions. For publishers, the same results reinforce the need to measure referrals separately from on-page answer success. A platform’s ability to satisfy a query and a publisher’s ability to fund the information behind it remain distinct outcomes.[2]

Report an error or suggest a correction