Ask a company database for “revenue last quarter” and the SQL can be syntactically perfect while the answer is commercially wrong. The organisation may mean booked contracts, recognised accounting revenue, invoices issued or cash collected. Those definitions usually live in policy, analyst practice and metric code—not in the column names visible to a language model.
AtScale’s new AI suite is built around that gap. Ava is meant to clarify an ambiguous question, ACE computes governed metrics, CoModel helps draft and maintain the semantic model, and Canary is designed to test answers against validated results. Ava and ACE are available; CoModel and Canary were announced as forthcoming.[1]
One word, four legitimate numbers
| Question asked | Possible definition | Why the answer changes |
|---|---|---|
| “What was revenue?” | Bookings | Counts contracted value even when service has not yet been delivered |
| “What was revenue?” | Recognised revenue | Follows accounting rules for when performance obligations are satisfied |
| “What was revenue?” | Billed revenue | Tracks invoices issued, which can lead or lag recognition |
| “What was revenue?” | Cash collected | Tracks payment timing rather than economic performance |
A semantic layer changes the task
Without governed semantics, the model has to infer joins, aggregation rules, reporting periods and exceptions from a raw schema. With a semantic layer, the model can ask for a named measure whose formula and permissions already exist. The model is no longer being rewarded for guessing the business’s private language.
AtScale reports a large change on BIRD-Interact. On 317 questions it judged to have verifiable answers, Claude Sonnet 5 scored 46% when writing SQL against the raw databases and 93% with an automatically generated semantic layer. Across the tested models, it reports improvements of 1.7 to 2.4 times.[2]
What the benchmark does—and does not—show
| Measure | Value | Context |
|---|---|---|
| Benchmark size | 600 tasks across 22 databases | The wider BIRD-Interact set includes business questions and database-management operations.Source 2 |
| Answerable business questions | 317 | AtScale excluded questions it judged to have flawed or non-verifiable answer keys.Source 2 |
| Claude Sonnet 5 result | 46% to 93% | Best-of-run result reported by AtScale with and without its semantic layer.Source 2 |
| Independent small study | +17 to +23 percentage points | A separate 100-question experiment found gains from adding a 4 KB hand-authored semantics document across three frontier models.Source 3 |
Ninety-three percent can still be fragile
If a workflow requires ten independent steps and each succeeds 93% of the time, the probability that all ten succeed is about 48%. At one hundred steps, it falls to roughly 0.07%. Real errors are not necessarily independent, but the calculation shows why a high single-answer score is not enough for unattended operations.
The definitions still need an owner
AI can propose a semantic model, but it cannot decide by itself which definition finance, sales and operations should share. A wrong rule encoded once can make every dashboard and agent consistently wrong. Human review, versioning and a record of the SQL behind each answer are therefore part of the product—not administrative cleanup.[1]
The larger lesson is simple. A more capable language model may improve query generation, but it cannot reason its way to a business definition that the organisation has never made explicit. Reliable analytics begins before the prompt, in the governance of meaning.
Sources & further reading
- AtScale Launches AtScale for AI to Close the Enterprise AI Accuracy Gap AtScale · 15 Oct 2026 · Accessed 16 Oct 2026
- BIRD-Interact Benchmark Results: Methodology and Analysis AtScale · 15 Oct 2026 · Accessed 16 Oct 2026
- Semantic Layers for Reliable LLM-Powered Data Analytics: A Paired Benchmark of Accuracy and Hallucination Across Three Frontier Models arXiv · 28 Apr 2026 · Accessed 16 Oct 2026