The same database question worked less often in Portuguese—even with an English schema

A user asks a question in ordinary language; a model writes SQL; the database runs the query. In a controlled WikiSQL experiment, that chain succeeded less often when the question was Portuguese rather than English—even though the database schema and inference configuration were held fixed.[1]

For LLaMA 3.3 70B in zero-shot testing, execution accuracy was 47.25% in English, 40.03% in Brazilian Portuguese and 39.34% in European Portuguese. “Execution accuracy” means the generated query returned the correct result. It does not require the model to reproduce one exact reference string.[1]

Relative to the English result, the Brazilian Portuguese score was about 15.3% lower and the European Portuguese score about 16.7% lower. Those percentages describe performance in this benchmark and setup; they are not universal estimates for every database, prompt or Portuguese-speaking user.[1]

The researchers checked whether translation quality explained the result

The translations were evaluated with a cosine-similarity score of 0.966, linguistic analysis of differences between the two Portuguese varieties and multi-model semantic judgments. The researchers also kept the database schemas in English. That design makes the input language the main changed factor, although translated benchmarks can never reproduce every feature of naturally authored questions.[1]

The hardest cases magnified the difference

Performance degraded more on semantically complex and quantified questions, particularly COUNT queries. English-only successes outnumbered Portuguese-only successes by nearly ten to one. A second model, Qwen 3.6 35B, reproduced the broader English–Portuguese gap but did not produce the same stable difference between Brazilian and European Portuguese.[1]

That distinction matters. The evidence supports a cross-language weakness. It does not support a fixed ranking in which one Portuguese variety is always easier than the other. The smaller intralinguistic effect appears to depend on the model.[1]

Why a translated interface is not enough

A database product can display Portuguese menus while its natural-language query layer remains less reliable in Portuguese. Evaluation therefore has to use the language people actually type, the schemas they actually query and the operations that matter to the organisation. A single English benchmark can hide errors in counting, filtering or aggregation that appear after localisation.[1][2]

Report an error or suggest a correction