How far do on-premises open LLMs get on Text-to-SQL? A cross-family size and technique frontier on BIRD

arXiv:2606.29733 · cs.CL, cs.DB, cs.LG · Submitted 2026-06-29 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Hi, I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "How far do on-premises open LLMs get on Text-to-SQL? A cross-family size and technique frontier on BIRD".

Jane: Here's the gist of the paper. The gist: Generation matters more than raw size, self-correction generalizes across three families and is significant, schema linking does not help even with a high-recall embedding retriever, and self-consistency is not worth its cost.

Tom: How it works The study evaluates Text–to–SQL performance on the BIRD development split using three open model families—Qwen2.5–Coder, CodeLlama, and Llama–3.x—across a ∼5× size range, serving them on-premises with vLLM

6: (Page 1).

Jane: The evaluation employs a matched protocol where model-agnostic recipes are ablated component by component: schema linking, self-correction, and self-consistency

7: (Page 2).

Tom: Execution Accuracy is measured based on whether the execution returns the gold result set

1: (Page 3).

Paper summary: Jane: So we're looking at this paper now titled "How far do on-premises open LLMs get on Text-to-SQL? A cross-family size and technique frontier on BIRD". The authors are really showing that when you’re stuck running AI locally without a cloud API, generation quality and specific recipe choices matter way more than just how big your model is or how many parameters it has.

Tom: They focused heavily on comparing different model families—Qwen2 point 5–Coder and CodeLlama against Llama-three variants—to see where the real performance gains come from when you're working on narrow, specific schemas like BIRD’s > <ref:2606.29733#pg1>

Lu: They organized the study around four research questions: RQ1 How do accuracy gains compare across model size and across model family on-prem? RQ2 Which recipe components improve accuracy, and do the gains generalize across families? RQ3 Does schema linking help if we replace a lexical linker with a stronger retrieval/embedding one? And RQ4 Where do configurations sit on the accuracy–cost frontier in real terms > <ref:2606.29733#pg2>

Meng: The main point they drive home is that self-correction is the only thing that consistently works across all those different model families they tested >

Jane: That means for a Text-to-SQL setup on private data, the best bet is to focus your compute on adding that self-correction step instead of chasing every little linking trick >

Lu: They also found that schema linking doesn't help much at all, even with a better embedding retriever with ninety-six point five percent gold-table recall, it was statistically indistinguishable from no linking > <ref:2606.29733#pg2>

Meng: That rules out the idea that you need to spend compute just to make your schema connections better >

Tom: And self-consistency only changed accuracy by zero point one three points at about five times the tokens, which is not significant >

Jane: So it means you don't need all those fancy recipe additions if your goal is just to get a usable result without spending extra compute time on things that don't actually help >

Lu: The paper is really important because it shows that when you’re dealing with specific, narrow schemas like BIRD’s, the prompt itself isn't usually the bottleneck; generation and self-correction are what actually make the difference in accuracy > <ref:2606.29733#pg5>

Meng: It also gives us a clear picture of where to focus our engineering efforts—on improving the base model family and adding that single self-correction step rather than chasing every little linking trick >

Tom: So we've covered how generation beats size, that self-correction is the key component, and that schema linking doesn't offer any real benefit >

Jane: We’re going to wrap up here with a look at what this all means for Text-to-SQL when you're running models locally. Next time we look at how these results change when we try to scale those self-correction steps up for production environments.

Conclusion: SEGMENT: Conclusion — Title and Authors**

Tom: So we've covered how generation beats size, that self-correction is the key component, and that schema linking doesn't offer any real benefit in this paper about Text-to-SQL.

Jane: Right. The authors of this study are looking at how far on-premises open LLMs can get on Text-to-SQL when you’re working with narrow schemas like BIRD’s.

Lu: They put a lot of work into comparing different model families—Qwen2 point five-Coder and CodeLlama against Llama three variants—to see where the actual performance gains happen when you're running things locally.

Meng: The main point they drive home is that self-correction is the only thing that consistently works across all those different model families they tested on.

Tom: That’s what I liked, seeing a component that works reliably across different architectures. So, if you’re deploying this, you can focus your compute there instead of chasing every little linking trick.

Jane: It shows that generation matters more than just raw size when you look at which recipe works best on those specific schemas.

Meng: That rules out the idea that you need to spend compute just to make your schema connections better. And self-consistency only changed accuracy by zero point one three points at about five times the tokens, which is not significant.

Tom: So the practical implication is that if you’re setting up a Text-to-SQL system on private data, the best bet is to pick a strong, recent family, run it at a mid-to-large size model, add that self-correction step we talked about and just skip both the schema linking and self-consistency work.

Jane: It means you don't need all those fancy recipe additions if your goal is just to get a usable result without spending extra compute time on things that don't actually help.

Tom: So we’ve covered how generation beats size, that self-correction is the key component, and that schema linking doesn't offer any real benefit.

Vladimir Beskorovainyi

Moscow Institute of Physics and Technology (MIPT)

cs.CL, cs.DB, cs.LG

Submitted: 2026-06-29

Updated: 2026-10-18

Comments: 18 pages, 4 figures. v2 corrects v1 (read-only databases, official BIRD metric, truncated FP8-API arm; v1's ~13-point quantization-penalty claim is withdrawn) and adds re-runs; code and data: https://doi.org/10.5281/zenodo.23445116

Code: https://github.com/beskvladimir-create/nl2sql-onprem-bench

Project page: https://bird-bench.github.io

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 92/100

The gist: The gist: Generation matters more than raw size, self-correction generalizes across three families and is significant, schema linking does not help even with a high-recall embedding retriever, and

Key concepts

Text-to-SQL: This task involves taking a natural language question and translating it into a precise SQL query that can be executed against a database. The study measures how accurately different AI models can perform this translation on specific database schemas.
Self-Correction: This technique involves having the model generate an initial answer, then critically review its own output and make corrections based on the execution results. The research found this method consistently improved accuracy across different model families, making it a robust way to boost performance.
Schema Linking: This is a process where models try to connect natural language terms in a question to specific database tables or columns. The study showed that even using advanced embedding linkers did not significantly improve accuracy on narrow BIRD schemas, meaning this technique is often unnecessary for these types of tasks.
Self-Consistency: This involves running the model multiple times with different sampling methods and aggregating the results to find a consensus. The research concluded that this method provided very little accuracy gain for a high computational cost, indicating it is not a worthwhile technique for improving Text-to-SQL performance.

Terminology

Accuracy — 2× in this paper · explained in 1 episode
Consistency — 2× in this paper · explained in 3 episodes
Self-consistency — 2× in this paper · explained in 3 episodes
Text-to-SQL — 2× in this paper · explained in 4 episodes
Benchmark — 1× in this paper · explained in 2 episodes
Model-Agnostic — 1× in this paper · explained in 2 episodes

Summary

The gist: Generation matters more than raw size, self-correction generalizes across three families and is significant, schema linking does not help even with a high-recall embedding retriever, and self-consistency is not worth its cost.

How it works

The study evaluates Text–to–SQL performance on the BIRD development split using three open model families—Qwen2.5–Coder, CodeLlama, and Llama–3.x—across a ∼5× size range, serving them on-premises with vLLM [6] (Page 1). The evaluation employs a matched protocol where model-agnostic recipes are ablated component by component: schema linking, self-correction, and self-consistency [7] (Page 2). Execution Accuracy is measured based on whether the execution returns the gold result set [1] (Page 3).

The models tested include Qwen2.5–Coder–Instruct (7B/14B/32B) and CodeLlama–Instruct (7B/13B/34B), alongside Llama-3.x variants like Llama-3.1–8B and Llama-3.3–70B [5] (Page 2). Serving was conducted on an OpenAI-compatible vLLM endpoint in fp16, with smaller models running on an L40S GPU and larger ones on an A100-80GB GPU [3] (Page 2). The comparison utilized the paired McNemar test to assess significance across model configurations [13] (Page 2).

The research addressed four primary research questions regarding accuracy gains across size and family [2] (Page 2). RQ1 investigated how accuracy gains compare across model size and family on-prem [3] (Page 2). RQ2 examined which recipe components improve accuracy and if the gains generalize across families [4] (Page 2). RQ3 tested whether schema linking helps when replacing a lexical linker with a stronger retrieval/embedding one [5] (Page 2). RQ4 sought to determine where configurations sit on the accuracy–cost frontier in real terms [6] (Page 2).

The study specifically analyzed the effect of self-correction, which was found to be a robust, near-free win at sizes with room for improvement, such as p < 10−10 on CodeLlama–7B and Llama–3.1–8B [4] (Page 2). The gain from self-correction shrinks for the strongest models, such as Llama-3.3–70B (+1.04 pp) [6] (Page 6).

The analysis of schema linking revealed that lexical linking significantly hurts performance on Qwen and Llama, and a stronger retrieval/embedding linker with 96.5% gold-table recall was statistically indistinguishable from no linking (p ≥ 0.55) [4] (Page 2). This finding ruled out the weak lexical strawman objection across three families and two generations [4] (Page 2). The discussion concluded that on BIRD’s narrow schemas, the prompt is rarely the bottleneck, meaning even a high-recall retriever offers no accuracy gain and risks removing a needed table [5] (Page 5).

The study also assessed self-consistency as an ablation component [7] (Page 2). Self-consistency was found to be poor value, yielding only +0.13 pp for ∼5× tokens, with a high p-value of 0.86 [4] (Page 2). Furthermore, the cost analysis showed that self-correction roughly doubles per-query time for CodeLlama–34B (from 0.123 to 0.215) for its significant gain, whereas self-consistency costs ∼5× the tokens and tail latency for a non-significant +0.13 pp [6] (Page 6).

The findings suggest that generation matters more than raw size, as a modern non-Qwen model (Llama–3.3–70B) was competitive with Qwen–32B on a matched FP8 serving, indicating that the large CodeLlama gap reflects model generation rather than being solely due to its 2023 release [4] (Page 4). The practical recommendation is to pick a strong, recent family, serve a mid–to–large size, add self–correction, and skip both schema linking and self-consistency for private Text-to-SQL on BIRD-class schemas [8] (Page 8).

The harness used for the study is fully reproducible, with all code, configurations, per-example predictions, and summaries released on GitHub and archived on Zenodo [7] (Page 7). The results are tested using the exact McNemar test on per-example correctness [13] (Page 3). The paper notes that the best on-prem base model sits just below strong general-purpose single models like Gemini 2.0 Flash, confirming the harness is sound rather than weak [8] (Page 8).

The final conclusion emphasizes that for private Text-to-SQL on BIRD–class schemas, practitioners should choose a strong, recent family and add self-correction while skipping schema linking and self-consistency [8] (Page 8). The study confirms that generation beats size, self-correction generalizes across families, schema linking does not help even with a high recall retriever, and self-consistency is not worth its cost [8] (Page 8). The natural next steps involve exploring more families and a wide-schema benchmark using the released harness [8] (Page 9).

The paper rigorously tested claims by including negative results in full, ensuring an honest assessment of the recipes' value on small on-prem models across different families [2] (Page 2). The protocol was kept minimal—zero-shot, single greedy call, no in-context examples, no fine-tuning—so the base EX should land below systems that add reasoning or retrieval [8] (Page 8). This controlled comparison of cheap recipe components across families is the core contribution [2] (Page 2). The research received no external funding; on-prem runs used rented GPUs at the author’s own expense [8] (Page 8). The paper declares no competing interests [9] (Page 9).

The BIRD dataset is third-party and obtained from its authors under their terms, and it is not redistributed here [7] (Page 7). The evaluation utilized the full BIRD dev split with n = 1534 examples, using Execution Accuracy (EX) where a prediction is correct if executing it returns the gold result set [1] (Page 3). The paper notes that BIRD labels contain documented annotation errors, necessitating paired significance rather than raw deltas [8] (Page 8). This rigorous approach ensures that every number can be regenerated using the released harness and code [7] (Page 7). The paper is available on GitHub and Zenodo DOI 10.5281/zenodo.20952794 [7] (Page 7). The data and code availability are provided to ensure reproducibility [8] (Page 9).

The study provides real per-stage cost, reported as real dollars per 1k queries from wall-clock time at a representative rate of an A100–80GB GPU [6] (Page 3). The cost for CodeLlama–34B base was reported as 376 / 1k queries, and self-correction increased this to 658 / 1k queries [6] (Page 4). This cost analysis is crucial for operators who pay per GPU-second [6] (Page 2). The paper notes that the Llama family and Qwen-32B embedding comparison ran on a quantized FP8 API endpoint, which was reported only on its own matched serving and supports relative claims only [6] (Page 6). This information is used to support relative claims only [6] (Page 6). The paper's best on-prem base sits just below systems that add reasoning or reinforcement learning, confirming the harness is sound rather than weak [8] (Page 8). The paper concludes by stating that the natural next steps fit the released harness [9] (Page 9).

The study's methodology involves ablating schema linking (lexical and embedding) and self-correction to isolate the value of each component [7] (Page 2). The results show that lexical linking significantly hurts performance on Qwen2.5–Coder–32B (−1.56 pp) [6] (Page 6). The final recommendation is to pick a strong, recent family, serve a mid-to-large size, add self-correction, and skip both schema linking and self-consistency [8] (Page 8). This configuration is identified as the cheapest configuration that clears the bar [8] (Page 8). The paper's findings are tested with the paired McNemar test on per-example correctness [3] (Page 3). The entire study is designed to answer how far on-prem open LLMs get on Text–to–SQL and which accuracy tricks are worth their compute [1] (Page 1).

Improvements for AI systems

Improve model selection by prioritizing recent generations over raw size, as Generation matters more than raw size, suggesting practitioners should choose a strong, recent model first and size second. This leads to better performance on on-prem systems because a 2024 model at a modest size beats a 2023 model at 34B.
Implement self-correction as the primary recipe component, as it is a robust, near-free win across families, with significant gains observed in models where feedback is possible (Self-correction helps on all three families and is statistically significant wherever the model is capable enough to act on feedback). This makes the system more accurate without incurring high computational overhead.
Eliminate schema linking and self-consistency from the deployment pipeline for typical BIRD schemas, as schema linking does not help — and a stronger linker does not rescue it and self-consistency is poor value (+0.13 pp for ∼5× tokens, p = 0.86). This reduces latency and cost while maintaining sufficient accuracy by focusing resources on the most effective component.
If schema linking is required for extremely wide schemas, utilize a stronger retrieval/embedding linker tuned to high recall (96.5%), though the paper suggests this does not help on BIRD's narrow schemas because the prompt is rarely the bottleneck. This allows for targeted improvement only when necessary, avoiding unnecessary complexity and potential performance degradation.