Confidence Estimation for Text-to-SQL in Large Language Models
By: Sepideh Entezari Maleki, Mohammadreza Pourreza, Davood Rafiei
Potential Business Impact:
Helps computers know if their database answers are right.
Confidence estimation for text-to-SQL aims to assess the reliability of model-generated SQL queries without having access to gold answers. We study this problem in the context of large language models (LLMs), where access to model weights and gradients is often constrained. We explore both black-box and white-box confidence estimation strategies, evaluating their effectiveness on cross-domain text-to-SQL benchmarks. Our evaluation highlights the superior performance of consistency-based methods among black-box models and the advantage of SQL-syntax-aware approaches for interpreting LLM logits in white-box settings. Furthermore, we show that execution-based grounding of queries provides a valuable supplementary signal, improving the effectiveness of both approaches.
Similar Papers
Confidence Estimation for Error Detection in Text-to-SQL Systems
Machine Learning (CS)
Helps computers understand questions to find database answers.
Cost-Aware Text-to-SQL: An Empirical Study of Cloud Compute Costs for LLM-Generated Queries
Databases
Saves money running computer questions on data.
Rethinking Agentic Workflows: Evaluating Inference-Based Test-Time Scaling Strategies in Text2SQL Tasks
Computation and Language
Lets computers answer questions from data.