Efficiently Detecting DBMS Bugs through Bottom-up Syntax-based SQL Generation
Read the paper · doi:10.14722/ndss.2026.240198
What this paper does with SQLancer
SQLancer is the paper's reference point for the template-based approach it is arguing against, and it is unusually direct about SQLancer's standing: the most advanced tool in the category, and the most popular platform for implementing the latest SQL testing techniques. Its baselines are two of those techniques rather than SQLancer's own oracles -- SQLancer+QPG on CockroachDB and SQLancer+DQP on MySQL and MariaDB -- and the paper notes both are built on top of SQLancer and use its SQL templates. The critique is that those hand-written Java templates bound which SQL features can appear at all, which is the limitation the grammar-based generation is meant to remove.
SQLBull generates SQL bottom-up from a DBMS's own grammar rather than from hand-written templates, on the argument that templates only cover the features whoever wrote them thought to include. Building queries upwards from grammar productions reaches syntax the template-based generators never emit, which is where a class of memory errors lives. It is evaluated on CockroachDB, MySQL and MariaDB.
How it was classified
uses infrastructure — no
SQLancer+QPG and SQLancer+DQP are run as baselines. M12 and M13 describe those tools as built on SQLancer and using its templates, but that is a property of the baselines, not of SQLBull, whose generator works from the DBMS grammar instead.
extends technique — no
SQLBull replaces template-based generation with bottom-up grammar-based generation. No SQLancer oracle is carried forward or generalised.
compares with — yes
M8 and M9 name SQLancer+QPG and SQLancer+DQP as the techniques included in the evaluation, M10 records which was run against which system, and M11 reports SQLancer's generated queries in the feature statistics.
Query Plan Guidance (QPG)Differential Query Plans (DQP)
In this evaluation, we include two of the latest and most advanced testing techniques implemented in SQLancer which are capable of detecting DBMS memory errors, i.
, SQLancer +QPG[52] and SQLancer +DQP[53].
We compare SQLBull with SQLancer +QPG onCockroachDB, and SQLancer +DQPonMySQL and MariaDB.
For completeness, we also include the statistics of these SQL features from the queries generated by other testing tools, including mutation-based fuzzer Squirrel [39] and templatebased SQL generator SQLancer [30].
describes as state of the art — yes
M6 calls SQLancer the most advanced tool in its category, M7 the most popular platform for implementing the latest SQL testing techniques, and M12 describes QPG and DQP as the latest and most advanced DBMS testing techniques, implemented on top of it.
The most advanced tool in this category is SQLancer.
Although SQLancer was first introduced in 2022, it has become the most popular platform for implementing the latest SQL testing techniques [52]–[56].
SQLancer +QPG and SQLancer +DQP are the latest and most advanced DBMS testing techniques implemented on top of SQLancer.
SQLancer publications it cites (8)
Bibliography entries that resolved to a SQLancer publication, or to a paper by one of the project's authors. A sentence citing one of these numbers is a reference to SQLancer even when it never writes the name.
| # | Entry | Matched as |
|---|---|---|
| 20 | M. Rigger, “Bugs Found in Database Management Systems,” https: //www.manuelrigger.at/dbms-bugs/, (visited in April 2025). | project authored |
| 30 | Manuel Rigger, “SQLancer,” https://github.com/sqlancer/sqlancer, (visited in April 2025). | sqlancer publication |
| 52 | J. Ba and M. Rigger, “Testing Database Engines via Query Plan Guidance,” inProceedings of International Conference on Software Engineering (ICSE), 2023. | sqlancer publication · QPG |
| 53 | ——, “Keep it simple: Testing databases via differential query plans,” 2, no. 3, May 2024. [Online]. Available: https://doi.org/10.1145/3654991 | sqlancer publication · DQP |
| 54 | M. Rigger and Z. Su, “Testing Database Engines via Pivoted Query Synthesis,” in Proceedings of the 14th USENIX Symposium on Operating Systems Design and Implementation (OSDI), Virtual, 2020. | sqlancer publication · PQS |
| 55 | ——, “Finding Bugs in Database Systems via Query Partitioning,” Proceedings of the ACM on Programming Languages, vol. 4, no. OOPSLA, pp. 1–30, 2020. | sqlancer publication · TLP |
| 56 | ——, “Detecting Optimization Bugs in Database Engines via Nonoptimizing Reference Engine Construction,” in Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foun... | sqlancer publication · NOREC |
| 76 | M. Raselimo, J. Taljaard, and B. Fischer, “Breaking Parsers: Mutationbased Generation of Programs with Guaranteed Syntax Errors,” in Proceedings of the 12th ACMSIGPLAN International Conference on Software Language Eng... | project authored |
Every place it refers to SQLancer (19)
19 sentences, each stored verbatim from the extracted text with where it was found and how. “Citation marker” means the sentence names no tool at all and was reached through a reference number that resolved to a SQLancer publication.
| Id | Sentence | Found by | Where |
|---|---|---|---|
| M1 | However, the high complexity of DBMSs makes them prone to bugs [17]–[20]. |
citation marker project authored |
page 1 |
| M2 | , syntax-based) [ 17], [30], [31]. |
citation marker |
page 1 |
| M3 | For example, SQLancer[30] is a representative syntax-based SQL testing platform that adopts hand-crafted SQL templates to generate testing SQL queries. |
name |
page 1 |
| M4 | This method is more commonly used by syntax-based testing tools that rely on handwritten SQL templates, such as SQLancer [30] and SQLsmith [31]. |
name |
page 2 |
| M5 | For example, SQLancer uses Java to implement its SQL templates, while SQLsmith uses C++. |
name |
page 2 |
| M6 | The most advanced tool in this category is SQLancer. |
name |
page 8 |
| M7 | Although SQLancer was first introduced in 2022, it has become the most popular platform for implementing the latest SQL testing techniques [52]–[56]. |
name |
page 8 |
| M8 | In this evaluation, we include two of the latest and most advanced testing techniques implemented in SQLancer which are capable of detecting DBMS memory errors, i. |
name |
page 8 |
| M9 | , SQLancer +QPG[52] and SQLancer +DQP[53]. |
name |
page 8 |
| M10 | We compare SQLBull with SQLancer +QPG onCockroachDB, and SQLancer +DQPonMySQL and MariaDB. |
name |
page 8 |
| M11 | For completeness, we also include the statistics of these SQL features from the queries generated by other testing tools, including mutation-based fuzzer Squirrel [39] and templatebased SQL generator SQLancer [30]. |
name |
page 8 |
| M12 | SQLancer +QPG and SQLancer +DQP are the latest and most advanced DBMS testing techniques implemented on top of SQLancer. |
name |
page 9 |
| M13 | Both tools utilize the same SQL templates from SQLancer to generate SQL queries but leverage different testing strategies to explore the grammar space of the DBMSs. |
name |
page 9 |
| M14 | Especially for SQLancer +DQP, it is designed to execute the same valid SQL query multiple times with different optimization settings to understand the impact of the query optimization on the query execution plan, which leads to a near-perfect query correctness rate. |
name |
page 9 |
| M15 | 4% SQLancer 3. |
name |
page 12 |
| M16 | We first discuss the existing syntax-based testing tools for the DBMS testing community [30], [39], [52] in § VI-A and § VI-B. |
citation marker |
page 13 |
| M17 | These templates are implemented with custom programming languages, such as C/C++ [31], [35], Java [27], [30], and languages designed for SAT solvers [67], [68]. |
citation marker |
page 13 |
| M18 | SQLancer [30] is another template-based DBMS testing tool that detects DBMS memory and logic bugs [54]–[56]. |
name |
page 13 |
| M19 | Its latest improvement, SQLancer +QPG[52] and SQLancer +DQP[53], leverage the DBMS query plan and optimization hints to guide the query generation. | name | page 13 |