LLMSQLMUTATOR: LLM-Powered Test Case Generation for Database Using Bug Reports
Read the paper · doi:10.1109/icde65706.2026.00240
What this paper does with SQLancer
SQLancer is both a baseline and a component. As a baseline it is one of four tools compared over 48 hours, and the paper is even-handed: SQLancer's syntactic correctness exceeds 90% on every system tested, higher than the mutation-based tools, while its coverage peaks around 52%. It is called the most effective black-box fuzzing tool, distinguished by its three complementary oracles. Those oracles are also reused directly: for test cases containing SELECT queries, LLMSQLMutator randomly selects one of TLP, NoREC or CERT as its oracle, describing them as the three most effective existing oracles for detecting result inconsistency. Of its 14 result-inconsistency bugs, TLP found 12 and CERT the other two.
LLMSQLMutator generates DBMS test cases by having a language model mutate SQL drawn from historical bug reports, on the argument that reports encode the shapes that have already broken systems. It reaches constructs grammar-based fuzzers avoid -- deep recursion, for instance, which they cap to stay syntactically correct -- and found bugs across six relational DBMSs.
How it was classified
uses infrastructure — yes (unclear)
M2 states that for SELECT test cases the tool selects one oracle at random from TLP, NoREC or CERT, M4 names those as the oracles chosen, and M5 reports which of them found each bug. The oracles are used as-is inside the tool, but the paper does not say whether they are invoked through SQLancer or reimplemented, so the form of the reuse is unclear.
2) For test cases containing SELECT queries, we employ result inconsistency detection by randomly selecting one test oracle from TLP [13], NoREC [12], or CERT [44].
We selected the three most effective existing oracles for detecting logic bugs through result inconsistency: NoREC [12], TLP [13], and CERT [44].
Of these 14 result inconsistency bugs, 12 bugs are detected by the TLP oracle, and the other two are found by the CERT oracle.
extends technique — no
The three oracles are used as they are; nothing is generalised or adapted. The contribution is the LLM-driven mutation that produces the test cases.
compares with — yes
M7 names SQLancer among the state-of-the-art approaches compared against, M10 gives the 48-hour bug and pattern table, and M12 reports its coverage.
2) Comparsion of baselines We compare LLMSQLM UTATOR with state-of-the-art DBMS testing approaches, including mutation-based methods SQLRight [15] and Squirrel [16], as well as grammar-based SQLancer [11] and random-based SQLsmith [6] to provide a comprehensive empirical comparison.
TABLE VI: The bug number and bug pattern covered by SQLsmith, SQLancer, SQLRight, Squirrel and LLMSQLM UTATOR in 48h DBMSSQLsmith SQLancer Squirrel SQLRight LLMSQLM UTATOR #Bug Pattern #Bug Pattern #Bug Pattern #Bug Pattern #Bug Pattern MySQL - - 1 23.
SQLancer attains only modest coverage (peaking at 52.
describes as state of the art — yes
M14 states that SQLancer has emerged as the most effective black-box fuzzing tool, distinguished by its adoption of three complementary oracles.
In recent years, SQLancer [11] has emerged as the most effective black-box fuzzing tool, distinguished by its adoption of three complementary oracles [14], [12], [13].
SQLancer publications it cites (5)
Bibliography entries that resolved to a SQLancer publication, or to a paper by one of the project's authors. A sentence citing one of these numbers is a reference to SQLancer even when it never writes the name.
| # | Entry | Matched as |
|---|---|---|
| 11 | “Sqlancer,” https://github.com/sqlancer/sqlancer, 2019, accessed on July, 7, 2025. | sqlancer publication |
| 12 | M. Rigger and Z. Su, “Detecting optimization bugs in database engines via non-optimizing reference engine construction,” in Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Sym... | sqlancer publication · NOREC |
| 13 | M. Rigger and Z. Su, “Finding bugs in database systems via query partitioning,” Proceedings of the ACM on Programming Languages, vol. 4, no. OOPSLA, pp. 1–30, 2020. | sqlancer publication · TLP |
| 14 | M. Rigger and Z. Su, “Testing database engines via pivoted query synthesis,” in 14th USENIX Symposium on Operating Systems Design and Implementation (OSDI 20), 2020, pp. 667–682. | sqlancer publication · PQS |
| 44 | J. Ba and M. Rigger, “Cert: Finding performance issues in database systems through the lens of cardinality estimation,” in Proceedings of the IEEE/ACM 46th International Conference on Software Engineering, 2024, pp. 1... | sqlancer publication · CERT |
Every place it refers to SQLancer (14)
14 sentences, each stored verbatim from the extracted text with where it was found and how. “Citation marker” means the sentence names no tool at all and was reached through a reference number that resolved to a SQLancer publication.
| Id | Sentence | Found by | Where |
|---|---|---|---|
| M1 | Generally, the automated DBMS testing process comprises two phases: test case generation [6], [11] and test oracle construction [12], [13], [14], [15]. |
citation marker |
I INTRODUCTION page 1 |
| M2 | 2) For test cases containing SELECT queries, we employ result inconsistency detection by randomly selecting one test oracle from TLP [13], NoREC [12], or CERT [44]. |
technique |
D DBMS Testing page 8 |
| M3 | It is important to note that we used the latest release of each RDBMS, which has been extensively tested by existing approaches [12], [13]. |
citation marker |
IV EVALUATION page 8 |
| M4 | We selected the three most effective existing oracles for detecting logic bugs through result inconsistency: NoREC [12], TLP [13], and CERT [44]. |
technique |
IV EVALUATION page 8 |
| M5 | Of these 14 result inconsistency bugs, 12 bugs are detected by the TLP oracle, and the other two are found by the CERT oracle. |
technique |
B Experiment results page 9 |
| M6 | Existing fuzzers typically restrict recursion depth to maintain syntactic correctness [15], [13], thereby failing to reach the parser’s threshold. |
citation marker |
B Experiment results page 9 |
| M7 | 2) Comparsion of baselines We compare LLMSQLM UTATOR with state-of-the-art DBMS testing approaches, including mutation-based methods SQLRight [15] and Squirrel [16], as well as grammar-based SQLancer [11] and random-based SQLsmith [6] to provide a comprehensive empirical comparison. |
name |
B Experiment results page 9 |
| M8 | 3259 TABLE IV: Bugs detected by LLMSQLM UTATOR SNID Target Type Bug pattern Root cause 1 118145 MySQL Error CREATE, INDEX, ENGINE=Archive Archive engine: Index validation missing during table creation 2 113176 MySQL Error CREATE, CHECK, NOT, IN, IS FALSE Error with null expression in check condition 3 109149 MySQL I... |
name |
B Experiment results page 9 |
| M9 | As a grammar-based tool, SQLancer demonstrates consistently high syntactic correctness across all six tested DBMSs,with rates exceeding 90% in every case and reaching 99. |
name |
B Experiment results page 10 |
| M10 | TABLE VI: The bug number and bug pattern covered by SQLsmith, SQLancer, SQLRight, Squirrel and LLMSQLM UTATOR in 48h DBMSSQLsmith SQLancer Squirrel SQLRight LLMSQLM UTATOR #Bug Pattern #Bug Pattern #Bug Pattern #Bug Pattern #Bug Pattern MySQL - - 1 23. |
name |
B Experiment results page 10 |
| M11 | In contrast, grammar-based SQLancer and random-based tool SQLsmith exhibit significant limitations. |
name |
B Experiment results page 11 |
| M12 | SQLancer attains only modest coverage (peaking at 52. |
name |
B Experiment results page 11 |
| M13 | Metamorphic testing is another mainstream approaches for RDB-MS testing [51], [54], [14], [12], [13], [55], [56], [15]. |
citation marker |
V RELATED WORK page 12 |
| M14 | In recent years, SQLancer [11] has emerged as the most effective black-box fuzzing tool, distinguished by its adoption of three complementary oracles [14], [12], [13]. |
name |
V RELATED WORK page 12 |