← Research building on SQLancer

Jiaqi Li, Ke Wang, Yaoguang Chen, Yajin Zhou, Lei Wu, Jiashui Wang. 2025. Computers & Security.

Read the paper · doi:10.1016/j.cose.2025.104564 · arXiv:2312.04941v1

What this paper does with SQLancer

Kangaroo argues that SQLancer's oracles constrain the SQL they can test -- NoREC needing a WHERE clause, all three putting limits on queries -- and positions multi-plan execution as free of that constraint. SQLancer is then one of three tools it is measured against over 24 hours, run with NoREC, or TLP where NoREC is unavailable. Kangaroo targets both memory and logic bugs in DBMSs with two techniques. Context-sensitive instantiation takes all static semantic requirements into account, not only identifier types as existing systems do, so generated queries are semantically valid. Multi-plan execution provides the oracle: rather than running only the optimizer's chosen plan, the DBMS executes all plans for a test case and the results are compared, with any difference indicating a logic bug. Applied to SQLite, PostgreSQL and MySQL it detected 50 new bugs. Written by claude-opus-5 from the 36 places this paper refers to SQLancer. The quotations below are the paper's own words, stored verbatim when the text was extracted.

How it was classified

uses infrastructure — no

SQLancer appears only as a tool run for comparison; nothing shown says Kangaroo is built on it.

extends technique — no

Multi-plan execution is presented as an alternative to the oracles rather than an extension of one; the paper's argument is that NoREC and TLP restrict the SQL they apply to.

compares with — yes

SQLancer is one of three tools compared over a 24-hour run, configured with NoREC, or TLP for MySQL where SQLancer does not implement NoREC, and bug counts and program states are reported.

Non-optimizing Reference Engine Construction (NoREC)Ternary Logic Partitioning (TLP)

We also compared Kangaroo with leading DBMS testing tools, such as Squirrel, SQLancer, and SQLRight. M4 · 1 Introduction · page 2
Specifically, after conducting a 24-hour test on the three DBMSs, Kangaroo successfully detected 17 bugs, while SQLancer, Squirrel, and SQLRight only identified 1, 3, and 6 bugs, respectively. M5 · 1 Introduction · page 2
We use NoREC for SQLancer and SQLRight. M14 · 5.2 Generating Valid Queries · page 11
Since SQLancer does not implement NoREC for MySQL, we use TLP instead. M15 · 5.2 Generating Valid Queries · page 11

describes as state of the art — yes

M12 calls NoREC one of the most effective DBMS logic bug detection oracles, and M16 calls SQLancer one of three state-of-the-art open-source systems.

Why Existing Works Cannot Detect the Bug NoREC is one of the most effective DBMS logic bug detection oracles which requires the SELECT statements to satisfy some predefined rules, e. M12 · 3.2 Multi-Plan Execution · page 5
Comparisons with Existing Tools We compare Kangaroo with three state-of-the-art and open source systems: Squirrel, SQLancer, and SQLRight. M16 · 5.3 Comparisons with Existing Tools · page 11

Where this differs from the pattern checks

The regular expressions that scan for these relationships are advisory. Where the reading above contradicts one, the reason is recorded.


SQLancer publications it cites (4)

Bibliography entries that resolved to a SQLancer publication, or to a paper by one of the project's authors. A sentence citing one of these numbers is a reference to SQLancer even when it never writes the name.

#EntryMatched as
14 M. Rigger and Z. Su, “Testing database engines via pivoted query synthesis,” in 14th USENIX Symposium on Operating Systems Design and Implementation (OSDI 20), 2020, pp. 667–682. sqlancer publication · PQS
15 ——, “Detecting optimization bugs in database engines via non-optimizing reference engine construction,” in Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Fou... sqlancer publication · NOREC
16 ——, “Finding bugs in database systems via query partitioning,” Proceedings of the ACM on Programming Languages, vol. 4, no. OOPSLA, pp. 1–30, 2020. sqlancer publication · TLP
28 “Bugs found in database management systems,” https: //www.manuelrigger.at/dbms-bugs/, 2022. project authored

Every place it refers to SQLancer (36)

36 sentences, each stored verbatim from the extracted text with where it was found and how. “Citation marker” means the sentence names no tool at all and was reached through a reference number that resolved to a SQLancer publication.

Id Sentence Found by Where
M1 Second, oracles used by existing systems [13–17] to detect logic bugs have strict requirements on the SQL statements. citation marker
motivation 		1 Introduction
page 1
M2 For example, NoREC [15] requires that the effective SQL query in a test case has WHERE clauses, thus it can only detect logic bugs due to the optimization of WHERE clauses. technique
motivation 		1 Introduction
page 1
M3 Furthermore, both NoREC and TQS attempt to explore multiple query plans of queries but are only capable of covering partial query plans. technique
motivation 		1 Introduction
page 1
M4 We also compared Kangaroo with leading DBMS testing tools, such as Squirrel, SQLancer, and SQLRight. name
baseline 		1 Introduction
page 2
M5 Specifically, after conducting a 24-hour test on the three DBMSs, Kangaroo successfully detected 17 bugs, while SQLancer, Squirrel, and SQLRight only identified 1, 3, and 6 bugs, respectively. name
result comparison 		1 Introduction
page 2
M6 3x more program states than the rule-based tool SQLancer. name
result comparison 		1 Introduction
page 2
M7 The rule-based ones [14–17, 24] generate test cases following a predefined model to ensurethe generated SQL queries can pass the SQL parser. citation marker 2.4 DBMS Testing
page 3
M8 NoREC [15] is an oracle for logic bug detection. technique
definition 		2.4 DBMS Testing
page 3
M9 Although this approach has been effective in detecting bugs in widely-used DBMS [28], it can only be applied to a subset of SQL that can be translated. citation marker project authored
definition 		2.4 DBMS Testing
page 3
M10 proposed three oracles for DBMS logic bug detection, including PQS [14], NoREC [15], and TLP [16], all of them put limitations on the SQL queries. technique
motivation 		3.2 Multi-Plan Execution
page 4
M11 In addition, both TQS and NoREC attempt to explore different query plans for queries. technique
definition 		3.2 Multi-Plan Execution
page 4
M12 NoREC only compares two distinct plans by transforming queries into semantically similar ones. technique
state of the art 		3.2 Multi-Plan Execution
page 4
M13 Why Existing Works Cannot Detect the Bug NoREC is one of the most effective DBMS logic bug detection oracles which requires the SELECT statements to satisfy some predefined rules, e. technique 3.2 Multi-Plan Execution
page 5
M14 in previous works [13–17], since other statements typically lack returning results to be checked. citation marker
baseline 		4.5 Multi-Plan Execution
page 9
M15 We use NoREC for SQLancer and SQLRight. name
baseline 		5.2 Generating Valid Queries
page 11
M16 Since SQLancer does not implement NoREC for MySQL, we use TLP instead. name
state of the art 		5.2 Generating Valid Queries
page 11
M17 Comparisons with Existing Tools We compare Kangaroo with three state-of-the-art and open source systems: Squirrel, SQLancer, and SQLRight. name 5.3 Comparisons with Existing Tools
page 11
M18 SQLancer is a generate-based tool that does not require any initial inputs. name 5.3 Comparisons with Existing Tools
page 11
M19 Squirrel and SQLancer detected five and one memory bugs, respectively. name 5.3 Comparisons with Existing Tools
page 11
M20 SQLancer, the only generationbased tool, found the least bugs across all comparisons, demonstrating the advantage of the mutation-based method to detect DBMS bugs. name 5.3 Comparisons with Existing Tools
page 11
M21 It explores 52%, 44%, and 14% more edges than SQLancer, Squirrel, and SQLRight on average, respectively. name 5.3 Comparisons with Existing Tools
page 11
M22 As shown in Figure 6def, SQLancer achieves the highest query validity. name 5.3 Comparisons with Existing Tools
page 11
M23 This result is reasonable because SQLancer follows very limited grammar rules to generate SQL statements. name 5.3 Comparisons with Existing Tools
page 11
M24 For example, SQLancer does not support generating subqueries that are prone to semantic errors. name 5.3 Comparisons with Existing Tools
page 11
M25 Oracle Comparsion To eliminate the benefit from context-sensitive instantiation, We build Kangaroo MPE by replacing the context-sensitive instantiation with type-sensitive instantiation in Kangaroo, and compare Kangaroo MPE to SQLRight NoREC and SQLRight TLP. technique 5.3 Comparisons with Existing Tools
page 12
M26 Doing an automatic comparison of the MPE and PQS is difficult because PQS requires a generation method to generate statements. technique 5.3 Comparisons with Existing Tools
page 12
M27 Considering that the generation of test cases is also one of the evaluation metrics of the oracle, we include SQLancer PQS as the comparison target. name 5.3 Comparisons with Existing Tools
page 12
M28 After 5 rounds of 24-hour testing, Kangaroo MPE finds three logic bugs in SQLite and two logic bugs in MySQL, and SQLRight NoREC finds one logic bug in SQLite and one in MySQL. technique 5.3 Comparisons with Existing Tools
page 12
M29 Besides, all logic bugs found by NoREC are covered by MPE. technique 5.3 Comparisons with Existing Tools
page 12
M30 SQLRight TLP and SQLancer PQS fail to detect any logic bug. name 5.3 Comparisons with Existing Tools
page 12
M31 We use Squirrel as a baseline since it performs better than SQLancer in detecting memory bugs. name 5.4 Benefits of the Two Key Techniques
page 12
M32 NoREC [15] generates equivalent queries by shifting the conditions in the WHERE clause to the SELECT expression. technique 7 Related Work
page 13
M33 TLP [16] partitions a query lacking where clause into three subqueries whose where clause are x IS TRUE ,x IS FALSE, and x IS NULL. technique 7 Related Work
page 13
M34 To simplify the ground truth generation, Pivoted Query Synthesis(PQS) [14] only partly validates a query’s result. technique 7 Related Work
page 13
M35 By checking whether this row is fetched, PQS can detect logic bugs in the DBMS. technique 7 Related Work
page 13
M36 The generation-based approach [7, 14–17, 27, 37, 38] is effective in generating syntax-correct test cases since it typically follows a grammar model that describes the format of the input. citation marker 7 Related Work
page 13

This page is rendered from _data/papers/paper_doi_10_1016_j_cose_2025_104564.json, extracted from arxiv. 15 pages, 46 references parsed.