A Comprehensive Survey on Database Management System Fuzzing: Techniques, Taxonomy and Evaluation
Read the paper · doi:10.1145/3799227 · arXiv:2311.06728
What this paper does with SQLancer
SQLancer's oracles are among the principal subjects of the survey: PQS, NoREC, TLP and QPG each get their own pipeline description, and the experimental section runs them against one another, reporting for instance that QPG detects bugs more efficiently than TLP within 240 minutes.
A survey of fuzzing for database management systems, arguing that DBMSs differ from general-purpose software in internal structure, input and output, and testing objectives, so they need strategies that existing fuzzing surveys do not cover. It defines a general fuzzing procedure, decomposes existing methods against it, and classifies them again by testing objective across DBMS components. It also presents an open-source toolkit, OpenDBFuzz, and uses it to compare the surveyed methods experimentally.
How it was classified
uses infrastructure — no
The survey describes SQLancer's methods and runs them through its own toolkit; nothing in the mentions says its toolkit is built on SQLancer's codebase.
extends technique — no
A survey classifies and compares; it does not claim to extend any of the techniques it reviews.
compares with — yes
The survey's experimental section runs SQLancer's oracles against one another and reports the outcome, comparing QPG's bug-detection efficiency with TLP's over 240 minutes. That is an empirical comparison rather than a description.
Ternary Logic Partitioning (TLP)Query Plan Guidance (QPG)
Within 240 minutes, QPG detects bugs more efficiently than TLP, and the number of detected bugs gradually converges in the later stage, demonstrating that query plan feedback effectively improves the efficiency of logic bug detection.
describes as state of the art — uncertain
The survey treats SQLancer's oracles as the central methods of the field and devotes several pipelines to them, but no mention shown calls them the state of the art in those words.
What could not be determined
- 39 further mentions were omitted from this task; the survey mentions SQLancer's techniques throughout, so the roles recorded here cover the mentions read rather than all of them.
SQLancer publications it cites (16)
Bibliography entries that resolved to a SQLancer publication, or to a paper by one of the project's authors. A sentence citing one of these numbers is a reference to SQLancer even when it never writes the name.
| # | Entry | Matched as |
|---|---|---|
| 13 | Jinsheng Ba and Manuel Rigger. 2023. Testing database engines via query plan guidance. In Proceedings of the International Conference on Software Engineering. ACM, 2060–2071. | sqlancer publication · QPG |
| 14 | Jinsheng Ba and Manuel Rigger. 2024. Cert: Finding performance issues in database systems through the lens of cardinality estimation. In Proceedings of the International Conference on Software Engineering. 1–13. | sqlancer publication · CERT |
| 15 | Jinsheng Ba and Manuel Rigger. 2024. Keep it simple: Testing databases via differential query plans. Proceedingsofthe ACM on Management of Data 2, 3 (2024), 1–26. | sqlancer publication · DQP |
| 30 | Wenjing Deng, Qiuyang Mang, Chengyu Zhang, and Manuel Rigger. 2024. Finding logic bugs in spatial database engines via affine equivalent inputs. Proceedings of the ACM on Management of Data 2, 6 (2024), 1–26. | project authored |
| 34 | Jingzhou Fu, Jie Liang, Zhiyong Wu, Yanyang Zhao, Shanshan Li, and Yu Jiang. 2025. Understanding and detecting sql function bugs: Using simple boundary arguments to trigger hundreds of DBMS bugs. In Proceedings of the... | project authored |
| 51 | Yuancheng Jiang, Jiahao Liu, Jinsheng Ba, Roland H. C. Yap, Zhenkai Liang, and Manuel Rigger. 2024. Detecting logic bugs in graph database management systems via injective and surjective graph query transformation. In... | project authored |
| 52 | Yuancheng Jiang, Jianing Wang, Chuqi Zhang, Roland Yap, Zhenkai Liang, and Manuel Rigger. 2025. Enhanced differential testing in emerging database systems. arXiv:2501.01236. Retrieved from https://arxiv.org/abs/2501.0... | project authored |
| 54 | Zu-Ming Jiang, Si Liu, Manuel Rigger, and Zhendong Su. 2023. Detecting transactional bugs in database engines via{graph-based }oracle construction. In Proceedings of the USENIX Symposium on Operating Systems Design an... | project authored |
| 57 | Matteo Kamm, Manuel Rigger, Chengyu Zhang, and Zhendong Su. 2023. Testing graph database engines via query partitioning. In Proceedings of the ACMSIGSOFT International Symposium on Software Testing and Analysis. | project authored |
| 75 | Qiuyang Mang, Jinsheng Ba, Pinjia He, and Manuel Rigger. 2025. Finding logic bugs in graph-processing systems via graph-cutting. Proceedings of the ACM on Management of Data 3, 3 (2025), 1–27. | project authored |
| 90 | Manuel Rigger and Zhendong Su. 2020. Detecting optimization bugs in database engines via non-optimizing reference engine construction. In Proceedings of the Joint Meeting on European Software Engineering Conference an... | sqlancer publication · NOREC |
| 91 | Manuel Rigger and Zhendong Su. 2020. Finding bugs in database systems via query partitioning. Programming Languages 4, OOPSLA (2020), 1–30. | sqlancer publication · TLP |
| 92 | Manuel Rigger and Zhendong Su. 2020. Testing database engines via pivoted query synthesis. In Proceedingsofthe USENIX Symposium on Operating Systems Design and Implementation. USENIX Association, 667–682. | sqlancer publication · PQS |
| 93 | ManuelRiggerandZhendongSu.2022.Intramorphictesting:Anewapproachtothetestoracleproblem.In Proceedings oftheInternationalSymposiumonNewIdeas,NewParadigms,andReflectionsonProgrammingandSoftware. ACM, 128–136. | project authored |
| 112 | Chi Zhang and Manuel Rigger. 2025. Constant optimization driven database system testing. ProceedingsoftheACM on Management of Data 3, 1 (2025), 1–24. | sqlancer publication · CODDTEST |
| 117 | SuyangZhongandManuelRigger.2025.TestingDatabaseSystemswithLargeLanguageModelSynthesizedFragments. arXiv:2505.02012. Retrieved from https://arxiv.org/abs/2505.02012 | sqlancer publication |
Every place it refers to SQLancer (73)
73 sentences, each stored verbatim from the extracted text with where it was found and how. “Citation marker” means the sentence names no tool at all and was reached through a reference number that resolved to a SQLancer publication.
| Id | Sentence | Found by | Where |
|---|---|---|---|
| M1 | proposed several new fuzzing methods, including NoREC [90], TLP [91], and PQS [ 92]. |
technique |
1 Introduction page 2 |
| M2 | NoREC and TLP apply metamorphic testing to the database domain, enabling fuzzing on a single DBMS. |
technique |
1 Introduction page 2 |
| M3 | PQS is a constraint-solving testing method, and its oracle only requires that the execution result contain one row of the solving outcome, thereby accelerating the solving process compared to the former approach, which requires matching all the rows. |
technique |
1 Introduction page 2 |
| M4 | respectively proposed DQP [ 15] and Mozi [ 67], further extending metamorphic testing to different execution paths of the same query. |
technique |
1 Introduction page 2 |
| M5 | A Comprehensive Survey on Database Management System Fuzzing 268:3 Applied metamorphic testing to DBMS (NOREC [91] and TLP [92]) The concept of test oracle (Howden et al. |
technique |
1 Introduction page 2 |
| M6 | [12]) Open-source rule-based statement generator (SQLSmith [4]) 2015 Execution feedback for crash detection (Squirrel [117]) 2020 Accelerated single-row DBMS constraint solving (PQS [93]) Sub-graph isomorphism search for constraint solving (TQS [102]) 2023 Differential Testing Metamorphic Testing Constraint-solving ... |
technique |
1 Introduction page 3 |
| M7 | [79]) 2024 Execution Path Manipulation for metamorphic testing (DQP[15] and Mozi[68]) Fig. |
technique |
1 Introduction page 3 |
| M8 | Performance bugs manifest themselves as significant differences in the execution time of the same query in different versions of DBMS, or large performance gaps between a query and its equivalent [ 35,37,40,43,92]. | citation marker | 2.2 Basic Definitions page 4 |
| M9 | For example, intramorphic testing [ 93] works by replacing a specific component in the system to get another version and then testing between the two versions to detect bugs in that component. |
citation marker project authored |
2.2 Basic Definitions page 5 |
| M10 | Comparison of Test Case Generators Fuzzer YearGenerator TypeGenerator StrategyFeedbackDatabase Instance RAGS[95]1998 Generation-based AST Model (Static Configuration) No Feedback (Black-Box)Existing DatabasesSQLsmith[ 4]2015 APOLLO[ 56]2019 AST Model (Dynamic Configuration) AMOEBA[ 72]2022 Go-Randgen[ 87]2019 AST Mo... | technique | 3.1 Test Case Generator page 7 |
| M11 | Ontheotherhand,fuzzersbasedonrandomdatabases [12,14,15,27,31,33,54,80,87,91,92,96,101,105,106,116] create random database instances from scratch. | citation marker | 3.1 Test Case Generator page 8 |
| M12 | A common practice is to first create tables, indexes, and views randomly and then populate them with data using INSERT, UPDATE, and DELETE statements [ 90]. | citation marker | 3.1 Test Case Generator page 8 |
| M13 | Without this limitation, numerous complex SQL statements would be generated, which may expand the search space but reduce the overall efficiency of bug detection [ 27,31, 90–92,96]. | citation marker | 3.1 Test Case Generator page 9 |
| M14 | Mutation-based generators can be divided into three major categories: SQL structure mutation [16,25,34,53,63,64,68,105,113,116], SQL sequence mutation [ 33,66], and DBMS state mutation [13]. | citation marker project authored | 3.1 Test Case Generator page 10 |
| M15 | Some generators of black-box fuzzers [ 13] obtain feedback such as the validity of query plans or test cases from the query or query plan interface provided by the DBMS, guiding the subsequent generation process. |
citation marker |
3.1 Test Case Generator page 11 |
| M16 | Fuzzing methods based on crash oracle [33,34,53,66,105,116] can only detect crashes. | citation marker project authored | 3.2 Oracle-based Comparator page 11 |
| M17 | Comparison of Oracles Fuzzer Oracle Type Feature Test Scope Squirrel[ 116] Crash Database Crashes Crash BugsSquill[105] Griffin[33] LEGO[66] DynSQL[ 53] SOFT[34] RAGS[95] DifferentialDifferent DBMSs Logic BugsSQLsmith[ 4] Go-Randgen[ 87] GARan[ 16] DT2[27] Radar[97]Different Database InstancesDDLCheck[ 98] APOLLO[ 5... | technique | 3.2 Oracle-based Comparator page 12 |
| M18 | Most relations are equivalence-based [ 15,25,54,55,68,72,90, 91,96,106,112,113], while only a small subset are non-equivalent [ 14,44]. | citation marker | 3.2 Oracle-based Comparator page 13 |
| M19 | Expression Rewriting refers to the process of rewriting logical or arithmetic expressions, such as adding predicates that always hold true to the WHERE clause or changing the string comparison operator from ‘=’ to ‘LIKE’, which will affect the result of the expression as expected [ 14,25,44,72,90,112,113]. | citation marker | 3.2 Oracle-based Comparator page 13 |
| M20 | For instance, query hints or system variables are employed to induce the same query to be executed using different plans, detecting performance or logic bugs [ 15,67]. | citation marker | 3.2 Oracle-based Comparator page 13 |
| M21 | —Query Partition: Query partition refers to dividing an original SQL query into multiple partitions, and the merged results of the partitioned queries should be consistent with the results of the original query [ 91]. | citation marker | 3.2 Oracle-based Comparator page 13 |
| M22 | partitioned into three queries: ‘ SELECT * FROM t1 WHERE c1=10 ’, and ‘SELECT * FROM t1 WHERE c1 IS NULL ’; —Transaction Splitting: Transaction splitting refers to splitting transactions into statements and then, by building a multiple version chain of data outside the database [ 31] or reorganizing the statement ex... | citation marker project authored | 3.2 Oracle-based Comparator page 14 |
| M23 | Constraint-solving fuzzing methods [ 12,80,92,101] generally rely on forward or backward solving to derive the ground truth of the execution results. |
citation marker |
3.2 Oracle-based Comparator page 14 |
| M24 | —Backward Solving: Backward solving entails initially selecting some tuples as ground truth at random and then using an SAT solver to work backward and obtain a statement whose execution results include these tuples [ 92]. | citation marker | 3.2 Oracle-based Comparator page 14 |
| M25 | Comparison of Fuzzer Execution Feedbacks Fuzzer Metamorphic Validation Coverage Query Plan Syntax and Semantics Error APOLLO[ 56] ✓ Squirrel[ 116] ✓ LEGO[66] ✓ SQLRight[ 68] ✓ QPG[13] ✓ AMOEBA[ 72]✓ ✓ GARan[ 16] ✓ DynSQL[ 53] ✓ ✓ Squill[105] ✓ ✓ ✓ include seven types: metamorphic feedback, validation feedback, cover... |
technique |
3.3 Execution Feedback page 15 |
| M26 | Query plan feedback [ 13] uses the emergence of new query plans to guide subsequent query generation. | citation marker | 3.3 Execution Feedback page 15 |
| M27 | Comparison of Fuzzer Query Reducers FuzzerReduce ExpressionDelete ClauseDelete SubqueryDelete IR NodeSemantics Preservation RAGS[95] ✓ ✓ SQLancer[ 92]✓ ✓ MutaSQL[ 25]✓ ✓ GARan[ 16] ✓ ✓ SQLRight[ 68] ✓ ✓ APOLLO[ 56]✓ ✓ ✓ ✓ DynSQL[ 53]✓ ✓ ✓ ✓ 3. |
name |
3.3 Execution Feedback page 16 |
| M28 | Reducing expressions [ 16,25,92,95] and deleting clauses refer to the process of reducing queries by simplifying WHERE clauses, arithmetic expressions, or logical expressions. |
citation marker |
3.4 Query Reducer page 16 |
| M29 | PQS [92] begins by randomly generating a pivot row as the ground truth and then constructs an SQL query that includes this pivot row in its result. |
technique |
4.1 Overall Fuzzing page 17 |
| M30 | Figure 11illustrates the entire PQS pipeline. |
technique |
4.1 Overall Fuzzing page 17 |
| M31 | The main challenge of PQS is how to generate an SQL query whose execution result contains a known pivot row. |
technique |
4.1 Overall Fuzzing page 17 |
| M32 | The solution to PQS is to first create predicates randomly and then use the AST interpreter to evaluate whether the pivot row satisfies the predicate conditions. |
technique |
4.1 Overall Fuzzing page 17 |
| M33 | By employing the above method, PQS ensures that any random predicate can produce a satisfactory SQL query after being queried. |
technique |
4.1 Overall Fuzzing page 17 |
| M34 | The core idea of TLP [ 91] is that the result of the predicate evaluation always falls within the values of True, False, and NULL. |
technique |
4.1 Overall Fuzzing page 17 |
| M35 | Pipeline of PQS. |
technique |
4.1 Overall Fuzzing page 18 |
| M36 | Pipeline of TLP. | technique | 4.1 Overall Fuzzing page 18 |
| M37 | Pipeline of QPG. | technique | 4.1 Overall Fuzzing page 18 |
| M38 | Figure 12illustrates the main process of TLP. | technique | 4.1 Overall Fuzzing page 18 |
| M39 | Using a generator similar to PQS, the original query is split into three equivalent partitioned queries, and metamorphic oracles are employed for result verification to detect logic bugs. | technique | 4.1 Overall Fuzzing page 18 |
| M40 | Neither PQS nor TLP adopts feedback but performs a random search throughout the state space. | technique | 4.1 Overall Fuzzing page 18 |
| M41 | QPG [13] mutates the state of the database to generate more unique query plans, as shown in Figure13. | technique | 4.1 Overall Fuzzing page 18 |
| M42 | QPG implements a generation-based generator based on PQS, TLP, and NoREC. | technique | 4.1 Overall Fuzzing page 18 |
| M43 | QPG also utilizes TLP and NoREC to validate query execution results. | technique | 4.1 Overall Fuzzing page 18 |
| M44 | Since different query plans represent different execution paths, QPG can enhance the code coverage of DBMS. | technique | 4.1 Overall Fuzzing page 18 |
| M45 | Pipeline of NoREC. | technique | 4.3 Optimizer Testing page 22 |
| M46 | TheNoRECpipeline[ 90]isillustratedinFigure 20. | citation marker | 4.3 Optimizer Testing page 22 |
| M47 | Pipeline of DQP. | technique | 4.4 Executor Testing page 23 |
| M48 | DQP [15] detects logic bugs by enforcing the DBMS to utilize distinct execution plans, 𝑃0and𝑃1, for the same query 𝑄and identifying inconsistencies in their execution results. | technique | 4.4 Executor Testing page 23 |
| M49 | Finally, DQP compares execution results under these plan manipulations with those from the default execution to determine whether a logic bug exists. | technique | 4.4 Executor Testing page 23 |
| M50 | Comparison of GDBMS Fuzzing Fuzzer YearGDBMS Type Language Oracle Feature Grand[115]2022PropertyGremlin Differential Different GDBMSs GDsmith[ 70]2023 Cypher RD2[109]2023 RDFSPARQL GDBMeter[ 57]2023 PropertyGremlin and Cypher MetamorphicQuery Partition Gamera[ 119]2023Statement Rewriting, Data Manipulation GraphGeni... | citation marker project authored | 5 Non-relational DBMS Fuzzing and New Techniques page 24 |
| M51 | More recent studies—including GDBMeter [ 57], Gamera [ 119], GraphGenie [ 51], GRev [ 76], GraspDB [ 71], QuDi [ 114], GQS [ 111], and Gslicer [ 75]—focus on in-depth exploration of test oracles. | citation marker project authored | 5 Non-relational DBMS Fuzzing and New Techniques page 24 |
| M52 | Spatter [ 30] focuses on SpatialDatabaseManagementSystems (SDBMSs ) and employs affine equivalent inputs as its test oracle. | citation marker project authored | 5 Non-relational DBMS Fuzzing and New Techniques page 24 |
| M53 | A Comprehensive Survey on Database Management System Fuzzing 268:25 Additionally, SQLxDiff [ 52] enables differential testing for emerging systems—including TSDBMSs, DDBMSs, and streaming DBMSs—by mapping their queries to equivalent constructs in PostgreSQL. | citation marker project authored | 5 Non-relational DBMS Fuzzing and New Techniques page 24 |
| M54 | Similarly, ShQveL [ 117] adopts an LLM-guided approach for multi-dialect testing by decoupling structural consistency from dialect-specific features, using SQL sketchesand LLM-basedfragmentsynthesistoinjectcomplexdialectconstructs. | name | 5.3 LLM in Fuzzing page 25 |
| M55 | In order to objectively evaluate the performance of each fuzzer, we present a comprehensive open-source toolkit, OpenDBFuzz,1which integrates five relational DBMS (SQLite, MySQL, TiDB, CockroachDB, and PostgreSQL) and eight fuzzing tools (SQLancer, DQE, AMOEBA, SQLRight, Squirrel, Troc, APOLLO, and Radar), and provi... | name | 6 Benchmarks and Comparison page 25 |
| M56 | SourceCode of Related Fuzzers Fuzzer Supported DBMS Link SQLancer[ 92] (PQS[92], NoREC[ 90], TLP[91], QPG[13], CERT[14], DQP[15], CODDTest[ 112])SQLite, MySQL, TiDB, MariaDB CockroachDB, DuckDB, OceanBasehttps://github. | name | 6.1 Database Instances and Test Cases page 26 |
| M57 | com/sqlancer/sqlancer DQE[96]SQLite, MySQL, MariaDB TiDB, CockroachDBhttps://github. | name | 6.1 Database Instances and Test Cases page 26 |
| M58 | ly/3I995jL TxCheck[ 54] TiDB, MySQL, MariaDB https://github. | citation marker project authored | 6.1 Database Instances and Test Cases page 26 |
| M59 | State & Resource MutaSQL ✓ ✓ ✓ Esql ✓ ✓ ✓ ✓ ✓ ADUSA ✓ NoREC ✓ ✓ ✓ ✓ ✓ SQLRight ✓ ✓ ✓ ✓ ✓ DQE ✓ ✓ ✓ ✓ TLP ✓ ✓ ✓ ✓ ✓ TQS ✓ ✓ ✓ ✓ ✓ PINOLO ✓ ✓ ✓ ✓ EET ✓ ✓ ✓ ✓ ✓ DQP ✓ ✓ ✓ ✓ ✓ ✓ CODD ✓ ✓ Radar ✓ ✓ ✓ ✓ ✓ DDLcheck ✓ ✓ ✓ Kangaroo ✓ ✓ ✓ PQS ✓ ✓ ✓ ✓ ✓ ✓ Mozi ✓ ✓ SQLsmith ✓ ✓ ✓ ✓ ✓ DT2 ✓ Troc ✓ Fucci ✓ TxCheck ✓ ✓ WriteCheck ... | technique | 6.4 Evaluation Metrics page 27 |
| M60 | Errors squirrel ✓ ✓ ✓ ✓ ✓ Squill ✓ ✓ ✓ ✓ Griffin ✓ ✓ ✓ ✓ LEGO ✓ ✓ ✓ ✓ DynSQL ✓ ✓ ✓ ✓ ✓ SOFT ✓ ✓ ✓ ✓ ✓ ✓ SQLRight ✓ ✓ ✓ SQLsmith ✓ ✓ ✓ ✓ ✓ ✓ TLP ✓ NoREC ✓ DQP ✓ ✓ CODD ✓ DDLcheck ✓ ✓ ✓ ✓ ✓ Kangaroo ✓ ✓ ✓ PQS ✓ ✓ Radar ✓ ✓ EET ✓ ✓ ✓ ✓ DQE ✓ TxCheck ✓ ✓ ✓ ✓ ✓ ✓ WriteCheck ✓ (c) Performance Bugs Detection Fuzzer Query R... | technique | 6.4 Evaluation Metrics page 27 |
| M61 | including PQS, NoREC, TLP, QPG, DQP, and DQE. | technique | 6.5 Logic Bugs Detection Comparison page 28 |
| M62 | Among them, TLP and QPG only differ in feedback, and the other modules are consistent. | technique | 6.5 Logic Bugs Detection Comparison page 28 |
| M63 | TLP and QPG generally report more bugs and achieve higher throughput, but at the cost of lower query validity, owing to their efficient metamorphic transformationsandbroadexplorationofqueryplans. | technique | 6.5 Logic Bugs Detection Comparison page 28 |
| M64 | Finally, DQP and PQS report fewer bugs, primarily due to their stricter and more conservative oracle validation, which reduces false positives. | technique | 6.5 Logic Bugs Detection Comparison page 28 |
| M65 | rates of DQE, TLP, and QPG on this platform. | technique | 6.5 Logic Bugs Detection Comparison page 29 |
| M66 | However, this does not imply that CockroachDB is more stable; rather, the result is largely due to compatibility issues with DQE and NoREC, which frequently cause failures during database instance construction and significantly reduce the number of valid queries. | technique | 6.5 Logic Bugs Detection Comparison page 29 |
| M67 | One exception is NoREC on CockroachDB, where validity is noticeably lower due to frequent errors caused by version incompatibilities. | technique | 6.5 Logic Bugs Detection Comparison page 29 |
| M68 | DQE and NoREC achieve only about one case per second on CockroachDB because of compatibility. | technique | 6.5 Logic Bugs Detection Comparison page 29 |
| M69 | In contrast, PQS exhibits high throughput because its oracle adopts a partial validation strategy, checking only row-count consistency rather than full result equivalence. | technique | 6.5 Logic Bugs Detection Comparison page 29 |
| M70 | We conducted comparative experiments using TLP, QPG, and SQLRight on another SQLite version (3. | technique | 6.5 Logic Bugs Detection Comparison page 29 |
| M71 | These three methods employ the same oracle: TLP does not incorporate feedback, QPG adopts query plan feedback, and SQLRight utilizes coverage feedback. |
technique |
6.5 Logic Bugs Detection Comparison page 29 |
| M72 | In addition, TLP and QPG share the same generation-based statement generator. | technique | 6.5 Logic Bugs Detection Comparison page 29 |
| M73 | Within 240 minutes, QPG detects bugs more efficiently than TLP, and the number of detected bugs gradually converges in the later stage, demonstrating that query plan feedback effectively improves the efficiency of logic bug detection. | technique | 6.5 Logic Bugs Detection Comparison page 29 |