Towards Generic Database Management System Fuzzing
What this paper does with SQLancer
SQLancer is one of BuzzBee's baselines for the relational case, run with its PQS oracle, and also on ArangoDB because SQLancer had recently added support for it. The comparison is reported per bug: on several Redis and RedisGraph defects SQLancer's column is empty, and the paper attributes that to scope -- it looks for logic errors through specific syntax structures rather than the crashes and semantic faults those systems exhibit. The paper is even-handed about the relational case, noting that none of BuzzBee, SQUIRREL or SQLancer found a bug in current PostgreSQL within 24 hours. SQLancer's design also informs BuzzBee indirectly through the IR line of work it belongs to.
BuzzBee fuzzes database systems beyond the relational model -- key-value, graph and document stores -- where SQL-specific fuzzers do not apply. It generalises the intermediate-representation approach that SQL fuzzers use for tracking syntax and semantics into a form covering several data models, so that semantic correctness survives mutation. Evaluated on Redis, RedisGraph, ArangoDB, MongoDB and relational systems, it found bugs the specialised tools miss.
How it was classified
uses infrastructure — no
SQLancer is executed as a baseline; BuzzBee's own implementation is an independent intermediate representation, not built on SQLancer.
extends technique — no
BuzzBee generalises the IR approach it credits to SQUIRREL and its successors, not a SQLancer oracle.
compares with — yes
M11 lists SQLancer among the fuzzers BuzzBee is compared with, M13 states the relational comparison is against SQUIRREL and SQLancer using PQS, M14 that SQLancer was additionally run on ArangoDB, and M16 reports a result covering all three tools.
Pivoted Query Synthesis (PQS)
We compare BUZZBEEwith general-purpose fuzzers AFL++ [ 14],REDQUEEN [4], syntax-aware fuzzers POLYGLOT [9], Grammarinator [ 22], and SQL-specialized SQUIR REL [56] and SQLANCER [43].
For relational DBMSs, we compare BUZZBEEwith SQUIRREL [56] and SQL ANCER (PQS [ 43]), two DBMS fuzzers specialized in SQL DBMS fuzzing.
We also evaluate SQLANCER on ArangoDB because SQLANCER recently adds support for it.
None of BUZZBEE,SQUIRREL, and SQLANCER can find bugs in the latest version of PostgreSQL within 24 hours.
describes as state of the art — no
SQLancer is called SQL-specialized and a well-known tool, but the paper does not describe it as the state of the art.
SQLancer publications it cites (4)
Bibliography entries that resolved to a SQLancer publication, or to a paper by one of the project's authors. A sentence citing one of these numbers is a reference to SQLancer even when it never writes the name.
| # | Entry | Matched as |
|---|---|---|
| 26 | Matteo Kamm, Manuel Rigger, Chengyu Zhang, and Zhendong Su. Testing Graph Database Engines via Query Partitioning. In René Just and Gordon Fraser, editors, Proceedings of the 32nd ACMSIGSOFT International Symposium on... | project authored |
| 42 | Manuel Rigger and Zhendong Su. Detecting Optimization Bugs in Database Engines via Non-optimizing Reference Engine Construction. In Prem Devanbu, Myra B. Cohen, and Thomas Zimmermann, editors, ESEC/FSE ’20: 28th ACM J... | sqlancer publication · NOREC |
| 43 | Manuel Rigger and Zhendong Su. Testing Database Engines via Pivoted Query Synthesis. In 14th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2020, Virtual Event, November 4-6, 2020, pages 667–682... | sqlancer publication · PQS |
| 56 | Rui Zhong, Yongheng Chen, Hong Hu, Hangfan Zhang, Wenke Lee, and Dinghao Wu. SQUIRREL: Testing Database Management Systems with Language Validity and Coverage Feedback. In Jay Ligatti, Xinming Ou, Jonathan Katz, and G... | sqlancer publication |
Every place it refers to SQLancer (21)
21 sentences, each stored verbatim from the extracted text with where it was found and how. “Citation marker” means the sentence names no tool at all and was reached through a reference number that resolved to a SQLancer publication.
| Id | Sentence | Found by | Where |
|---|---|---|---|
| M1 | Fuzzing frameworks and research related to relational DBMSs [27,28,43,44,49,56] have been developed and advanced extensively over the years, contributing to more secure and trustworthy systems in the relational DBMS venue. |
citation marker |
1 Introduction page 2 |
| M2 | Current research has made significant advancements in relational DBMS testing [ 27,28,43,44,49,56]. |
citation marker |
2.2 Existing Challenges and Limitations page 3 |
| M3 | SQUIRREL [56] and later works [ 27,28] advance by introducing an intermediate representation (IR) that effectively incorporates SQL syntax and semantics, allowing it to adapt to multiple SQL databases. |
citation marker |
2.2 Existing Challenges and Limitations page 4 |
| M4 | Recent studies [ 27,28,56] highlight the significance of maintaining the semantic correctness of the test cases in mutation-based DBMS fuzzing. |
citation marker |
2.2 Existing Challenges and Limitations page 4 |
| M5 | To model the constraints, the approach used by existing fuzzing frameworks [ 9,27,28,56] is HMSET k1 k1_field1 1HMSET k1 k1_field1 1APPEND x 413413HMSET k1 k1_field1 1XGROUP CREATs g 0HMSET k1 k1_field1 1HRANDFIELD key1 1. |
citation marker |
2.2 Existing Challenges and Limitations page 4 |
| M6 | Existing mutation-based DBMS fuzzers [ 27,28,56] perform mutations randomly and rely on code coverage to explore more program behaviors. |
citation marker |
2.2 Existing Challenges and Limitations page 5 |
| M7 | Inspired by existing works [ 27,28,56], we design an IR to incorporate both the syntactic structures and the abstract semantics of the inputs. |
citation marker |
2.3 Our Insights and Solutions page 5 |
| M8 | 4 Generalization Mutation can easily break the semantic correctness of a test case, as mentioned in many related works [ 27,56]. |
citation marker |
2.3 Our Insights and Solutions page 6 |
| M9 | Inspired by existing works [ 27,28,56],BUZZBEE’s IR is a tree structure with a one-to-one mapping to the abstract syntax tree of the original test case. |
citation marker |
2.3 Our Insights and Solutions page 6 |
| M10 | SQUIRREL SQLANCER redis 2 Sem ✔ ✔ ✗ ✗ ✗ ✗ ✗ ✗ ⊖ ⊖ redis 3 Data ✔ ✔ ✔ ✗ ✗ ✗ ✗ ✗ ⊖ ⊖ redis 5 Data ✔ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ⊖ ⊖ redis 6 Data ✔ ✔ ✗ ✗ ✗ ✗ ✗ ✗ ⊖ ⊖ redis 7 Data ✔ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ⊖ ⊖ redis 8 Syn ✗ ✗ ✗ ✔ ✗ ✔ ✗ ✗ ⊖ ⊖ redis 12 Sem ✔ ✔ ✔ ✗ ✗ ✗ ✗ ✗ ⊖ ⊖ RedisGraph 19 Data ✔ ✔ ✗ ✗ ✗ ✗ ✗ ✗ ⊖ ⊖ RedisGraph 20 Sem ✔ ✔ ✗ ✗ ✗ ✗ ✗... |
name |
8.4 Contributions of the Solutions page 13 |
| M11 | We compare BUZZBEEwith general-purpose fuzzers AFL++ [ 14],REDQUEEN [4], syntax-aware fuzzers POLYGLOT [9], Grammarinator [ 22], and SQL-specialized SQUIR REL [56] and SQLANCER [43]. |
name |
8.5 Comparison with Existing Tools page 13 |
| M12 | AFL++ and REDQUEEN are widely used coverage-guided fuzzers that perform syntax912 33rd USENIX Security Symposium USENIX Association 0 4 8 12 16 2001224364860EdgeCov (1E+02) BuzzBee AFL++ RedQueen PolyGlot Grammarinator 0 4 8 12 16 2001224364860EdgeCov (1E+02) (a)redis 0 4 8 12 16 2001632486480EdgeCov (1E+02) (b)Redi... |
name |
8.5 Comparison with Existing Tools page 13 |
| M13 | For relational DBMSs, we compare BUZZBEEwith SQUIRREL [56] and SQL ANCER (PQS [ 43]), two DBMS fuzzers specialized in SQL DBMS fuzzing. |
technique |
8.5 Comparison with Existing Tools page 14 |
| M14 | We also evaluate SQLANCER on ArangoDB because SQLANCER recently adds support for it. |
name |
8.5 Comparison with Existing Tools page 14 |
| M15 | Referring to SQUIRREL ’s paper [ 56], it achieves a semantic correctness rate of 11. |
citation marker |
8.5 Comparison with Existing Tools page 14 |
| M16 | None of BUZZBEE,SQUIRREL, and SQLANCER can find bugs in the latest version of PostgreSQL within 24 hours. |
name |
8.5 Comparison with Existing Tools page 14 |
| M17 | SQLANCER focuses on finding logic errors using specific syntax structures and does not discover the bug as well. |
name |
8.5 Comparison with Existing Tools page 14 |
| M18 | 3 DBMSs Fuzzing Fuzzing DBMSs has been an active research area in recent years [ 15,25–27,43,44,49,56]. |
citation marker |
10.3 DBMSs Fuzzing page 15 |
| M19 | Tools like SQLancer [ 42], SQLsmith [ 44], and Squirrel [ 56] have emerged to test relational DBMSs. |
name |
10.3 DBMSs Fuzzing page 15 |
| M20 | SQLancer uses differential testing techniques to report inconsistencies in query results. |
name |
10.3 DBMSs Fuzzing page 15 |
| M21 | Researchers also propose some solutions targeting non-relational DBMSs [ 26,55]. |
citation marker project authored |
10.3 DBMSs Fuzzing page 15 |