Review of Fuzz Testing Techniques for Database Management Systems
Read the paper · doi:10.1007/978-981-96-4506-0_18
What this paper does with SQLancer
SQLancer is one of the survey's principal subjects, given its own subsection as a well-known tool for detecting logical vulnerabilities and covered oracle by oracle with the bug counts each produced: over 60 logic bugs from PQS across MySQL, PostgreSQL and SQLite; 51 from NoREC across PostgreSQL, MariaDB, SQLite and CockroachDB; 175 errors from TLP across MySQL, TiDB, SQLite and CockroachDB, 77 of them logic bugs. It also appears in the survey's comparative tables, both in its own coverage row and as the baseline in Unicorn's reported figures.
A survey of fuzz testing for database management systems, classifying tools by testing type, generation method, target vulnerability class and supported systems, and tracing the field from RAGS in 1998 through the generation-based and coverage-guided tools that followed. It tabulates comparative coverage figures for the major tools and discusses how test coverage might be improved.
How it was classified
uses infrastructure — no
A survey describes tools rather than using them; nothing here reports running or building on SQLancer.
extends technique — no
No technique is proposed or extended.
compares with — no
The tabulated coverage figures are collected from the surveyed papers rather than measured here, so the survey reports comparisons rather than conducting one.
describes as state of the art — no
SQLancer is called well-known and its results are reported at length, but the survey does not claim it is the state of the art.
SQLancer publications it cites (4)
Bibliography entries that resolved to a SQLancer publication, or to a paper by one of the project's authors. A sentence citing one of these numbers is a reference to SQLancer even when it never writes the name.
| # | Entry | Matched as |
|---|---|---|
| 48 | Rigger, M., Su, Z.: Testing database engines via pivoted query synthesis. In: 14th USENIX Symposium on Operating Systems Design and Implementation (OSDI 20), pp. 667–682 (2020) | sqlancer publication · PQS |
| 49 | Rigger, M., Su, Z.: Finding bugs in database systems via query partitioning. Proc. ACM Program. Lang. 4, 1–30 (2020). https://doi.org/10.1145/3428279 | sqlancer publication · TLP |
| 54 | Rigger, M., Su, Z.: Detecting optimization bugs in database engines via non-optimizing ref-erence engine construction. In: Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symp... | sqlancer publication · NOREC |
| 55 | Rigger, M.: Sqlancer: detecting logic bugs in dbms (2020) Review of Fuzz Testing Techniques for DBMS 301 | sqlancer publication |
Every place it refers to SQLancer (13)
13 sentences, each stored verbatim from the extracted text with where it was found and how. “Citation marker” means the sentence names no tool at all and was reached through a reference number that resolved to a SQLancer publication.
| Id | Sentence | Found by | Where |
|---|---|---|---|
| M1 | Many tools have proposed innovative testing criteria, such as the PQS [ 48] and TLP [ 49]o f 288 Y. |
technique |
2.4 Application of Fuzz Testing in Database Systems page 5 |
| M2 | SQLancer. |
name |
2.4 Application of Fuzz Testing in Database Systems page 6 |
| M3 | Typical DBMS Fuzz Testing Tools and Their Support Capabilities Across Different Dimensions Tool Year Type Method Vulnerability Type Currently Supported RAGS 1998 black-box testing genertion-based logic Microsoft SQL Server SQLsmith 2015 black-box testing genertion-based crash PostgreSQL SQLite SQLancer 2020 black-bo... |
name |
2.4 Application of Fuzz Testing in Database Systems page 6 |
| M4 | (continued) Tool Year Type Method Vulnerability TypeCurrently Supported Ratel 2021 grey-box testing mutation-based crash GaussDB PostgreSQL Comdb2 Amoeba 2022 black-box testing generation-based performances PostgreSQL CocroachDB Unicorn 2022 grey-box testing mutation-based crash IoTDB KairosDB GridDB SQRight 2022 gr... |
technique |
2.4 Application of Fuzz Testing in Database Systems page 7 |
| M5 | Black-box testing does not require knowledge of the internal implementation details of the database; it focuses solely on the relationship between input and output [ 48, 50–58]. |
citation marker |
3.1 Classification Based on DBMS Testing Methodology page 7 |
| M6 | (1) SQLancer: SQLancer [ 48, 54, 55] is a well-known tool for detecting logical vulnerabilities in database management systems. |
name |
3.2 Classification of DBMS Fuzz Testing Based on Testing Objectives page 9 |
| M7 | Using the PQS method, SQLancer has conducted extensive testing on databases such as MySQL, PostgreSQL, and SQLite, ultimately discovering over 60 related logical vulnerabilities. |
name |
3.2 Classification of DBMS Fuzz Testing Based on Testing Objectives page 9 |
| M8 | The NoREC (Non-Optimizing Reference Engine Construction) criterion transforms an optimizable query into one that cannot be effectively optimized and then compares the result sets of both queries. |
technique |
3.2 Classification of DBMS Fuzz Testing Based on Testing Objectives page 9 |
| M9 | SQLancer has identified 51 logical vulnerabilities using the NoREC method in PostgreSQL, MariaDB, SQLite, and CockroachDB. |
name |
3.2 Classification of DBMS Fuzz Testing Based on Testing Objectives page 9 |
| M10 | The TLP (Ternary Logic Partitioning) criterion divides a query into multiple subqueries, each producing a portion of the original query’s results. |
technique |
3.2 Classification of DBMS Fuzz Testing Based on Testing Objectives page 9 |
| M11 | SQLancer discov-ered 175 errors using the TLP method across databases like MySQL, TiDB, SQLite, and CockroachDB, with 77 identified as logical issues. |
name |
3.2 Classification of DBMS Fuzz Testing Based on Testing Objectives page 9 |
| M12 | Specifically, Unicorn identified 21%-199% and 34%-693% more basic blocks than SQLsmith and SQLancer, respectively, across widely used time-series databases such as IoTDB, KairosDB, TimescaleDB, TDEngine, QuestDB, and GridDB. |
name |
3.3 Classification by Database Type page 11 |
| M13 | Test Tools Target Program SQLite PostgreSQL rows branch rows branch SQLsmith 27 22 33 25 Squirrel 59 49 15 11 Griffin 59 52 50 37 SQLancer 52 44 28 20 SQLRight 60 50 25 18 Apollo 26 20 – – 4. | name | 4.2 Enhancing Test Coverage page 14 |