DynSQL: Stateful Fuzzing for Database Management Systems with Complex and Valid SQL Query Generation

Zu-Ming Jiang, Jia-Ju Bai, Zhendong Su. 2023. USENIX Security Symposium.

What this paper does with SQLancer

SQLancer appears as related work rather than as a baseline, and the paper is explicit about why: it states that DynSQL and SQLancer are designed for different research problems and that no comparison experiment against SQLancer was run. Its baselines are SQLsmith and SQUIRREL. SQLancer is described as a well-known logic-bug detector integrating PQS, NoREC and TLP, each summarised in the related-work section, and one design point is credited to it -- DynSQL's scheduler checks whether the DBMS reports an error in the way SQLancer does. The paper also cites SQLancer's own observation that the common core of SQL features across dialects is small, using it to justify not relying on a single grammar template. DynSQL fuzzes DBMSs with multi-statement SQL queries, targeting the trade-off that earlier generators face between query complexity and validity: single-statement tools sidestep inter-statement dependencies, while SQUIRREL's static intermediate representation still yields over 50% invalid queries. DynSQL instead builds a query one statement at a time, querying the live DBMS state after each, so dependencies are resolved dynamically. It found 40 bugs in six DBMSs, 38 confirmed and 21 fixed. Written by claude-opus-5 from the 25 places this paper refers to SQLancer. The quotations below are the paper's own words, stored verbatim when the text was extracted.

How it was classified

uses infrastructure — no

SQLancer is cited and compared against, not reused; nothing describes building on its code.

extends technique — no

No SQLancer technique is extended; the work takes a different approach.

compares with — no

M15 states outright that the authors do not conduct a comparison experiment against SQLancer, on the grounds that the two address different research problems. The evaluated baselines are SQLsmith and SQUIRREL.

describes as state of the art — no

SQLancer is called a well-known DBMS testing tool in M14, which is recognition but not a claim that it is the state of the art; the paper reserves that description for the tools it measures against.

SQLancer publications it cites (5)

Bibliography entries that resolved to a SQLancer publication, or to a paper by one of the project's authors. A sentence citing one of these numbers is a reference to SQLancer even when it never writes the name.

#	Entry	Matched as
35	Manuel Rigger and Zhendong Su. Detecting optimization bugs in database engines via non-optimizing reference engine construction. In Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference...	sqlancer publication · NOREC
36	Manuel Rigger and Zhendong Su. Finding bugs in database systems via query partitioning. Proceedings of the ACM on Programming Languages, 4(OOPSLA):1– 30, 2020.	sqlancer publication · TLP
37	Manuel Rigger and Zhendong Su. Testing database engines via pivoted query synthesis. In Proceedings of the 14th USENIX Symposium on Operating Systems Design and Implementation (OSDI), pages 667–682, 2020.	sqlancer publication · PQS
41	SQLancer. https://github.com/sqlancer/sqla ncer.	sqlancer publication
48	Rui Zhong, Yongheng Chen, Hong Hu, Hangfan Zhang, Wenke Lee, and Dinghao Wu. SQUIRREL: testing database management systems with language validity and coverage feedback. In Proceedings of the 2020 International Confere...	project authored

Every place it refers to SQLancer (25)

25 sentences, each stored verbatim from the extracted text with where it was found and how. “Citation marker” means the sentence names no tool at all and was reached through a reference number that resolved to a SQLancer publication.

Id	Sentence	Found by	Where
M1	Fuzzing is a promising technique for bug detection [2,4,15, 17, 26, 30, 47], and it is applied to testing DBMSs [18, 43, 45, 48] by generating SQL (Structured Query Language) queries that contain a series of SQL statements [48].	citation marker project authored background	1 Introduction page 2
M2	To improve bug detection in DBMSs, several approaches [45, 48] further involve feedback mechanisms.	citation marker project authored background	1 Introduction page 2
M3	For example, SQLsmith [43] generates only one statement in each query, avoiding the analysis of dependencies among statements, which sacriﬁces complexity for validity; SQUIRREL [48] uses an intermediate representation (IR) model to infer dependencies and generate queries that contain multiple statements, but it prod...	citation marker project authored background	1 Introduction page 2
M4	SQUIRREL [48]) but suffer from soundness and completeness issues.	citation marker project authored background	1 Introduction page 2
M5	Instead, they either generate only one complex statement in each query to avoid the analysis of state changes [18, 43], or combine multiple relatively simple statements where the state changes can be easily inferred [48].	citation marker project authored background	2 Background and Motivation page 4
M6	Using an intermediate representation (IR) to maintain query structures, SQUIRREL [48] is aware of state changes caused by its generated statements.	citation marker project authored background	2 Background and Motivation page 4
M7	Even so, SQUIRREL still generates over 50% invalid queries [48].	citation marker project authored background	2 Background and Motivation page 4
M8	Typically, AST-based tools [39, 41, 43] generate random SQL statements according to their random seed (e.	citation marker background	3.1 Dynamic Query Interaction page 5
M9	Similar to SQLancer [37], Scheduler also checks whether the target DBMS reports any error.	name definition	3.1 Dynamic Query Interaction page 6
M10	Because many DBMSs use their own SQL dialects and the common core of their SQL features is small [37, 39], it is difﬁcult to use one grammar template to test all DBMSs effectively.	citation marker motivation	4 Framework and Implementation page 7
M11	We choose these DBMSs because they are widely used according to DB-Engines Ranking [12] and extensively tested [18, 24, 41, 43, 48].	citation marker background	5.1 Experimental Setup page 8
M12	2 Runtime Testing Following the evaluation setup of SQUIRREL [48] and the recommendations of Klees et al.	citation marker project authored background	5.2 Runtime Testing page 8
M13	5 Comparison to Existing DBMS Fuzzers We compare DynSQL to two state-of-the-art DBMS fuzzers, SQLsmith [43] and SQUIRREL [48].	citation marker project authored background	5.5 Comparison to Existing DBMS Fuzzers page 11
M14	SQLancer [35 –37] is also a well-known DBMS testing tool, but it mainly focuses on test oracles, which require test cases with speciﬁc patterns to 4958 32nd USENIX Security Symposium USENIX Association Table 7: Results of comparison DBMSSQLsmith SQUIRREL DynSQL Statement Query Bug Statement Query Bug Statement Query...	name definition	5.5 Comparison to Existing DBMS Fuzzers page 11
M15	Considering DynSQL and SQLancer are designed for different research problems, we do not conduct the comparison experiment to SQLancer.	name incidental	5.5 Comparison to Existing DBMS Fuzzers page 12
M16	Several existing DBMS testing works [35 –37] use special oracles to detect such bugs.	citation marker background	6 Limitations and Future Work page 13
M17	They either discover speciﬁc kinds of bugs [18,23,25,34 –37], or detect common bugs using general techniques [19, 20, 39].	citation marker background	7.1 DBMS Testing page 14
M18	SQLancer [41] is designed to detect logic bugs in DBMSs, and it integrates several novel approaches [35 –37].	name definition	7.1 DBMS Testing page 14
M19	PQS [37] can generate queries that require the target DBMS to return a result set where a speciﬁc row should be included.	technique definition	7.1 DBMS Testing page 14
M20	NoREC [35] is a metamorphic testing approach.	technique definition	7.1 DBMS Testing page 14
M21	NoREC identiﬁes a bug when these two queries make the DBMS return different results.	technique definition	7.1 DBMS Testing page 14
M22	However, this approach is limited because the common part of supported SQL features in different DBMSs is small [37, 39].	citation marker motivation	7.1 DBMS Testing page 14
M23	However, existing work [48] proves that general fuzzers cannot effectively test DBMSs.	citation marker project authored background	7.1 DBMS Testing page 14
M24	To test DBMSs more effectively, several approaches [18, 43, 45, 48] combine fuzzing with grammarbased generation techniques.	citation marker project authored background	7.1 DBMS Testing page 14
M25	To generate queries with multiple statements, SQUIRREL [48] uses a new intermediate representation (IR) to model SQL queries and statically infers DBMS state changes caused by the generated statements.	citation marker project authored background	7.1 DBMS Testing page 14

This page is rendered from _data/papers/paper_s2_3a7373bd891702ae93d7e058241a7e8be25e89eb.json, extracted from usenix. 18 pages, 48 references parsed.