SQLPass: A Semantic Effective Fuzzing Method for DBMS

Yu Li, Yixiao Yang, Yong Guan, Zhiping Shi, Rui Wang. 2024. Annual International Computer Software and Applications Conference.

Read the paper · doi:10.1109/compsac61105.2024.00141

What this paper does with SQLancer

SQLancer is one of four baselines, compared on semantic correctness, line coverage and bug replication speed. The paper's account of it is two-sided: because SQLancer is generation-based it cannot replicate existing bugs from seed test cases, which is one of the three measures; but it also observes that although SQLancer generates a limited range of SQL test cases, their syntactic and semantic accuracy is high, so it reaches deeper code logic and achieves higher coverage than the tools whose generated statements are less well formed. PQS's pivot-row construction is named as the reason for that accuracy. SQLPass fuzzes DBMSs with an operator selection algorithm aimed at keeping generated SQL semantically valid, on the grounds that semantically invalid statements are rejected early and never reach the deeper code. Over 24 hours on SQLite, MySQL, MariaDB and PostgreSQL it reports higher semantic correctness, higher line coverage and faster bug replication than its baselines. Written by claude-opus-5 from the 12 places this paper refers to SQLancer. The quotations below are the paper's own words, stored verbatim when the text was extracted.

How it was classified

uses infrastructure — no

SQLancer is run as a baseline; SQLPass's operator selection algorithm is its own.

extends technique — no

No SQLancer oracle is extended. SQLPass addresses how statements are generated, not what makes a result wrong.

compares with — yes

M3 states the 24-hour experiments compare SQLPass with AFL, SQLsmith, SQLancer and Squirrel on four systems, M5 reports the coverage increases against each, and M12 explains SQLancer's relative standing on accuracy and depth.

b) C omparison with benchmark testing tools: We conduct 24 hours experiments on SQLPass and benchmark testing tools AFL, SQLsmith, SQLancer, and Squirrel on SQLite, MySQL, MariaDB, and PostgreSQL, respectively, and compare and evaluate three ind icators: semantic correctness, code (line) coverage, and bug replication speed. M3 · B Experimental Comparison · page 7

The paper sets this passage with letter-spacing, so extraction returns it one character at a time and there is no readable form to quote. M5 · B Experimental Comparison · page 7

Therefore, the code coverage generated by the SQL test cases it generates is also not high; Although SQLancer generates limited types of SQL test cases, the syntax and semantic accuracy of the generated SQL statements are high, so it can trigger deeper code logic and generate higher code coverage; The syntax and semantic accuracy of SQL test cases generated by Squirrel mutation are not high, but the number of SQL statements that can mutate within the same time is relatively high, and the types of SQL statements that can mutat e are relatively diverse, resulting in a relatively high coverage rate; In contrast, SQLPass does not mutate as many SQL statements as Squirrel in the same amount of time, but its generated SQL statements have higher syntax and semantic correctness, and there are rich types of mutable SQL statements, resulting in higher code coverage. M12 · B Experimental Comparison · page 9

describes as state of the art — no

SQLancer is called a benchmark testing tool and described by its generation characteristics; no state-of-the-art claim is made.

SQLancer publications it cites (2)

Bibliography entries that resolved to a SQLancer publication, or to a paper by one of the project's authors. A sentence citing one of these numbers is a reference to SQLancer even when it never writes the name.

#	Entry	Matched as
11	M. Rigger and Z. Su, “Testing database engines via pivoted query synthesis ,” in Proceedings of the 14th USENIX Symposium on Operating Systems Design and Implementation. Alberta, Canada, pp. 667-682, 2020.	sqlancer publication · PQS
31	PostgreSQL. https://www.postgresql.org/. CREATE TABLE t1(c1,c2); CREATE TABLE t2(c1,c2); WITH t2(c1) AS (SELECT 111) INSERT INTO t1 (c1, c2) SELECT c1, 123 FROM t2; CREATE TABLE t1(c1,c2); CREATE TABLE t2(c1,c2); WITH...	project authored

Every place it refers to SQLancer (12)

12 sentences, each stored verbatim from the extracted text with where it was found and how. “Citation marker” means the sentence names no tool at all and was reached through a reference number that resolved to a SQLancer publication.

Id	Sentence	Found by	Where
M1	57b, SQLsmith, SQLancer, and Squirrel.	name baseline	A Experimental Environment page 7
M2	Experimental Comparison a) Comparison with Disab led Operator Selection Algorithm: To evaluate the contribution of the “optimal ” operator selection algorithm proposed in this paper to the performance of SQLPass, we compare the code (line) coverage of SQLite3 [8], MySQL [30], MariaDB [9], and PostgreSQL [31] when th...	citation marker project authored incidental	B Experimental Comparison page 7
M3	b) C omparison with benchmark testing tools: We conduct 24 hours experiments on SQLPass and benchmark testing tools AFL, SQLsmith, SQLancer, and Squirrel on SQLite, MySQL, MariaDB, and PostgreSQL, respectively, and compare and evaluate three ind icators: semantic correctness, code (line) coverage, and bug replicatio...	name baseline	B Experimental Comparison page 7
M4	Because SQLsmith and SQLancer are generation-based, they cannot replicate existing bugs based on seed test cases.	name result comparison	B Experimental Comparison page 7
M5	The paper sets this passage with letter-spacing, which extraction cannot undo — the sentence is there, but no readable form of it is.	name result comparison	B Experimental Comparison page 7
M6	6% respectively on MySQL; Compared to AFL, SQLancer, and Squirrel, it has increased by 83.	name result comparison	B Experimental Comparison page 7
M7	4% respectively on MariaDB; Compared to AFL, SQLsmith, SQLancer, and Squirrel, PostgreSQL has increased by 86.	name result comparison	B Experimental Comparison page 7
M8	However, due to the complex structure of the generated SQL statements, the final semantic correctness of the SQL statement s cannot be guaranteed during semantic data filling; SQLancer constructs SQL statements based on selected pivot rows.	name definition	B Experimental Comparison page 7
M9	012345678910 0 3 6 9 12 15 18 2124vulnerability number SQLite’s test time(h)0102030405060708090100 0 3 6 9 12 15 18 21 24code coverage(%) SQLite’s test time(h) 0102030405060708090100 0 3 6 9 12 15 18 21 24Semantic correctness (%) MySQL ’s test time(h)0102030405060708090100 0 3 6 9 12 15 18 21 24Semantic correctness(...	name result comparison	B Experimental Comparison page 8
M10	3% compared to the four benchmarks on SQLite3, respectively; Compared to AFL, SQLancer, and Squirrel, it has increased by 35.	name result comparison	B Experimental Comparison page 9
M11	9% compared to AFL, SQLancer, and Squirrel, respectively; Compared to the four benchmarks, PostgreSQL has increased by 36.	name result comparison	B Experimental Comparison page 9
M12	Therefore, the code coverage generated by the SQL test cases it generates is also not high; Although SQLancer generates limited types of SQL test cases, the syntax and semantic accuracy of the generated SQL statements are high, so it can trigger deeper code logic and generate higher code coverage; The syntax and sem...	name result comparison	B Experimental Comparison page 9

This page is rendered from _data/papers/paper_doi_10_1109_compsac61105_2024_00141.json, extracted from supplied pdf. 10 pages, 31 references parsed.