← Research building on SQLancer

Shutao Chu, Yongjun Wang, Haoran Xu, Zhiyuan Jiang, Yongxin Chen. 2024. IEEE International Conference on Tools with Artificial Intelligence.

Read the paper · doi:10.1109/ictai62512.2024.00085

What this paper does with SQLancer

SQLancer is one of NNTailor's two baselines, compared on code coverage and on the syntactic and semantic correctness of generated queries. The paper attributes SQLancer's coverage ceiling to template reliance -- predefined rules achieve considerable coverage but bound how much of the code space can be explored -- while acknowledging the other side of that trade: because SQLancer's oracles require expressions built to predefined semantic rules, it achieves high semantic correctness on SQLite. The comparison is reported as mixed rather than one-sided, with the paper noting that the two tools have their own advantages on different systems. NNTailor drives DBMS fuzzing with a neural network: a model trained on SQL learns how to mutate queries so they stay syntactically and semantically valid, instead of relying on hand-written templates. On the latest PostgreSQL and SQLite it reports 35% higher line coverage than SQLsmith and 10% higher than SQLancer, along with better validity of the generated queries. Written by claude-opus-5 from the 15 places this paper refers to SQLancer. The quotations below are the paper's own words, stored verbatim when the text was extracted.

How it was classified

uses infrastructure — no

SQLancer is run as a baseline; NNTailor's generation is driven by a neural network model of its own.

extends technique — no

No SQLancer oracle is extended. NNTailor changes how queries are produced, not what makes a result wrong.

compares with — yes

M1 states the evaluation compares NNTailor's performance with SQLsmith and SQLancer, M6 that the comparison covers coverage and query correctness, and M2 gives the coverage result.

We conducted evaluations of NNTailor on the latest versions of PostgreSQL and SQLite, comparing its performance with SQLsmith[5] and SQLancer[6, 25, 26, 27]. M1 · I INTRODUCTION · page 2
With the neural network model-driven mutation approach, NNTailor achieves 35% and 10% higher code line coverage compared to SQLsmith and SQLancer. M2 · I INTRODUCTION · page 2
EVALUATION This section compares NNTailor with typical DBMS blackbox fuzzing tools SQLsmith and SQLancer in terms of coverage on the target DBMS, syntactic and semantic correctness of synthetic queries. M6 · G Test · page 6

describes as state of the art — no

SQLancer is called an open-source black box fuzzing tool; the paper makes no state-of-the-art claim about it.

SQLancer publications it cites (5)

Bibliography entries that resolved to a SQLancer publication, or to a paper by one of the project's authors. A sentence citing one of these numbers is a reference to SQLancer even when it never writes the name.

#EntryMatched as
6 Jinsheng Ba and Manuel Rigger. Testing database engines via query plan guidance. In The 45th International Conference on Software Engineering (ICSE’23), May 2023. sqlancer publication · QPG
8 Chris Cummins, Pavlos Petoumenos, Alastair Murray, and Hugh Leather. Compiler fuzzing through deep learning. In Proceedings of the 27th ACMSIGSOFT In570 Fuzzer PostgreSQL SQLite NNTailor 34% 37% SQLsmith 29% 24% SQLan... sqlancer publication
25 Manuel Rigger and Zhendong Su. Detecting Optimization Bugs in Database Engines via Non-Optimizing Reference Engine Construction. In Proceedings of the 2020 28th ACM Joint Meeting on European Software Engineering Confe... sqlancer publication · NOREC
26 Manuel Rigger and Zhendong Su. Finding bugs in database systems via query partitioning. Proc. ACM Program. Lang., 4(OOPSLA), 2020. doi: 10.1145/3428279. sqlancer publication · TLP
27 Manuel Rigger and Zhendong Su. Testing database engines via pivoted query synthesis. In Proceedings of the 14th USENIX Symposium on Operating Systems Design and Implementation OSDI 20, page 667–682, 2020. sqlancer publication · PQS

Every place it refers to SQLancer (15)

15 sentences, each stored verbatim from the extracted text with where it was found and how. “Citation marker” means the sentence names no tool at all and was reached through a reference number that resolved to a SQLancer publication.

Id Sentence Found by Where
M1 We conducted evaluations of NNTailor on the latest versions of PostgreSQL and SQLite, comparing its performance with SQLsmith[5] and SQLancer[6, 25, 26, 27]. name
baseline 		I INTRODUCTION
page 2
M2 With the neural network model-driven mutation approach, NNTailor achieves 35% and 10% higher code line coverage compared to SQLsmith and SQLancer. name
result comparison 		I INTRODUCTION
page 2
M3 NNTailor and SQLancer have their own advantages in different DBMS. name
result comparison 		I INTRODUCTION
page 2
M4 Experimental results demonstrate that NNTailor surpasses open-source black box fuzzing tools such as SQLsmith and SQLancer in terms of code coverage and test case effectiveness. name
result comparison 		I INTRODUCTION
page 2
M5 [8] introduced a method leveraging Long Short-Term Memory (LSTM)[14] models to autonomously learn syntax specifications of the OpenCL programming language. citation marker
background 		II BACKGROUND
page 3
M6 EVALUATION This section compares NNTailor with typical DBMS blackbox fuzzing tools SQLsmith and SQLancer in terms of coverage on the target DBMS, syntactic and semantic correctness of synthetic queries. name
baseline 		G Test
page 6
M7 SQLsmith detects database system crashes,while SQLancer detects database system logical problems. name
definition 		G Test
page 6
M8 At the time of writing, SQLsmith’s official support for PostgreSQL and SQLite was notably robust,and SQLancer has not yet implemented the NoREC test oracle on MySQL. name
incidental 		A Experimental Setup
page 6
M9 SQLsmith and SQLancer achieves considerable coverage using predefined rules and extensive executions, yet its reliance on templates limits their ability to fully explore the code space. name
result comparison 		B Code Coverage
page 6
M10 This strategy improves semantic validity and results in superior coverage performance compared to SQLsmith and SQLancer. name
result comparison 		B Code Coverage
page 6
M11 SQLancer is not fully adapted to the latest version of PostgreSQL, so its grammatical correctness is lower than that of other tools. name
result comparison 		C Validity
page 7
M12 SQLancer’s test oracle require it to generate SQL expressions according to predefined semantic rules, so it can achieve high semantic correctness on SQLite. name
result comparison 		C Validity
page 7
M13 In recent years, several DBMS fuzzing tools like Sqlsmith, Sqlancer, Squirrel [28], SQLRight [22], DynSQL [17], Apollo[18], Griffin[9], and LEGO[21] have emerged, effectively uncovering numerous issues in popular DBMS and bolstering their security. name C Validity
page 7
M14 Other types of vulnerabilities are addressed with innovative oracle like PQS and TLP introduced by SQLancer. name C Validity
page 7
M15 Experimental results demonstrate that, compared to the classic black-box DBMS fuzzing tools SQLsmith and SQLancer, NNTailor outperforms SQLsmith and SQLancer in terms of comprehensive performance in code coverage and test case validity. name C Validity
page 7

This page is rendered from _data/papers/paper_doi_10_1109_ictai62512_2024_00085.json, extracted from supplied pdf. 9 pages, 34 references parsed.