Industry Practice of Coverage-Guided Enterprise-Level DBMS Fuzzing

Mingzhe Wang, Zhiyong Wu, Xinyi Xu, Jie Liang, Chijin Zhou, Huafeng Zhang, Yu Jiang. 2021. 2021 IEEE/ACM 43rd International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP).

Read the paper · doi:10.1109/icse-seip52600.2021.00042 · arXiv:2103.00804

What this paper does with SQLancer

SQLancer is one of the three fuzzers the practice was built around and measured against, described as a black-box logic-bug fuzzer that combines three oracle strategies and credited with finding over 400 bugs in two years. An industry report on bringing coverage-guided fuzzing to enterprise DBMSs at Huawei and Bloomberg. The authors note that industry has used black-box fuzzing for decades while coverage guidance has produced impressive research gains that rarely reach enterprise systems, given their complexity and distributed nature. Testing GaussDB and Comdb2 they hit challenges at all three stages -- collecting precise coverage, optimising fuzzing performance, and analysing root causes -- and built Ratel to address them, discovering 32, 42 and 5 unknown bugs in GaussDB, Comdb2 and PostgreSQL. Written by claude-opus-5 from the 33 places this paper refers to SQLancer. The quotations below are the paper's own words, stored verbatim when the text was extracted.

How it was classified

uses infrastructure — no

SQLancer is one of the fuzzers run for comparison; Ratel is its own coverage-guided tool.

extends technique — no

Nothing shown claims to extend PQS, NoREC or TLP.

compares with — yes

SQLancer is one of three fuzzers Ratel is measured against, with basic block coverage reported relative to each.

Compared to industrial black box fuzzers SQLsmith and SQLancer, as well as coverage-guided academic fuzzer Squirrel, RATEL covered 38. M1 · page 1

Compared to SQLsmith, SQLancer, and Squirrel, it covered 38. M5 · I INTRODUCTION · page 2

SQLsmith, SQLancer, and Squirrel, which are chosen for our industry practice. M6 · II BACKGROUND · page 2

describes as state of the art — no

SQLancer is described as an industrial black-box fuzzer and credited with its bug count, but not called the state of the art.

SQLancer publications it cites (5)

Bibliography entries that resolved to a SQLancer publication, or to a paper by one of the project's authors. A sentence citing one of these numbers is a reference to SQLancer even when it never writes the name.

#	Entry	Matched as
3	M. Rigger, “Sqlancer: detecting logic bugs in dbms,” 2020. [Online]. Available: https://github.com/sqlancer/sqlancer	sqlancer publication
4	M. Rigger and Z. Su, “Detecting Optimization Bugs in Database Engines via Non-Optimizing Reference Engine Construction,” in Proceedings of the 2020 28th ACM Joint Meeting on European Software Engineering Conference an...	sqlancer publication · NOREC
5	“Testing database engines via pivoted query synthesis,” in 14th USENIX Symposium on Operating Systems Design and Implementation (OSDI 20). Banff, Alberta: USENIX Association, Nov. 2020. [Online]. Available: https://ww...	sqlancer publication · PQS
14	M. Rigger and Z. Su, “Finding bugs in database systems via query partitioning,” Proc. ACM Program. Lang., no. OOPSLA, 2020.	sqlancer publication · TLP
17	M. Rigger, “Sqlancer: Bugs found in database management systems,” 2020. [Online]. Available: https://www.manuelrigger.at/dbms-bugs	sqlancer publication

Every place it refers to SQLancer (33)

33 sentences, each stored verbatim from the extracted text with where it was found and how. “Citation marker” means the sentence names no tool at all and was reached through a reference number that resolved to a SQLancer publication.

Id	Sentence	Found by	Where
M1	Compared to industrial black box fuzzers SQLsmith and SQLancer, as well as coverage-guided academic fuzzer Squirrel, RATEL covered 38.	name result comparison	page 1
M2	For example, SQLsmith [1] triggers system bugs by continuously generating random SQL queries; RAGS [2] detects logic bugs by comparing the results of a query on multiple DBMSs; SQLancer [3] detects logic bugs by constructing an invariant oracle from different angles [4], BYu Jiang is the correspondence author.	name background	I INTRODUCTION page 1
M3	[5].	citation marker background	I INTRODUCTION page 1
M4	The effectiveness of blackbox fuzzing has been proven by many previously-unknown bugs from the industry’s practice: more than 100 bugs were found by SQLsmith, and more than 400 bugs were found by SQLancer.	name background	I INTRODUCTION page 1
M5	Compared to SQLsmith, SQLancer, and Squirrel, it covered 38.	name result comparison	I INTRODUCTION page 2
M6	SQLsmith, SQLancer, and Squirrel, which are chosen for our industry practice.	name baseline	II BACKGROUND page 2
M7	In addition, SQLancer [3] integrates three different strategies [4], [5], [14] to construct invirant oracle to detect logic bugs.	name definition	B Fuzzing DBMSs page 2
M8	For example, its pivoted query synthesis strategy [5] generates queries of which corresponding result table is supposed to include a speciﬁc row, and if the DBMS fails to fetch the row, a logic bug is discovered.	technique definition	B Fuzzing DBMSs page 2
M9	For example, SQLsmith has found 118 bugs in the past ﬁve years [16], SQLancer has found over 400 bugs in the past two years [17].	name incidental	B Fuzzing DBMSs page 3
M10	•SQLancer is a fuzzer for hunting logic bugs in DBMS.	name definition	C Fuzzers Chosen by This Paper page 3
M11	It detects logic bugs by constructing invariant oracle [4], [5], [14] and checking whether results violate semantic logic.	citation marker definition	C Fuzzers Chosen by This Paper page 3
M12	TABLE IFEATURES OFCHOSEN FUZZERS Fuzzer SQLsmith SQLancer Squirrel Syntax Validity√ √ √ Semantic Validity ×√ √ Logic Check ×√× Coverage-Guided × ×√ Adaption Difﬁculty Easy Medium Hard III.	name baseline	C Fuzzers Chosen by This Paper page 3
M13	Many tools are implemented following this method, including SQLsmith, SQLancer, and Squirrel.	name background	C Fuzzers Chosen by This Paper page 4
M14	To detect semantic bugs, tools such as SQLancer implement a test oracle by verifying the returned values.	name definition	C Fuzzers Chosen by This Paper page 4
M15	SQLsmith, SQLancer, and Squirrel to test enterprise-level DBMSs.	name	C Fuzzers Chosen by This Paper page 4
M16	We observed similar results on SQLancer.	name	A Imprecise Coverage Collection page 4
M17	For example, SQLsmith and SQLancer test DBMS with blackbox methods.	name	A Imprecise Coverage Collection page 5
M18	These generation-based methods have shown signiﬁcant effectiveness in DBMS bug discovery [8], [17].	citation marker	B Fragile Input Generation page 6
M19	Take PostgreSQL as an example: SQLsmith only targets PostgreSQL’s SELECT statement, and its grammar model already has 42 elements; SQLancer use over 8,000 lines of Java code to generate syntactically-correct test cases; Squirrel uses over 33,000 lines of C++ code to translate between SQL statement and AST.	name	B Fragile Input Generation page 6
M20	As Table I shows, besides RATEL, we used SQLsmith and SQLancer for grammar-based techniques with and without semantic checks.	name	A Analysis of Results page 7
M21	TABLE IICOVERAGE OF DBMS FOREACH FUZZER DBMS SQLsmith SQLancer Squirrel RATEL GaussDB 50,172 2,513 N/A 69,432 PostgreSQL 42,563 39,913 16,954 87,739 Comdb2 N/A 2,773 N/A 18,941 Table II presents the number of covered basic blocks for each fuzzer.	name	A Analysis of Results page 7
M22	For example, SQLancer covered more basic blocks than Squirrel on PostgreSQL but under-performed SQLsmith on GaussDB and PostgreSQL.	name	A Analysis of Results page 7
M23	The reason is that SQLancer is specialized in logic bug hunting and only uses a part of SQL features to construct its oracle violation.	name	A Analysis of Results page 7
M24	SQLancer also uses advanced test oracles to detect bugs, thus the CREATE TABLE statements generated by it have many ﬁne-tuned parameters.	name	A Analysis of Results page 7
M25	63x more basic blocks than SQLsmith and SQLancer, respectively; On PostgreSQL, it covers 1.	name	A Analysis of Results page 7
M26	29x more basic blocks than SQLsmith, SQLancer, and Squirrel, respectively; On Comdb2, it covers 5.	name	A Analysis of Results page 7
M27	83x than SQLancer.	name	A Analysis of Results page 7
M28	100101102103 GaussDB(Paths)0200040006000800010000120001400016000 100101102103 GaussDB(Basic Blocks)010000200003000040000500006000070000 100101102103 PostgreSQL(Paths)0200040006000800010000120001400016000 100101102103 PostgreSQL(Basic Blocks)0100002000030000400005000060000700008000090000 100101102103 Comdb2(Paths)020...	name	A Analysis of Results page 7
M29	In contrast, SQLancer, SQLsmith, and Squirrel did not ﬁnd them.	name	B Analysis of Bugs page 8
M30	Both SQLancer and SQLsmith did not ﬁnd this bug.	name	B Analysis of Bugs page 8
M31	Due to the incomplete syntax modeling, SQLancer and SQLsmith can never generate such queries.	name	B Analysis of Bugs page 8
M32	For blackbox fuzzers like SQLsmith and SQLancer, they have a low possibility of generating SQL statements to cover these states without feedback.	name	B Analysis of Bugs page 9
M33	For example, Squirrel and SQLancer drop databases after each query.	name	B Analysis of Bugs page 9

This page is rendered from _data/papers/paper_doi_10_1109_icse_seip52600_2021_00042.json, extracted from arxiv. 10 pages, 26 references parsed.