Thanos: DBMS Bug Detection via Storage Engine Rotation Based Differential Testing

Ying Fu, Zhiyong Wu, Yuanliang Zhang, Jie Liang, Jingzhou Fu, Yu Jiang, Shanshan Li, Xiangke Liao. 2025. International Conference on Software Engineering.

Read the paper · doi:10.1109/icse55347.2025.00257

What this paper does with SQLancer

SQLancer is one of Thanos's three baselines and is named a state-of-the-art DBMS testing tool throughout. The comparison is reported per system in both bugs and branch coverage, and SQLancer found none of the 14 bugs Thanos did. The paper's argument for why is about where the oracles apply: SQLancer's rely on specially crafted rules -- NoREC on optimizer behaviour, and it describes TLP as needing a pivot row -- which leaves storage-engine differences outside their reach, and its use of standard SQL syntax bounds the code it exercises. SQLancer's construction of functionally equivalent queries is also cited as the metamorphic-testing line Thanos departs from. Thanos tests DBMSs that support several storage engines by running the same workload under each in turn and comparing the results. Because the engines sit beneath a shared SQL layer, any difference between them is a defect in one of them, which gives an oracle without needing to know the correct answer. On MySQL, MariaDB and Percona it found 14 bugs in 24 hours and covered substantially more of the storage-engine code than the existing tools. Written by claude-opus-5 from the 20 places this paper refers to SQLancer. The quotations below are the paper's own words, stored verbatim when the text was extracted.

How it was classified

uses infrastructure — no

SQLancer is run as a baseline; Thanos rotates storage engines beneath a shared SQL layer and nothing indicates its implementation reuses SQLancer's code.

extends technique — no

Thanos's oracle is differential comparison across storage engines, which the paper contrasts with SQLancer's rule-based oracles rather than deriving from them.

compares with — yes

M8 states the comparative study pits Thanos against SQLancer, and M10, M11 and M12 give the per-DBMS bug counts, branch coverage and totals.

Non-optimizing Reference Engine Construction (NoREC)Ternary Logic Partitioning (TLP)

Comparison with Existing Techniques To assess the effectiveness of THANOS, we conducted a comparative study that pitted THANOS against contemporary state-of-the-art DBMS testing methods, namely SQLancer, SQLsmith, and SQUIRREL. M8 · C Comparison with Existing Techniques · page 8

DBMS SQLancer SQLsmith SQUIRREL THANOS MySQL 0 1 1 6 MariaDB 0 0 1 5 Percona 0 0 1 3 Total 0 1 3 14 Increment 14↑ 13↑ 11↑ – TABLE IV: Number of branches covered by THANOS, SQLancer, SQLsmith, and SQUIRREL on 3 DBMSs in 24 hours. M10 · C Comparison with Existing Techniques · page 8

DBMS SQLancer SQLsmith SQUIRREL THANOS MySQL 59,242 93,742 109,323 120,156 MariaDB 60,293 88,923 100,920 132,532 Percona 63,829 89,987 109,823 143,293 Total 183,364 272,652 320,066 395,981 Increment 115. M11 · C Comparison with Existing Techniques · page 8

Specifically, THANOS found 14, 13, and 11 more bugs than SQLancer, SQLsmith, and SQUIRREL, respectively. M12 · C Comparison with Existing Techniques · page 9

describes as state of the art — yes

M1, M2 and M5 each introduce SQLancer as a state-of-the-art DBMS fuzzer or testing tool.

We evaluate THANOS on three widely used and extensively tested DBMSs, namely MySQL, MariaDB, and Percona against state-of-the-art fuzzers SQLancer, SQLsmith, and SQUIRREL. M1 · page 1

To assess the effectiveness of THANOS, we compare THANOS against contemporary state-of-the-art DBMS testing methods, namely SQLancer, SQLsmith, and SQUIRREL. M2 · I INTRODUCTION · page 2

To evaluate the effectiveness of THANOS, we compared THANOS with the state-of-art DBMS test tools, SQUIRREL [22, 23], SQLancer [24], and SQLsmith [25, 26]. M5 · A Evaluation Setup · page 7

SQLancer publications it cites (5)

Bibliography entries that resolved to a SQLancer publication, or to a paper by one of the project's authors. A sentence citing one of these numbers is a reference to SQLancer even when it never writes the name.

#	Entry	Matched as
24	M. Rigger, “Sqlancer website,” https://github .com/ sqlancer/sqlancer, accessed: August 16, 2024.	sqlancer publication
30	M. Rigger and Z. Su, “Finding bugs in database systems via query partitioning,” Proceedings of the ACM on 665 Programming Languages, vol. 4, no. OOPSLA, pp. 1– 30, 2020.	sqlancer publication · TLP
31	Rigger, Manuel and Su, Zhendong, “Detecting optimization bugs in database engines via non-optimizing reference engine construction,” in Proceedings of the 28th ACM Joint Meeting on European Software Engineering Confer...	sqlancer publication · NOREC
33	M. Rigger and Z. Su, “Testing database engines via pivoted query synthesis,” in 14th USENIX Symposium on Operating Systems Design and Implementation (OSDI 20), 2020, pp. 667–682.	sqlancer publication · PQS
34	Z.-M. Jiang, S. Liu, M. Rigger, and Z. Su, “Detecting transactional bugs in database engines via GraphBased oracle construction,” in 17th USENIX Symposium on Operating Systems Design and Implementation (OSDI 23). Bost...	project authored

Every place it refers to SQLancer (20)

20 sentences, each stored verbatim from the extracted text with where it was found and how. “Citation marker” means the sentence names no tool at all and was reached through a reference number that resolved to a SQLancer publication.

Id	Sentence	Found by	Where
M1	We evaluate THANOS on three widely used and extensively tested DBMSs, namely MySQL, MariaDB, and Percona against state-of-the-art fuzzers SQLancer, SQLsmith, and SQUIRREL.	name state of the art	page 1
M2	To assess the effectiveness of THANOS, we compare THANOS against contemporary state-of-the-art DBMS testing methods, namely SQLancer, SQLsmith, and SQUIRREL.	name state of the art	I INTRODUCTION page 2
M3	72% more branches, and finds 14, 13, and 11 more bugs in 24 hours on three DBMSs than SQLancer, SQLsmith, and SQUIRREL, respectively.	name result comparison	I INTRODUCTION page 2
M4	Similarly, metamorphic testing tools like SQLancer, which rely on specially crafted rules still prove ineffective in detecting this particular problem.	name motivation	II BACKGROUND ANDMOTIVATION page 3
M5	To evaluate the effectiveness of THANOS, we compared THANOS with the state-of-art DBMS test tools, SQUIRREL [22, 23], SQLancer [24], and SQLsmith [25, 26].	name state of the art	A Evaluation Setup page 7
M6	SQLancer generates SQL queries and detects the logic bugs of DBMSs.	name definition	A Evaluation Setup page 7
M7	Or they require adherence to specific rules, as seen in SQLancer where the NOREC test oracle is optimizer-related, and TLP necessitates a pivot row.	name motivation	B DBMS Bug Detection page 7
M8	Comparison with Existing Techniques To assess the effectiveness of THANOS, we conducted a comparative study that pitted THANOS against contemporary state-of-the-art DBMS testing methods, namely SQLancer, SQLsmith, and SQUIRREL.	name baseline	C Comparison with Existing Techniques page 8
M9	Each DBMS underwent a 24hour testing period using these tools, and we documented TABLE III: Number of bugs detected by THANOS, SQLancer, SQLsmith and SQUIRREL on 3 DBMSs in 24 hours.	name result comparison	C Comparison with Existing Techniques page 8
M10	DBMS SQLancer SQLsmith SQUIRREL THANOS MySQL 0 1 1 6 MariaDB 0 0 1 5 Percona 0 0 1 3 Total 0 1 3 14 Increment 14↑ 13↑ 11↑ – TABLE IV: Number of branches covered by THANOS, SQLancer, SQLsmith, and SQUIRREL on 3 DBMSs in 24 hours.	name result comparison	C Comparison with Existing Techniques page 8
M11	DBMS SQLancer SQLsmith SQUIRREL THANOS MySQL 59,242 93,742 109,323 120,156 MariaDB 60,293 88,923 100,920 132,532 Percona 63,829 89,987 109,823 143,293 Total 183,364 272,652 320,066 395,981 Increment 115.	name result comparison	C Comparison with Existing Techniques page 8
M12	Specifically, THANOS found 14, 13, and 11 more bugs than SQLancer, SQLsmith, and SQUIRREL, respectively.	name result comparison	C Comparison with Existing Techniques page 9
M13	72% more branches than SQLancer, SQLsmith, and SQUIRREL, respectively.	name result comparison	C Comparison with Existing Techniques page 9
M14	Specifically, the NoREC test oracle in SQLancer detects logical errors in the DBMS optimizer by constructing equivalent optimized and unoptimized queries based on op-timizer rules.	name definition	C Comparison with Existing Techniques page 9
M15	In comparison, tools like SQLancer and SQLsmith use standard SQL syntax to build test cases, resulting in THANOS covering more branches and detecting more bugs.	name result comparison	C Comparison with Existing Techniques page 9
M16	5: Number of branches covered by THANOS, SQLancer, SQLsmith, and SQUIRREL in the Storage Engine component of each DBMS in 24 hours.	name result comparison	D Effectiveness of Feature-Oriented Test Case Synthesis page 9
M17	Metamorphic testing in DBMS involves transforming SQL queries and verifying if the resulting output changes align with expected behavior [28, 29, 30, 31].	citation marker background	D Effectiveness of Feature-Oriented Test Case Synthesis page 10
M18	SQLancer proposes constructing functionally equivalent queries to test one DBMS [30, 31, 33].	name definition	D Effectiveness of Feature-Oriented Test Case Synthesis page 10
M19	TxCheck [34] detects transactional bugs of DBMSs through graph-based oracle construction.	citation marker project authored background	D Effectiveness of Feature-Oriented Test Case Synthesis page 10
M20	DBMS fuzzers [22, 25, 33, 36, 37, 38, 39, 40, 41, 42], automate this process, focusing on creating complex SQL queries to uncover memory safety issues.	citation marker background	D Effectiveness of Feature-Oriented Test Case Synthesis page 10

This page is rendered from _data/papers/paper_doi_10_1109_icse55347_2025_00257.json, extracted from supplied pdf. 12 pages, 42 references parsed.