Hulk: Exploring Data-Sensitive Performance Anomalies in DBMSs via Data-Driven Analysis

Zhiyong Wu, Jie Liang, Jingzhou Fu, Mingzhe Wang, Yu Jiang. 2025. Proc. ACM Softw. Eng..

What this paper does with SQLancer

CERT is the closest prior work and the baseline, and the paper notes it is implemented in SQLancer, referring to it throughout as SQLancer-CERT -- one of only two open-source tools available for the comparison. SQLancer's own oracles are described in related work as the logic-bug counterpart to Hulk's performance focus. Hulk looks for performance anomalies that depend on the data rather than the query. The authors argue developers diagnose performance issues by intuition or by comparing execution time against a baseline DBMS, and both overlook how the dataset affects performance. Hulk instead watches performance as the dataset grows, estimating a reasonable response-time range for each data volume to locate performance cliffs, then checks whether a cliff deviates from expectation. Across six DBMSs it reported 135 anomalies, 129 confirmed as new bugs including 14 CVEs. Written by claude-opus-5 from the 10 places this paper refers to SQLancer. The quotations below are the paper's own words, stored verbatim when the text was extracted.

How it was classified

uses infrastructure — no

M3, which a pattern fired on, says CERT is implemented in SQLancer -- a fact about the baseline, not about Hulk. Hulk itself is not built on SQLancer.

extends technique — no

No technique is extended; the citation is background.

compares with — yes

M4 states SQLancer-CERT is one of two open-source tools Hulk was compared against, and M5 reports the anomaly, crash and branch figures for each.

Cardinality Estimation Restriction Testing (CERT)

In addition, we compare Hulk with the state-of-the-art DBMS validation tools in industry, including both DBMS performance testing tool APOLLO [ 30] and SQLancer𝐶𝐸𝑅𝑇[11], as well as DBMS fuzzing tools Sqirrel [60]. M2 · 1 Introduction · page 3

Since only APOLLO [30] and SQLancer𝐶𝐸𝑅𝑇[11] are open-source tools, as a remedy, we still compared Hulk with other state-of-the-art SQL fuzzer ( Sqirrel [60]) to evaluate the capability to explore the state space on DBMSs. M4 · 5.1 Evaluation Setup · page 13

Number of confirmed performance anomalies and crashes, and covered branches on six DBMSs Performance Bugs Crashes Branches APOLLO 5 6 281,056 SQLancer 7 3 273,467 Sqirrel 0 9 312,713 Hulk 42 19 331,909 data-sensitive clauses on average (which will be discussed in Section 5. M5 · 5.3 Comparison With Other Techniques · page 17

describes as state of the art — yes

M2 calls the tools Hulk is compared with, CERT among them, the state-of-the-art DBMS validation tools in industry.

SQLancer publications it cites (4)

Bibliography entries that resolved to a SQLancer publication, or to a paper by one of the project's authors. A sentence citing one of these numbers is a reference to SQLancer even when it never writes the name.

#	Entry	Matched as
11	Jinsheng Ba and Manuel Rigger. 2024. CERT: Finding Performance Issues in Database Systems Through the Lens of Cardinality Estimation. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineerin...	sqlancer publication · CERT
45	M Rigger and Z Su. [n. d.]. Finding Bugs in Database Systems via Query Partitioning. PACMPL 4 (OOPSLA)(Nov 2020).	sqlancer publication · TLP
46	Manuel Rigger and Zhendong Su. 2020. Detecting optimization bugs in database engines via non-optimizing reference engine construction. In Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conf...	sqlancer publication · NOREC
47	Manuel Rigger and Zhendong Su. 2020. Testing database engines via pivoted query synthesis. In 14th USENIX Symposium on Operating Systems Design and Implementation OSDI 20). 667–682.	sqlancer publication · PQS

Every place it refers to SQLancer (10)

10 sentences, each stored verbatim from the extracted text with where it was found and how. “Citation marker” means the sentence names no tool at all and was reached through a reference number that resolved to a SQLancer publication.

Id	Sentence	Found by	Where
M1	Additionally, CERT1[11] tests cardinality estimation to find performance anomalies.	citation marker background	1 Introduction page 3
M2	In addition, we compare Hulk with the state-of-the-art DBMS validation tools in industry, including both DBMS performance testing tool APOLLO [ 30] and SQLancer𝐶𝐸𝑅𝑇[11], as well as DBMS fuzzing tools Sqirrel [60].	citation marker baseline	1 Introduction page 3
M3	1It is implemented in SQLancer and will be referred to as SQLancer𝐶𝐸𝑅𝑇hereafter.	name definition	1 Introduction page 3
M4	Since only APOLLO [30] and SQLancer𝐶𝐸𝑅𝑇[11] are open-source tools, as a remedy, we still compared Hulk with other state-of-the-art SQL fuzzer ( Sqirrel [60]) to evaluate the capability to explore the state space on DBMSs.	citation marker baseline	5.1 Evaluation Setup page 13
M5	Number of confirmed performance anomalies and crashes, and covered branches on six DBMSs Performance Bugs Crashes Branches APOLLO 5 6 281,056 SQLancer 7 3 273,467 Sqirrel 0 9 312,713 Hulk 42 19 331,909 data-sensitive clauses on average (which will be discussed in Section 5.	name result comparison	5.3 Comparison With Other Techniques page 17
M6	To accommodate the majority of queries generated by popular DBMS fuzzers like Sqirrel and SQLancer, we set the minimum average clause number for evaluation to 5.	name background	5.5 Importance of Data-Sensitivity page 18
M7	Generally, current DBMS fuzzers are used to detect crash bugs, memory safety bugs with AddressSanitizer [ 50], or logic bugs with the test oracle [ 36,47].	citation marker background	7 Related Work page 19
M8	SQLancer [ 47] detects logic bugs of DBMSs by generating queries to fetch an existing row from databases.	name definition	7 Related Work page 19
M9	Its following works [ 45,46] also utilize the similar idea by generating semantically equivalent queries and comparing the results.	citation marker definition	7 Related Work page 19
M10	CERT [11] identifies performance issues stemming from unexpectedly estimated cardinalities, which indicate the projected number of rows returned by a query.	technique background	7 Related Work page 20

This page is rendered from _data/papers/paper_doi_10_1145_3728973.json, extracted from supplied pdf. 22 pages, 60 references parsed.