Finding Performance Issues in Database Systems by Exploiting Dormant Code Paths
Read the paper · arXiv:2605.22992
What this paper does with SQLancer
SQLancer is used two ways. It supplies the workload: the authors collected ten thousand randomly generated small workloads from it, described as a popular DBMS testing tool, and those are the queries QueryZen runs against the flipped branches. Separately, CERT is one of four prior performance-testing methods QueryZen is measured against, and the comparison is reported issue by issue -- of the 21 issues found, 4 could potentially have been found by CERT, and 21 of 21 already existed when CERT was published. Note that SQLancer here generates input for a different technique rather than being the thing QueryZen is built on.
QueryZen looks for performance issues in code the optimiser rarely reaches. Its starting observation is that 8 of 10 historical performance issues found by APOLLO, AMOEBA, CERT and PUPPY could have been reached by flipping a branch, so instead of manipulating the input it manipulates the program: branches are flipped to force execution down dormant paths, and the resulting slowdowns are examined. On PostgreSQL, MySQL, MariaDB and CockroachDB it found 21 issues, most of which the prior methods cannot reach and which pre-dated their publication.
How it was classified
uses infrastructure — yes (workload)
M7 states that ten thousand randomly generated small workloads were collected from SQLancer. That is reuse, but only as a source of queries for a technique that operates on the DBMS's own code -- QueryZen is not built on SQLancer.
In this paper, we collected 10 thousand randomly generated small workloads from SQLancer[27, 28], which is a popular DBMS testing tool.
extends technique — no
Flipping branches to reach dormant code is not a development of any SQLancer oracle; the paper positions it against input-manipulating approaches including CERT.
compares with — yes
M5 and M9 name CERT among the four prior performance-testing methods considered, and M12 reports how many of QueryZen's issues each could potentially have found.
Cardinality Estimation Restriction Testing (CERT)
We considered the performance issues found by the four existing performance-testing methods:APOLLO[16],AMOEBA[21],CERT[1], and PUPPY[40].
We considered the same performancetesting methods in Section 3:APOLLO[16],AMOEBA[21], CERT[1], and PUPPY[40].
Overall, 8, 0, 4, and 1 issues have the potential to be found by APOLLO,AMOEBA, CERT, and PUPPY, respectively.
describes as state of the art — no
SQLancer is called a popular DBMS testing tool and CERT is described by what it does; neither is presented as the state of the art.
SQLancer publications it cites (5)
Bibliography entries that resolved to a SQLancer publication, or to a paper by one of the project's authors. A sentence citing one of these numbers is a reference to SQLancer even when it never writes the name.
| # | Entry | Matched as |
|---|---|---|
| 1 | Jinsheng Ba and Manuel Rigger. Cert: Finding performance issues in database systems through the lens of cardinality estimation. InProceedings of the IEEE/ACM 46th International Conference on Software Engineering, page... | sqlancer publication · CERT |
| 2 | Jinsheng Ba and Manuel Rigger. Keep it simple: Testing databases via differential query plans.Proceedings of the ACM on Management of Data, 2(3):1–26, 2024. | sqlancer publication · DQP |
| 27 | Manuel Rigger and Zhendong Su. Detecting optimization bugs in database engines via non-optimizing reference engine construction. In Prem Devanbu, Myra B. Cohen, and Thomas Zimmermann, editors,ESEC/FSE ’20: 28th ACM Jo... | sqlancer publication · NOREC |
| 28 | Manuel Rigger and Zhendong Su. Finding bugs in database systems via query partitioning.Proc. ACM Program. Lang., 4(OOPSLA):211:1–211:30, 2020. | sqlancer publication · TLP |
| 29 | Manuel Rigger and Zhendong Su. Testing database engines via pivoted query synthesis. In14th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2020, Virtual Event, November 4-6, 2020, pages 667–682.... | sqlancer publication · PQS |
Every place it refers to SQLancer (18)
18 sentences, each stored verbatim from the extracted text with where it was found and how. “Citation marker” means the sentence names no tool at all and was reached through a reference number that resolved to a SQLancer publication.
| Id | Sentence | Found by | Where |
|---|---|---|---|
| M1 | CERT[1] finds performance issues by finding inconsistent cardinality estimation, which is typically deemed as the most critical component for query optimization [17]. |
technique |
1 Introduction page 1 |
| M2 | To manipulate code, we propose flipping branches, which is motivated by our observation that 8 of 10 historical performance issues from APOLLO,AMOEBA,CERT, and PUPPYcould be found by flipping branches. |
technique |
1 Introduction page 2 |
| M3 | We evaluatedQueryZenon PostgreSQL, MySQL, MariaDB, and CockroachDB, which are widely used in previous performance testing works [1, 16, 40]. |
citation marker |
1 Introduction page 2 |
| M4 | QueryZenis necessary to find these issues as most of them cannot be found by APOLLO,CERT,PUPPY, and AMOEBA. |
technique |
1 Introduction page 2 |
| M5 | We considered the performance issues found by the four existing performance-testing methods:APOLLO[16],AMOEBA[21],CERT[1], and PUPPY[40]. |
technique |
3 Motivating Study page 3 |
| M6 | Table 1 shows the 10 performance issues from APOLLO,CERT,PUPPY, and AMOEBA. |
technique |
3 Motivating Study page 3 |
| M7 | In this paper, we collected 10 thousand randomly generated small workloads from SQLancer[27, 28], which is a popular DBMS testing tool. |
citation marker |
4 Approach page 5 |
| M8 | These DBMSs have been extensively tested by previous works about finding performance issues [1, 16, 20, 21, 40]. |
citation marker |
6 Evaluation page 7 |
| M9 | We considered the same performancetesting methods in Section 3:APOLLO[16],AMOEBA[21], CERT[1], and PUPPY[40]. |
technique |
6 Evaluation page 8 |
| M10 | We evaluated whether the performance issues found byQueryZencan be potentially found by prior performance-testing methods APOLLO,AMOEBA,CERT, and PUPPY. |
technique |
6 Evaluation page 8 |
| M11 | DBMS Issue IDAPOLLO AMOEBA CERTPUPPY PostgreSQL 1 PostgreSQL 2 PostgreSQL 3✓ PostgreSQL 4✓ PostgreSQL 5 MySQL 116456 MySQL 116484✓ MySQL 116309✓ MySQL 116534 MySQL 116773 MySQL 116774✓ MySQL 116775 MySQL 116776✓ ✓ MySQL 116777✓ CockorachDB 134803 CockorachDB 135001✓ CockorachDB 136350✓ ✓ MariaDB 35280 MariaDB 35332 ... |
citation marker |
6 Evaluation page 9 |
| M12 | Overall, 8, 0, 4, and 1 issues have the potential to be found by APOLLO,AMOEBA, CERT, and PUPPY, respectively. |
technique |
6 Evaluation page 9 |
| M13 | Additionally, we evaluated whether these performance issues existed when prior performancetesting methods APOLLO,AMOEBA,CERT, and PUPPY were proposed. |
technique |
6 Evaluation page 9 |
| M14 | In total, 14, 19, 21, and 21 of 21 issues existed before the publication date of APOLLO,AMOEBA,CERT, and PUPPY. |
technique |
6 Evaluation page 9 |
| M15 | Most issues found byQueryZencannot be found by APOLLO,AMOEBA,CERT, and PUPPY, and existed before the publication of prior methods. |
technique |
6 Evaluation page 9 |
| M16 | Unlike existing approaches [1, 21, 40], which focus on manipulating external inputs,BFAoperates on the program code itself. |
citation marker |
7 Discussion page 11 |
| M17 | proposed CERT[1] to find performance issues by testing specifically cardinality estimation. |
citation marker |
8 Related Work page 11 |
| M18 | Oracles such as PQS [29], NoREC [27],and TLP [28] detect logic bugs in SELECT statement implementations, while DQE [31] targets logic errors in UPDATE and INSERT statements. |
technique |
8 Related Work page 12 |