PUPPY: Finding Performance Degradation Bugs in DBMSs via Limited-Optimization Plan Construction
Read the paper · doi:10.1109/icse55347.2025.00045
What this paper does with SQLancer
SQLancer is one of three widely used tools PUPPY is measured against over 48 hours, finding 6 bugs to PUPPY's 35. Query plan guidance is discussed as the closest prior idea, with PUPPY arguing that the sequence of optimisation operations inside a plan is a finer-grained signal than the plan itself.
PUPPY finds performance degradation bugs, where a DBMS's fully optimised plan runs slower than a plan built with only some optimisations enabled, because interactions between optimisations are complex and some cases are overlooked. PUPPY generates queries covering sequences of optimisation operations, then selectively disables optimisations to build a limited-optimisation plan; if the restricted plan is faster, that indicates a bug. Across five DBMSs it reported 62 such bugs, 54 confirmed as previously unknown.
How it was classified
uses infrastructure — no
SQLancer is cited, not used; nothing in the mentions describes reusing its code.
extends technique — no
No technique is extended; the citation is background.
compares with — yes
M2 names SQLancer among the tools PUPPY is compared with, and M3 and M4 report the outcome: 35 bugs to SQLancer's 6 over 48 hours across five systems.
Query Plan Guidance (QPG)
To further evaluate the performance of PUPPY, we also compare PUPPY with SQLancer, SQLsmith, and SQUIRREL, which are widely used in industry.
It shows that PUPPY outperforms SQLancer and SQLsmith in detecting bugs.
Specifically, PUPPY detect detected a total of 35 bugs (including 30 performance bugs and 5 crash bugs) in 48 hours, while SQLancer, SQLsmith and SQUIRREL only detected 29, 31, and 30 bugs in total.
describes as state of the art — no
The citation does not characterise SQLancer as the state of the art.
SQLancer publications it cites (5)
Bibliography entries that resolved to a SQLancer publication, or to a paper by one of the project's authors. A sentence citing one of these numbers is a reference to SQLancer even when it never writes the name.
| # | Entry | Matched as |
|---|---|---|
| 5 | B A, J., ANDRIGGER, M. Testing database engines via query plan guidance. In Proceedings of International Conference on Software Engineering (ICSE) (2023). | sqlancer publication · QPG |
| 37 | RIGGER, M. Bugs found in database management systems. https://www.manuelrigger.at/dbms-bugs, 2024. Accessed: August 19, 2024. | project authored |
| 38 | RIGGER, M., ANDSU, Z. Finding bugs in database systems via query partitioning. pacmpl 4 (oopsla)(nov 2020). | sqlancer publication · TLP |
| 39 | RIGGER, M., ANDSU, Z. Detecting optimization bugs in database engines via non-optimizing reference engine construction. In Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference andSympo... | sqlancer publication · NOREC |
| 40 | RIGGER, M., ANDSU, Z. Testing database engines via pivoted query synthesis. In 14th USENIX Symposium on Operating Systems Design and Implementation OSDI 20) (2020), pp. 667–682. | sqlancer publication · PQS |
Every place it refers to SQLancer (9)
9 sentences, each stored verbatim from the extracted text with where it was found and how. “Citation marker” means the sentence names no tool at all and was reached through a reference number that resolved to a SQLancer publication.
| Id | Sentence | Found by | Where |
|---|---|---|---|
| M1 | Ensuring the efficiency of DBMSs is crucial as it directly influences the responsiveness, scalability, and user satisfaction of dependent applications [6, 8, 37, 9, 33]. |
citation marker project authored |
I INTRODUCTION page 1 |
| M2 | To further evaluate the performance of PUPPY, we also compare PUPPY with SQLancer, SQLsmith, and SQUIRREL, which are widely used in industry. |
name |
D Efficiency of the Optimization Guided Algorithm page 10 |
| M3 | It shows that PUPPY outperforms SQLancer and SQLsmith in detecting bugs. |
name |
D Efficiency of the Optimization Guided Algorithm page 10 |
| M4 | Specifically, PUPPY detect detected a total of 35 bugs (including 30 performance bugs and 5 crash bugs) in 48 hours, while SQLancer, SQLsmith and SQUIRREL only detected 29, 31, and 30 bugs in total. |
name |
D Efficiency of the Optimization Guided Algorithm page 10 |
| M5 | TABLE VIIDETECTED BUGS BY SQLANCER, SQLSMITH ANDPUPPY IN 48HOURS. |
name |
D Efficiency of the Optimization Guided Algorithm page 10 |
| M6 | DBMS SQLancer SQLsmith SQUIRREL PUPPY MySQL 2 1 2 11 Percona 2 1 1 9 T iDB 1 0 0 5 PolarDB 1 1 2 8 PostgreSQL 0 1 0 2 T otal 6 4 5 35 VII. |
name |
D Efficiency of the Optimization Guided Algorithm page 10 |
| M7 | SQLancer [40, 39, 38] aims to find logic bugs and it generates queries based on the test oracle it builds. |
name |
A DBMS Fuzzing page 10 |
| M8 | QPG [5] gradually mutates DDL and DML statements to change database states, aiming to cover more unique query plans to cover more DBMS logic. |
technique |
A DBMS Fuzzing page 10 |
| M9 | Compared to QPG which utilizes plan-based guidance, using the sequence of optimization operations within the plan offers a finer-grained form of guidance. |
technique |
A DBMS Fuzzing page 10 |