SQLaw: Detecting Bugs in GPU Database Management Systems via Rule-Based Differential Execution
Read the paper · doi:10.1109/tse.2025.3574328
What this paper does with SQLancer
SQLaw imitates SQLancer's statement-level, syntax-based reduction to simplify its own bug reports, and takes NoREC and TLP as two of its three baselines, describing them as representative state-of-the-art approaches that have each found over a hundred bugs in non-GPU DBMSs.
SQLaw tests GPU-based DBMSs, which the authors say had received no systematic bug-detection effort despite growing interest. It combines offline rule learning, which automatically extracts differential execution rules used to guide synthesis of configurations and queries, with an online interpreter that mutates those statements to generalise them. Evaluated on three major GPU DBMSs it outperformed state-of-the-art approaches by up to 2.22x and detected 51 previously unknown GPU-related bugs, 37 confirmed or fixed.
How it was classified
uses infrastructure — uncertain (unclear)
M3 says they imitated SQLancer's statement-level, syntax-based reduction to simplify test cases. Imitating a technique is not necessarily reusing the code, and no mention read settles which.
To identify unique GPU-related bugs, we imitated the statement-level and syntax-based reduction in SQLancer [47] to automatically simplify bug reports.
extends technique — no
A pattern fired on M9, but the rule-based differential execution SQLaw contributes is its own; NoREC and TLP are described as baselines.
compares with — yes
NoREC and TLP are two of the three representative approaches selected for comparison.
Non-optimizing Reference Engine Construction (NoREC)Ternary Logic Partitioning (TLP)
We selected three representative state-ofthe-art approaches, including SQLsmith ,NoREC and TLP .
•NoREC [52] translates query statements into nonoptimized queries in the optimizer and compares the results to identify logic bugs.
•TLP [53] partitions an original query into three separated queries by decomposing its predicate, and triggers a logic bug when the union result of these separated queries mis-matches the original one.
describes as state of the art — yes
M4 calls the three selected approaches state-of-the-art and M5 says NoREC and TLP represent the most effective for logic bugs.
We selected three representative state-ofthe-art approaches, including SQLsmith ,NoREC and TLP .
I n particular, SQLsmith is one of the most effective approaches for detecting crash and error bugs, while both NoREC and TLP represent advanced approaches for detecting logic bugs.
Where this differs from the pattern checks
The regular expressions that scan for these relationships are advisory. Where the reading above contradicts one, the reason is recorded.
What could not be determined
- Whether SQLaw reuses SQLancer's reduction code or reimplements the idea.
SQLancer publications it cites (5)
Bibliography entries that resolved to a SQLancer publication, or to a paper by one of the project's authors. A sentence citing one of these numbers is a reference to SQLancer even when it never writes the name.
| # | Entry | Matched as |
|---|---|---|
| 16 | J. Ba and M. Rigger, “Keep it simple: Testing databases via differential query plans,” in Proc. SIGMOD, 2024, pp. 38–52. | sqlancer publication · DQP |
| 47 | M. Rigger, “SQLancer,” 2024. [Online]. Available: https://github.com/ sqlancer/sqlancer | sqlancer publication |
| 52 | M. Rigger and Z. Su, “Detecting optimization bugs in database engines via non-optimizing reference engine construction,” in Proc. ESEC/FSE, 2020, pp. 1140–1152. | sqlancer publication · NOREC |
| 53 | M. Rigger and Z. Su, “Finding bugs in database systems via query partitioning,” Oopsla’20, 2020. | sqlancer publication · TLP |
| 65 | M. Kamm, M. Rigger, C. Zhang, and Z. Su, “Testing graph database engines via query partitioning,” in Proc. ISSTA, 2023, pp. 140–149. | project authored |
Every place it refers to SQLancer (23)
23 sentences, each stored verbatim from the extracted text with where it was found and how. “Citation marker” means the sentence names no tool at all and was reached through a reference number that resolved to a SQLancer publication.
| Id | Sentence | Found by | Where |
|---|---|---|---|
| M1 | Inspired by the widely adopted differential testing approach forbug detection in non-GPU DBMSs [13] ,[14] ,[15] ,[16] and GPU-related software [17] ,[18], we borrow its core concept of comparing the results of identical test cases executed ondifferent software systems to establish a test oracle [19] . |
citation marker |
I INTRODUCTION page 1 |
| M2 | First, the existing approaches for non-GPU DBMSs are limited to detecting bugs in non-GPU-specific fea-tures [13] ,[14] ,[15] ,[16]. |
citation marker |
I INTRODUCTION page 1 |
| M3 | To identify unique GPU-related bugs, we imitated the statement-level and syntax-based reduction in SQLancer [47] to automatically simplify bug reports. |
citation marker |
IV IMPLEMENTATION page 10 |
| M4 | We selected three representative state-ofthe-art approaches, including SQLsmith ,NoREC and TLP . |
technique |
A Evaluation Step page 10 |
| M5 | I n particular, SQLsmith is one of the most effective approaches for detecting crash and error bugs, while both NoREC and TLP represent advanced approaches for detecting logic bugs. |
technique |
A Evaluation Step page 10 |
| M6 | Each of them has found a total of over a hundred previously unknown bugs in various non-GPU DBMSs [51] ,[52], [53]. |
citation marker |
A Evaluation Step page 10 |
| M7 | •NoREC [52] translates query statements into nonoptimized queries in the optimizer and compares the results to identify logic bugs. |
technique |
A Evaluation Step page 10 |
| M8 | •TLP [53] partitions an original query into three separated queries by decomposing its predicate, and triggers a logic bug when the union result of these separated queries mis-matches the original one. |
technique |
A Evaluation Step page 10 |
| M9 | Consequently, we refer to the adapted approaches as SQLsmith ∗,NoREC∗and TLP∗. | technique | A Evaluation Step page 10 |
| M10 | HU AND WU: SQLAW: DETECTING BUGS IN GPUDA TABASE MANAGEMENT SYSTEMS 2155 TABLE VIIIUNIQUE BUGSDETECTED INTHREE GPU DBMS SUNDER TESTWITHIN 24 HOURS DBMS SQLaw SQLsmith∗TLP∗NoREC∗ PG-Strom 22 10 7 5 HeavyDB 15 6 6 4 Dask-sql 14 7 4 6 Total 51 23 17 15 Improvement-122% ↑ 200%↑ 240%↑ C. | technique | B Effectiveness and Efficiency of SQLaw page 11 |
| M11 | For a more comprehensive evaluation of SQLaw, we compared it with the three stateof-the-art approaches with GPU DBMSs adaptation, including SQLsmith∗, which is good at detecting GPU crash bugs and GPU error bugs, NoREC∗and TLP∗, which are proficient at finding GPU logic bugs. | technique | C Comparison With Baselines page 12 |
| M12 | As shown in Table VIII, for PG-Strom, SQLaw detected 22 GPU-related bugs, while SQLsmith∗,TLP∗and NoREC∗only found 10, 7, and 5 bugs, respectively. | technique | C Comparison With Baselines page 12 |
| M13 | For HeavyDB, SQLaw detected 15 GPU-related bugs, while SQLsmith∗,TLP∗and NoREC∗only discovered 6, 6, 4 bugs, respectively. | technique | C Comparison With Baselines page 12 |
| M14 | For Dask-sql, SQLaw detected 14 GPU-related bugs, while SQLsmith∗,TLP∗and NoREC∗only detected 7, 4, 6 bugs, respectively. | technique | C Comparison With Baselines page 12 |
| M15 | It is considerably more than 2 and 1 GPU logic bugs detected by TLP∗in PG-Strom and HeavyDB, and only 1 GPU logic bug detected by NoREC∗in PG-Strom . | technique | C Comparison With Baselines page 12 |
| M16 | The paper sets this passage with letter-spacing, which extraction cannot undo — the sentence is there, but no readable form of it is. | technique | C Comparison With Baselines page 12 |
| M17 | Neither can NoREC∗find any GPU logic bugs in either HeavyDB orDask-sql. | technique | C Comparison With Baselines page 12 |
| M18 | It should be notedthat SQLaw can only begin bug detection after completing the TABLE IXPOWER ANDMEMORY CONSUMPTION ,INCLUDINGS AVERAGE CPU POWER ,AVERAGE CPU MEMORY USAGE ,AVERAGE GPUPOWER ANDAVERAGE GPU MEMORY USAGE DBMS PG-Strom HeavyDB Dask-sql SQLaw 40W 457MB 58W 1222MB32W 1130MB 79W 2551MB62W 1407MB 81W 1980MB ... | technique | C Comparison With Baselines page 12 |
| M19 | In terms of DBMS types, significant research efforts have been devoted to relational DBMSs [14] ,[52] ,[53] ,[61] ,[62] ,[63] ,[64] ,g r a p hD B M S s [13] ,[65] ,[66] ,a n dR D Fs t o r e s [67]. | citation marker | VI RELATED WORK page 15 |
| M20 | Many studies [52] ,[53] ,[61] ,[62], [64] ,[65] ,[66] employ metamorphic testing to evaluate DBMSs. | citation marker | VI RELATED WORK page 15 |
| M21 | TLP (Ternary Logic Partitioning) is an effective metamorphic relation for detecting logic bugs in both relational DBMSs [53] and graph DBMSs [65]. | technique | VI RELATED WORK page 15 |
| M22 | To uncover logic bugs in relational DBMSs, NoREC [52] verifies the equivalence between original queriesand their non-optimized translations; PINOLO [62] employs the approximation relation; TQS [61] utilizes join optimization relations; and EET [64] constructs equivalent transformations through expression-level manipu... | technique | VI RELATED WORK page 15 |
| M23 | To detect correctness bugs, Grand [13] and RD2 [67] employ randomized differential testing for multiple graph DBMSs and RDF stores, respectively; THANOS [14] compares test cases on differential storage engines; and DQP [16] leverages differential query plans for simple yet effective bug detection. | technique | VI RELATED WORK page 15 |