Automated Discovery of Test Oracles for Database Management Systems Using LLMs
Read the paper · arXiv:2510.06663
What this paper does with SQLancer
Argus uses SQLancer++'s query generator to produce seed queries while deliberately not using its predefined oracles, since discovering oracles is the paper's own subject. It measures itself against SQLancer, SQLancer++ and EET on coverage, and calls SQLancer a state-of-the-art open-source DBMS testing framework. Its artifact carries SQLancer's own source files.
Argus automates what has been the manual part of DBMS test oracle design: inventing mechanisms that generate equivalent query pairs. It builds on the Constrained Abstract Query, a SQL skeleton with placeholders and conditions on how they may be filled; a language model proposes pairs of skeletons, their equivalence is formally proven with a SQL equivalence solver, and only then are the placeholders instantiated with synthesised snippets. On five extensively tested DBMSs it found 41 previously unknown bugs, 36 of them logic bugs.
How it was classified
uses infrastructure — yes (generator)
M11 states that SQLancer++'s query generator produces the seed queries while its predefined oracles are deliberately not used, and M14 that SQLancer can be used directly for schema and data generation. So a component is reused rather than the framework being built on. The artifact carrying SQLancer source files agrees.
These snippets can be generated offline by a hybrid approach combining an LLM to cover diverse database features and a high-throughput generator, such as SQLancer [ 49], even though the prover currently cannot reason about them.
Specifically, we use SQLancer++ [ 82]’s query generator to produce seed queries, but we do not use their predefined test oracles (such as TLP [48] and NoREC [47]) for bug detection.
,SQLancer can be directly used for schema and concrete 4, No.
extends technique — no
Argus discovers new oracles rather than extending an existing one; TLP appears as the worked example of what a Constrained Abstract Query can represent, not as something being generalised.
compares with — yes
Coverage is compared against SQLancer and SQLancer++ over 24-hour runs, with line, function and branch figures reported for each.
2 Code Coverage We comparedArguswith three DBMS logic bugs finding tools, SQLancer [ 3,47–49,74], SQLancer++ [ 82] and EET [ 24] in multiple coverage metrics.
1% higher line and branch coverage than SQLancer++, respectively, and 11.
571×more line, function, and branch coverage than SQLancer, respectively.
describes as state of the art — yes
M31 calls SQLancer a state-of-the-art, open-source DBMS testing framework supporting most of the latest test oracles.
As a state-of-the-art, open-source DBMS testing framework, SQLancer supports most of the latest test oracles [ 4,57,74].
Its artifact
joyemang33/Argus carries sqlancer source in nested artifact.
Repository names this paper.
source
# Argus: Automated Discovery of Test Oracles for Database Management Systems Using LLMs Argus is a novel framework for automatically discovering and instantiating test oracles to find logic bugs in Database Management Systems (DBMSs) using Large Language Models (LLMs).
source
MetamorphicCoverageArtifact/Code/mc-guided_fuzzing/src/sqlancer/Randomly.java is SQLancer's Randomly.java, but it sits under MetamorphicCoverageArtifact, which looks like a bundled copy of another project
source
private static StringGenerationStrategy stringGenerationStrategy = StringGenerationStrategy.SOPHISTICATED;
source
SQLancer publications it cites (14)
Bibliography entries that resolved to a SQLancer publication, or to a paper by one of the project's authors. A sentence citing one of these numbers is a reference to SQLancer even when it never writes the name.
| # | Entry | Matched as |
|---|---|---|
| 2 | Jinsheng Ba, Yuancheng Jiang, and Manuel Rigger. 2025. Metamorphic Coverage.arXiv preprint arXiv:2508.16307 (2025). | project authored |
| 3 | Jinsheng Ba and Manuel Rigger. 2023. Testing database engines via query plan guidance. In2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). IEEE, 2060–2071. | sqlancer publication · QPG |
| 4 | Jinsheng Ba and Manuel Rigger. 2024. Cert: Finding performance issues in database systems through the lens of cardinality estimation. InProceedings of the IEEE/ACM 46th International Conference on Software Engineering... | sqlancer publication · CERT |
| 5 | Jinsheng Ba and Manuel Rigger. 2024. Keep it simple: Testing databases via differential query plans.Proceedings of the ACM on Management of Data2, 3 (2024), 1–26. | sqlancer publication · DQP |
| 12 | Wenjing Deng, Qiuyang Mang, Chengyu Zhang, and Manuel Rigger. 2024. Finding logic bugs in spatial database engines via affine equivalent inputs.Proceedings of the ACM on Management of Data2, 6 (2024), 1–26. | project authored |
| 23 | Yuancheng Jiang, Jiahao Liu, Jinsheng Ba, Roland HC Yap, Zhenkai Liang, and Manuel Rigger. 2024. Detecting logic bugs in graph database management systems via injective and surjective graph query transformation. InPro... | project authored |
| 37 | Qiuyang Mang, Jinsheng Ba, Pinjia He, and Manuel Rigger. 2025. Finding Logic Bugs in Graph-processing Systems via Graph-cutting.Proceedings of the ACM on Management of Data3, 3 (2025), 1–27. | project authored |
| 47 | Manuel Rigger and Zhendong Su. 2020. Detecting optimization bugs in database engines via non-optimizing reference engine construction. InProceedings of the 28th ACM Joint Meeting on European Software Engineering Confe... | sqlancer publication · NOREC |
| 48 | Manuel Rigger and Zhendong Su. 2020. Finding bugs in database systems via query partitioning.Proceedings of the ACM on Programming Languages4, OOPSLA (2020), 1–30. | sqlancer publication · TLP |
| 49 | Manuel Rigger and Zhendong Su. 2020. Testing database engines via pivoted query synthesis. In14th USENIX Symposium on Operating Systems Design and Implementation (OSDI 20). 667–682. 4, No. 3 (SIGMOD), Article 140. Pub... | sqlancer publication · PQS |
| 74 | Chi Zhang and Manuel Rigger. 2025. Constant Optimization Driven Database System Testing.Proceedings of the ACM on Management of Data3, 1 (2025), 1–24. 4, No. 3 (SIGMOD), Article 140. Publication date: June 2026. 140:2... | sqlancer publication · CODDTEST |
| 81 | Suyang Zhong and Manuel Rigger. 2025. Testing Database Systems with Large Language Model Synthesized Fragments. arXiv preprint arXiv:2505.02012(2025). | sqlancer publication |
| 82 | Suyang Zhong and Manuel Rigger. 2026. Scaling Automated Database System Testing. InProceedings of the 31st ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume ... | sqlancer publication |
| 87 | Ándré Albrecht and Contributors. 2025. sqlparse: A non-validating SQL parser module for Python. https://github.com/ andialbrecht/sqlparse. Accessed: 2025-10-15. 4, No. 3 (SIGMOD), Article 140. Publication date: June 2... | sqlancer publication |
Every place it refers to SQLancer (59)
59 sentences, each stored verbatim from the extracted text with where it was found and how. “Citation marker” means the sentence names no tool at all and was reached through a reference number that resolved to a SQLancer publication.
| Id | Sentence | Found by | Where |
|---|---|---|---|
| M1 | Logic bugs are particularly insidious; they cause a DBMS to return incorrect results without raising errors, therefore silently corrupting downstream applications [ 47]. | citation marker | 1 Introduction page 2 |
| M2 | In response, the research community has developed automated testing techniques [ 24,47–49,74] that have discovered hundreds of bugs in real-world systems. | citation marker | 1 Introduction page 2 |
| M3 | For example, Ternary Logic Partitioning (TLP) [ 48] is a highly effective oracle that partitions a query𝑄based on a predicate 𝑃and then checks equivalence between 𝑄and the union of its three-way partition 𝑄WHERE𝑃 ,𝑄WHERE NOT𝑃, and𝑄WHERE𝑃IS NULL. |
technique |
1 Introduction page 2 |
| M4 | Compared to traditional SQL generators [ 16,53], this is infeasible for modern DBMS testing, which often requires executing thousands of queries per minute to find bugs efficiently, such as in SQLancer [3]. |
name |
1 Introduction page 2 |
| M5 | An example of representing and instantiating TLP [48] oracle in CAQ. |
technique |
1 Introduction page 3 |
| M6 | For instance, TLP [ 48] can be represented as an equivalent CAQ pair,i. |
technique |
1 Introduction page 3 |
| M7 | These snippets can be generated offline by a hybrid approach combining an LLM to cover diverse database features and a high-throughput generator, such as SQLancer [ 49], even though the prover currently cannot reason about them. |
name |
1 Introduction page 4 |
| M8 | 43 ×in metamorphic coverage [ 2], a recently developed coverage indicator to assess the ability to find logic bugs. | citation marker project authored | 1 Introduction page 4 |
| M9 | 33 ×more unique logic bugs than the sum of prior oracles from these works [5, 24, 47, 48] within 6 hours of testing on Dolt [ 14]. | citation marker | 1 Introduction page 4 |
| M10 | In the context of DBMS testing, test oracles typically operate by transforming a given SQL query into a semantically equivalent variant, as in TLP [ 48], NoREC [ 47], and EET [ 24]. | technique | 2 Background page 5 |
| M11 | Specifically, we use SQLancer++ [ 82]’s query generator to produce seed queries, but we do not use their predefined test oracles (such as TLP [48] and NoREC [47]) for bug detection. |
name |
3 Argus Overview page 5 |
| M12 | Note that these are also widely adopted constraints in prior work on manually designed test oracles [ 3,24,48,74]. | citation marker | 4 Constrained Abstract Query page 7 |
| M13 | ,SQLancer [3]). |
name |
5 Test Oracle Discovery page 8 |
| M14 | ,SQLancer can be directly used for schema and concrete 4, No. |
name |
5.1 Database Seeding page 8 |
| M15 | Instead of pursuing syntactic differences, our goal is to produce queries with variedquery plans, as this is a key principle to creating effective test oracles in previous works [ 3,5,74]. | citation marker | 5.2 CAQ Pairs Generation page 9 |
| M16 | The first constraint is also widely used in previous works [ 47–49] and can be checked by applying a regular expression to identify non-deterministic features. | citation marker | 6.2 Query Instantiation page 13 |
| M17 | We basically follow the widely-used random data generation method [ 47–49] to populate the tables in the instantiated schema 𝑠with random tuples, and then create random indices on those tables to diversify the execution plans. | citation marker | 6.3 Database Instantiation and Bug Reporting page 13 |
| M18 | Note that, though our test oracles can also detect performance issues similar to [ 25,36], we do not detect them in a large scale due to those issues are typically regarded as expected behaviors by developers [4]. | citation marker | 6.3 Database Instantiation and Bug Reporting page 13 |
| M19 | DBMS Tested by GitHub stars Released LOC Dolt [82] 19. | citation marker | 7 Evaluation page 14 |
| M20 | 1k 2018 380k DuckDB [15, 16, 48, 74, 82] 32. | citation marker | 7 Evaluation page 14 |
| M21 | 7k 2019 1,496k MySQL [24, 31, 49, 57, 58, 60, 61, 74, 80, 82] 11. | citation marker | 7 Evaluation page 14 |
| M22 | 7k 1995 5,532k PostgreSQL [16, 24, 30, 47, 49, 58, 80, 82] 18. | citation marker | 7 Evaluation page 14 |
| M23 | 5k 1995 938k TiDB [3, 24, 57, 58, 60, 61, 74, 82] 39. | citation marker | 7 Evaluation page 14 |
| M24 | ,reporting 21 [ 5], 24 [74], and 35 [ 24] logic bugs),Argusfinds more logic bugs in comparison, even though the DBMSs 4, No. | citation marker | 7.1 New Bugs and Oracles page 14 |
| M25 | While several recent database testing studies have targeted PostgreSQL [ 58,71,81,82], none reported finding new bugs, which attests to the effectiveness of our approach. | citation marker | 7.1 New Bugs and Oracles page 15 |
| M26 | We also find that prior test oracles, such as TLP [ 48], struggle to detect this bug. | technique | 7.1 New Bugs and Oracles page 15 |
| M27 | As a result, TLP does not report a bug in this case. | technique | 7.1 New Bugs and Oracles page 15 |
| M28 | Argus SQLancer SQLancer++ EET 0 4 8 12 16 20 24 Hours3031323334353637Line Coverage (%) (a) DuckDB line 0 4 8 12 16 20 24 Hours18192021Branch Coverage (%) (b) DuckDB branch 0 4 8 12 16 20 24 Hours20222426283032Line Coverage (%) (c) PostgreSQL line 0 4 8 12 16 20 24 Hours141618202224Branch Coverage (%) (d) PostgreSQL ... |
name |
7.1 New Bugs and Oracles page 17 |
| M29 | Code coverage achieved by Argus, SQLancer, and SQLancer++ on DuckDB and PostgreSQL over 24-hour runs. |
name |
7.1 New Bugs and Oracles page 17 |
| M30 | 2 Code Coverage We comparedArguswith three DBMS logic bugs finding tools, SQLancer [ 3,47–49,74], SQLancer++ [ 82] and EET [ 24] in multiple coverage metrics. |
name |
7.2 Code Coverage page 17 |
| M31 | As a state-of-the-art, open-source DBMS testing framework, SQLancer supports most of the latest test oracles [ 4,57,74]. |
name |
7.2 Code Coverage page 17 |
| M32 | SQLancer++ is a scalable variant that can be easily extended to multiple DBMSs with only lightweight modifications. | name | 7.2 Code Coverage page 17 |
| M33 | Approach Lines Functions Branches SQLancer 3. | name | 7.2 Code Coverage page 18 |
| M34 | Average metamorphic coverage on DuckDB of 10 test suites for Argus and SQLancer. | name | 7.2 Code Coverage page 18 |
| M35 | to find logic bugs [ 82], it is still a fair metric to evaluate the diversity of test cases generated by Argusand the effectiveness of its corpus synthesis. | citation marker | 7.2 Code Coverage page 18 |
| M36 | Meanwhile,metamorphic coverage[ 2] is a more relevant metric for evaluating the effectiveness of test oracles and is highly related to logic bug finding abilities according to the historical bug study in SQLite and DuckDB [ 2]. | citation marker project authored | 7.2 Code Coverage page 18 |
| M37 | 1% higher line and branch coverage than SQLancer++, respectively, and 11. |
name |
7.2 Code Coverage page 18 |
| M38 | 4% higher than SQLancer. | name | 7.2 Code Coverage page 18 |
| M39 | In PostgreSQL,Argusslightly underperforms SQLancer overall but outperforms SQLancer++ (by 19. | name | 7.2 Code Coverage page 18 |
| M40 | This suboptimal performance is expected, as SQLancer has been extensively optimized specifically for PostgreSQL by the open-source community over many years. | name | 7.2 Code Coverage page 18 |
| M41 | Specifically, SQLancer supports 22 types of DDL statements (e. | name | 7.2 Code Coverage page 18 |
| M42 | For a finer-grained comparison, we additionally evaluatedArgusand SQLancer on PostgreSQL with respect to optimizer code coverage, since the optimizer is a core component that is closely tied to SELECT queries and widely studied in prior work [ 47,60], along with the diversity of features exercised by SELECT queries ... | name | 7.2 Code Coverage page 18 |
| M43 | 17% by SQLancer. | name | 7.2 Code Coverage page 18 |
| M44 | To evaluate feature diversity, we randomly sampled ten SELECT queries from each tool and counted their unique features using two third-party parsers:pglast[ 17] (counting number of unique PostgreSQL types) andsqlparse[ 87] (counting number of unique AST nodes). | citation marker | 7.2 Code Coverage page 18 |
| M45 | We found thatArguscovered 23 features in pglast and 151 features in sqlparse, while SQLancer only covered 15 and 76, respectively. | name | 7.2 Code Coverage page 18 |
| M46 | For metamorphic coverage, we comparedArguswith SQLancer on DuckDB across a fixed number of test cases, following the same setting in [ 2]. | name | 7.2 Code Coverage page 19 |
| M47 | We did not evaluate metamorphic coverage on PostgreSQL, which has not been supported by [ 2]’s implementation. | citation marker project authored | 7.2 Code Coverage page 19 |
| M48 | Following the experimental setup in [ 2], we generated 10 test suites for SQLancer, each with 100 test cases. | name | 7.2 Code Coverage page 19 |
| M49 | Specifically, SQLancer produce 50% of the test cases from TLP [ 48] and 50% from NoREC [ 47]. | name | 7.2 Code Coverage page 19 |
| M50 | 571×more line, function, and branch coverage than SQLancer, respectively. |
name |
7.2 Code Coverage page 19 |
| M51 | The baseline combines 11 test oracles from four previous works: TLP [ 48], NoREC [ 47], EET [ 24], and DQP [ 5]. | technique | 7.3 Effect of Test Oracles page 19 |
| M52 | We select a 6-hour time window for a fair comparison, following common practice in fuzz testing [ 26] and prior studies on DBMS logic bug detection [ 81], which typically fix the testing duration within a range of 1 – 24 hours. | citation marker | 7.3 Effect of Test Oracles page 19 |
| M53 | Argus SQLancer SQLancer++ Argus-grammar-based 0 4 8 12 16 20 24 Hours283032343638Line Coverage (%) (a) DuckDB line 0 4 8 12 16 20 24 Hours18192021Branch Coverage (%) (b) DuckDB branch Fig. | name | 7.6 Component-wise Analysis page 22 |
| M54 | Ablation study of code coverage achieved by Argus and a variant using only SQLancer’s grammarbased generator on DuckDB over 24-hour runs. | name | 7.6 Component-wise Analysis page 22 |
| M55 | We evaluate the effectiveness of LLM-powered SQL snippet generation by comparingArgus’s method against a variant that only uses SQLancer’s grammarbased generator to build the snippet corpus during the CAQ instantiation phase. | name | 7.6 Component-wise Analysis page 22 |
| M56 | (1)Arguscurrently focuses on relational DBMS and SELECT queries like prior works [ 3,5,24,47– 49,74], but the underlying principles can be extended to other types of databases and query languages, such as graph DBMS [23, 34, 37, 86] and spatial DBMS testing [12]. | citation marker | 8 Discussion page 23 |
| M57 | Similarly, ShQvel [ 81] uses LLMs to generate feature-rich SQL queries for testing DBMS with manually crafted test oracles, while SQLStorm [ 52] employs LLMs to synthesize a large-scale benchmark for evaluating performance of DBMSs. | name | 9 Related Work page 23 |
| M58 | Test query generation focuses on automatically generating complex SQL queries to find crashes, with techniques being either grammar-based, exemplified by tools like SQLsmith [ 53] and SQLancer [ 49], or mutationbased, as seen in Griffin [ 16]. | name | 9 Related Work page 24 |
| M59 | For instance, oracles like TLP [ 48], NoREC [ 47], and EET [ 24] were designed for logic bugs, while others such as CERT [ 4] and Apollo [ 25] target performance issues. | technique | 9 Related Work page 24 |