Reproducibility Report for ACM SIGMOD 2025 Paper: ‘Constant Optimization Driven Database System Testing’

Yuvaraj Chesetti, Chi Zhang. 2025.

Read the paper · doi:10.1145/3785021.3787993

What this paper does with SQLancer

This paper is a reproduction of SQLancer work rather than a citation of it. CODDTest is implemented inside SQLancer -- the report states the tool works by integrating the method with SQLancer, and that the SQLancer repository acknowledges the work as a supported testing method -- so reproducing it meant running a version of SQLancer that supports the method, from the authors' artifacts. The comparison it verifies is between three oracles in that one tool: CODDTest, NoREC and query partitioning, all executed through SQLancer against SQLite. An independent reproducibility report on the SIGMOD 2025 paper introducing CODDTest. The reviewer re-ran the authors' artifacts to check the claims: an oracle comparison verifying CODDTest finds bugs other oracles do not, an efficiency test measuring its throughput, and a script identifying the earliest SQLite version carrying each bug. The reported counts were CODDTest 1297 bugs, NoREC 644, with query partitioning producing 8280 database logs. Written by claude-opus-5 from the 13 places this paper refers to SQLancer. The quotations below are the paper's own words, stored verbatim when the text was extracted.

How it was classified

uses infrastructure — yes (implementation)

M4 states the tool under review works by integrating its method with SQLancer, M5 that the SQLancer repository acknowledges it as a supported testing method, and M10 that the reproduction ran a version of SQLancer supporting the method. The whole exercise is conducted through SQLancer.

•The tool works by integrating their method with SQLancer [ 1], a tool to automatically test databases. M4 · 1 Introduction · page 1

The SQLancer github repo also acknowledges the author’s work as a supported testing method. M5 · 1 Introduction · page 1

We downloaded the artifacts provided by the author that contain a version of SQLancer that supports their method. M10 · 4.2 Reproducibility Process · page 2

extends technique — no

The report reproduces CODDTest rather than proposing anything; the extension is the reproduced paper's, not this one's.

compares with — yes

M11 records running the scripts for CODDTest, NoREC and query partitioning, and M13 gives the counts each reported. The comparison is between oracles within SQLancer, which is what the reproduction set out to verify.

Non-optimizing Reference Engine Construction (NoREC)Ternary Logic Partitioning (TLP)

We ran the scripts for CODDTest (the author’s method), NoREC and Query Partitioning. M11 · 4.2 Reproducibility Process · page 2

•CODDTest reported 1297 bugs, NoRec reported 644 bugs, and query partitioning generated 8280 database logs. M13 · 4.2 Reproducibility Process · page 2

describes as state of the art — no

The report is an evaluation of reproducibility and makes no claim about standing.

SQLancer publications it cites (2)

Bibliography entries that resolved to a SQLancer publication, or to a paper by one of the project's authors. A sentence citing one of these numbers is a reference to SQLancer even when it never writes the name.

#	Entry	Matched as
1	[n. d.]. https://github.com/sqlancer/sqlancer.	sqlancer publication
2	Chi Zhang and Manuel Rigger. 2025. Constant Optimization Driven Database System Testing. 2025), 24 pages. doi:10.1145/3709674 58	sqlancer publication · CODDTEST

Every place it refers to SQLancer (13)

13 sentences, each stored verbatim from the extracted text with where it was found and how. “Citation marker” means the sentence names no tool at all and was reached through a reference number that resolved to a SQLancer publication.

Id	Sentence	Found by	Where
M1	Reproducibility Report for ACMSIGMOD 2025 Paper: “Constant Optimization Driven Database System Testing” Yuvaraj Chesetti chesetti.	technique incidental	page 1
M2	Reproducibility Report for ACMSIGMOD 2025 Paper: “Constant Optimization Driven Database System Testing”.	technique incidental	page 1
M3	3787993 1 Introduction The original paper presented a new methodology for automated bug finding in production databases [ 2].	citation marker background	page 1
M4	•The tool works by integrating their method with SQLancer [ 1], a tool to automatically test databases.	name reuse implementation	1 Introduction page 1
M5	The SQLancer github repo also acknowledges the author’s work as a supported testing method.	name reuse implementation	1 Introduction page 1
M6	The script takes the result of the SQLancer logs and attempts to find the earliest version of SQLite that had the bug.	name reuse component	1 Introduction page 1
M7	(2)Oracle comparision test A script that parses output logs from an SQLancer run on SQLite3 from various oracles to verify that their method finds bugs that other methods did not.	name reuse component	4.1 Artifacts and Experiments page 1
M8	(3)Efficiency test SQLancer commands in various configurations to measure the efficacy of their method.	name reuse component	4.1 Artifacts and Experiments page 1
M9	The bugs are filed against bug trackers of various databases 57 SIGMOD ARI Reports ’25, June 22–27, 2025, Berlin, Germany Yuvaraj Chesetti and Chi Zhang Method # of tests # successful tests # unsuccesful tests CODD 233,985k 780,488k 25,828k NoRec 658,042k 1,326,809k 47,170k Table 2: Number of tests conducted by CODD...	technique result comparison	4.2 Reproducibility Process page 1
M10	We downloaded the artifacts provided by the author that contain a version of SQLancer that supports their method.	name reuse implementation	4.2 Reproducibility Process page 2
M11	We ran the scripts for CODDTest (the author’s method), NoREC and Query Partitioning.	technique result comparison	4.2 Reproducibility Process page 2
M12	The efficiency test uses similar scripts but with flags to SQLancer to perform additional logging.	name reuse component	4.2 Reproducibility Process page 2
M13	•CODDTest reported 1297 bugs, NoRec reported 644 bugs, and query partitioning generated 8280 database logs.	technique result comparison	4.2 Reproducibility Process page 2

This page is rendered from _data/papers/paper_doi_10_1145_3785021_3787993.json, extracted from supplied pdf. 2 pages, 2 references parsed.