Data-Oriented Differential Testing of Object-Relational Mapping Systems
Read the paper · doi:10.1109/icse43902.2021.00137
What this paper does with SQLancer
All three SQLancer oracles are described in detail in the discussion -- PQS forming its oracle by fetching a specific record, NoREC's semantics-preserving transformation, TLP deriving partial-result queries -- as the DBMS-level precedent for Cynthia's own differential testing one layer above.
Cynthia is, as far as the authors know, the first systematic approach to testing Object-Relational Mapping systems. It generates random relational schemas, sets up the corresponding databases, and queries them through the APIs of the ORMs under test. Because ORMs share no common input language, queries are written in an abstract query language and translated into concrete executable ORM queries for differential comparison. A solver-based approach produces records targeted at each query's constraints. Cynthia found 28 bugs in five popular ORMs.
How it was classified
uses infrastructure — no
SQLancer is cited, not used; nothing in the mentions describes reusing its code.
extends technique — no
No technique is extended; the citation is background.
compares with — no
No empirical comparison against SQLancer is reported in the mentions.
describes as state of the art — no
The citation does not characterise SQLancer as the state of the art.
SQLancer publications it cites (3)
Bibliography entries that resolved to a SQLancer publication, or to a paper by one of the project's authors. A sentence citing one of these numbers is a reference to SQLancer even when it never writes the name.
| # | Entry | Matched as |
|---|---|---|
| 21 | M. Rigger and Z. Su, “Testing database engines via pivoted query synthesis,” in 14th USENIX Symposium on Operating Systems Design and Implementation (OSDI 20). USENIX Association, Nov. 2020, pp. 667-682. [Online]. Ava... | sqlancer publication · PQS |
| 27 | M. Rigger and Z. Su, “Detecting optimization bugs in database engines via non-optimizing reference engine construction,” in Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Sym... | sqlancer publication · NOREC |
| 38 | M. Rigger and Z. Su, “Finding bugs in database systems via query partitioning,” Proc. ACM Program. Lang., vol. 4, no. OOPSLA, Nov. 2020. [Online]. Available: https://doi.org/10.1145/3428279 | sqlancer publication · TLP |
Every place it refers to SQLancer (11)
11 sentences, each stored verbatim from the extracted text with where it was found and how. “Citation marker” means the sentence names no tool at all and was reached through a reference number that resolved to a SQLancer publication.
| Id | Sentence | Found by | Where |
|---|---|---|---|
| M1 | Although DBMSs share common functionality, they differ significantly from each other [21]. |
citation marker |
page 2 |
| M2 | To demonstrate the effectiveness of our solver-based data generation approach and its suitability for differential testing, we compare it against a simplistic approach that populates the database with random records a-priori [21], [27], i. |
citation marker |
D RQ3: Effectiveness of Solver-Based Data Generation page 10 |
| M3 | [21] proposed the Pivoted Query Synthesis (PQS) technique for testing database engines. |
technique |
E Discussion & Threats to Validity page 11 |
| M4 | PQS generates SQL queries so that they fetch a specific record from the database. |
technique |
E Discussion & Threats to Validity page 11 |
| M5 | In this way, PQS forms the test oracle: failing to fetch the expected record reveals a potential bug in DBMS. |
technique |
E Discussion & Threats to Validity page 11 |
| M6 | In an attempt to find optimization bugs in database systems, their subsequent work introduced a metamorphic testing technique called Non-Optimizing Reference Engine Construction (NoREC) [27]. |
technique |
E Discussion & Threats to Validity page 11 |
| M7 | At a high-level, NoREC applies a semanticspreserving transformation to a given SQL query in way that the various optimizations performed by the DBMS are disabled. |
technique |
E Discussion & Threats to Validity page 11 |
| M8 | Finally, NoREC compares the results of the original and the resulting queries for mismatches. |
technique |
E Discussion & Threats to Validity page 11 |
| M9 | In their most recent work, they propose Ternary Logic Partitioning (TLP) [38]. |
technique |
E Discussion & Threats to Validity page 11 |
| M10 | Given an SQL query, TLP derives multiple queries that compute a partial result of the initial query, and then combines the results of each individual query using a UNION operation. |
technique |
E Discussion & Threats to Validity page 11 |
| M11 | TLP is suitable for testing the implementation of the WHERE, HAVING, DISTI NCT clauses, or aggregate functions. | technique | E Discussion & Threats to Validity page 11 |