Retromorphic Testing: A New Approach to the Test Oracle Problem

Boxi Yu, Qiuyang Mang, Qingshuo Guo, Pinjia He. 2023. arXiv.org.

What this paper does with SQLancer

PQS is the paper's worked example of its own concept: the authors argue that although PQS was described as a technique for a specific system, it is an instance of retromorphic testing, and record that an author of PQS agreed with that categorisation. Retromorphic Testing proposes a black-box test oracle inspired by inverse functions. It pairs the program under test with an auxiliary program to form a forward and a backward program: input data is processed by the forward program and its output converted back into the original input format by the backward program, after which the relation between the initial input and the transformed output is checked. Either program can play either role, giving different testing modes. Written by claude-opus-5 from the 6 places this paper refers to SQLancer. The quotations below are the paper's own words, stored verbatim when the text was extracted.

How it was classified

uses infrastructure — no

SQLancer is cited, not used; nothing in the mentions describes reusing its code.

extends technique — yes

M2 reframes PQS as an instance of the general retromorphic testing concept the paper introduces, and M3 records that an author of PQS agreed. Generalising an existing technique into a broader class is the paper's contribution, not a baseline it runs.

Pivoted Query Synthesis (PQS)

Although the original paper describes PQS as a testing technique for a specific system, we think it can be regarded as an instance of the general Retromorphic Testing methodology. M2 · 4 TESTING SCOPES · page 7

We also discussed with one author of PQS and he agrees that PQS should be categorized as a Retromorphic Testing technique. M3 · 4 TESTING SCOPES · page 7

compares with — no

No empirical comparison against SQLancer is reported in the mentions.

describes as state of the art — no

The citation does not characterise SQLancer as the state of the art.

SQLancer publications it cites (5)

Bibliography entries that resolved to a SQLancer publication, or to a paper by one of the project's authors. A sentence citing one of these numbers is a reference to SQLancer even when it never writes the name.

#	Entry	Matched as
13	Matteo Kamm, Manuel Rigger, Chengyu Zhang, and Zhendong Su. 2023. Testing Graph Database Engines via Query Partitioning. In Proceedings of the 32nd ACMSIGSOFT International Symposium on Software Testing and Analysis (...	project authored
19	Manuel Rigger and Zhendong Su. 2020. Detecting Optimization Bugs in Database Engines via Non-Optimizing Reference Engine Construction. In Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conf...	sqlancer publication · NOREC
20	Manuel Rigger and Zhendong Su. 2020. Finding Bugs in Database Systems via Query Partitioning. Proc. ACM Program. Lang. 4, OOPSLA, Article 211 (nov 2020), 30 pages. https://doi.org/10.1145/3428279	sqlancer publication · TLP
21	Manuel Rigger and Zhendong Su. 2020. Testing Database Engines via Pivoted Query Synthesis. In 14th USENIX Symposium on Operating Systems Design and Implementation (OSDI 2020). USENIX Association, 667–682. https://www....	sqlancer publication · PQS
22	Manuel Rigger and Zhendong Su. 2022. Intramorphic Testing: A New Approach to the Test Oracle Problem. In Proceedings of the 2022 ACMSIGPLAN International Symposium on New Ideas, New Paradigms, and Reflections on Progr...	project authored

Every place it refers to SQLancer (6)

6 sentences, each stored verbatim from the extracted text with where it was found and how. “Citation marker” means the sentence names no tool at all and was reached through a reference number that resolved to a SQLancer publication.

Id	Sentence	Found by	Where
M1	In addition to these black-box methodologies, Rigger and Su [ 22] recently introduced intramorphic testing, a white-box automated testing methodology, which modifies a system component and constructs a test oracle that can relate the output of the original and modified systems given the same input.	citation marker project authored definition	1 INTRODUCTION page 1
M2	Metamorphic testing is also used for testing various systems such as compilers [ 14,15], database engines [ 13,19,20], SMT solvers [ 34], Android apps [ 28,29], quantum computing platforms [ 1,17], and AI systems [ 30,32,33,36,37].	citation marker extension	2.2 Terminology page 2
M3	In the context of Database Management Systems (DBMS), Rigger and Su introduced the concept of Pivoted Query Synthesis (PQS) [21] as an effective testing approach to uncover logic bugs in these systems.	technique background	4 TESTING SCOPES page 7
M4	Although the original paper describes PQS as a testing technique for a specific system, we think it can be regarded as an instance of the general Retromorphic Testing methodology.	technique background	4 TESTING SCOPES page 7
M5	We also discussed with one author of PQS and he agrees that PQS should be categorized as a Retromorphic Testing technique.	technique	4 TESTING SCOPES page 7
M6	The PQS testing approach initiates with the generation of a pivot row (e.	technique	4 TESTING SCOPES page 7

This page is rendered from _data/papers/paper_arxiv_2310_06433.json, extracted from arxiv. 9 pages, 38 references parsed.