Unicorn: detect runtime errors in time-series databases with hybrid input synthesis
Read the paper · doi:10.1145/3533767.3534364
What this paper does with SQLancer
SQLancer is Unicorn's principal baseline. The authors adapted it to each of the six time-series systems and report coverage and bug counts side by side across the whole evaluation, and they call it a state-of-the-art database fuzzer when introducing that comparison. Their explanation for the gap is about scope rather than quality: SQLancer targets the relational model, so its generated queries lack time-series elements and most are invalid on these systems, and it cannot address KairosDB at all because that system takes JSON over HTTP rather than SQL. They also cite SQLancer's three test oracles as the prior work on oracle design.
Unicorn fuzzes time-series databases, whose inputs combine SQL-like queries with time-series-specific schema and data constraints that relational fuzzers do not model. It synthesises inputs from a hybrid of grammar-based generation and dependency handling, and adds exception handling so that a rejected statement does not end a run. Across IoTDB, KairosDB, QuestDB, TDEngine, TimescaleDB and GridDB it covered 21-199% more basic blocks than SQLsmith and 34-693% more than SQLancer, and found 42 bugs.
How it was classified
uses infrastructure — no
SQLancer was adapted so it could be run as a baseline, which is comparison rather than reuse: nothing in Unicorn's own implementation is built on SQLancer's code.
extends technique — no
No SQLancer technique is extended; the work takes a different approach.
compares with — yes
M2 states the authors adapted SQLancer for comparison, M5 names it as one of the fuzzers Unicorn is evaluated against, and M9 and M22 report the outcome in both coverage and bugs found.
We also adapted the industrial fuzzers SQLancer [ 25] and SQLsmith [ 31] for comparison.
6 EVALUATION In this section, we evaluate the effectiveness of Unicorn in terms of coverage and bug discovery against state-of-the-art database fuzzers—SQLsmith and SQLancer.
2 Overall Performance Unicorn performs better than SQLsmith and SQLancer both in the number of basic blocks covered and bugs triggered.
Furthermore, 2 bugs were also found by SQLsmith, while SQLancer did not detect any bug.
describes as state of the art — yes
M3 and M5 both introduce SQLancer as one of the state-of-the-art fuzzers the evaluation measures against.
•We evaluate Unicorn on 6 popular time-series databases against state-of-the-art fuzzers SQLsmith and SQLancer.
6 EVALUATION In this section, we evaluate the effectiveness of Unicorn in terms of coverage and bug discovery against state-of-the-art database fuzzers—SQLsmith and SQLancer.
SQLancer publications it cites (5)
Bibliography entries that resolved to a SQLancer publication, or to a paper by one of the project's authors. A sentence citing one of these numbers is a reference to SQLancer even when it never writes the name.
| # | Entry | Matched as |
|---|---|---|
| 24 | Manuel Rigger. 2019. Apache IoTDB: Database for Internet of Things. https: //iotdb .apache .org/ | project authored |
| 25 | Manuel Rigger. 2020. SQLancer: detecting logic bugs in DBMS. https: //github .com/sqlancer/sqlancer | sqlancer publication |
| 26 | Manuel Rigger and Zhendong Su. 2020. Detecting Optimization Bugs in Database Engines via Non-Optimizing Reference Engine Construction. In Proceedings of the 2020 28th ACM Joint Meeting on European Software Engineering... | sqlancer publication · NOREC |
| 27 | Manuel Rigger and Zhendong Su. 2020. Finding bugs in database systems via query partitioning. Proc. ACM Program. Lang. 4, OOPSLA (2020), 211:1–211:30. https://doi .org/10 .1145/3428279 | sqlancer publication · TLP |
| 28 | Manuel Rigger and Zhendong Su. 2020. Testing Database Engines via Pivoted Query Synthesis. In 14th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2020, Virtual Event, November 4-6, 2020. USENIX ... | sqlancer publication · PQS |
Every place it refers to SQLancer (26)
26 sentences, each stored verbatim from the extracted text with where it was found and how. “Citation marker” means the sentence names no tool at all and was reached through a reference number that resolved to a SQLancer publication.
| Id | Sentence | Found by | Where |
|---|---|---|---|
| M1 | Specifically, Unicorn outperforms SQLsmith and SQLancer on widely used time-series databases IoTDB, KairosDB, TimescaleDB, TDEngine, QuestDB, and GridDB in the number of basic blocks by 21%-199% and 34%-693%, respectively. |
name |
page 1 |
| M2 | We also adapted the industrial fuzzers SQLancer [ 25] and SQLsmith [ 31] for comparison. |
name |
1 INTRODUCTION page 2 |
| M3 | •We evaluate Unicorn on 6 popular time-series databases against state-of-the-art fuzzers SQLsmith and SQLancer. |
name |
1 INTRODUCTION page 2 |
| M4 | 3 CHALLENGES INFUZZING TIME-SERIES DATABASES Fuzzing has achieved remarkable results in the field of software testing [ 4,7,15,25,31]. |
citation marker |
2.2 Handling Exceptions Implicitly page 3 |
| M5 | 6 EVALUATION In this section, we evaluate the effectiveness of Unicorn in terms of coverage and bug discovery against state-of-the-art database fuzzers—SQLsmith and SQLancer. |
name |
4.2.1 Agent. As Figure 2 shows, time-series databases register a page 7 |
| M6 | Table 1: Features of Chosen Time-series Databases Target Language Schema Grammar Dependency IoTDB Java tree-based SQL-like N/A KairosDB Java tag-based Http API Cassandra QuestDB Java column-based SQL-like N/A TDEngine C++ table-based SQL-like N/A TimescaleDB C relational SQL-like PostgreSQL GridDB C++ tag-based SQL-... |
name |
6.1 Evaluation Setup page 7 |
| M7 | •SQLancer [25] is a fuzzer for hunting logic bugs in DBMS. |
name |
6.1 Evaluation Setup page 7 |
| M8 | It detects logic bugs by constructing invariant oracles [ 26– 28] and checking whether results violate semantic logic. |
citation marker |
6.1 Evaluation Setup page 7 |
| M9 | 2 Overall Performance Unicorn performs better than SQLsmith and SQLancer both in the number of basic blocks covered and bugs triggered. |
name |
6.2 Overall Performance page 7 |
| M10 | Figure 7: The growing trend of the number of basic blocks covered when fuzzing each time-series database by Unicorn, SQLsmith, and SQLancer over 5 runs in 12 hours. |
name |
6.2 Overall Performance page 8 |
| M11 | Compared to SQLsmith and SQLancer, on the four successfully adapted databases for both of them (i. |
name |
6.2 Overall Performance page 8 |
| M12 | In particular, Unicorn covered 690% more basic blocks than SQLancer in GridDB. |
name |
6.2 Overall Performance page 8 |
| M13 | In contrast, SQLsmith and SQLancer do not match the time-series characteristics because SQLsmith and SQLancer only focus on the relational query in the relational model. |
name |
6.2 Overall Performance page 8 |
| M14 | In addition, Unicorn performs well on all tested time-series databases while SQLancer and SQLsmith both can hardly adapt to KairosDB. |
name |
6.2 Overall Performance page 8 |
| M15 | Because KairosDB does not accept the SQL-like inputs but the JSON format input by HTTP-API, the generation model of SQLsmith and SQLancer can hardly generate inputs like that. |
name |
6.2 Overall Performance page 8 |
| M16 | Target SQLsmith SQLancer Unicorn IoTDB 4, 857 3, 754 14, 532 KairosDB N/A N/A 26, 489 QuestDB 5, 742 3, 298 10, 486 TDEngine 3, 245 1, 423 8, 473 TimescaleDB 26, 423 13, 485 32, 124 GridDB N/A 2, 286 18, 145 Total 40, 276 24, 246 110, 249Figure 7 shows the growth trends of the number of basic blocks for each fuzzer ... |
name |
6.2 Overall Performance page 8 |
| M17 | By comparing the curves of Unicorn, SQLsmith and SQLancer in each sub-figure, we find that Unicorn is faster to cover basic blocks on target time-series databases than both SQLsmith and SQLancer. |
name |
6.2 Overall Performance page 8 |
| M18 | Specifically, in Figure 7 (a), Unicorn covers more basic blocks in IoTDB than SQLsmith and SQLancer from start to the end. |
name |
6.2 Overall Performance page 8 |
| M19 | The main reason is that the inputs generated by SQLsmith and SQLancer lack time-series elements, so most of them are invalid. |
name |
6.2 Overall Performance page 8 |
| M20 | Target SQLsmith SQLancer Unicorn IoTDB 1 0 18 KairosDB N/A N/A 2 QuestDB 0 0 3 TDEngine 0 0 6 TimescaleDB 1 1 3 GridDB N/A 0 10 Total 2 1 42 Bugs. |
name |
6.2 Overall Performance page 8 |
| M21 | Table 3 shows the number of bugs discovered in six timeseries databases by Unicorn, SQLsmith, and SQLancer. |
name |
6.2 Overall Performance page 8 |
| M22 | Furthermore, 2 bugs were also found by SQLsmith, while SQLancer did not detect any bug. |
name |
6.2 Overall Performance page 9 |
| M23 | For example, SQLancer uses 8,134 lines of Java code to support generating syntax-correct queries. |
name |
6.4 Effectiveness of Exception Detection page 10 |
| M24 | Various test oracles have been proposed to detect various bugs in databases and have achieved good results [ 13,25–27,31,33,38]. |
citation marker |
6.4 Effectiveness of Exception Detection page 11 |
| M25 | SQLancer [ 25] designs three test oracles according to the logic characteristics of relational databases. |
name |
6.4 Effectiveness of Exception Detection page 11 |
| M26 | In addition to performance testing, many time-series databases developers [ 2,6,11,23,24] also use a large amount of unit tests to ensure their security. |
citation marker project authored |
6.4 Effectiveness of Exception Detection page 11 |