Keynote 1: DuckDB Testing - Present and Future

Mark Raasveldt · DBTest 2022 · 2022 · DBTest Workshop

Watch the talk

What it does with SQLancer

DuckDB's testing, from one of its authors: the keynote credits SQLancer with around eighty bugs that the SQLite and Postgres test suites DuckDB had borrowed all missed, argues for running it beside SQLsmith because each finds what the other does not, and says the robot in DuckDB's CI runs both.

recognitionreports adoptioncompares

Where SQLancer comes up (6)

• 35:44

  

  The talk at 35:44. The slide behind the story: “and then Dr. Rigger came along!”, over a DuckDB bug report.

  A frame from the recording. Nothing is transcribed from it: what the slide says is shown, not quoted.
• 35:54
  thought we had like a pretty robust system right we had like um thousands of tests from various systems like sqlite postgres hundreds of our own tests and then this guy called dr rieger came along and he uh unleashed his creation upon us and started furiously opening bug reports so uh using sql answer manual found around 80 bugs inductive and those were bugs that were not found using the test suites of the other systems right so we ran the sql light tests they did not find these bugs we ran the postgres test they didn't find these bugs and it turns out this kind of thing where database systems are complex surprise surprise and each system has their own
  From the talk's automatic captions — a machine transcription of speech, not the speaker's words as written: it renders the name as “sql answer”. Follow the link to hear what was said.
• 36:10

  

  The talk at 36:10. The next slide puts a number on it: the bugs SQLancer found in DuckDB, which the test suites DuckDB had borrowed from other systems did not.

  A frame from the recording. Nothing is transcribed from it: what the slide says is shown, not quoted.
• 37:48
  important not to just run one fuzzer but to run many different types of buzzers because these fuzzers they all they have like different domains so if you run you shouldn't just run sql lancer you should also run sql smith because sql smith will find bug sequel answer will not and sql answer will find bugs sql smith so uh it's like it's very powerful you should run as many as possible i think um also kind of a lesson we learned is that
  From the talk's automatic captions — a machine transcription of speech, not the speaker's words as written: it renders the name as “sql lancer”. Follow the link to hear what was said.
  Nothing to show from this moment: the slide makes the general case for running many kinds of fuzzer; sqlancer is named only in what is said.
• 40:51
  just file github issues and so the the fuzzer of docs robot was born and this is what runs now in our ci it currently runs sql answer and sql smith we also run oss fuzz but that's managed by google that's not by fuzzer ducks and what it does is if a bug is found it creates a reproducible test case based on the output of the fuzzer it does test case reduction and then it files an
  From the talk's automatic captions — a machine transcription of speech, not the speaker's words as written: it renders the name as “sql answer”. Follow the link to hear what was said.
• 40:55

  

  The talk at 40:55. The robot that runs fuzzers in DuckDB's CI, and what it was running at the time: SQLancer and SQLsmith.

  A frame from the recording. Nothing is transcribed from it: what the slide says is shown, not quoted.

What was read

• captions — extracted, 481 caption segments
  Windows around candidate moments, read out of YouTube's transcript panel in a browser. Not the whole talk: only the segments a loose prefilter flagged, so the matcher decides on text a person can check.
• frame — extracted
  Captured at 35:44.
• frame — extracted
  Captured at 36:10.
• frame — extracted
  Captured at 40:55.

This record is talks.json, under the id talk:youtube:BgC79Zt2fPs.