verified_2026_v2
Verified campaign with memory measurements, September 2026. This campaign holds the measurements of the paper Database System Performance on Recursive Queries (Section 4). It is a complete re-run of the first 2026 campaign (verified_2026) with the present harness, in which every run is a separate process with a 600 s limit and every result is checked. Unlike the first campaign, it also records the memory each query used and the kind of every failure.
Outcome by series
Verification
- duckdb/double_recursion: binary_tree, cycle, grid, multi_path, path, reverse_binary_tree, y
- mariadb/right_recursion: scale_free
Environment
- date
- 2026-09-29T02:26:35Z
- machine
- Apple M3 Pro, 11 cores, 18 GB
- os
- macOS 27.0 (26A428)
- python
- Python 3.12.14
- xsb
- not captured
- postgresql
- PostgreSQL 17.11 (Homebrew) on aarch64-apple-darwin25.6.0, compiled by Apple clang version 21.0.0...
- mariadb
- /opt/homebrew/opt/mariadb/bin/mariadbd Ver 13.0.2-MariaDB for osx10.23 on arm64 (Homebrew)
- cockroachdb
- v26.3.2
- mongodb
- db version v8.3.11
- neo4j
- 2026.09.0, APOC: apoc-2026.09.0-core.jar
- java
- openjdk version "21.0.12.1" 2026-08-18
- trans-bench commit
- bc0149ff17ddc440f7395b09b553ae8da20452b5 + uncommitted changes (code.patch)
- singlestore image
- ghcr.io/singlestore-labs/singlestoredb-dev:0.2.62...
Mean time per system as the graph grows (log scale, the same scale at every n). A hatched bar hit the 600 s limit or ran out of memory; the larger sizes were then skipped.
Mean of the runs per size. Legend in the order the curves end (highest first); ✕ marks a failure, drawn at the 600 s limit.
| Series | Topology | Mode | First failed n | Kind | Message |
|---|---|---|---|---|---|
| CockroachDB | Binary Tree | double | 100 | Unsupported | CockroachDB experiment error: recursive reference to query "tc" must not appear more than once RUN_ONE_RESULT {"timing": {"CreateTableRealTime": 0.03189158299937844, "CreateTableCPUTime": 3.199999999997649e-05, "LoadDataRealTime": 0.2792650409974158, "LoadDataCPUTime": 0.00014999999999998348, "Creat |
| CockroachDB | Complete Graph | double | 100 | Unsupported | CockroachDB experiment error: recursive reference to query "tc" must not appear more than once RUN_ONE_RESULT {"timing": {"CreateTableRealTime": 0.027074708021245897, "CreateTableCPUTime": 3.599999999998049e-05, "LoadDataRealTime": 0.5481122910277918, "LoadDataCPUTime": 0.00018799999999996597, "Crea |
| CockroachDB | Cycle Graph | double | 100 | Unsupported | CockroachDB experiment error: recursive reference to query "tc" must not appear more than once RUN_ONE_RESULT {"timing": {"CreateTableRealTime": 0.03310920798685402, "CreateTableCPUTime": 3.6000000000036e-05, "LoadDataRealTime": 0.3618408340262249, "LoadDataCPUTime": 0.00016300000000002424, "CreateI |
| CockroachDB | Cycle with Shortcuts | double | 100 | Unsupported | CockroachDB experiment error: recursive reference to query "tc" must not appear more than once RUN_ONE_RESULT {"timing": {"CreateTableRealTime": 0.031644999980926514, "CreateTableCPUTime": 3.4000000000034e-05, "LoadDataRealTime": 0.349134000018239, "LoadDataCPUTime": 0.00011600000000000499, "CreateI |
| CockroachDB | Grid Graph | double | 100 | Unsupported | CockroachDB experiment error: recursive reference to query "tc" must not appear more than once RUN_ONE_RESULT {"timing": {"CreateTableRealTime": 0.03297233406919986, "CreateTableCPUTime": 3.799999999998249e-05, "LoadDataRealTime": 0.3098867500666529, "LoadDataCPUTime": 0.00014599999999997948, "Creat |
| CockroachDB | Max Acyclic Graph | double | 100 | Unsupported | CockroachDB experiment error: recursive reference to query "tc" must not appear more than once RUN_ONE_RESULT {"timing": {"CreateTableRealTime": 0.03162058407906443, "CreateTableCPUTime": 3.700000000000925e-05, "LoadDataRealTime": 0.4062533749965951, "LoadDataCPUTime": 0.00016699999999997273, "Creat |
| CockroachDB | Multi-Path Graph | double | 100 | Unsupported | CockroachDB experiment error: recursive reference to query "tc" must not appear more than once RUN_ONE_RESULT {"timing": {"CreateTableRealTime": 0.032281249994412065, "CreateTableCPUTime": 3.999999999998449e-05, "LoadDataRealTime": 0.6707835000706837, "LoadDataCPUTime": 9.900000000001574e-05, "Creat |
| CockroachDB | Path Graph | double | 100 | Unsupported | CockroachDB experiment error: recursive reference to query "tc" must not appear more than once RUN_ONE_RESULT {"timing": {"CreateTableRealTime": 0.03193904203362763, "CreateTableCPUTime": 3.4999999999951736e-05, "LoadDataRealTime": 0.32047558401245624, "LoadDataCPUTime": 0.00016100000000002224, "Cre |
| CockroachDB | Reverse Binary Tree | double | 100 | Unsupported | CockroachDB experiment error: recursive reference to query "tc" must not appear more than once RUN_ONE_RESULT {"timing": {"CreateTableRealTime": 0.030532584059983492, "CreateTableCPUTime": 2.999999999997449e-05, "LoadDataRealTime": 0.2760142079787329, "LoadDataCPUTime": 0.00015399999999998748, "Crea |
| CockroachDB | Scale-Free Directed | left | 20,000 | Timeout | |
| CockroachDB | Scale-Free Directed | right | 30,000 | Timeout | |
| CockroachDB | W Graph | double | 100 | Unsupported | CockroachDB experiment error: recursive reference to query "tc" must not appear more than once RUN_ONE_RESULT {"timing": {"CreateTableRealTime": 0.032516540959477425, "CreateTableCPUTime": 3.6000000000036e-05, "LoadDataRealTime": 0.32177883305121213, "LoadDataCPUTime": 0.00014100000000000223, "Creat |
| CockroachDB | X Graph | double | 100 | Unsupported | CockroachDB experiment error: recursive reference to query "tc" must not appear more than once RUN_ONE_RESULT {"timing": {"CreateTableRealTime": 0.030495082959532738, "CreateTableCPUTime": 3.4000000000034e-05, "LoadDataRealTime": 0.28300308308098465, "LoadDataCPUTime": 0.000142000000000031, "CreateI |
| CockroachDB | Y Graph | double | 100 | Unsupported | CockroachDB experiment error: recursive reference to query "tc" must not appear more than once RUN_ONE_RESULT {"timing": {"CreateTableRealTime": 0.033549249987117946, "CreateTableCPUTime": 3.500000000000725e-05, "LoadDataRealTime": 0.275977540994063, "LoadDataCPUTime": 0.00013499999999999623, "Creat |
| DuckDB | Scale-Free Directed | left | 90,000 | Out of memory | DuckDB command 4 error: Out of Memory Error: failed to allocate data of size 8.0 GiB (9.7 GiB/14.3 GiB used) Possible solutions: * Reducing the number of threads (SET threads=X) * Disabling insertion-order preservation (SET preserve_insertion_order=false) * Increasing the memory limit (SET memory_li |
| DuckDB | Scale-Free Directed | right | 90,000 | Out of memory | DuckDB command 4 error: Out of Memory Error: failed to allocate data of size 8.0 GiB (11.5 GiB/14.3 GiB used) Possible solutions: * Reducing the number of threads (SET threads=X) * Disabling insertion-order preservation (SET preserve_insertion_order=false) * Increasing the memory limit (SET memory_l |
| MariaDB | Complete Graph | double | 700 | Timeout | |
| MariaDB | Complete Graph | left | 700 | Timeout | |
| MariaDB | Complete Graph | right | 700 | Timeout | |
| MariaDB | Cycle Graph | double | 700 | Timeout | |
| MariaDB | Cycle with Shortcuts | double | 700 | Timeout | |
| MariaDB | Max Acyclic Graph | double | 1,000 | Timeout | |
| MariaDB | Max Acyclic Graph | left | 1,000 | Timeout | |
| MariaDB | Max Acyclic Graph | right | 1,000 | Timeout | |
| MariaDB | Multi-Path Graph | double | 300 | Timeout | |
| MariaDB | Path Graph | double | 1,000 | Timeout | |
| MariaDB | Scale-Free Directed | left | 20,000 | Timeout | |
| MariaDB | Scale-Free Directed | right | 30,000 | Timeout | |
| MongoDB | Barabási-Albert (Scale-Free) | left | 40,000 | Timeout | |
| MongoDB | Barabási-Albert (Scale-Free) | right | 40,000 | Timeout | |
| MongoDB | Complete Graph | left | 200 | Timeout | |
| MongoDB | Max Acyclic Graph | left | 200 | Timeout | |
| MongoDB | Scale-Free Directed | left | 30,000 | Timeout | |
| MongoDB | Scale-Free Directed | right | 30,000 | Timeout | |
| Neo4j | Scale-Free Directed | left | 30,000 | Out of memory | Neo4j experiment error: {neo4j_code: Neo.TransientError.General.MemoryPoolOutOfMemoryError} {message: The allocation of an extra 2.0 MiB would use more than the limit 3.1 GiB. Currently using 3.1 GiB. dbms.memory.transaction.total.max threshold reached} {gql_status: 51N72} {gql_status_description: e |
| PostgreSQL | Binary Tree | double | 100 | Unsupported | PostgreSQL experiment error: recursive reference to query "tc" must not appear more than once LINE 6: SELECT tc1.x, tc2.y FROM tc AS tc1, tc AS tc2 WH... ^ RUN_ONE_RESULT {"timing": {"CreateTableRealTime": 0.004498291993513703, "CreateTableCPUTime": 4.8000000000048004e-05, "LoadDataRealTime": 0.0012 |
| PostgreSQL | Complete Graph | double | 100 | Unsupported | PostgreSQL experiment error: recursive reference to query "tc" must not appear more than once LINE 6: SELECT tc1.x, tc2.y FROM tc AS tc1, tc AS tc2 WH... ^ RUN_ONE_RESULT {"timing": {"CreateTableRealTime": 0.003761625033803284, "CreateTableCPUTime": 2.0999999999993246e-05, "LoadDataRealTime": 0.0127 |
| PostgreSQL | Cycle Graph | double | 100 | Unsupported | PostgreSQL experiment error: recursive reference to query "tc" must not appear more than once LINE 6: SELECT tc1.x, tc2.y FROM tc AS tc1, tc AS tc2 WH... ^ RUN_ONE_RESULT {"timing": {"CreateTableRealTime": 0.004268334014341235, "CreateTableCPUTime": 1.8999999999991246e-05, "LoadDataRealTime": 0.0011 |
| PostgreSQL | Cycle with Shortcuts | double | 100 | Unsupported | PostgreSQL experiment error: recursive reference to query "tc" must not appear more than once LINE 6: SELECT tc1.x, tc2.y FROM tc AS tc1, tc AS tc2 WH... ^ RUN_ONE_RESULT {"timing": {"CreateTableRealTime": 0.003810291993431747, "CreateTableCPUTime": 2.0999999999993246e-05, "LoadDataRealTime": 0.0065 |
| PostgreSQL | Grid Graph | double | 100 | Unsupported | PostgreSQL experiment error: recursive reference to query "tc" must not appear more than once LINE 6: SELECT tc1.x, tc2.y FROM tc AS tc1, tc AS tc2 WH... ^ RUN_ONE_RESULT {"timing": {"CreateTableRealTime": 0.004037417005747557, "CreateTableCPUTime": 2.0999999999993246e-05, "LoadDataRealTime": 0.0013 |
| PostgreSQL | Max Acyclic Graph | double | 100 | Unsupported | PostgreSQL experiment error: recursive reference to query "tc" must not appear more than once LINE 6: SELECT tc1.x, tc2.y FROM tc AS tc1, tc AS tc2 WH... ^ RUN_ONE_RESULT {"timing": {"CreateTableRealTime": 0.004549916018731892, "CreateTableCPUTime": 1.6999999999989246e-05, "LoadDataRealTime": 0.0112 |
| PostgreSQL | Multi-Path Graph | double | 100 | Unsupported | PostgreSQL experiment error: recursive reference to query "tc" must not appear more than once LINE 6: SELECT tc1.x, tc2.y FROM tc AS tc1, tc AS tc2 WH... ^ RUN_ONE_RESULT {"timing": {"CreateTableRealTime": 0.004003042005933821, "CreateTableCPUTime": 2.0999999999993246e-05, "LoadDataRealTime": 0.0066 |
| PostgreSQL | Path Graph | double | 100 | Unsupported | PostgreSQL experiment error: recursive reference to query "tc" must not appear more than once LINE 6: SELECT tc1.x, tc2.y FROM tc AS tc1, tc AS tc2 WH... ^ RUN_ONE_RESULT {"timing": {"CreateTableRealTime": 0.003950083046220243, "CreateTableCPUTime": 1.999999999996449e-05, "LoadDataRealTime": 0.00156 |
| PostgreSQL | Reverse Binary Tree | double | 100 | Unsupported | PostgreSQL experiment error: recursive reference to query "tc" must not appear more than once LINE 6: SELECT tc1.x, tc2.y FROM tc AS tc1, tc AS tc2 WH... ^ RUN_ONE_RESULT {"timing": {"CreateTableRealTime": 0.004015290993265808, "CreateTableCPUTime": 4.799999999999249e-05, "LoadDataRealTime": 0.00126 |
| PostgreSQL | Scale-Free Directed | left | 60,000 | Timeout | |
| PostgreSQL | Scale-Free Directed | right | 60,000 | Timeout | |
| PostgreSQL | W Graph | double | 100 | Unsupported | PostgreSQL experiment error: recursive reference to query "tc" must not appear more than once LINE 6: SELECT tc1.x, tc2.y FROM tc AS tc1, tc AS tc2 WH... ^ RUN_ONE_RESULT {"timing": {"CreateTableRealTime": 0.004089166992343962, "CreateTableCPUTime": 4.599999999999049e-05, "LoadDataRealTime": 0.00742 |
| PostgreSQL | X Graph | double | 100 | Unsupported | PostgreSQL experiment error: recursive reference to query "tc" must not appear more than once LINE 6: SELECT tc1.x, tc2.y FROM tc AS tc1, tc AS tc2 WH... ^ RUN_ONE_RESULT {"timing": {"CreateTableRealTime": 0.003955792053602636, "CreateTableCPUTime": 2.0999999999993246e-05, "LoadDataRealTime": 0.0010 |
| PostgreSQL | Y Graph | double | 100 | Unsupported | PostgreSQL experiment error: recursive reference to query "tc" must not appear more than once LINE 6: SELECT tc1.x, tc2.y FROM tc AS tc1, tc AS tc2 WH... ^ RUN_ONE_RESULT {"timing": {"CreateTableRealTime": 0.0036302499938756227, "CreateTableCPUTime": 1.999999999996449e-05, "LoadDataRealTime": 0.0011 |
| SingleStore | Binary Tree | double | 100 | Unsupported | SingleStore experiment error: (2709, 'recursive reference to query "tc" must not appear more than once') RUN_ONE_RESULT {"timing": {"CreateTableRealTime": 0.015756624983623624, "CreateTableCPUTime": 3.6000000000036e-05, "LoadDataRealTime": 0.010151415946893394, "LoadDataCPUTime": 0.00017799999999995 |
| SingleStore | Complete Graph | double | 100 | Unsupported | SingleStore experiment error: (2709, 'recursive reference to query "tc" must not appear more than once') RUN_ONE_RESULT {"timing": {"CreateTableRealTime": 0.021897583035752177, "CreateTableCPUTime": 6.099999999997774e-05, "LoadDataRealTime": 0.04317258298397064, "LoadDataCPUTime": 0.0003979999999999 |
| SingleStore | Complete Graph | left | 100 | Out of memory | SingleStore experiment error: (1712, "Leaf Error (127.0.0.1:3307): Memory used by MemSQL (5800.00 Mb) has reached the 'maximum_memory' setting (6000 Mb) on this node. Possible causes include (1) available query execution memory has been used up for table memory (in use table memory: 196.12 Mb) and ( |
| SingleStore | Complete Graph | right | 100 | Out of memory | SingleStore experiment error: (1712, "Leaf Error (127.0.0.1:3307): Memory used by MemSQL (5800.00 Mb) has reached the 'maximum_memory' setting (6000 Mb) on this node. Possible causes include (1) available query execution memory has been used up for table memory (in use table memory: 197.38 Mb) and ( |
| SingleStore | Cycle Graph | double | 100 | Unsupported | SingleStore experiment error: (2709, 'recursive reference to query "tc" must not appear more than once') RUN_ONE_RESULT {"timing": {"CreateTableRealTime": 0.018449833034537733, "CreateTableCPUTime": 4.099999999995774e-05, "LoadDataRealTime": 0.017228417098522186, "LoadDataCPUTime": 0.000130999999999 |
| SingleStore | Cycle Graph | left | 100 | Iteration limit | SingleStore experiment error: (2741, "The query failed because a recursive common table expression exceeded the max number of iterations: 10000, which can be configured by the parameter 'max_recursive_cte_iterations'.") RUN_ONE_RESULT {"timing": {"CreateTableRealTime": 0.016487207962200046, "CreateT |
| SingleStore | Cycle Graph | right | 100 | Iteration limit | SingleStore experiment error: (2741, "The query failed because a recursive common table expression exceeded the max number of iterations: 10000, which can be configured by the parameter 'max_recursive_cte_iterations'.") RUN_ONE_RESULT {"timing": {"CreateTableRealTime": 0.017285167006775737, "CreateT |
| SingleStore | Cycle with Shortcuts | double | 100 | Unsupported | SingleStore experiment error: (2709, 'recursive reference to query "tc" must not appear more than once') RUN_ONE_RESULT {"timing": {"CreateTableRealTime": 0.023539334069937468, "CreateTableCPUTime": 0.00011500000000003174, "LoadDataRealTime": 0.025938958977349102, "LoadDataCPUTime": 0.00024799999999 |
| SingleStore | Cycle with Shortcuts | left | 100 | Out of memory | SingleStore experiment error: (1712, "Leaf Error (127.0.0.1:3307): Memory used by MemSQL (5800.00 Mb) has reached the 'maximum_memory' setting (6000 Mb) on this node. Possible causes include (1) available query execution memory has been used up for table memory (in use table memory: 197.62 Mb) and ( |
| SingleStore | Cycle with Shortcuts | right | 100 | Out of memory | SingleStore experiment error: (1712, "Leaf Error (127.0.0.1:3307): Memory used by MemSQL (5800.00 Mb) has reached the 'maximum_memory' setting (6000 Mb) on this node. Possible causes include (1) available query execution memory has been used up for table memory (in use table memory: 196.75 Mb) and ( |
| SingleStore | Grid Graph | double | 100 | Unsupported | SingleStore experiment error: (2709, 'recursive reference to query "tc" must not appear more than once') RUN_ONE_RESULT {"timing": {"CreateTableRealTime": 0.02760250004939735, "CreateTableCPUTime": 0.0001000000000000445, "LoadDataRealTime": 0.027740290970541537, "LoadDataCPUTime": 0.0003080000000000 |
| SingleStore | Grid Graph | left | 300 | Out of memory | SingleStore experiment error: (1712, "Leaf Error (127.0.0.1:3307): Memory used by MemSQL (5800.00 Mb) has reached the 'maximum_memory' setting (6000 Mb) on this node. Possible causes include (1) available query execution memory has been used up for table memory (in use table memory: 199.88 Mb) and ( |
| SingleStore | Grid Graph | right | 300 | Out of memory | SingleStore experiment error: (1712, "Leaf Error (127.0.0.1:3307): Memory used by MemSQL (5800.00 Mb) has reached the 'maximum_memory' setting (6000 Mb) on this node. Possible causes include (1) available query execution memory has been used up for table memory (in use table memory: 199.88 Mb) and ( |
| SingleStore | Max Acyclic Graph | double | 100 | Unsupported | SingleStore experiment error: (2709, 'recursive reference to query "tc" must not appear more than once') RUN_ONE_RESULT {"timing": {"CreateTableRealTime": 0.01836908294353634, "CreateTableCPUTime": 4.900000000002125e-05, "LoadDataRealTime": 0.028465625015087426, "LoadDataCPUTime": 0.0002739999999999 |
| SingleStore | Max Acyclic Graph | left | 100 | Out of memory | SingleStore experiment error: (1712, "Leaf Error (127.0.0.1:3307): Memory used by MemSQL (5800.00 Mb) has reached the 'maximum_memory' setting (6000 Mb) on this node. Possible causes include (1) available query execution memory has been used up for table memory (in use table memory: 197.88 Mb) and ( |
| SingleStore | Max Acyclic Graph | right | 100 | Out of memory | SingleStore experiment error: (1712, "Leaf Error (127.0.0.1:3307): Memory used by MemSQL (5800.00 Mb) has reached the 'maximum_memory' setting (6000 Mb) on this node. Possible causes include (1) available query execution memory has been used up for table memory (in use table memory: 197.62 Mb) and ( |
| SingleStore | Multi-Path Graph | double | 100 | Unsupported | SingleStore experiment error: (2709, 'recursive reference to query "tc" must not appear more than once') RUN_ONE_RESULT {"timing": {"CreateTableRealTime": 0.019096208037808537, "CreateTableCPUTime": 5.6000000000000494e-05, "LoadDataRealTime": 0.021259082946926355, "LoadDataCPUTime": 0.00022899999999 |
| SingleStore | Path Graph | double | 100 | Unsupported | SingleStore experiment error: (2709, 'recursive reference to query "tc" must not appear more than once') RUN_ONE_RESULT {"timing": {"CreateTableRealTime": 0.019045958993956447, "CreateTableCPUTime": 4.599999999999049e-05, "LoadDataRealTime": 0.015842875000089407, "LoadDataCPUTime": 0.000233999999999 |
| SingleStore | Reverse Binary Tree | double | 100 | Unsupported | SingleStore experiment error: (2709, 'recursive reference to query "tc" must not appear more than once') RUN_ONE_RESULT {"timing": {"CreateTableRealTime": 0.015460875001735985, "CreateTableCPUTime": 4.199999999998649e-05, "LoadDataRealTime": 0.012708374997600913, "LoadDataCPUTime": 0.000238000000000 |
| SingleStore | Scale-Free Directed | left | 10,000 | Out of memory | SingleStore experiment error: (1712, "Leaf Error (127.0.0.1:3307): Memory used by MemSQL (5800.00 Mb) has reached the 'maximum_memory' setting (6000 Mb) on this node. Possible causes include (1) available query execution memory has been used up for table memory (in use table memory: 202.38 Mb) and ( |
| SingleStore | Scale-Free Directed | right | 10,000 | Out of memory | SingleStore experiment error: (1712, "Leaf Error (127.0.0.1:3307): Memory used by MemSQL (5800.12 Mb) has reached the 'maximum_memory' setting (6000 Mb) on this node. Possible causes include (1) available query execution memory has been used up for table memory (in use table memory: 201.62 Mb) and ( |
| SingleStore | W Graph | double | 100 | Unsupported | SingleStore experiment error: (2709, 'recursive reference to query "tc" must not appear more than once') RUN_ONE_RESULT {"timing": {"CreateTableRealTime": 0.016029125079512596, "CreateTableCPUTime": 3.700000000000925e-05, "LoadDataRealTime": 0.009594792034476995, "LoadDataCPUTime": 0.000183000000000 |
| SingleStore | X Graph | double | 100 | Unsupported | SingleStore experiment error: (2709, 'recursive reference to query "tc" must not appear more than once') RUN_ONE_RESULT {"timing": {"CreateTableRealTime": 0.019379999954253435, "CreateTableCPUTime": 4.900000000002125e-05, "LoadDataRealTime": 0.018509040935896337, "LoadDataCPUTime": 0.000234999999999 |
| SingleStore | Y Graph | double | 100 | Unsupported | SingleStore experiment error: (2709, 'recursive reference to query "tc" must not appear more than once') RUN_ONE_RESULT {"timing": {"CreateTableRealTime": 0.01611620804760605, "CreateTableCPUTime": 3.999999999998449e-05, "LoadDataRealTime": 0.010757916956208646, "LoadDataCPUTime": 0.0002050000000000 |
| XSB | Scale-Free Directed | left | 70,000 | Timeout |
No failure matches
Change the kinds or the filter.
Pick a file
README, versions, the code patch the campaign ran with, and the analysis outputs.