Campaign

verified_2026

Verified campaign, September 2026. This is the first campaign in which every run of trans-bench was timed in its own process and its result checked against an independently computed closure. It covers the same systems, topologies, modes and sizes as the paper, with five runs per configuration and a 600 s limit per run. The paper's figures come from verified_2026_v2, a complete re-run of this campaign that also records memory; the two agree on every result.

summary.csv Run again

Runs executed

12,279

9 series, plus 530 skipped

Completed and verified

12,205

99.40% of executed runs

Failed

23 timeout, 11 out of memory, 36 unsupported, 2 iteration limit, 2 error

Wall-clock span

—

2026-09-26 13:42 → 2026-09-28 05:16 UTC

Outcome by series

completedtimeoutout of memory / errorunsupported / limitskipped

CockroachDB

1,450

DuckDB

2,582

MariaDB

1,802

MariaDB (4 GB tmp)

1,200

MongoDB

598

Neo4j

667

PostgreSQL

1,478

SingleStore

1,054

XSB

1,978

Verification

Incorrect results (row count or hash differs from the reference closure):

duckdb/double_recursion: binary_tree, cycle, grid, multi_path, path, reverse_binary_tree, y
mariadb/right_recursion: scale_free

Scale-free and Barabási–Albert graphs (no reference closure): the systems agree on 17 of 19 graph sizes.

Environment

date: 2026-09-26T13:41:40Z
machine: Apple M3 Pro, 11 cores, 18 GB
os: macOS 27.0 (26A428)
python: Python 3.12.14
xsb: XSB Version 5.0.0 (Green Tea) of May 15, 2022
postgresql: PostgreSQL 17.11 (Homebrew) on aarch64-apple-darwin25.6.0, compiled by Apple clang version 21.0.0...
mariadb: /opt/homebrew/opt/mariadb/bin/mariadbd Ver 13.0.2-MariaDB for osx10.23 on arm64 (Homebrew)
cockroachdb: v26.3.2
mongodb: db version v8.3.11
neo4j: 2026.09.0, APOC: apoc-2026.09.0-core.jar
java: openjdk version "21.0.12.1" 2026-08-18
singlestore image: ghcr.io/singlestore-labs/singlestoredb-dev:0.2.62...
trans-bench commit: 15d040e1458a83d57c6ae4ef403e52cd059c0975

Series	Topology	Mode	First failed n	Kind	Message
CockroachDB	Binary Tree	double	100	Unsupported	CockroachDB error: recursive reference to query "tc" must not appear more than once 2026-09-26 23:58:04,751 - INFO: Timing results saved to: /private/tmp/claude-501/-Users-johnidogun-Downloads-DBSysPerformance/210578b2-2f50-4fd6-8875-f449fb528170/scratchpad/results_2026/cockroachdb/timing/cockroachd
CockroachDB	Complete Graph	double	100	Unsupported	CockroachDB error: recursive reference to query "tc" must not appear more than once 2026-09-26 23:00:38,579 - INFO: Timing results saved to: /private/tmp/claude-501/-Users-johnidogun-Downloads-DBSysPerformance/210578b2-2f50-4fd6-8875-f449fb528170/scratchpad/results_2026/cockroachdb/timing/cockroachd
CockroachDB	Cycle Graph	double	100	Unsupported	CockroachDB error: recursive reference to query "tc" must not appear more than once 2026-09-26 23:17:38,361 - INFO: Timing results saved to: /private/tmp/claude-501/-Users-johnidogun-Downloads-DBSysPerformance/210578b2-2f50-4fd6-8875-f449fb528170/scratchpad/results_2026/cockroachdb/timing/cockroachd
CockroachDB	Cycle with Shortcuts	double	100	Unsupported	CockroachDB error: recursive reference to query "tc" must not appear more than once 2026-09-26 23:22:18,039 - INFO: Timing results saved to: /private/tmp/claude-501/-Users-johnidogun-Downloads-DBSysPerformance/210578b2-2f50-4fd6-8875-f449fb528170/scratchpad/results_2026/cockroachdb/timing/cockroachd
CockroachDB	Grid Graph	double	100	Unsupported	CockroachDB error: recursive reference to query "tc" must not appear more than once 2026-09-26 23:55:00,190 - INFO: Timing results saved to: /private/tmp/claude-501/-Users-johnidogun-Downloads-DBSysPerformance/210578b2-2f50-4fd6-8875-f449fb528170/scratchpad/results_2026/cockroachdb/timing/cockroachd
CockroachDB	Max Acyclic Graph	double	100	Unsupported	CockroachDB error: recursive reference to query "tc" must not appear more than once 2026-09-26 23:13:28,965 - INFO: Timing results saved to: /private/tmp/claude-501/-Users-johnidogun-Downloads-DBSysPerformance/210578b2-2f50-4fd6-8875-f449fb528170/scratchpad/results_2026/cockroachdb/timing/cockroachd
CockroachDB	Multi-Path Graph	double	100	Unsupported	CockroachDB error: recursive reference to query "tc" must not appear more than once 2026-09-26 23:51:46,538 - INFO: Timing results saved to: /private/tmp/claude-501/-Users-johnidogun-Downloads-DBSysPerformance/210578b2-2f50-4fd6-8875-f449fb528170/scratchpad/results_2026/cockroachdb/timing/cockroachd
CockroachDB	Path Graph	double	100	Unsupported	CockroachDB error: recursive reference to query "tc" must not appear more than once 2026-09-26 23:25:57,233 - INFO: Timing results saved to: /private/tmp/claude-501/-Users-johnidogun-Downloads-DBSysPerformance/210578b2-2f50-4fd6-8875-f449fb528170/scratchpad/results_2026/cockroachdb/timing/cockroachd
CockroachDB	Reverse Binary Tree	double	100	Unsupported	CockroachDB error: recursive reference to query "tc" must not appear more than once 2026-09-27 00:01:13,949 - INFO: Timing results saved to: /private/tmp/claude-501/-Users-johnidogun-Downloads-DBSysPerformance/210578b2-2f50-4fd6-8875-f449fb528170/scratchpad/results_2026/cockroachdb/timing/cockroachd
CockroachDB	Scale-Free Directed	left	20,000	Timeout
CockroachDB	Scale-Free Directed	right	30,000	Timeout
CockroachDB	W Graph	double	100	Unsupported	CockroachDB error: recursive reference to query "tc" must not appear more than once 2026-09-27 00:10:48,570 - INFO: Timing results saved to: /private/tmp/claude-501/-Users-johnidogun-Downloads-DBSysPerformance/210578b2-2f50-4fd6-8875-f449fb528170/scratchpad/results_2026/cockroachdb/timing/cockroachd
CockroachDB	X Graph	double	100	Unsupported	CockroachDB error: recursive reference to query "tc" must not appear more than once 2026-09-27 00:04:24,952 - INFO: Timing results saved to: /private/tmp/claude-501/-Users-johnidogun-Downloads-DBSysPerformance/210578b2-2f50-4fd6-8875-f449fb528170/scratchpad/results_2026/cockroachdb/timing/cockroachd
CockroachDB	Y Graph	double	100	Unsupported	CockroachDB error: recursive reference to query "tc" must not appear more than once 2026-09-27 00:07:36,125 - INFO: Timing results saved to: /private/tmp/claude-501/-Users-johnidogun-Downloads-DBSysPerformance/210578b2-2f50-4fd6-8875-f449fb528170/scratchpad/results_2026/cockroachdb/timing/cockroachd
DuckDB	Scale-Free Directed	left	90,000	Out of memory	Error executing command: CREATE TABLE tc_result AS WITH RECURSIVE tc AS ( SELECT x, y FROM edge UNION SELECT tc.x, edge.y FROM tc JOIN edge ON tc.y = edge.x ) SELECT * FROM tc;. Error: Out of Memory Error: failed to allocate data of size 8.0 GiB (9.7 GiB/14.3 GiB used) Possible solutions: * Reducing
DuckDB	Scale-Free Directed	right	90,000	Out of memory	Error executing command: CREATE TABLE tc_result AS WITH RECURSIVE tc AS ( SELECT x, y FROM edge UNION SELECT edge.x, tc.y FROM edge JOIN tc ON edge.y = tc.x ) SELECT * FROM tc;. Error: Out of Memory Error: failed to allocate data of size 8.0 GiB (11.5 GiB/14.3 GiB used) Possible solutions: * Reducin
MariaDB	Complete Graph	double	700	Timeout
MariaDB	Complete Graph	left	700	Timeout
MariaDB	Complete Graph	right	700	Timeout
MariaDB	Cycle Graph	double	700	Timeout
MariaDB	Cycle with Shortcuts	double	700	Timeout
MariaDB	Max Acyclic Graph	double	1,000	Timeout
MariaDB	Max Acyclic Graph	left	1,000	Timeout
MariaDB	Max Acyclic Graph	right	1,000	Timeout
MariaDB	Multi-Path Graph	double	300	Timeout
MariaDB	Path Graph	double	1,000	Timeout
MariaDB	Scale-Free Directed	left	20,000	Timeout
MariaDB	Scale-Free Directed	right	30,000	Timeout
MongoDB	Barabási-Albert (Scale-Free)	left	30,000	Timeout
MongoDB	Barabási-Albert (Scale-Free)	right	30,000	Timeout
MongoDB	Complete Graph	left	200	Timeout
MongoDB	Max Acyclic Graph	left	200	Timeout
MongoDB	Scale-Free Directed	left	30,000	Timeout
MongoDB	Scale-Free Directed	right	30,000	Timeout
Neo4j	Scale-Free Directed	left	30,000	Out of memory	Penultimate Neo4J query error: {neo4j_code: Neo.TransientError.General.MemoryPoolOutOfMemoryError} {message: The allocation of an extra 2.0 MiB would use more than the limit 3.1 GiB. Currently using 3.1 GiB. dbms.memory.transaction.total.max threshold reached} {gql_status: 51N72} {gql_status_descrip
PostgreSQL	Binary Tree	double	100	Unsupported	PostgreSQL error: recursive reference to query "tc" must not appear more than once LINE 6: SELECT tc1.x, tc2.y FROM tc AS tc1, tc AS tc2 WH... ^ 2026-09-26 19:23:21,328 - INFO: Timing results saved to: /private/tmp/claude-501/-Users-johnidogun-Downloads-DBSysPerformance/210578b2-2f50-4fd6-8875-f449f
PostgreSQL	Complete Graph	double	100	Unsupported	PostgreSQL error: recursive reference to query "tc" must not appear more than once LINE 6: SELECT tc1.x, tc2.y FROM tc AS tc1, tc AS tc2 WH... ^ 2026-09-26 19:02:45,003 - INFO: Timing results saved to: /private/tmp/claude-501/-Users-johnidogun-Downloads-DBSysPerformance/210578b2-2f50-4fd6-8875-f449f
PostgreSQL	Cycle Graph	double	100	Unsupported	PostgreSQL error: recursive reference to query "tc" must not appear more than once LINE 6: SELECT tc1.x, tc2.y FROM tc AS tc1, tc AS tc2 WH... ^ 2026-09-26 19:12:13,461 - INFO: Timing results saved to: /private/tmp/claude-501/-Users-johnidogun-Downloads-DBSysPerformance/210578b2-2f50-4fd6-8875-f449f
PostgreSQL	Cycle with Shortcuts	double	100	Unsupported	PostgreSQL error: recursive reference to query "tc" must not appear more than once LINE 6: SELECT tc1.x, tc2.y FROM tc AS tc1, tc AS tc2 WH... ^ 2026-09-26 19:14:35,391 - INFO: Timing results saved to: /private/tmp/claude-501/-Users-johnidogun-Downloads-DBSysPerformance/210578b2-2f50-4fd6-8875-f449f
PostgreSQL	Grid Graph	double	100	Unsupported	PostgreSQL error: recursive reference to query "tc" must not appear more than once LINE 6: SELECT tc1.x, tc2.y FROM tc AS tc1, tc AS tc2 WH... ^ 2026-09-26 19:22:31,205 - INFO: Timing results saved to: /private/tmp/claude-501/-Users-johnidogun-Downloads-DBSysPerformance/210578b2-2f50-4fd6-8875-f449f
PostgreSQL	Max Acyclic Graph	double	100	Unsupported	PostgreSQL error: recursive reference to query "tc" must not appear more than once LINE 6: SELECT tc1.x, tc2.y FROM tc AS tc1, tc AS tc2 WH... ^ 2026-09-26 19:10:41,264 - INFO: Timing results saved to: /private/tmp/claude-501/-Users-johnidogun-Downloads-DBSysPerformance/210578b2-2f50-4fd6-8875-f449f
PostgreSQL	Multi-Path Graph	double	100	Unsupported	PostgreSQL error: recursive reference to query "tc" must not appear more than once LINE 6: SELECT tc1.x, tc2.y FROM tc AS tc1, tc AS tc2 WH... ^ 2026-09-26 19:21:29,767 - INFO: Timing results saved to: /private/tmp/claude-501/-Users-johnidogun-Downloads-DBSysPerformance/210578b2-2f50-4fd6-8875-f449f
PostgreSQL	Path Graph	double	100	Unsupported	PostgreSQL error: recursive reference to query "tc" must not appear more than once LINE 6: SELECT tc1.x, tc2.y FROM tc AS tc1, tc AS tc2 WH... ^ 2026-09-26 19:15:57,150 - INFO: Timing results saved to: /private/tmp/claude-501/-Users-johnidogun-Downloads-DBSysPerformance/210578b2-2f50-4fd6-8875-f449f
PostgreSQL	Reverse Binary Tree	double	100	Unsupported	PostgreSQL error: recursive reference to query "tc" must not appear more than once LINE 6: SELECT tc1.x, tc2.y FROM tc AS tc1, tc AS tc2 WH... ^ 2026-09-26 19:24:11,421 - INFO: Timing results saved to: /private/tmp/claude-501/-Users-johnidogun-Downloads-DBSysPerformance/210578b2-2f50-4fd6-8875-f449f
PostgreSQL	Scale-Free Directed	left	50,000	Timeout
PostgreSQL	Scale-Free Directed	right	70,000	Timeout
PostgreSQL	W Graph	double	100	Unsupported	PostgreSQL error: recursive reference to query "tc" must not appear more than once LINE 6: SELECT tc1.x, tc2.y FROM tc AS tc1, tc AS tc2 WH... ^ 2026-09-26 19:26:42,026 - INFO: Timing results saved to: /private/tmp/claude-501/-Users-johnidogun-Downloads-DBSysPerformance/210578b2-2f50-4fd6-8875-f449f
PostgreSQL	X Graph	double	100	Unsupported	PostgreSQL error: recursive reference to query "tc" must not appear more than once LINE 6: SELECT tc1.x, tc2.y FROM tc AS tc1, tc AS tc2 WH... ^ 2026-09-26 19:25:01,743 - INFO: Timing results saved to: /private/tmp/claude-501/-Users-johnidogun-Downloads-DBSysPerformance/210578b2-2f50-4fd6-8875-f449f
PostgreSQL	Y Graph	double	100	Unsupported	PostgreSQL error: recursive reference to query "tc" must not appear more than once LINE 6: SELECT tc1.x, tc2.y FROM tc AS tc1, tc AS tc2 WH... ^ 2026-09-26 19:25:51,898 - INFO: Timing results saved to: /private/tmp/claude-501/-Users-johnidogun-Downloads-DBSysPerformance/210578b2-2f50-4fd6-8875-f449f
SingleStore	Binary Tree	double	100	Unsupported	SingleStore error: (2709, 'recursive reference to query "tc" must not appear more than once') 2026-09-27 21:10:56,863 - INFO: Timing results saved to: /private/tmp/claude-501/-Users-johnidogun-Downloads-DBSysPerformance/210578b2-2f50-4fd6-8875-f449fb528170/scratchpad/results_2026/singlestore/timing/
SingleStore	Complete Graph	double	100	Unsupported	SingleStore error: (2709, 'recursive reference to query "tc" must not appear more than once') 2026-09-27 20:11:28,368 - INFO: Timing results saved to: /private/tmp/claude-501/-Users-johnidogun-Downloads-DBSysPerformance/210578b2-2f50-4fd6-8875-f449fb528170/scratchpad/results_2026/singlestore/timing/
SingleStore	Complete Graph	left	100	Out of memory	SingleStore error: (1712, "Leaf Error (127.0.0.1:3307): Memory used by MemSQL (5800.00 Mb) has reached the 'maximum_memory' setting (6000 Mb) on this node. Possible causes include (1) available query execution memory has been used up for table memory (in use table memory: 189.38 Mb) and (2) the quer
SingleStore	Complete Graph	right	100	Out of memory	SingleStore error: (1712, "Leaf Error (127.0.0.1:3307): Memory used by MemSQL (5807.12 Mb) has reached the 'maximum_memory' setting (6000 Mb) on this node. Possible causes include (1) available query execution memory has been used up for table memory (in use table memory: 191.12 Mb) and (2) the quer
SingleStore	Cycle Graph	double	100	Unsupported	SingleStore error: (2709, 'recursive reference to query "tc" must not appear more than once') 2026-09-27 20:14:32,984 - INFO: Timing results saved to: /private/tmp/claude-501/-Users-johnidogun-Downloads-DBSysPerformance/210578b2-2f50-4fd6-8875-f449fb528170/scratchpad/results_2026/singlestore/timing/
SingleStore	Cycle Graph	left	100	Iteration limit	SingleStore error: (2741, "The query failed because a recursive common table expression exceeded the max number of iterations: 10000, which can be configured by the parameter 'max_recursive_cte_iterations'.") 2026-09-27 20:14:01,557 - INFO: Timing results saved to: /private/tmp/claude-501/-Users-joh
SingleStore	Cycle Graph	right	100	Iteration limit	SingleStore error: (2741, "The query failed because a recursive common table expression exceeded the max number of iterations: 10000, which can be configured by the parameter 'max_recursive_cte_iterations'.") 2026-09-27 20:14:31,746 - INFO: Timing results saved to: /private/tmp/claude-501/-Users-joh
SingleStore	Cycle with Shortcuts	double	100	Unsupported	SingleStore error: (2709, 'recursive reference to query "tc" must not appear more than once') 2026-09-27 20:16:08,252 - INFO: Timing results saved to: /private/tmp/claude-501/-Users-johnidogun-Downloads-DBSysPerformance/210578b2-2f50-4fd6-8875-f449fb528170/scratchpad/results_2026/singlestore/timing/
SingleStore	Cycle with Shortcuts	left	100	Out of memory	SingleStore error: (1712, "Leaf Error (127.0.0.1:3307): Memory used by MemSQL (5800.00 Mb) has reached the 'maximum_memory' setting (6000 Mb) on this node. Possible causes include (1) available query execution memory has been used up for table memory (in use table memory: 191.12 Mb) and (2) the quer
SingleStore	Cycle with Shortcuts	right	100	Out of memory	SingleStore error: (1712, "Leaf Error (127.0.0.1:3307): Memory used by MemSQL (5800.00 Mb) has reached the 'maximum_memory' setting (6000 Mb) on this node. Possible causes include (1) available query execution memory has been used up for table memory (in use table memory: 190.50 Mb) and (2) the quer
SingleStore	Grid Graph	double	100	Unsupported	SingleStore error: (2709, 'recursive reference to query "tc" must not appear more than once') 2026-09-27 21:09:00,450 - INFO: Timing results saved to: /private/tmp/claude-501/-Users-johnidogun-Downloads-DBSysPerformance/210578b2-2f50-4fd6-8875-f449fb528170/scratchpad/results_2026/singlestore/timing/
SingleStore	Grid Graph	left	300	Out of memory	SingleStore error: (1712, "Leaf Error (127.0.0.1:3307): Memory used by MemSQL (5800.00 Mb) has reached the 'maximum_memory' setting (6000 Mb) on this node. Possible causes include (1) available query execution memory has been used up for table memory (in use table memory: 199.12 Mb) and (2) the quer
SingleStore	Grid Graph	right	300	Out of memory	SingleStore error: (1712, "Leaf Error (127.0.0.1:3307): Memory used by MemSQL (5800.00 Mb) has reached the 'maximum_memory' setting (6000 Mb) on this node. Possible causes include (1) available query execution memory has been used up for table memory (in use table memory: 199.25 Mb) and (2) the quer
SingleStore	Max Acyclic Graph	double	100	Unsupported	SingleStore error: (2709, 'recursive reference to query "tc" must not appear more than once') 2026-09-27 20:13:30,835 - INFO: Timing results saved to: /private/tmp/claude-501/-Users-johnidogun-Downloads-DBSysPerformance/210578b2-2f50-4fd6-8875-f449fb528170/scratchpad/results_2026/singlestore/timing/
SingleStore	Max Acyclic Graph	left	100	Out of memory	SingleStore error: (1712, "Leaf Error (127.0.0.1:3307): Memory used by MemSQL (5800.00 Mb) has reached the 'maximum_memory' setting (6000 Mb) on this node. Possible causes include (1) available query execution memory has been used up for table memory (in use table memory: 191.38 Mb) and (2) the quer
SingleStore	Max Acyclic Graph	right	100	Out of memory	SingleStore error: (1712, "Leaf Error (127.0.0.1:3307): Memory used by MemSQL (5800.00 Mb) has reached the 'maximum_memory' setting (6000 Mb) on this node. Possible causes include (1) available query execution memory has been used up for table memory (in use table memory: 190.88 Mb) and (2) the quer
SingleStore	Multi-Path Graph	double	100	Unsupported	SingleStore error: (2709, 'recursive reference to query "tc" must not appear more than once') 2026-09-27 20:40:39,748 - INFO: Timing results saved to: /private/tmp/claude-501/-Users-johnidogun-Downloads-DBSysPerformance/210578b2-2f50-4fd6-8875-f449fb528170/scratchpad/results_2026/singlestore/timing/
SingleStore	Path Graph	double	100	Unsupported	SingleStore error: (2709, 'recursive reference to query "tc" must not appear more than once') 2026-09-27 20:22:06,009 - INFO: Timing results saved to: /private/tmp/claude-501/-Users-johnidogun-Downloads-DBSysPerformance/210578b2-2f50-4fd6-8875-f449fb528170/scratchpad/results_2026/singlestore/timing/
SingleStore	Reverse Binary Tree	double	100	Unsupported	SingleStore error: (2709, 'recursive reference to query "tc" must not appear more than once') 2026-09-27 21:12:53,387 - INFO: Timing results saved to: /private/tmp/claude-501/-Users-johnidogun-Downloads-DBSysPerformance/210578b2-2f50-4fd6-8875-f449fb528170/scratchpad/results_2026/singlestore/timing/
SingleStore	Scale-Free Directed	left	10,000	Out of memory	SingleStore error: (1712, "Leaf Error (127.0.0.1:3307): Memory used by MemSQL (5800.00 Mb) has reached the 'maximum_memory' setting (6000 Mb) on this node. Possible causes include (1) available query execution memory has been used up for table memory (in use table memory: 201.25 Mb) and (2) the quer
SingleStore	Scale-Free Directed	right	10,000	Out of memory	SingleStore error: (1712, "Leaf Error (127.0.0.1:3307): Memory used by MemSQL (5800.00 Mb) has reached the 'maximum_memory' setting (6000 Mb) on this node. Possible causes include (1) available query execution memory has been used up for table memory (in use table memory: 200.25 Mb) and (2) the quer
SingleStore	W Graph	double	100	Unsupported	SingleStore error: (2709, 'recursive reference to query "tc" must not appear more than once') 2026-09-27 21:18:57,143 - INFO: Timing results saved to: /private/tmp/claude-501/-Users-johnidogun-Downloads-DBSysPerformance/210578b2-2f50-4fd6-8875-f449fb528170/scratchpad/results_2026/singlestore/timing/
SingleStore	X Graph	double	100	Unsupported	SingleStore error: (2709, 'recursive reference to query "tc" must not appear more than once') 2026-09-27 21:14:52,738 - INFO: Timing results saved to: /private/tmp/claude-501/-Users-johnidogun-Downloads-DBSysPerformance/210578b2-2f50-4fd6-8875-f449fb528170/scratchpad/results_2026/singlestore/timing/
SingleStore	Y Graph	double	100	Unsupported	SingleStore error: (2709, 'recursive reference to query "tc" must not appear more than once') 2026-09-27 21:16:52,467 - INFO: Timing results saved to: /private/tmp/claude-501/-Users-johnidogun-Downloads-DBSysPerformance/210578b2-2f50-4fd6-8875-f449fb528170/scratchpad/results_2026/singlestore/timing/
XSB	Scale-Free Directed	left	70,000	Timeout

verified_2026

Outcome by series

Verification

Environment

No failure matches

Pick a file