The always-on vote fix on Hyperdisk Balanced at 20,000 IOPS

2026-11-13 · started 13:51 UTC · ended 16:22 UTC · merged mixed window 40 min 27 s

telcoin-network 2df831e3c · bench/bench6-2026-11 · commit verified from node startup logs

24 node starts across 11 instances logged telcoin-network 0.1.0 (2df831e3ca34c06a361a7acca10397787e7b332e) The node image probe recorded the same commit before deploy Node logs: fork schedule: seed_signature, multi_workers, prevrandao, leader_seeded_ordering, and subsecond_timestamp active from genesis Each of the 24 node starts logged its node switches line, and each matched the switch state the manifest records for that start Commit 2df831e3c is the head of the branch bench/bench6-2026-11, built for this benchmark: a base commit 165e3fe01 plus 98 commits, in which every fix for bench 5's concerns is a node switch, off by default. The twelve switches are quorum_exec_block (the vote fix, whose form the setting quorum_exec_block_starved_rounds picks), vote_queue_fix, epoch_record_state_check, mdbx_random_advice, mdbx_no_writemap, sealed_tx_keep, digest_dedupe, recovery_threads, epoch_close_task_drain, pipelined_persist, rpc_admission_backlog_outputs and parent_grace. A node logs its switches at every start ('node switches') and exports them as tn_node_switch. The image was named by registry digest (adiri@sha256:874537ff…852745) and ran all four runs of this program unchanged.

We ran the chaos run of 7 November (r20261107-1504, bench 5) again in its exact shape, on the same fleet and the same disk: ten c3-highcpu-8 validators in ten regions and an observer, each on Hyperdisk Balanced provisioned at 20,000 IOPS. The node is a new build (2df831e3c) in which every fix for bench 5's problems is a switch. This run turned on four of them: the vote fix in its always-on form, the voter's lock-wait fix, a state check on epoch records and the database's random-access hint, all on from the coordinated start. Average chain throughput with all ten generators running was 5,322.1 tx/s, 54.3 % above bench 5's 3,448.7 tx/s. The median time until a generator saw its receipt was 12.99 s (14.18 s on bench 5), and the floor, with 8 senders per generator, stayed at 1.65 s (1.63 s). Bench 5 lost own-header certification on its three fastest validators (0.0 to 16.3 % before the kill). Here every validator certified 99.7 % or more of its own headers over the load blast and 99.1 % or more in every five-minute bucket, and no vote was refused as execution-behind under load. Bench 5's extra disk bytes are gone: the validators wrote 1.186 to 1.193 times the database pages they copied and the observer 1.064, against an estimated 10 to 13 times on bench 5. What binds now is the disk's write count. The validators other than validator-05 ran at 0.998 of their 20,000 writes a second (the median; 19,946 to 19,984) for the whole 40 minutes, so the rate is 20,000 divided by the writes per transaction, which rose as the database grew. Throughput fell from 6,092 tx/s in the first five minutes to 4,719 in the last full five. Some accepted transactions still landed only after their generator resent them 300 s later (29,546 resends, on three generators), the reordering tracked as issue #1807. The killed validator rejoined once, 21 min 40.7 s after the kill (28 min 05.8 s on bench 5). While it followed the chain its RPC refused every transaction, so generator 05 landed nothing for the last 949 s of the blast. No node logged a fork, a halt, an unexpected execution result, a fatal database error or a panic. This run is one of four in one program, one node build with two forms of the vote fix on two disks: the always-on form as r20261113-1351 (20,000 IOPS) and r20261113-1630 (50,000 IOPS), the adaptive form as r20261113-2118 (20,000 IOPS) and r20261114-0000 (50,000 IOPS). Across them the form changed neither throughput nor what binds; the always-on form certified slightly more own headers at 20,000 IOPS, neither form held certification at 50,000 IOPS (the adaptive run certified more), the adaptive form's senders paid more on both disks through the same reordering (issue #1807), and which form goes upstream is not decided.

Compared against r20261107-1504.

Key figures

5,322.1tx/s
Average chain throughput, all load generators running
all generators active
7,327.4tx/s
Best 60 seconds, all load generators running
all generators active
14.66s
Time to land, p95
whole run
0.98–0.98s
Commit latency, p95
merged mixed window

How it was run

Software

Upstream committelcoin-network 2df831e3ca34c06a361a7acca10397787e7b332e · bench/bench6-2026-11 · commit verified from node startup logs
Node imageus-docker.pkg.dev/telcoin-network/tn-public/adiri@sha256:874537ff1cd8d496140154837e7a36524978bdbf3b0440e48972b073d9852745
Node image digestsha256:874537ff1cd8d496140154837e7a36524978bdbf3b0440e48972b073d9852745
Node version0.1.0
Fork schedulesubsecond-from-genesis
Load generator commit62bbd5f
Load generator image digestsha256:7cefcc505201b2e40361a7d8cfe034778090b76c568cb2400369bf566c605a59
Analysis commit62bbd5f
Harness commit74e9456ae6287c3ef324eb114930c72a7be711ad
Harness submodulescommon: 1a6955d8a2c6f19e53cf7aaad5e32a9107a21a7d
tn-node-deployment: d9446f720026fa5ce8b64fce9074e56b05b3f4eb

Fleet

Fleetc3-highcpu-8 + Hyperdisk Balanced at 20,000 IOPS (c3-hdb), ten regions, validator-05 in Ashburn
Validators10
Observers1
Machine typec3-highcpu-8
vCPUs per node8
Memory per node16 GB
Diskhyperdisk-balanced, 100 GB, 20,000 IOPS, 800 MiB/s
Regionsasia-east2 (Hong Kong), asia-northeast1 (Tokyo), asia-northeast3 (Seoul), asia-south1 (Mumbai), asia-southeast1 (Singapore), australia-southeast1 (Sydney), europe-west2 (London), europe-west4 (Netherlands), us-east4 (Ashburn), us-west2 (Los Angeles) (10)
Load generators10 generators × e2-standard-2 × 600 senders each
Load placementone blaster per validator, in that validator's zone, sending only to it

Chain

Chain ID32286
Networkbench
Epoch length1,200 s (20 min)

Consensus parameters

Max header delay1,000 ms
Min header delay350 ms
Max batch delay250 ms
Vote timeout5,000 ms
Parameters fileconsensus-parameters.yaml
Confirmed bynode startup logs

Parameter arms

ArmFrom (UTC)To (UTC)Values
final2026-11-13 14:10:04—all 11 nodes: epoch_record_state_check=true, mdbx_random_advice=true, quorum_exec_block=true, vote_queue_fix=true

Workload

Warm-up120 s (2 min)
Mixed phase2,400 s (40 min)
Cool-down180 s (3 min)
Transaction typesbatched_self_send, contract_deploy, faucet_drip, native_transfer, stablecoin_approve, stablecoin_transfer, uniswap_v2_add_liquidity, uniswap_v2_remove_liquidity, uniswap_v2_swap, uniswap_v3_swap, uniswap_v4_swap, wtel_unwrap, wtel_wrap

Chaos events

MarkerTime (UTC)NodeActionDetail
C12026-11-13 15:40:59bench-validator-05docker kill -s KILL telcoinsystemctl stop holds it down (unit has Restart=on-failure). bench-blaster-05 targets ONLY this node, so its sends fail for the outage by design; the generator resends on connection errors
C22026-11-13 15:45:59bench-validator-05systemctl start telcoindocker start is impossible: the unit runs docker run --rm
C32026-11-13 15:50:59bench-observer-11systemctl restart telcoinobserver restart

Steps

  1. All four runs of this program used one node image, built from the branch bench/bench6-2026-11 at commit 2df831e3c (a base commit 165e3fe01 plus 98 commits), with sub-second timestamps from genesis, and named by registry digest (adiri@sha256:874537ff…852745). Nothing was rebuilt between the runs. Every fix in that build is a node switch, off by default; a node logs its switches at every start ('node switches') and exports them as the metric tn_node_switch. The load generator was built from tn-transaction-generator commit 62bbd5f.
  2. The vote fix is the switch quorum_exec_block. Instead of its own newest executed block, a proposer names in each header a block that a quorum of validators should already have executed; a voter signs only after it finds the named block in its own chain. The setting quorum_exec_block_starved_rounds picks the form. With 0, the always-on form, every header takes the quorum value over the author's own executed tip and the blocks its parent headers named, capped at the own tip and never more than 8 outputs behind it (the floor); in practice nearly every header named the author's tip minus 8 outputs. With 4, the adaptive form, a header names the author's own tip until one of the author's headers is cancelled uncertified; the author then uses the always-on rule inside a starved window of up to 128 rounds, and afterwards tries its own tip again.
  3. Three more switches were on in all four runs: vote_queue_fix (the voter's lock wait), epoch_record_state_check (a check of the closing state when an epoch record is built) and mdbx_random_advice (the MDBX database's random-access hint over the whole file). The other eight stayed at their defaults: mdbx_no_writemap, sealed_tx_keep, digest_dedupe, recovery_threads, epoch_close_task_drain, pipelined_persist, rpc_admission_backlog_outputs and parent_grace. The switch set came from an arm file applied at deployment, before the nodes' coordinated start: no arm walk, no cache eviction and no restart before the load. Each node first ran once, alone, during staging (17 to 90 s, every switch at its default, no peer reached, no output executed); the switches of the run were on from the coordinated start, the first process that executed blocks.
  4. Each run created ten validators and one observer on c3-highcpu-8 (8 vCPU, 16 GiB), each node with one 100 GB Hyperdisk Balanced disk for the root and the data directory at 800 MiB/s, provisioned at 20,000 IOPS (bench 5's disk) or 50,000 IOPS. One validator ran in each of bench 5's ten regions (Los Angeles, London, Sydney, Tokyo, Ashburn, the Netherlands, Singapore, Hong Kong, Seoul and Mumbai), the observer in Los Angeles. Epochs last 1,200 s.
  5. Each run had three blasts. Ten generators on e2-standard-2, one per validator in that validator's zone and sending only to it, ran blast 0, the floor, with 8 senders each and up to 16 transactions in flight per sender (8 x 16); blast 1 with 600 senders and a window of 4 (600 x 4, at most 24,000 in flight); and blast 2, the load blast, with 600 senders and a window of 16 (600 x 16, at most 96,000 in flight) and phases of 120 s warm-up, 2,400 s mixed load and 180 s cool-down, the shape of bench 5's load blast.
  6. The chaos plan was bench 5's, inside blast 2: validator-05's container was killed with SIGKILL shortly before an epoch boundary and held down for 300 s, so the outage crossed the boundary; then it was started, and the observer was restarted 300 s after that. Generator 05 sends only to validator-05 and does not fail over to another endpoint, so it is cut off for the outage and for as long as its validator follows the chain afterwards.
  7. Headline figures keep bench 5's definitions and code; bench 5's throughput, latency and landing figures quoted on these pages were recomputed exactly from bench 5's own generator documents with the same code, while its certification, disk, stage and rejoin figures come from bench 5's report or from this program's reading of bench 5's logs and model. Throughput is the chain's transactions per second over the window in which all ten generators ran, with the outage and generator 05 inside it. Time to seen runs from submission until the generator saw the receipt, from the generators' summed histograms (the bundle method); time to land is to the block's timestamp, in whole seconds, so about 0.5 s low. Own-header certification is the share of a validator's own headers that were certified, over blast 2's mixed phase unless another window is named.
  8. The analysis read every node's whole main log (the node's stderr file was not pulled) and the node metrics exported from Prometheus before each teardown. The comparison of the two forms ran the same scripts on both runs of a disk and reproduced the always-on runs' figures.
  9. This run is the program's headline. Against r20261107-1504 the node build and the generator build differ; the fleet, the zones, the disk, the load blast and the chaos plan are bench 5's. The arm file final set quorum_exec_block_starved_rounds to 0, the always-on form.
  10. Before the nodes started, the harness probed each host: the same kernel (7.0.0-1017-gcp) and read-ahead (128 KB) on every node, and an fsync probe median of 1,060 µs, with validators 04 and 05 flagged slow (1,285 and 1,284 µs at the median). Before the first blast it found, on all eleven nodes, the four switches on and the rest off, the random-access hint covering the whole database map, and the database's prefault writes at 0.806 of newly allocated pages.
  11. A first deployment, r20261113-1327, stopped at key generation: the remote session to one validator dropped in the middle of its setup, the harness's retry met the first attempt's lock on the node, and the resume could not reuse the keys. A headline deployment was not repaired by hand. The fleet was torn down and the run deployed fresh under its own id, which cost 24 minutes.
  12. Times are UTC on 13 November. Deployment ran from 13:51:36 to 14:33:39, with the arm in place at 14:10:04. The mixed phases ran 14:44:38-14:51:02 (blast 0), 14:57:06-15:03:54 (blast 1) and 15:16:22-15:56:49 (blast 2). Epoch boundaries fell at 15:01:58, 15:21:58 and 15:41:58. Validator-05 was killed at 15:40:59 and started at 15:45:59, and the observer was restarted at 15:50:59.
  13. All ten generators ended every blast with rc 0, every fund was complete, and no generator started more than 3 s late. The teardown deleted all 12 instances (16:13:13-16:23:02); the fleet was up 2 h 31 min. The results directory holds 648 files, 2.44 GB, and the export before the teardown holds 382 node metrics (21,926 series).

The harness scripts live in a private repository; the steps above describe what they do.

Results

MetricValueWindowSourceMethod
Average chain throughput, all load generators running5,322.1 tx/sall generators activecombined.series.per_second[chain_tx]Chain transactions per block-timestamp second, summed over the 2373-s window when every load generator was in its mixed phase and divided by its length.
Best 60 seconds, all load generators running7,327.4 tx/sall generators activecombined.series.per_second[chain_tx]Highest mean chain tx/s over any 60 consecutive seconds inside the all-generators window (aggregate.py best_window_mean).
Transactions landed over the whole blast14,620,952 txwhole runcombined.totals.landed_chainSum of tx_count over every block the load generators saw during the blast, own and foreign transactions alike.
Blocks per second23.7 blocks/smerged mixed windowcombined.blocks.rows[timestamp]Chain blocks whose timestamp falls in the merged mixed window, divided by its length (aggregate.py landed.capacity.blocks_per_sec over the same window).
Time to land, median0.68 swhole runcombined.latency.ts_ms.p50Block timestamp minus submit time over every landed own transaction in the blast, from the load generators' merged latency histograms (aggregate.py merge_latency), in seconds.
Time to land, p9514.66 swhole runcombined.latency.ts_ms.p95Block timestamp minus submit time over every landed own transaction in the blast, from the load generators' merged latency histograms (aggregate.py merge_latency), in seconds.
Time to land, median, whole-second bias removed1.18 swhole runcombined.latency.ts_ms.p50 + 0.5 sTime to land, median, plus 0.5 s. The block timestamp is the committing leader's header creation time cut to the whole second, so every sample reads 0 to 999 ms low, 0.5 s on average.
Time to seen, median12.99 swhole runcombined.latency.seen_ms.p50Submit time to the moment the load generator's block scanner had the block holding the transaction (block and receipts fetched from the generator's own node), over every landed own transaction in the blast, from the load generators' merged latency histograms (aggregate.py merge_latency), in seconds.
Time to seen, p9530.89 swhole runcombined.latency.seen_ms.p95Submit time to the moment the load generator's block scanner had the block holding the transaction (block and receipts fetched from the generator's own node), over every landed own transaction in the blast, from the load generators' merged latency histograms (aggregate.py merge_latency), in seconds.
Time to land at light load, median0.26 slight load, 8 senders per generatorblast-0-20261113T145346Z/blasters latency.ts_ms.p50Block timestamp minus submit time over every landed own transaction submitted in the mixed phase of the light-load blast light8 (index 0: 8 senders per load generator at window 16, chaos off), pooled from the generators' latency histograms with the aggregate.py merge_latency method, in seconds; not bias-corrected.
Time to seen at light load, median1.65 slight load, 8 senders per generatorblast-0-20261113T145346Z/blasters latency.seen_ms.p50Submit time to the generator's block scanner having the block over every landed own transaction submitted in the mixed phase of the light-load blast light8 (index 0: 8 senders per load generator at window 16, chaos off), pooled from the generators' latency histograms with the aggregate.py merge_latency method, in seconds; not bias-corrected.
Commit latency, p950.98–0.98 smerged mixed windowmetrics.prometheus.series.tn_primary_commit_latency_seconds.by_node[healthy validators].p95For each healthy validator, the nearest-rank p95 over the merged mixed window of its 15-s histogram_quantile(0.95) commit-latency series; reported as the range over validators.
Execution time per consensus output0.72–0.72 smerged mixed windowmetrics.prometheus.series.tn_engine_execution_duration_seconds.by_node[healthy validators].meanFor each healthy validator, the nearest-rank median over the merged mixed window of its 15-s mean execution-duration series; reported as the range over validators.
Write throughput, p95, busiest validator104.5 MiB/smerged mixed windowmetrics.gcp[bench-validator-03].disk_write_bytes_per_sec p95Nearest-rank p95 of the 60-s mean write-byte rate over the merged mixed window for each healthy validator; the highest (bench-validator-03) is reported.
Write operations, p95, busiest validator20,146 IOPSmerged mixed windowmetrics.gcp[bench-validator-06].disk_write_ops_per_sec p95Nearest-rank p95 of the 60-s mean write-operation rate over the merged mixed window for each healthy validator; the highest (bench-validator-06) is reported.
CPU, p95, busiest validator36 %merged mixed windowmetrics.gcp[bench-validator-03].cpu_utilization p95Nearest-rank p95 of the 60-s CPU utilisation over the merged mixed window for each healthy validator; the highest (bench-validator-03) is reported.
Node resident memory, p95, busiest validator6.4 GiBmerged mixed windowmetrics.gcp[bench-validator-06].telcoin_rss_bytes p95Nearest-rank p95 of the 60-s telcoin process resident memory over the merged mixed window for each healthy validator; the highest (bench-validator-06) is reported.
IO pressure, highest reading, range over validators13.0–14.9 %merged mixed windowsampler[healthy validators].psi_io_some_avg10 maxFor each healthy validator, the highest 10-s 'some' IO pressure reading the on-node sampler logged inside the merged mixed window; reported as the range over validators.

Recovery

Nodebench-validator-05
Killed2026-11-13 15:40:59 UTC
Restarted2026-11-13 15:45:59 UTC
Rejoined2026-11-13 16:02:44 UTC
Catch-up time990 s (16 min 30 s)
Failed batch reportsnot recorded

Chain TPS per 5-minute interval

Chain TPS per 5-minute interval 9 intervals from 15:16 to 15:56 UTC; average 4,947 TPS, highest 6,092 TPS. The merged mixed window ran from 15:16 to 15:56 UTC. TPS 0 2,000 4,000 6,000 8,000 15:16–15:21 UTC: 6,092 TPS, 9.5 generators active 15:21–15:26 UTC: 5,548 TPS, 10.0 generators active 15:26–15:31 UTC: 5,617 TPS, 10.0 generators active 15:31–15:36 UTC: 5,482 TPS, 10.0 generators active 15:36–15:41 UTC: 5,178 TPS, 10.0 generators active 15:41–15:46 UTC: 5,356 TPS, 10.0 generators active 15:46–15:51 UTC: 4,963 TPS, 10.0 generators active 15:51–15:56 UTC: 4,719 TPS, 10.0 generators active 15:56–15:56 UTC: 1,569 TPS, 5.0 generators active avg 4,947 15:16 UTC 15:26 UTC 15:36 UTC 15:46 UTC 15:56 UTC C1 2026-11-13 15:40:59 UTC: docker kill -s KILL telcoin on bench-validator-05 (systemctl stop holds it down (unit has Restart=on-failure). bench-blaster-05 targets ONLY this node, so its sends fail for the outage by design; the generator resends on connection errors) C1 C2 2026-11-13 15:45:59 UTC: systemctl start telcoin on bench-validator-05 (docker start is impossible: the unit runs docker run --rm) C2 C3 2026-11-13 15:50:59 UTC: systemctl restart telcoin on bench-observer-11 (observer restart) C3

Each bar is the average chain TPS over one 5-minute interval of the merged mixed window (15:16–15:56 UTC); fainter bars had fewer generators active than the maximum, the dashed line is the average of all bars and amber lines mark the chaos events listed below.

  • C1 2026-11-13 15:40:59 UTC: docker kill -s KILL telcoin on bench-validator-05 (systemctl stop holds it down (unit has Restart=on-failure). bench-blaster-05 targets ONLY this node, so its sends fail for the outage by design; the generator resends on connection errors)
  • C2 2026-11-13 15:45:59 UTC: systemctl start telcoin on bench-validator-05 (docker start is impossible: the unit runs docker run --rm)
  • C3 2026-11-13 15:50:59 UTC: systemctl restart telcoin on bench-observer-11 (observer restart)
5-minute data (9 rows)
start_utcend_utcgenerators_activechain_tps
2026-11-13T15:16:22Z2026-11-13T15:21:22Z9.56,091.7
2026-11-13T15:21:22Z2026-11-13T15:26:22Z10.05,547.9
2026-11-13T15:26:22Z2026-11-13T15:31:22Z10.05,617.0
2026-11-13T15:31:22Z2026-11-13T15:36:22Z10.05,481.6
2026-11-13T15:36:22Z2026-11-13T15:41:22Z10.05,177.7
2026-11-13T15:41:22Z2026-11-13T15:46:22Z10.05,356.2
2026-11-13T15:46:22Z2026-11-13T15:51:22Z10.04,962.7
2026-11-13T15:51:22Z2026-11-13T15:56:22Z10.04,718.8
2026-11-13T15:56:22Z2026-11-13T15:56:49Z5.01,569.3

The fine-grained chain throughput series (its step_s field gives the seconds per point) is published as series/tps-1s.json.

Other networks

MeasureTelcoin Network (this run)Tempo
Throughput5,322.1 tx/s (Avg chain TPS)16,455 TPS, shown as "Settled TPS" for the nightly run of 5 Oct 2026 (commit de5f5d3). perf.tempo.xyz describes it as "Median settled throughput across 130 measured blocks."
How throughput is measuredChain transactions per block-timestamp second, summed over the 2373-s window when every load generator was in its mixed phase and divided by its length.The methodology page defines TPS as "Transactions per second (derived from gas / avg tx gas)": gas used divided by the average gas of one transaction, reported as the median over the measured blocks.
Run length40.5 min merged mixed window; each generator was set to a 40 min mixed phase2m 39s for the 5 Oct 2026 nightly run (130 measured blocks at a 687 ms average block time), per perf.tempo.xyz.
Validators and placement10 validators and 1 observer across 10 regions: asia-east2 (Hong Kong), asia-northeast1 (Tokyo), asia-northeast3 (Seoul), asia-south1 (Mumbai), asia-southeast1 (Singapore), australia-southeast1 (Sydney), europe-west2 (London), europe-west4 (Netherlands), us-east4 (Ashburn), us-west2 (Los Angeles); load generators: one blaster per validator, in that validator's zone, sending only to itperf.tempo.xyz labels the scenario "Public mix · multi-region". The methodology page says "Benchmarks run two local validators in consensus on a dedicated bare-metal server with 100 GiB pre-bloated state." The public tempoxyz/tempo repo has nightly workflows for both kinds of setup: bench-e2e runs on one bare-metal runner, and bench-e2e-multi-region defaults to 10 validators on cloud VMs (its regions input reads "Empty uses the selected cloud's five defaults."). The run page does not say which setup produced this number.
Hardwarec3-highcpu-8, 8 vCPUs, 16 GB memory; disk: hyperdisk-balanced, 100 GB, 20,000 IOPS, 800 MiB/sThe methodology page lists a dedicated bare-metal server: "AMD EPYC 4585PX (16C/32T, 4.3/5.7 GHz)", "128 GB 3600 MHz" memory and "2×960 GB NVMe + 2×1.92 TB NVMe via schelk". The repo's multi-region workflow uses cloud VMs instead (c4d-standard-48-lssd on GCP, c6id.8xlarge on AWS).
Workloadbatched_self_send, contract_deploy, faucet_drip, native_transfer, stablecoin_approve, stablecoin_transfer, uniswap_v2_add_liquidity, uniswap_v2_remove_liquidity, uniswap_v2_swap, uniswap_v3_swap, uniswap_v4_swap, wtel_unwrap, wtel_wrap; 10 generators × 600 senders eachperf.tempo.xyz labels the scenario "Public mix"; the methodology page says "100% TIP-20 token transfers (ERC-20 equivalent)". The run used 1,000 accounts and 4 TIP-20 tokens.
StateFresh chain from genesis for every runThe methodology page says "100 GiB pre-bloated state".
Latency (as each source defines it)Time to land, median: 0.68 s; Time to land, p95: 14.66 s; Commit latency, p95: 0.98–0.98 sBlock time of 687 ms, which the methodology page defines as "Average wall-clock time between blocks". The run page gives no per-transaction latency. Separately, tempo.xyz says "Blocks finalize in ~0.6 seconds with no re-orgs." (a network claim, not a figure from this benchmark).
Source and fetch dateThis page; every figure names its source field.perf.tempo.xyz nightly run of 5 Oct 2026 (commit de5f5d3), the perf.tempo.xyz methodology page, tempo.xyz, and the benchmark workflows in the tempoxyz/tempo repo. All fetched on 5 Oct 2026.

Sources for Tempo (fetched 2026-10-05):

Tempo's setup may not match ours: its methodology page describes two validators on one bare-metal server, while our runs use ten validators across five regions, so read this table as context, not a ranking.

This table is context, not a ranking.

Caveats

  • Against r20261107-1504 the fleet, zones, disk, load blast and chaos plan are the same, and the node and generator builds differ. Throughput rose from 3,448.7 to 5,322.1 tx/s, the best 60 s from 4,939.2 to 7,327.4 tx/s, blocks a second from 12.96 to 23.70 and transactions landed in blast 2 from 9,288,504 to 14,620,952. Time to land p50/p95/p99 went from 2.81/14.40/402.56 s to 0.68/14.66/61.91 s, and time to seen p50/p95 from 14.18/53.39 s to 12.99/30.89 s. The two runs are bound by different things: bench 5 by the bytes its disk wrote, this run by the count of writes, so 5,322 against 3,449 compares a run bound by write count with one bound by bytes.
  • Generator 05's outage weighs more on bench 5's headline: generator 05 was silent for 65 % of bench 5's window against 40 % here. The other nine generators carried 5,066.9 tx/s. The headline is a 40-minute mean of a falling rate (6,091.7 tx/s in the first five-minute bucket, 4,718.8 in the eighth); the fall is not the outage, since the other nine carried 4,699 tx/s in the bucket that ends 23 s after the kill and 4,963 and 4,719 in the two after the outage.
  • At 20,000 IOPS the disk's write count binds. Over blast 2 the median validator, validator-05 aside, ran 19,967 device writes a second (0.998 of 20,000; 19,946 to 19,984 across the nine), 5.0 KiB each, 3.74 writes per executed transaction, and every engine was 981 to 999 ms busy each second with its queue at the cap of 8 in 95 to 100 % of samples. The executed rate fell from 6,701 tx/s in minute 0 to 4,508-4,846 in minutes 35 to 39 while writes per transaction rose from 2.94 to 4.1-4.5. From the first five-minute bucket to the last the database commit rose from 54 to 98 µs per transaction and outputs shrank from 4,160 to 3,361 transactions. Writes per transaction follow output size, database size and key density; that is a fit across the arm-walk run's blasts, not a mechanism, and no switch moved them by more than about 7 % alone.
  • Per executed transaction the engine spent 187.2 µs: execute 90.2, persist 95.2 (save 20.9, commit 74.2) and announce 0.7. In this build execute and persist run in series. Bench 5's commit was 178.0 µs per transaction.
  • This program explains bench 5's extra disk bytes. In an earlier arm-walk run of the same build (r20261112-1520), every node's first process that executed blocks entered one state: pages ahead of the database's allocation frontier were already in the page cache, the database library skipped its prefault writes, and new pages landed in large page-cache folios that were written back whole. A restart ends the state; the one node not restarted there, the observer, wrote 6.5 to 10.9 times its page bytes, like bench 5. With the random-access hint on from the coordinated start, the first process that executed blocks, every node of this run, the observer included, wrote single pages from genesis to the end. Each step is measured, but what starts the state is a correlation, and whether bench 5's nodes blasted in their first process that executed blocks was not checked; that bench 5's amplification was this state is the explanation that fits its signature on every node.
  • The vote fix held while the disk bound: 0 votes refused as execution-behind and 0 vote timeouts in blast 2, the voter's lock wait 0 at the 95th percentile on every voter, and execution lag even across the nodes at 35 to 45 rounds (bench 5: 47,614 refused votes). The disk is the reason: it gave every engine the same cost, 187.0 to 187.3 µs per transaction, so no author ran far ahead of the voters. On the same build at 50,000 IOPS (r20261113-1630) the engines spread and four validators lost certification, so this pass holds only while the disk binds.
  • Accepted transactions can still land late, or not at all without a resend. 40,263 landings were seen 300 s or more after submission: 30,497 gap resends by generators 02, 06 and 10, and 9,766 that generator 05's scanner saw late although they had landed in time. 29,546 transactions were resent after the generators' gap check, and 183 never landed after the 180-s tail scan; landed over accepted was 99.9987 %. Bench 5 did not record these counts. The mechanism is the same in all four runs of this program and is tracked as issue #1807: when an own header is cancelled or committed late, its batches are queued again behind batches a newer own header already carried, so the sender's later transactions are skipped at execution as 'nonce too high', sit in no pool, and land only after the generator resends its window at its 300-s gap check. The pool switches sealed_tx_keep and digest_dedupe were off in all four runs; whether they change this cost was not measured in these runs (in the earlier arm-walk run, sealed_tx_keep with three other switches on left no gap resends, a reading confounded with the other switches and an epoch boundary).
  • The floor did not move: seen p50 at 8 x 16 was 1.65 s (1.63 s on bench 5), and by stage on the 1-in-100 sample 1.50 s, as in bench 5. This run's floor seen p95 was 2.27 s, against 2.02 to 2.03 s in the arm-walk run and r20261113-1630; the stage that holds it was not traced. London's and the Netherlands' generators (02 and 06) landed 11 % and 12 % of the others' median at the floor, and their certificates were 2.1 to 2.7 % of the committed ones against a fair 10 %; the parent grace that addresses this was off.
  • Under load the consensus part of the path halved against bench 5 (commit p50 1.73 s against 3.78 s before the kill) and the wait for the validator's own execution disappeared (its p95 0 against 16.9 s), while the engine queue grew (12.5 s against 9.6 s at the p50), because every node now carries the queue that bench 5's slow executors carried. So seen p50 at 600 x 16 barely moved. The loop is closed: with 96,000 transactions in flight the mean time to seen is in flight over the rate (18.02 s against the measured 17.82 s), so seen p50 should be read beside the mean.
  • The 600 x 4 point (blast 1) carried 6,380.8 tx/s (3,868.5 on bench 5) with seen p50 3.21 s (3.22 s). It is now bound by the window: the implied in flight is 105 % of the 24,000 cap. The load blast landed 17 % less than this run's own 600 x 4 blast.
  • Validator-05 rejoined once and held: 21 min 40.7 s from the kill to active and 16 min 40.8 s from its start (bench 5: 28 min 05.8 s and 23 min 05.7 s), by the node's log. It replayed the already-closed epoch, then followed epoch 5 under load at 1.332 outputs a second while the chain made 1.409, so its deficit grew to 527 outputs until the load ended, and it promoted at the start of epoch 6. This is the rejoin rule working as written: in this build a follower may promote only within 20 rounds of the newest known header and replays no faster than it executes, so under a load that holds the committee at its execution limit it cannot close the gap. That it would never rejoin under a load that does not stop is inferred from the trend. While it followed, its RPC refused every transaction with '-32603 validator is inactive; retry another endpoint'; generator 05 dropped 369,678 sends after three attempts each and landed nothing for the last 949 s of the blast. Validator-05 refused 20,402 votes as inactive and signed neither epoch record after its kill. The observer followed the chain again 23.3 s after its restart.
  • All six epoch closes certified on one record hash and one closing block on every node, with at least 7 of 10 signers; the minimum signers per epoch were 10, 10, 7, 7, 9 and 9. The margin is thin at loaded closes: at 15:21:58 the 7th and 8th votes came 0.45 s apart, with a commit gap of 14.49 s. The error 'failed to reach quorum on epoch close' is a false alarm: across the arm-walk run (r20261112-1520), this run and r20261113-1630 it fired 11 times, each time on a record the network had certified. No test in this build (2df831e3c) crosses an epoch close with the always-on form.
  • No hard check fired: no halt, fork, unexpected execution result, fatal database error or panic in any node's whole main log. The nodes logged 67,650 warnings and 5,370 errors, mostly from the outage (17,578 dial failures), validator-05 as a follower (19,134 certificate-manager warnings in the last 12 minutes of the blast) and the observer's sync retries until its restart. The node's stderr file was not pulled.
  • Two statements made before this run was analysed were corrected. The expected fall through a 40-minute blast at this disk was put at about a third; the measured fall was 15 % from minutes 1-4 to 35-39 (29 % from minute 0). And the arm-walk run's 'no whole-block writes anywhere' held for its validators only; its observer reproduced bench 5.
  • The teardown's provenance check warns 'node startup provenance mismatch' in all four runs: each node's start during staging ran before the arm file was applied. It is known and not a finding.
  • Not measured: when a vote reaches its author (not logged), what the age of the named block costs, a deeper floor in a real run, and the senders' cost with sealed_tx_keep or digest_dedupe on.
  • Each configuration ran once (one run per form and disk), so run-to-run variation is not measured. The form of the vote fix is not decided: at 20,000 IOPS the always-on form certified a little more and cost the senders less, at 50,000 IOPS neither form held certification, and the floor's depth has been tested only as arithmetic. A further run of both forms is planned on a build of the newer upstream main branch, which uses a different execution storage engine (tn-lmdb) than these runs (MDBX), with a candidate fix for issue #1807 added; its result is not known.
  • Per-validator rows cover 9 validators; bench-validator-05 excluded as chaos target(s).
  • Recovery: demoted / rejoined are the first 15-s Prometheus samples showing the change, so they trail the node's own log times by up to a scrape interval plus one step.
  • Recovery: no tn_batch_builder_seal_failures_total count for bench-validator-05; failed_reports unavailable.
  • The harness working tree had uncommitted changes when the run was deployed.
  • Before the coordinated start each node ran once during staging with every node switch at its default (11 start(s) on 11 node(s)); the run's switches were on from the coordinated start at 2026-11-13T14:10:04Z.

Full report

Full report HTML · 1.6 MB

Changes from the original report:

  • The site adds a banner at the top of the report that links back to this page.

SHA-256 of the committed file, before the banner is added: a0a2a3a1c1a612b308dd83d0f8769e0868e9934402d4032f212ff1165ad32393