The always-on vote fix on Hyperdisk Balanced at 50,000 IOPS

2026-11-13 · started 16:30 UTC · ended 19:08 UTC · merged mixed window 40 min 47 s

telcoin-network 2df831e3c · bench/bench6-2026-11 · commit verified from node startup logs

24 node starts across 11 instances logged telcoin-network 0.1.0 (2df831e3ca34c06a361a7acca10397787e7b332e) The node image probe recorded the same commit before deploy Node logs: fork schedule: seed_signature, multi_workers, prevrandao, leader_seeded_ordering, and subsecond_timestamp active from genesis Each of the 24 node starts logged its node switches line, and each matched the switch state the manifest records for that start Commit 2df831e3c is the head of the branch bench/bench6-2026-11, built for this benchmark: a base commit 165e3fe01 plus 98 commits, in which every fix for bench 5's concerns is a node switch, off by default. The twelve switches are quorum_exec_block (the vote fix, whose form the setting quorum_exec_block_starved_rounds picks), vote_queue_fix, epoch_record_state_check, mdbx_random_advice, mdbx_no_writemap, sealed_tx_keep, digest_dedupe, recovery_threads, epoch_close_task_drain, pipelined_persist, rpc_admission_backlog_outputs and parent_grace. A node logs its switches at every start ('node switches') and exports them as tn_node_switch. The image was named by registry digest (adiri@sha256:874537ff…852745) and ran all four runs of this program unchanged.

We repeated r20261113-1351 with one change: each node's Hyperdisk Balanced disk was provisioned at 50,000 IOPS instead of bench 5's 20,000. The build (2df831e3c), the four switches (the vote fix in its always-on form, the voter's lock-wait fix, the epoch-record state check and the database's random-access hint), the fleet, the load and the chaos plan were the same. With the disk off its limit, average chain throughput with all ten generators running was 7,329.0 tx/s, 112.5 % above bench 5's 3,448.7 tx/s and 37.7 % above r20261113-1351's 5,322.1 (derived), and it held through the blast at 7,077 to 7,684 tx/s in each five minutes. The median time until a generator saw its receipt fell from 12.99 s to 8.32 s; the floor stayed at 1.63 s. The disk no longer binds: the median validator's disk ran 24,837 writes a second, half its limit, and the database commit fell from 74.2 to 22.4 µs per transaction. The engine binds instead. In this build executing and persisting a consensus output run in series, and the costliest engine (validator-01, 136.9 µs per transaction) has a capacity of 7,287 tx/s, the fleet's executed rate. The vote fix did not hold at this disk. Through the light blasts and the first 120 s of the load it certified as well as at 20,000 IOPS, but over the load blast four validators lost certification: validator-06 certified 44.5 % of its own headers, validator-10 68.2 %, validator-02 78.6 % and validator-09 95.8 %, while the other six stayed at 99.9 to 100 %. The always-on form names its author's executed tip minus 8 outputs. Once the disk no longer gave every engine the same cost, the engines spread by 18 to 45 outputs, so a fast author named blocks that four or five voters had not executed yet, and those voters answered after the author had moved on to its next header. It is a liveness and throughput problem, not a safety one: no node forked or halted. More accepted transactions landed only after a resend (133,975 resends on eight generators; 14,928 never landed), the reordering tracked as issue #1807. Validator-05 rejoined once, 21 min 30.2 s after its kill. This run is one of four in one program, one node build with two forms of the vote fix on two disks: the always-on form as r20261113-1351 (20,000 IOPS) and r20261113-1630 (50,000 IOPS), the adaptive form as r20261113-2118 (20,000 IOPS) and r20261114-0000 (50,000 IOPS). Across them the form changed neither throughput nor what binds; the always-on form certified slightly more own headers at 20,000 IOPS, neither form held certification at 50,000 IOPS (the adaptive run certified more), the adaptive form's senders paid more on both disks through the same reordering (issue #1807), and which form goes upstream is not decided.

Compared against r20261113-1351.

Key figures

7,329tx/s
Average chain throughput, all load generators running
all generators active
8,256.4tx/s
Best 60 seconds, all load generators running
all generators active
3.53s
Time to land, p95
whole run
0.99–1.30s
Commit latency, p95
merged mixed window

How it was run

Software

Upstream committelcoin-network 2df831e3ca34c06a361a7acca10397787e7b332e · bench/bench6-2026-11 · commit verified from node startup logs
Node imageus-docker.pkg.dev/telcoin-network/tn-public/adiri@sha256:874537ff1cd8d496140154837e7a36524978bdbf3b0440e48972b073d9852745
Node image digestsha256:874537ff1cd8d496140154837e7a36524978bdbf3b0440e48972b073d9852745
Node version0.1.0
Fork schedulesubsecond-from-genesis
Load generator commit62bbd5f
Load generator image digestsha256:7cefcc505201b2e40361a7d8cfe034778090b76c568cb2400369bf566c605a59
Analysis commit62bbd5f
Harness commit4b60934ba6ee780ca4f42b7aa46791e641d61d09
Harness submodulescommon: 1a6955d8a2c6f19e53cf7aaad5e32a9107a21a7d
tn-node-deployment: d9446f720026fa5ce8b64fce9074e56b05b3f4eb

Fleet

Fleetc3-highcpu-8 + Hyperdisk Balanced at 50,000 IOPS (c3-hdb), ten regions, validator-05 in Ashburn
Validators10
Observers1
Machine typec3-highcpu-8
vCPUs per node8
Memory per node16 GB
Diskhyperdisk-balanced, 100 GB, 50,000 IOPS, 800 MiB/s
Regionsasia-east2 (Hong Kong), asia-northeast1 (Tokyo), asia-northeast3 (Seoul), asia-south1 (Mumbai), asia-southeast1 (Singapore), australia-southeast1 (Sydney), europe-west2 (London), europe-west4 (Netherlands), us-east4 (Ashburn), us-west2 (Los Angeles) (10)
Load generators10 generators × e2-standard-2 × 600 senders each
Load placementone blaster per validator, in that validator's zone, sending only to it

Chain

Chain ID32286
Networkbench
Epoch length1,200 s (20 min)

Consensus parameters

Max header delay1,000 ms
Min header delay350 ms
Max batch delay250 ms
Vote timeout5,000 ms
Parameters fileconsensus-parameters.yaml
Confirmed bynode startup logs

Parameter arms

ArmFrom (UTC)To (UTC)Values
final2026-11-13 16:50:28—all 11 nodes: epoch_record_state_check=true, mdbx_random_advice=true, quorum_exec_block=true, vote_queue_fix=true

Workload

Warm-up120 s (2 min)
Mixed phase2,400 s (40 min)
Cool-down180 s (3 min)
Transaction typesbatched_self_send, contract_deploy, faucet_drip, native_transfer, stablecoin_approve, stablecoin_transfer, uniswap_v2_add_liquidity, uniswap_v2_remove_liquidity, uniswap_v2_swap, uniswap_v3_swap, uniswap_v4_swap, wtel_unwrap, wtel_wrap

Chaos events

MarkerTime (UTC)NodeActionDetail
C12026-11-13 18:21:20bench-validator-05docker kill -s KILL telcoinsystemctl stop holds it down (unit has Restart=on-failure). bench-blaster-05 targets ONLY this node, so its sends fail for the outage by design; the generator resends on connection errors
C22026-11-13 18:26:20bench-validator-05systemctl start telcoindocker start is impossible: the unit runs docker run --rm
C32026-11-13 18:31:20bench-observer-11systemctl restart telcoinobserver restart

Steps

  1. All four runs of this program used one node image, built from the branch bench/bench6-2026-11 at commit 2df831e3c (a base commit 165e3fe01 plus 98 commits), with sub-second timestamps from genesis, and named by registry digest (adiri@sha256:874537ff…852745). Nothing was rebuilt between the runs. Every fix in that build is a node switch, off by default; a node logs its switches at every start ('node switches') and exports them as the metric tn_node_switch. The load generator was built from tn-transaction-generator commit 62bbd5f.
  2. The vote fix is the switch quorum_exec_block. Instead of its own newest executed block, a proposer names in each header a block that a quorum of validators should already have executed; a voter signs only after it finds the named block in its own chain. The setting quorum_exec_block_starved_rounds picks the form. With 0, the always-on form, every header takes the quorum value over the author's own executed tip and the blocks its parent headers named, capped at the own tip and never more than 8 outputs behind it (the floor); in practice nearly every header named the author's tip minus 8 outputs. With 4, the adaptive form, a header names the author's own tip until one of the author's headers is cancelled uncertified; the author then uses the always-on rule inside a starved window of up to 128 rounds, and afterwards tries its own tip again.
  3. Three more switches were on in all four runs: vote_queue_fix (the voter's lock wait), epoch_record_state_check (a check of the closing state when an epoch record is built) and mdbx_random_advice (the MDBX database's random-access hint over the whole file). The other eight stayed at their defaults: mdbx_no_writemap, sealed_tx_keep, digest_dedupe, recovery_threads, epoch_close_task_drain, pipelined_persist, rpc_admission_backlog_outputs and parent_grace. The switch set came from an arm file applied at deployment, before the nodes' coordinated start: no arm walk, no cache eviction and no restart before the load. Each node first ran once, alone, during staging (17 to 90 s, every switch at its default, no peer reached, no output executed); the switches of the run were on from the coordinated start, the first process that executed blocks.
  4. Each run created ten validators and one observer on c3-highcpu-8 (8 vCPU, 16 GiB), each node with one 100 GB Hyperdisk Balanced disk for the root and the data directory at 800 MiB/s, provisioned at 20,000 IOPS (bench 5's disk) or 50,000 IOPS. One validator ran in each of bench 5's ten regions (Los Angeles, London, Sydney, Tokyo, Ashburn, the Netherlands, Singapore, Hong Kong, Seoul and Mumbai), the observer in Los Angeles. Epochs last 1,200 s.
  5. Each run had three blasts. Ten generators on e2-standard-2, one per validator in that validator's zone and sending only to it, ran blast 0, the floor, with 8 senders each and up to 16 transactions in flight per sender (8 x 16); blast 1 with 600 senders and a window of 4 (600 x 4, at most 24,000 in flight); and blast 2, the load blast, with 600 senders and a window of 16 (600 x 16, at most 96,000 in flight) and phases of 120 s warm-up, 2,400 s mixed load and 180 s cool-down, the shape of bench 5's load blast.
  6. The chaos plan was bench 5's, inside blast 2: validator-05's container was killed with SIGKILL shortly before an epoch boundary and held down for 300 s, so the outage crossed the boundary; then it was started, and the observer was restarted 300 s after that. Generator 05 sends only to validator-05 and does not fail over to another endpoint, so it is cut off for the outage and for as long as its validator follows the chain afterwards.
  7. Headline figures keep bench 5's definitions and code; bench 5's throughput, latency and landing figures quoted on these pages were recomputed exactly from bench 5's own generator documents with the same code, while its certification, disk, stage and rejoin figures come from bench 5's report or from this program's reading of bench 5's logs and model. Throughput is the chain's transactions per second over the window in which all ten generators ran, with the outage and generator 05 inside it. Time to seen runs from submission until the generator saw the receipt, from the generators' summed histograms (the bundle method); time to land is to the block's timestamp, in whole seconds, so about 0.5 s low. Own-header certification is the share of a validator's own headers that were certified, over blast 2's mixed phase unless another window is named.
  8. The analysis read every node's whole main log (the node's stderr file was not pulled) and the node metrics exported from Prometheus before each teardown. The comparison of the two forms ran the same scripts on both runs of a disk and reproduced the always-on runs' figures.
  9. Against r20261113-1351 only the disk changed: 50,000 provisioned IOPS instead of 20,000, at the same 800 MiB/s. The arm file final, the image and the generator are the same, so the always-on form ran here too.
  10. Before the nodes started, the harness probed each host: the same kernel (7.0.0-1017-gcp) and read-ahead (128 KB) on every node, and an fsync probe median of 1,006 µs, with validator-06 flagged slow (1,226 µs). Before the first blast it found, on all eleven nodes, the four switches on and the rest off, the random-access hint covering the whole database map, and the database's prefault writes at 0.830 of newly allocated pages.
  11. Times are UTC on 13 November. Deployment ran from 16:30:48 to 17:14:11, with the arm in place at 16:50:28. The mixed phases ran 17:24:45-17:31:02 (blast 0), 17:37:13-17:43:22 (blast 1) and 17:55:32-18:36:19 (blast 2). Epoch boundaries fell at 17:42:19, 18:02:19 and 18:22:19. Validator-05 was killed at 18:21:20 and started at 18:26:20, and the observer was restarted at 18:31:20.
  12. All ten generators ended every blast with rc 0, every fund was complete, and no generator started more than 2 s late; the run's log records no friction. The teardown deleted all 12 instances (18:56:08-19:11:16); the fleet was up 2 h 40 min from the deploy's launch to the teardown's end. The results directory holds 651 files, 2.67 GB, and the export before the teardown holds 382 node metrics (21,950 series).

The harness scripts live in a private repository; the steps above describe what they do.

Results

MetricValueWindowSourceMethod
Average chain throughput, all load generators running7,329 tx/sall generators activecombined.series.per_second[chain_tx]Chain transactions per block-timestamp second, summed over the 2353-s window when every load generator was in its mixed phase and divided by its length.
Best 60 seconds, all load generators running8,256.4 tx/sall generators activecombined.series.per_second[chain_tx]Highest mean chain tx/s over any 60 consecutive seconds inside the all-generators window (aggregate.py best_window_mean).
Transactions landed over the whole blast19,484,670 txwhole runcombined.totals.landed_chainSum of tx_count over every block the load generators saw during the blast, own and foreign transactions alike.
Blocks per second27.72 blocks/smerged mixed windowcombined.blocks.rows[timestamp]Chain blocks whose timestamp falls in the merged mixed window, divided by its length (aggregate.py landed.capacity.blocks_per_sec over the same window).
Time to land, median0.86 swhole runcombined.latency.ts_ms.p50Block timestamp minus submit time over every landed own transaction in the blast, from the load generators' merged latency histograms (aggregate.py merge_latency), in seconds.
Time to land, p953.53 swhole runcombined.latency.ts_ms.p95Block timestamp minus submit time over every landed own transaction in the blast, from the load generators' merged latency histograms (aggregate.py merge_latency), in seconds.
Time to land, median, whole-second bias removed1.36 swhole runcombined.latency.ts_ms.p50 + 0.5 sTime to land, median, plus 0.5 s. The block timestamp is the committing leader's header creation time cut to the whole second, so every sample reads 0 to 999 ms low, 0.5 s on average.
Time to seen, median8.32 swhole runcombined.latency.seen_ms.p50Submit time to the moment the load generator's block scanner had the block holding the transaction (block and receipts fetched from the generator's own node), over every landed own transaction in the blast, from the load generators' merged latency histograms (aggregate.py merge_latency), in seconds.
Time to seen, p9525.74 swhole runcombined.latency.seen_ms.p95Submit time to the moment the load generator's block scanner had the block holding the transaction (block and receipts fetched from the generator's own node), over every landed own transaction in the blast, from the load generators' merged latency histograms (aggregate.py merge_latency), in seconds.
Time to land at light load, median0.24 slight load, 8 senders per generatorblast-0-20261113T173352Z/blasters latency.ts_ms.p50Block timestamp minus submit time over every landed own transaction submitted in the mixed phase of the light-load blast light8 (index 0: 8 senders per load generator at window 16, chaos off), pooled from the generators' latency histograms with the aggregate.py merge_latency method, in seconds; not bias-corrected.
Time to seen at light load, median1.63 slight load, 8 senders per generatorblast-0-20261113T173352Z/blasters latency.seen_ms.p50Submit time to the generator's block scanner having the block over every landed own transaction submitted in the mixed phase of the light-load blast light8 (index 0: 8 senders per load generator at window 16, chaos off), pooled from the generators' latency histograms with the aggregate.py merge_latency method, in seconds; not bias-corrected.
Commit latency, p950.99–1.30 smerged mixed windowmetrics.prometheus.series.tn_primary_commit_latency_seconds.by_node[healthy validators].p95For each healthy validator, the nearest-rank p95 over the merged mixed window of its 15-s histogram_quantile(0.95) commit-latency series; reported as the range over validators.
Execution time per consensus output0.75–0.79 smerged mixed windowmetrics.prometheus.series.tn_engine_execution_duration_seconds.by_node[healthy validators].meanFor each healthy validator, the nearest-rank median over the merged mixed window of its 15-s mean execution-duration series; reported as the range over validators.
Write throughput, p95, busiest validator129.8 MiB/smerged mixed windowmetrics.gcp[bench-validator-09].disk_write_bytes_per_sec p95Nearest-rank p95 of the 60-s mean write-byte rate over the merged mixed window for each healthy validator; the highest (bench-validator-09) is reported.
Write operations, p95, busiest validator27,058 IOPSmerged mixed windowmetrics.gcp[bench-validator-09].disk_write_ops_per_sec p95Nearest-rank p95 of the 60-s mean write-operation rate over the merged mixed window for each healthy validator; the highest (bench-validator-09) is reported.
CPU, p95, busiest validator43.6 %merged mixed windowmetrics.gcp[bench-validator-09].cpu_utilization p95Nearest-rank p95 of the 60-s CPU utilisation over the merged mixed window for each healthy validator; the highest (bench-validator-09) is reported.
Node resident memory, p95, busiest validator6.55 GiBmerged mixed windowmetrics.gcp[bench-validator-10].telcoin_rss_bytes p95Nearest-rank p95 of the 60-s telcoin process resident memory over the merged mixed window for each healthy validator; the highest (bench-validator-10) is reported.
IO pressure, highest reading, range over validators2.7–4.1 %merged mixed windowsampler[healthy validators].psi_io_some_avg10 maxFor each healthy validator, the highest 10-s 'some' IO pressure reading the on-node sampler logged inside the merged mixed window; reported as the range over validators.

Recovery

Nodebench-validator-05
Killed2026-11-13 18:21:20 UTC
Restarted2026-11-13 18:26:20 UTC
Rejoined2026-11-13 18:42:58 UTC
Catch-up time975 s (16 min 15 s)
Failed batch reportsnot recorded

Chain TPS per 5-minute interval

Chain TPS per 5-minute interval 9 intervals from 17:55 to 18:36 UTC; average 6,756 TPS, highest 7,684 TPS. The merged mixed window ran from 17:55 to 18:36 UTC. TPS 0 2,000 4,000 6,000 8,000 17:55–18:00 UTC: 7,343 TPS, 9.4 generators active 18:00–18:05 UTC: 7,415 TPS, 10.0 generators active 18:05–18:10 UTC: 7,625 TPS, 10.0 generators active 18:10–18:15 UTC: 7,684 TPS, 10.0 generators active 18:15–18:20 UTC: 7,377 TPS, 10.0 generators active 18:20–18:25 UTC: 7,126 TPS, 10.0 generators active 18:25–18:30 UTC: 7,239 TPS, 10.0 generators active 18:30–18:35 UTC: 7,077 TPS, 10.0 generators active 18:35–18:36 UTC: 1,918 TPS, 3.8 generators active avg 6,756 17:55 UTC 18:05 UTC 18:15 UTC 18:25 UTC 18:35 UTC C1 2026-11-13 18:21:20 UTC: docker kill -s KILL telcoin on bench-validator-05 (systemctl stop holds it down (unit has Restart=on-failure). bench-blaster-05 targets ONLY this node, so its sends fail for the outage by design; the generator resends on connection errors) C1 C2 2026-11-13 18:26:20 UTC: systemctl start telcoin on bench-validator-05 (docker start is impossible: the unit runs docker run --rm) C2 C3 2026-11-13 18:31:20 UTC: systemctl restart telcoin on bench-observer-11 (observer restart) C3

Each bar is the average chain TPS over one 5-minute interval of the merged mixed window (17:55–18:36 UTC); fainter bars had fewer generators active than the maximum, the dashed line is the average of all bars and amber lines mark the chaos events listed below.

  • C1 2026-11-13 18:21:20 UTC: docker kill -s KILL telcoin on bench-validator-05 (systemctl stop holds it down (unit has Restart=on-failure). bench-blaster-05 targets ONLY this node, so its sends fail for the outage by design; the generator resends on connection errors)
  • C2 2026-11-13 18:26:20 UTC: systemctl start telcoin on bench-validator-05 (docker start is impossible: the unit runs docker run --rm)
  • C3 2026-11-13 18:31:20 UTC: systemctl restart telcoin on bench-observer-11 (observer restart)
5-minute data (9 rows)
start_utcend_utcgenerators_activechain_tps
2026-11-13T17:55:32Z2026-11-13T18:00:32Z9.47,342.9
2026-11-13T18:00:32Z2026-11-13T18:05:32Z10.07,414.9
2026-11-13T18:05:32Z2026-11-13T18:10:32Z10.07,625.4
2026-11-13T18:10:32Z2026-11-13T18:15:32Z10.07,683.8
2026-11-13T18:15:32Z2026-11-13T18:20:32Z10.07,377.3
2026-11-13T18:20:32Z2026-11-13T18:25:32Z10.07,126.0
2026-11-13T18:25:32Z2026-11-13T18:30:32Z10.07,239.4
2026-11-13T18:30:32Z2026-11-13T18:35:32Z10.07,077.1
2026-11-13T18:35:32Z2026-11-13T18:36:19Z3.81,918.4

The fine-grained chain throughput series (its step_s field gives the seconds per point) is published as series/tps-1s.json.

Other networks

MeasureTelcoin Network (this run)Tempo
Throughput7,329 tx/s (Avg chain TPS)16,455 TPS, shown as "Settled TPS" for the nightly run of 5 Oct 2026 (commit de5f5d3). perf.tempo.xyz describes it as "Median settled throughput across 130 measured blocks."
How throughput is measuredChain transactions per block-timestamp second, summed over the 2353-s window when every load generator was in its mixed phase and divided by its length.The methodology page defines TPS as "Transactions per second (derived from gas / avg tx gas)": gas used divided by the average gas of one transaction, reported as the median over the measured blocks.
Run length40.8 min merged mixed window; each generator was set to a 40 min mixed phase2m 39s for the 5 Oct 2026 nightly run (130 measured blocks at a 687 ms average block time), per perf.tempo.xyz.
Validators and placement10 validators and 1 observer across 10 regions: asia-east2 (Hong Kong), asia-northeast1 (Tokyo), asia-northeast3 (Seoul), asia-south1 (Mumbai), asia-southeast1 (Singapore), australia-southeast1 (Sydney), europe-west2 (London), europe-west4 (Netherlands), us-east4 (Ashburn), us-west2 (Los Angeles); load generators: one blaster per validator, in that validator's zone, sending only to itperf.tempo.xyz labels the scenario "Public mix · multi-region". The methodology page says "Benchmarks run two local validators in consensus on a dedicated bare-metal server with 100 GiB pre-bloated state." The public tempoxyz/tempo repo has nightly workflows for both kinds of setup: bench-e2e runs on one bare-metal runner, and bench-e2e-multi-region defaults to 10 validators on cloud VMs (its regions input reads "Empty uses the selected cloud's five defaults."). The run page does not say which setup produced this number.
Hardwarec3-highcpu-8, 8 vCPUs, 16 GB memory; disk: hyperdisk-balanced, 100 GB, 50,000 IOPS, 800 MiB/sThe methodology page lists a dedicated bare-metal server: "AMD EPYC 4585PX (16C/32T, 4.3/5.7 GHz)", "128 GB 3600 MHz" memory and "2×960 GB NVMe + 2×1.92 TB NVMe via schelk". The repo's multi-region workflow uses cloud VMs instead (c4d-standard-48-lssd on GCP, c6id.8xlarge on AWS).
Workloadbatched_self_send, contract_deploy, faucet_drip, native_transfer, stablecoin_approve, stablecoin_transfer, uniswap_v2_add_liquidity, uniswap_v2_remove_liquidity, uniswap_v2_swap, uniswap_v3_swap, uniswap_v4_swap, wtel_unwrap, wtel_wrap; 10 generators × 600 senders eachperf.tempo.xyz labels the scenario "Public mix"; the methodology page says "100% TIP-20 token transfers (ERC-20 equivalent)". The run used 1,000 accounts and 4 TIP-20 tokens.
StateFresh chain from genesis for every runThe methodology page says "100 GiB pre-bloated state".
Latency (as each source defines it)Time to land, median: 0.86 s; Time to land, p95: 3.53 s; Commit latency, p95: 0.99–1.30 sBlock time of 687 ms, which the methodology page defines as "Average wall-clock time between blocks". The run page gives no per-transaction latency. Separately, tempo.xyz says "Blocks finalize in ~0.6 seconds with no re-orgs." (a network claim, not a figure from this benchmark).
Source and fetch dateThis page; every figure names its source field.perf.tempo.xyz nightly run of 5 Oct 2026 (commit de5f5d3), the perf.tempo.xyz methodology page, tempo.xyz, and the benchmark workflows in the tempoxyz/tempo repo. All fetched on 5 Oct 2026.

Sources for Tempo (fetched 2026-10-05):

Tempo's setup may not match ours: its methodology page describes two validators on one bare-metal server, while our runs use ten validators across five regions, so read this table as context, not a ranking.

This table is context, not a ranking.

Caveats

  • Against r20261113-1351 only the disk changed. Throughput rose from 5,322.1 to 7,329.0 tx/s, the best 60 s from 7,327.4 to 8,256.4 tx/s, blocks a second from 23.70 to 27.72 and transactions landed in blast 2 from 14,620,952 to 19,484,670. Time to land p50/p95/p99 went from 0.68/14.66/61.91 s to 0.86/3.53/31.45 s, and time to seen p50/p95 from 12.99/30.89 s to 8.32/25.74 s. The mean time to seen fell 4.54 s: the engine queue gave 6.05 s, partly given back by the wait for the validator's own execution (+1.28 s) and the pool wait (+0.41 s). Against bench 5 (r20261107-1504) throughput is 112.5 % higher, but bench 5 was bound by the bytes its disk wrote and this run by its engines.
  • Generator 05 was silent for 38 % of the headline window (65 % in bench 5), and the other nine generators carried 7,203.1 tx/s. The rate held through the blast: 7,342.9 tx/s in the first five-minute bucket, 7,683.8 at its best, and 7,077.1 in the eighth, 8 % under the best.
  • The disk no longer binds. Over blast 2 the median validator, validator-05 aside, ran 24,837 device writes a second (0.497 of 50,000), 5.1 KiB each, 3.41 writes per transaction, 123 MiB/s on the mean against 800 (busiest second 260 MiB/s), and the device was busy 0.17 of the time against 0.39 at 20,000 IOPS. The database commit took 22.4 µs per transaction against 74.2. Device bytes per page byte were 1.156 to 1.170 on the validators and 1.062 on the observer.
  • The engine binds instead. Per executed transaction it spent 133.9 µs: execute 87.7, persist 44.5 (save 21.9, commit 22.4) and announce 0.75. Against 20,000 IOPS the task fell 53.3 µs per transaction, 51.8 of it in the commit, and outputs grew 50 %. Execute is now the larger part (639 of 975 busy ms a second): state root 42.4 µs, EVM 27.5, commit 22.4, save_blocks 21.9, finish 16.6. The fleet's engines were 975 ms busy each second with the queue at its cap in 42 % of samples (3 to 98 % by validator). Validator-01, the costliest engine at 136.9 µs, ran 997 ms a second busy with a flat backlog of 14 to 18 outputs, and its capacity, 7,287 tx/s, is the fleet's executed rate (7,277 to 7,293 on the others). Validator-06, the cheapest at 130.6 µs, idled 50 ms a second. What held the input at validator-01's capacity was not measured; the pool held 477 transactions on the median validator, so it was not the pool.
  • An estimate made before this run put the ceiling at this disk at 8,600 to 10,300 tx/s. It was built on the arm-walk run's execute cost alone and left out the persist, which runs in series in this build. With the persist included, the same execute costs give 6,161 to 6,969 tx/s and this run's own costs 7,469 tx/s at full busy, against 7,287 measured; with the persist free, 11,397 tx/s. These figures are arithmetic, not runs. The persist is busy about 324 ms of each second here, so pipelined persist has room at this disk, but it was off and is untested at 50,000 IOPS, and it lowers each engine's cost, which may widen or narrow the spread behind the certification loss.
  • Certification held through both light blasts, the warm-up and blast 2's first 120 s (99.97 % of the fleet's own headers in that window). Over the mixed phase the fleet certified 88.26 % (7,025 of 59,826 own headers lost): validator-06 44.5 % (its lowest five-minute bucket 18.7 %), validator-10 68.2 %, validator-02 78.6 %, validator-09 95.8 %, the other six 99.9 to 100 %. Before the kill validator-05 answered 'superseded' to 33,414 vote requests and voted 2,908 times, validator-01 20,507 and 16,153 times. The fleet refused 265 votes as execution-behind and answered 'superseded' 93,730 times, and the median round slowed from 353 to 374 ms.
  • The mechanism, step by step. The always-on form names its author's tip minus 8 executed outputs on 97 to 99 % of headers, because its inputs are the parents' earlier picks and the floor is the only way up. Validator-06 ran about 18 outputs ahead of validator-01 and 45 ahead of validator-05, and 7 to 9 ahead of validators 04 and 02 (rounds converted to outputs at two rounds per output, an inference), so the block a fast author named was one that four or five voters had not executed. A voter without the block waits up to 1 s; the author's next header comes about 360 ms later, so the voter answers 'superseded', and neither that nor 'behind' is a vote. With four voters out a header cannot certify (none of validator-06's 1,643 headers with five or fewer timely votes did); with exactly six it certified only when the six round trips fit inside the round: 99 % for validator-09 in Seoul, whose six are near, 39 % for validator-06 and 48 % for validator-10. The spread came from the engines: with the disk off its limit each ran at its own cost (130.6 to 136.9 µs per transaction), validators 01, 04 and 05 kept full queues while validator-06 drained to about one output, and validators 05 and 01 fell steadily behind. At 20,000 IOPS the disk had given every engine the same cost (187.0 to 187.3 µs). The round-trip step is inferred, because vote arrival is not logged.
  • Safety held: a vote is signed only after the voter finds the named block number and hash in its own chain, the subscriber checks each committed leader's block the same way, and no node logged an unexpected execution result, a fork or a halt. Recomputing the rule on this run's measured tips before the kill, with rounds and leaders held fixed, gives validator-06 40.5 % at the floor of 8 as run, 90.4 % with a floor of 16 and 100 % with a floor of 32 (validator-10: 52.2, 80.5 and 99.7 %), for a named block about 6, 12 or 24 s old; a quorum over fresh tips gives validator-06 60.0 %, and the switch off 2.7 %. This is arithmetic, not a result: a different floor would move the tips and rounds it holds fixed, and no run has tested one. An older named block costs detection time, not detection: a diverged node is caught up to N outputs later, about 24 s at N = 32.
  • 130,554 landings were seen 300 s or more after submission: 120,778 gap resends and 9,776 that generator 05's scanner saw late. 133,975 transactions were resent after the generators' gap check, by eight generators, and 14,928 never landed after the 180-s tail scan, 7,674 of them sent to validator-02 and 5,715 to validator-06; landed over accepted was 99.9232 %. The faster disk multiplied the always-on form's resends by 4.4 against r20261113-1351. The mechanism is the same in all four runs of this program and is tracked as issue #1807: when an own header is cancelled or committed late, its batches are queued again behind batches a newer own header already carried, so the sender's later transactions are skipped at execution as 'nonce too high', sit in no pool, and land only after the generator resends its window at its 300-s gap check. The pool switches sealed_tx_keep and digest_dedupe were off in all four runs; whether they change this cost was not measured in these runs (in the earlier arm-walk run, sealed_tx_keep with three other switches on left no gap resends, a reading confounded with the other switches and an epoch boundary).
  • Validator-05 rejoined once and held: 21 min 30.2 s from the kill to active and 16 min 30.1 s from its start, by the node's log. It replayed epoch 5 under load at 1.180 outputs a second while the chain made 1.243, so its deficit grew to 585 outputs until the load ended, by the same rejoin rule as in r20261113-1351. While it followed, its RPC refused every transaction; generator 05 dropped 314,010 sends and landed nothing for the last 899 s of the blast. In the five minutes before the kill generator 05 landed only 131 to 163 tx/s against the others' median of 639 to 917, following its validator's backlog. Validator-05 refused 17,586 votes as inactive and signed neither epoch record after its kill. Nineteen minutes before the kill, at the 18:02:19 close, it had been demoted twice for 4.05 s in all ('epoch_behind'), with one failed promotion. The observer followed the chain again 8.0 s after its restart.
  • All six epoch closes certified on one record hash and one closing block on every node. The close inside the outage (18:22:19) is the thinnest close of the arm-walk run, r20261113-1351 and this run: with validator-05 down and validators 01 and 02 still draining 15-output backlogs, the seventh vote went out at 11.30 s and no eighth came, with a commit gap of 10.31 s. Validator-01 was the late validator at every loaded close. 'failed to reach quorum on epoch close' fired once, on validator-04 72.79 s after the 18:02:19 close, on a record the network had certified. No test in this build (2df831e3c) crosses an epoch close with the always-on form.
  • No hard check fired: no halt, fork, unexpected execution result, fatal database error or panic in any node's whole main log. The nodes logged 73,535 warnings and 17,833 errors, driven by the outage (20,875 dial failures), validator-05 as a follower (16,912 certificate-manager warnings to the blast's end and 7,569 'processed certificate that is too new' errors after it) and the observer's sync retries. Against r20261113-1351, 'failed to get vote for header … Canceled' rose from 45 to 1,132 lines and QuorumRejected from 39 to 142, both tied to the certification loss. The node's stderr file was not pulled.
  • The pre-start fsync probe flagged validator-06 as slow; it was the cheapest engine, so the flag did not predict the slow node.
  • The adaptive form on this disk (r20261114-0000) certified more own headers (94.82 % against 88.26 %), but inside its starved windows its fast proposers named the same floor-8 block, it still lost certification, and its senders paid more. With one run per form, the form cannot be told apart from which validators lagged.
  • The teardown's provenance check warns 'node startup provenance mismatch' in all four runs: each node's start during staging ran before the arm file was applied. It is known and not a finding.
  • Not measured: when a vote reaches its author (not logged), what held the input at validator-01's capacity, why blast 1's execute cost (112 to 113 µs per transaction) exceeded blast 2's (88 to 90), what the rise of save_blocks is made of, a second cause of slower rounds within the blast (0.352 to 0.392 s with almost no rounds waiting for under-held headers), and a deeper floor in a real run.
  • Each configuration ran once (one run per form and disk), so run-to-run variation is not measured. The form of the vote fix is not decided: at 20,000 IOPS the always-on form certified a little more and cost the senders less, at 50,000 IOPS neither form held certification, and the floor's depth has been tested only as arithmetic. A further run of both forms is planned on a build of the newer upstream main branch, which uses a different execution storage engine (tn-lmdb) than these runs (MDBX), with a candidate fix for issue #1807 added; its result is not known.
  • Per-validator rows cover 9 validators; bench-validator-05 excluded as chaos target(s).
  • Recovery: demoted / rejoined are the first 15-s Prometheus samples showing the change, so they trail the node's own log times by up to a scrape interval plus one step.
  • Recovery: no tn_batch_builder_seal_failures_total count for bench-validator-05; failed_reports unavailable.
  • The harness working tree had uncommitted changes when the run was deployed.
  • Validators bench-validator-02, bench-validator-06 and bench-validator-10 certified fewer than 90 % of their own headers in blast 2 mixed and run; their logs are preserved under results/r20261113-1630/logs/ (see INDEX.md).
  • Before the coordinated start each node ran once during staging with every node switch at its default (11 start(s) on 11 node(s)); the run's switches were on from the coordinated start at 2026-11-13T16:50:28Z.

Full report

Full report HTML · 1.3 MB

Changes from the original report:

  • The site adds a banner at the top of the report that links back to this page.

SHA-256 of the committed file, before the banner is added: 939a0be4e97f9f6919eafecedf8e64cb1dcdad982d952b6a97463b62b95cfe26