The adaptive vote fix on Hyperdisk Balanced at 50,000 IOPS

2026-11-14 · started 00:00 UTC · ended 03:23 UTC · merged mixed window 40 min 39 s

telcoin-network 2df831e3c · bench/bench6-2026-11 · commit verified from node startup logs

24 node starts across 11 instances logged telcoin-network 0.1.0 (2df831e3ca34c06a361a7acca10397787e7b332e) The node image probe recorded the same commit before deploy Node logs: fork schedule: seed_signature, multi_workers, prevrandao, leader_seeded_ordering, and subsecond_timestamp active from genesis Each of the 24 node starts logged its node switches line, and each matched the switch state the manifest records for that start Commit 2df831e3c is the head of the branch bench/bench6-2026-11, built for this benchmark: a base commit 165e3fe01 plus 98 commits, in which every fix for bench 5's concerns is a node switch, off by default. The twelve switches are quorum_exec_block (the vote fix, whose form the setting quorum_exec_block_starved_rounds picks), vote_queue_fix, epoch_record_state_check, mdbx_random_advice, mdbx_no_writemap, sealed_tx_keep, digest_dedupe, recovery_threads, epoch_close_task_drain, pipelined_persist, rpc_admission_backlog_outputs and parent_grace. A node logs its switches at every start ('node switches') and exports them as tn_node_switch. The image was named by registry digest (adiri@sha256:874537ff…852745) and ran all four runs of this program unchanged.

We repeated r20261113-1630 with the vote fix in its adaptive form instead of the always-on form. In the adaptive form a header names the author's own executed tip until one of the author's headers is cancelled uncertified; the author then uses the always-on quorum rule, never more than 8 outputs behind its tip, inside a starved window of up to 128 rounds, and afterwards tries its own tip again. The build (2df831e3c), the other three switches, the fleet, the disk (Hyperdisk Balanced at 50,000 provisioned IOPS), the load and the chaos plan were the same. One deviation: this fleet idled 46 minutes between deployment and its first blast, which moved the epoch boundaries and the kill to other points of the load blast. Throughput and what binds did not change: average chain throughput with all ten generators running was 7,286.1 tx/s against 7,329.0 (-0.6 %), the engines bound both runs and the disks ran at about half their limit. The median time to seen was 6.88 s against 8.32 s, lower in 7 of 8 five-minute blocks; why was not measured. Neither form held certification at this disk. This run certified 94.82 % of own headers over the load blast against 88.26 %, and its lowest validator, validator-06, 84.1 % against 44.5 %. But inside their starved windows the fast proposers named the same floor-8 block as the always-on form, and 92 % of the losses before the kill sat on those headers. The gain came on headers whose author led the seventh validator by 9 outputs or more (75.4 % certified against 22.7 %), alongside rounds that waited for under-held headers (17.1 % of rounds against 0.46 %) and 6.8 % fewer rounds a second before the kill. One run per form cannot separate this from which validators lagged (validator-04 and then validator-02 here, validator-01 there), and the gap lies inside the always-on run's five-minute range by one measure and outside the spread of its two windows by another. The adaptive form also lowered three validators that held under the always-on form, and validator-02 left the active set for 4 minutes near the end of the load. The senders paid more: 255,453 gap resends against 133,975 and 20,768 unlanded transactions against 14,928, through the same reordering (issue #1807). This run is one of four in one program, one node build with two forms of the vote fix on two disks: the always-on form as r20261113-1351 (20,000 IOPS) and r20261113-1630 (50,000 IOPS), the adaptive form as r20261113-2118 (20,000 IOPS) and r20261114-0000 (50,000 IOPS). Across them the form changed neither throughput nor what binds; the always-on form certified slightly more own headers at 20,000 IOPS, neither form held certification at 50,000 IOPS (the adaptive run certified more), the adaptive form's senders paid more on both disks through the same reordering (issue #1807), and which form goes upstream is not decided.

Compared against r20261113-1630.

Key figures

7,286.1tx/s
Average chain throughput, all load generators running
all generators active
8,256.5tx/s
Best 60 seconds, all load generators running
all generators active
3.88s
Time to land, p95
whole run
1.88–1.92s
Commit latency, p95
merged mixed window

How it was run

Software

Upstream committelcoin-network 2df831e3ca34c06a361a7acca10397787e7b332e · bench/bench6-2026-11 · commit verified from node startup logs
Node imageus-docker.pkg.dev/telcoin-network/tn-public/adiri@sha256:874537ff1cd8d496140154837e7a36524978bdbf3b0440e48972b073d9852745
Node image digestsha256:874537ff1cd8d496140154837e7a36524978bdbf3b0440e48972b073d9852745
Node version0.1.0
Fork schedulesubsecond-from-genesis
Load generator commit62bbd5f
Load generator image digestsha256:7cefcc505201b2e40361a7d8cfe034778090b76c568cb2400369bf566c605a59
Analysis commit62bbd5f
Harness commitb71e82f72bc61a49a9c536539b11f59bc1478124
Harness submodulescommon: 1a6955d8a2c6f19e53cf7aaad5e32a9107a21a7d
tn-node-deployment: d9446f720026fa5ce8b64fce9074e56b05b3f4eb

Fleet

Fleetc3-highcpu-8 + Hyperdisk Balanced at 50,000 IOPS (c3-hdb), ten regions, validator-05 in Ashburn
Validators10
Observers1
Machine typec3-highcpu-8
vCPUs per node8
Memory per node16 GB
Diskhyperdisk-balanced, 100 GB, 50,000 IOPS, 800 MiB/s
Regionsasia-east2 (Hong Kong), asia-northeast1 (Tokyo), asia-northeast3 (Seoul), asia-south1 (Mumbai), asia-southeast1 (Singapore), australia-southeast1 (Sydney), europe-west2 (London), europe-west4 (Netherlands), us-east4 (Ashburn), us-west2 (Los Angeles) (10)
Load generators10 generators × e2-standard-2 × 600 senders each
Load placementone blaster per validator, in that validator's zone, sending only to it

Chain

Chain ID32286
Networkbench
Epoch length1,200 s (20 min)

Consensus parameters

Max header delay1,000 ms
Min header delay350 ms
Max batch delay250 ms
Vote timeout5,000 ms
Parameters fileconsensus-parameters.yaml
Confirmed bynode startup logs

Parameter arms

ArmFrom (UTC)To (UTC)Values
final-adaptive2026-11-14 00:19:46—all 11 nodes: epoch_record_state_check=true, mdbx_random_advice=true, quorum_exec_block=true, quorum_exec_block_starved_rounds=4, vote_queue_fix=true

Workload

Warm-up120 s (2 min)
Mixed phase2,400 s (40 min)
Cool-down180 s (3 min)
Transaction typesbatched_self_send, contract_deploy, faucet_drip, native_transfer, stablecoin_approve, stablecoin_transfer, uniswap_v2_add_liquidity, uniswap_v2_remove_liquidity, uniswap_v2_swap, uniswap_v3_swap, uniswap_v4_swap, wtel_unwrap, wtel_wrap

Chaos events

MarkerTime (UTC)NodeActionDetail
C12026-11-14 02:30:22bench-validator-05docker kill -s KILL telcoinsystemctl stop holds it down (unit has Restart=on-failure). bench-blaster-05 targets ONLY this node, so its sends fail for the outage by design; the generator resends on connection errors
C22026-11-14 02:35:22bench-validator-05systemctl start telcoindocker start is impossible: the unit runs docker run --rm
C32026-11-14 02:40:22bench-observer-11systemctl restart telcoinobserver restart

Steps

  1. All four runs of this program used one node image, built from the branch bench/bench6-2026-11 at commit 2df831e3c (a base commit 165e3fe01 plus 98 commits), with sub-second timestamps from genesis, and named by registry digest (adiri@sha256:874537ff…852745). Nothing was rebuilt between the runs. Every fix in that build is a node switch, off by default; a node logs its switches at every start ('node switches') and exports them as the metric tn_node_switch. The load generator was built from tn-transaction-generator commit 62bbd5f.
  2. The vote fix is the switch quorum_exec_block. Instead of its own newest executed block, a proposer names in each header a block that a quorum of validators should already have executed; a voter signs only after it finds the named block in its own chain. The setting quorum_exec_block_starved_rounds picks the form. With 0, the always-on form, every header takes the quorum value over the author's own executed tip and the blocks its parent headers named, capped at the own tip and never more than 8 outputs behind it (the floor); in practice nearly every header named the author's tip minus 8 outputs. With 4, the adaptive form, a header names the author's own tip until one of the author's headers is cancelled uncertified; the author then uses the always-on rule inside a starved window of up to 128 rounds, and afterwards tries its own tip again.
  3. Three more switches were on in all four runs: vote_queue_fix (the voter's lock wait), epoch_record_state_check (a check of the closing state when an epoch record is built) and mdbx_random_advice (the MDBX database's random-access hint over the whole file). The other eight stayed at their defaults: mdbx_no_writemap, sealed_tx_keep, digest_dedupe, recovery_threads, epoch_close_task_drain, pipelined_persist, rpc_admission_backlog_outputs and parent_grace. The switch set came from an arm file applied at deployment, before the nodes' coordinated start: no arm walk, no cache eviction and no restart before the load. Each node first ran once, alone, during staging (17 to 90 s, every switch at its default, no peer reached, no output executed); the switches of the run were on from the coordinated start, the first process that executed blocks.
  4. Each run created ten validators and one observer on c3-highcpu-8 (8 vCPU, 16 GiB), each node with one 100 GB Hyperdisk Balanced disk for the root and the data directory at 800 MiB/s, provisioned at 20,000 IOPS (bench 5's disk) or 50,000 IOPS. One validator ran in each of bench 5's ten regions (Los Angeles, London, Sydney, Tokyo, Ashburn, the Netherlands, Singapore, Hong Kong, Seoul and Mumbai), the observer in Los Angeles. Epochs last 1,200 s.
  5. Each run had three blasts. Ten generators on e2-standard-2, one per validator in that validator's zone and sending only to it, ran blast 0, the floor, with 8 senders each and up to 16 transactions in flight per sender (8 x 16); blast 1 with 600 senders and a window of 4 (600 x 4, at most 24,000 in flight); and blast 2, the load blast, with 600 senders and a window of 16 (600 x 16, at most 96,000 in flight) and phases of 120 s warm-up, 2,400 s mixed load and 180 s cool-down, the shape of bench 5's load blast.
  6. The chaos plan was bench 5's, inside blast 2: validator-05's container was killed with SIGKILL shortly before an epoch boundary and held down for 300 s, so the outage crossed the boundary; then it was started, and the observer was restarted 300 s after that. Generator 05 sends only to validator-05 and does not fail over to another endpoint, so it is cut off for the outage and for as long as its validator follows the chain afterwards.
  7. Headline figures keep bench 5's definitions and code; bench 5's throughput, latency and landing figures quoted on these pages were recomputed exactly from bench 5's own generator documents with the same code, while its certification, disk, stage and rejoin figures come from bench 5's report or from this program's reading of bench 5's logs and model. Throughput is the chain's transactions per second over the window in which all ten generators ran, with the outage and generator 05 inside it. Time to seen runs from submission until the generator saw the receipt, from the generators' summed histograms (the bundle method); time to land is to the block's timestamp, in whole seconds, so about 0.5 s low. Own-header certification is the share of a validator's own headers that were certified, over blast 2's mixed phase unless another window is named.
  8. The analysis read every node's whole main log (the node's stderr file was not pulled) and the node metrics exported from Prometheus before each teardown. The comparison of the two forms ran the same scripts on both runs of a disk and reproduced the always-on runs' figures.
  9. Against r20261113-1630 the form of the vote fix changed: the arm file final-adaptive set quorum_exec_block_starved_rounds to 4 instead of 0, with the same other switches. The image, the generator, the fleet, the disk, the load and the chaos plan are the same.
  10. One deviation: the fleet idled 46 minutes between the end of its deployment (00:45:38) and the launch of its blast segment (01:32:05), an operational pause; r20261113-1630 went from deployment to blast in about one minute. The fleet was checked healthy at 01:31 (11 of 11 nodes, no fatal database line). The idle made the chain about 46 minutes longer, with three more idle epoch closes (nine against six) and epoch numbers two higher. The kill came 8 min 39 s earlier in blast 2's mixed phase (1,029 s in against 1,548 s); epoch 8's boundary fell 2.5 minutes before the mixed end, where r20261113-1630 had no boundary after 1,607 s in; and the database ended 0.08 GiB larger, grown during the blasts, not the idle. Whether any other difference comes from the idle cannot be told from one run.
  11. Times are UTC on 14 November. Deployment ran from 00:00:29 to 00:45:38, on the first try; all eleven nodes started at one due time, with 0 s spread. The fsync probe median was 1,046 µs, with validator-04 flagged slow. The mixed phases ran 01:41:59-01:48:20 (blast 0), 01:54:26-02:01:42 (blast 1) and 02:13:13-02:53:52 (blast 2). Validator-05 was killed at 02:30:22 and started at 02:35:22, epoch 7 opened during its outage at 02:31:21, and the observer was restarted at 02:40:22.
  12. The teardown ended with rc 0 at 03:24:37, 12 instances deleted and none left. The archive holds 627 files, 3.12 GB; the metric export (00:15:29-03:08:44) holds 382 metrics (21,010 series), with the vote-stage series on all eleven nodes. Before the kill the mixed phase ran 1,029 s here against 1,548 s in r20261113-1630, so only rates and shares are compared across the kill.

The harness scripts live in a private repository; the steps above describe what they do.

Results

MetricValueWindowSourceMethod
Average chain throughput, all load generators running7,286.1 tx/sall generators activecombined.series.per_second[chain_tx]Chain transactions per block-timestamp second, summed over the 2361-s window when every load generator was in its mixed phase and divided by its length.
Best 60 seconds, all load generators running8,256.5 tx/sall generators activecombined.series.per_second[chain_tx]Highest mean chain tx/s over any 60 consecutive seconds inside the all-generators window (aggregate.py best_window_mean).
Transactions landed over the whole blast19,240,775 txwhole runcombined.totals.landed_chainSum of tx_count over every block the load generators saw during the blast, own and foreign transactions alike.
Blocks per second28.05 blocks/smerged mixed windowcombined.blocks.rows[timestamp]Chain blocks whose timestamp falls in the merged mixed window, divided by its length (aggregate.py landed.capacity.blocks_per_sec over the same window).
Time to land, median0.99 swhole runcombined.latency.ts_ms.p50Block timestamp minus submit time over every landed own transaction in the blast, from the load generators' merged latency histograms (aggregate.py merge_latency), in seconds.
Time to land, p953.88 swhole runcombined.latency.ts_ms.p95Block timestamp minus submit time over every landed own transaction in the blast, from the load generators' merged latency histograms (aggregate.py merge_latency), in seconds.
Time to land, median, whole-second bias removed1.49 swhole runcombined.latency.ts_ms.p50 + 0.5 sTime to land, median, plus 0.5 s. The block timestamp is the committing leader's header creation time cut to the whole second, so every sample reads 0 to 999 ms low, 0.5 s on average.
Time to seen, median6.88 swhole runcombined.latency.seen_ms.p50Submit time to the moment the load generator's block scanner had the block holding the transaction (block and receipts fetched from the generator's own node), over every landed own transaction in the blast, from the load generators' merged latency histograms (aggregate.py merge_latency), in seconds.
Time to seen, p9523.15 swhole runcombined.latency.seen_ms.p95Submit time to the moment the load generator's block scanner had the block holding the transaction (block and receipts fetched from the generator's own node), over every landed own transaction in the blast, from the load generators' merged latency histograms (aggregate.py merge_latency), in seconds.
Time to land at light load, median0.25 slight load, 8 senders per generatorblast-0-20261114T015107Z/blasters latency.ts_ms.p50Block timestamp minus submit time over every landed own transaction submitted in the mixed phase of the light-load blast light8 (index 0: 8 senders per load generator at window 16, chaos off), pooled from the generators' latency histograms with the aggregate.py merge_latency method, in seconds; not bias-corrected.
Time to seen at light load, median1.64 slight load, 8 senders per generatorblast-0-20261114T015107Z/blasters latency.seen_ms.p50Submit time to the generator's block scanner having the block over every landed own transaction submitted in the mixed phase of the light-load blast light8 (index 0: 8 senders per load generator at window 16, chaos off), pooled from the generators' latency histograms with the aggregate.py merge_latency method, in seconds; not bias-corrected.
Commit latency, p951.88–1.92 smerged mixed windowmetrics.prometheus.series.tn_primary_commit_latency_seconds.by_node[healthy validators].p95For each healthy validator, the nearest-rank p95 over the merged mixed window of its 15-s histogram_quantile(0.95) commit-latency series; reported as the range over validators.
Execution time per consensus output0.82–0.87 smerged mixed windowmetrics.prometheus.series.tn_engine_execution_duration_seconds.by_node[healthy validators].meanFor each healthy validator, the nearest-rank median over the merged mixed window of its 15-s mean execution-duration series; reported as the range over validators.
Write throughput, p95, busiest validator127.2 MiB/smerged mixed windowmetrics.gcp[bench-validator-08].disk_write_bytes_per_sec p95Nearest-rank p95 of the 60-s mean write-byte rate over the merged mixed window for each healthy validator; the highest (bench-validator-08) is reported.
Write operations, p95, busiest validator26,414 IOPSmerged mixed windowmetrics.gcp[bench-validator-08].disk_write_ops_per_sec p95Nearest-rank p95 of the 60-s mean write-operation rate over the merged mixed window for each healthy validator; the highest (bench-validator-08) is reported.
CPU, p95, busiest validator43.3 %merged mixed windowmetrics.gcp[bench-validator-03].cpu_utilization p95Nearest-rank p95 of the 60-s CPU utilisation over the merged mixed window for each healthy validator; the highest (bench-validator-03) is reported.
Node resident memory, p95, busiest validator6.49 GiBmerged mixed windowmetrics.gcp[bench-validator-06].telcoin_rss_bytes p95Nearest-rank p95 of the 60-s telcoin process resident memory over the merged mixed window for each healthy validator; the highest (bench-validator-06) is reported.
IO pressure, highest reading, range over validators2.6–4.0 %merged mixed windowsampler[healthy validators].psi_io_some_avg10 maxFor each healthy validator, the highest 10-s 'some' IO pressure reading the on-node sampler logged inside the merged mixed window; reported as the range over validators.

Recovery

Nodebench-validator-05
Killed2026-11-14 02:30:22 UTC
Restarted2026-11-14 02:35:22 UTC
Rejoined2026-11-14 03:00:35 UTC
Catch-up time1,485 s (24 min 45 s)
Failed batch reportsnot recorded

Chain TPS per 5-minute interval

Chain TPS per 5-minute interval 9 intervals from 02:13 to 02:53 UTC; average 6,591 TPS, highest 7,558 TPS. The merged mixed window ran from 02:13 to 02:53 UTC. TPS 0 2,000 4,000 6,000 8,000 02:13–02:18 UTC: 7,558 TPS, 9.6 generators active 02:18–02:23 UTC: 7,309 TPS, 10.0 generators active 02:23–02:28 UTC: 7,480 TPS, 10.0 generators active 02:28–02:33 UTC: 7,343 TPS, 10.0 generators active 02:33–02:38 UTC: 7,269 TPS, 10.0 generators active 02:38–02:43 UTC: 7,395 TPS, 10.0 generators active 02:43–02:48 UTC: 7,254 TPS, 10.0 generators active 02:48–02:53 UTC: 6,877 TPS, 10.0 generators active 02:53–02:53 UTC: 833 TPS, 3.0 generators active avg 6,591 02:13 UTC 02:23 UTC 02:33 UTC 02:43 UTC 02:53 UTC C1 2026-11-14 02:30:22 UTC: docker kill -s KILL telcoin on bench-validator-05 (systemctl stop holds it down (unit has Restart=on-failure). bench-blaster-05 targets ONLY this node, so its sends fail for the outage by design; the generator resends on connection errors) C1 C2 2026-11-14 02:35:22 UTC: systemctl start telcoin on bench-validator-05 (docker start is impossible: the unit runs docker run --rm) C2 C3 2026-11-14 02:40:22 UTC: systemctl restart telcoin on bench-observer-11 (observer restart) C3

Each bar is the average chain TPS over one 5-minute interval of the merged mixed window (02:13–02:53 UTC); fainter bars had fewer generators active than the maximum, the dashed line is the average of all bars and amber lines mark the chaos events listed below.

  • C1 2026-11-14 02:30:22 UTC: docker kill -s KILL telcoin on bench-validator-05 (systemctl stop holds it down (unit has Restart=on-failure). bench-blaster-05 targets ONLY this node, so its sends fail for the outage by design; the generator resends on connection errors)
  • C2 2026-11-14 02:35:22 UTC: systemctl start telcoin on bench-validator-05 (docker start is impossible: the unit runs docker run --rm)
  • C3 2026-11-14 02:40:22 UTC: systemctl restart telcoin on bench-observer-11 (observer restart)
5-minute data (9 rows)
start_utcend_utcgenerators_activechain_tps
2026-11-14T02:13:13Z2026-11-14T02:18:13Z9.67,557.9
2026-11-14T02:18:13Z2026-11-14T02:23:13Z10.07,309.0
2026-11-14T02:23:13Z2026-11-14T02:28:13Z10.07,480.3
2026-11-14T02:28:13Z2026-11-14T02:33:13Z10.07,343.4
2026-11-14T02:33:13Z2026-11-14T02:38:13Z10.07,268.9
2026-11-14T02:38:13Z2026-11-14T02:43:13Z10.07,394.6
2026-11-14T02:43:13Z2026-11-14T02:48:13Z10.07,254.3
2026-11-14T02:48:13Z2026-11-14T02:53:13Z10.06,876.8
2026-11-14T02:53:13Z2026-11-14T02:53:52Z3.0833.2

The fine-grained chain throughput series (its step_s field gives the seconds per point) is published as series/tps-1s.json.

Other networks

MeasureTelcoin Network (this run)Tempo
Throughput7,286.1 tx/s (Avg chain TPS)16,455 TPS, shown as "Settled TPS" for the nightly run of 5 Oct 2026 (commit de5f5d3). perf.tempo.xyz describes it as "Median settled throughput across 130 measured blocks."
How throughput is measuredChain transactions per block-timestamp second, summed over the 2361-s window when every load generator was in its mixed phase and divided by its length.The methodology page defines TPS as "Transactions per second (derived from gas / avg tx gas)": gas used divided by the average gas of one transaction, reported as the median over the measured blocks.
Run length40.6 min merged mixed window; each generator was set to a 40 min mixed phase2m 39s for the 5 Oct 2026 nightly run (130 measured blocks at a 687 ms average block time), per perf.tempo.xyz.
Validators and placement10 validators and 1 observer across 10 regions: asia-east2 (Hong Kong), asia-northeast1 (Tokyo), asia-northeast3 (Seoul), asia-south1 (Mumbai), asia-southeast1 (Singapore), australia-southeast1 (Sydney), europe-west2 (London), europe-west4 (Netherlands), us-east4 (Ashburn), us-west2 (Los Angeles); load generators: one blaster per validator, in that validator's zone, sending only to itperf.tempo.xyz labels the scenario "Public mix · multi-region". The methodology page says "Benchmarks run two local validators in consensus on a dedicated bare-metal server with 100 GiB pre-bloated state." The public tempoxyz/tempo repo has nightly workflows for both kinds of setup: bench-e2e runs on one bare-metal runner, and bench-e2e-multi-region defaults to 10 validators on cloud VMs (its regions input reads "Empty uses the selected cloud's five defaults."). The run page does not say which setup produced this number.
Hardwarec3-highcpu-8, 8 vCPUs, 16 GB memory; disk: hyperdisk-balanced, 100 GB, 50,000 IOPS, 800 MiB/sThe methodology page lists a dedicated bare-metal server: "AMD EPYC 4585PX (16C/32T, 4.3/5.7 GHz)", "128 GB 3600 MHz" memory and "2×960 GB NVMe + 2×1.92 TB NVMe via schelk". The repo's multi-region workflow uses cloud VMs instead (c4d-standard-48-lssd on GCP, c6id.8xlarge on AWS).
Workloadbatched_self_send, contract_deploy, faucet_drip, native_transfer, stablecoin_approve, stablecoin_transfer, uniswap_v2_add_liquidity, uniswap_v2_remove_liquidity, uniswap_v2_swap, uniswap_v3_swap, uniswap_v4_swap, wtel_unwrap, wtel_wrap; 10 generators × 600 senders eachperf.tempo.xyz labels the scenario "Public mix"; the methodology page says "100% TIP-20 token transfers (ERC-20 equivalent)". The run used 1,000 accounts and 4 TIP-20 tokens.
StateFresh chain from genesis for every runThe methodology page says "100 GiB pre-bloated state".
Latency (as each source defines it)Time to land, median: 0.99 s; Time to land, p95: 3.88 s; Commit latency, p95: 1.88–1.92 sBlock time of 687 ms, which the methodology page defines as "Average wall-clock time between blocks". The run page gives no per-transaction latency. Separately, tempo.xyz says "Blocks finalize in ~0.6 seconds with no re-orgs." (a network claim, not a figure from this benchmark).
Source and fetch dateThis page; every figure names its source field.perf.tempo.xyz nightly run of 5 Oct 2026 (commit de5f5d3), the perf.tempo.xyz methodology page, tempo.xyz, and the benchmark workflows in the tempoxyz/tempo repo. All fetched on 5 Oct 2026.

Sources for Tempo (fetched 2026-10-05):

Tempo's setup may not match ours: its methodology page describes two validators on one bare-metal server, while our runs use ten validators across five regions, so read this table as context, not a ranking.

This table is context, not a ranking.

Caveats

  • Against r20261113-1630 (ranges are that run's own variation): blast 2 throughput 7,286.1 against 7,329.0 tx/s (-0.6 %; its five-minute blocks 7,077.1 to 7,683.8) and 7,134.0 against 7,203.1 tx/s without generator 05; blast 1 (600 x 4) 6,311.7 against 6,508.3 tx/s over the window in which all ten generators sent (-3.0 %), but 6,549.9 against 6,542.5 tx/s summing each generator's own rate over its own mixed phase; blast 0 701.2 against 704.5 tx/s; time to seen p50/p95 6.88/23.15 s against 8.32/25.74 s; time to land p95/p99 3.88/334.92 s against 3.53/31.45 s, the p99 following the landings 300 s or more late; executed rate per node 7,210 against 7,287 tx/s; engine task 133.8 against 133.9 µs per transaction; device writes 0.480 against 0.497 of 50,000 a second; transactions per output 6,234 against 5,781 (+7.8 %). At matched output size this run's engine was 0.4 to 1.8 µs per transaction dearer from 3,000 transactions per output up; the cause was not measured, and the longer chain is an untested candidate.
  • At this disk the adaptive form does not remove the floor problem of r20261113-1630. Six or seven proposers were inside a starved window for 996 of the 1,029 seconds before the kill, and there they named a block 7.94 to 7.97 outputs behind their tip, the always-on floor; 92 % of the pre-kill losses sit inside windows on those headers, and the floor-8 rule on this run's tips predicts 93.2 % against the 92.8 % measured before the kill (arithmetic). The three slowest validators (01, 04 and 05) named their own tip on 99.9 to 100 % of their headers and certified 100 %; under the always-on form the same three certified 99.9 to 100 % at the floor, so the own-tip naming gained nothing over the floor here. On the measured tips before the kill, with tips and rounds held fixed, a floor of 16 outputs gives 100.0 % on this run's tips and 97.0 % on r20261113-1630's, and a floor of 32 gives 100.0 and 99.9 %, at the price of a named block 16 or 32 outputs old on every header that uses the rule. No run has tested a deeper floor, and a different floor would move the tips and rounds the arithmetic holds fixed.
  • Over blast 2's mixed phase the fleet certified 94.82 % (2,807 of 54,226 own headers lost) against 88.26 % (7,025 of 59,826), higher in 7 of 8 five-minute buckets; the teardown's own check found validators 06 and 10 at 84.1 and 85.7 %. Before the kill validators 06, 10 and 02 certified 69.2, 72.0 and 99.4 % against 40.5, 52.3 and 84.1 %. But the adaptive form also lowered three validators that held under the always-on form: validators 08, 03 and 07 at 95.9, 98.5 and 98.5 % against 100 %, outside that run's range. In blast 2's first 120 s the fleet certified 98.02 % against 99.97 %, with seven validators under 99 % against none. Votes refused as execution-behind numbered 1,180 against 265, and there were six vote timeouts, all on validator-04 at the kill and the epoch-7 open (02:30:22-02:32:22), not traced.
  • The gain sits on headers past the floor: with a lead over the seventh validator of 9 outputs or more, 75.4 % certified against 22.7 %, and at the same lead this run's headers lived longer before the author's next header. Rounds with 4 or more under-held headers (held by at most 5 other voters at the proposal) were 17.1 % of rounds against 0.46 %, and 76.5 to 81.3 % of those headers certified against 8.9 to 32.7 %. In this build a round's next header needs 7 parent certificates (read from the code); that this wait is why this run certified more is one analysis's reading. Rounds were 6.8 % fewer a second before the kill, a tail rather than a shift (p50 +4 ms, p99 +497 ms), and outputs were 6 to 8 % larger; output size accounts for 1.4 to 1.8 points of the 5.77-point gap before the kill (arithmetic).
  • Whether the gain belongs to the form is not settled. Which validators lagged differs between the runs (validator-04 and then validator-02 here, validator-01 in r20261113-1630), and one run per form cannot separate that from the form. The analyses disagree on the noise: the 6.4-point gap between the five-minute bucket means lies inside the always-on run's own 10.6-point bucket range, but outside the spread of its two windows (87.0 and 87.3 %). They also disagree on why the always-on run's fast validators led by 10 or more outputs more often: one infers each node's engine rate and backlog; the other measured the authors' engines at 7,400 to 7,460 tx/s in both runs and puts it in the fleet's spread and the rounds that wait for under-held headers.
  • Blast 2: gap resends 255,453 against 133,975, from all ten generators against eight; 'nonce too high' skips 132,743 against 69,647; validator-minutes with an exposed lost own header 226 against 81; transactions unlanded after the 180-s tail scan 20,768 against 14,928; landings seen 300 s or more after submission 259,414 against 130,554; landed over accepted 99.8591 % against 99.9232 %. The adaptive form had 1.96 times the resends (generator 05 left out), 1.91 times the skips and 1.4 times the unlanded. Blast 1 had 3,452 gap resends and 3,361 unlanded against 0 and 1,410. Generator 02 also retried 24,205 sends that validator-02 refused while inactive; they are in none of these counts. The mechanism is the same in all four runs of this program and is tracked as issue #1807: when an own header is cancelled or committed late, its batches are queued again behind batches a newer own header already carried, so the sender's later transactions are skipped at execution as 'nonce too high', sit in no pool, and land only after the generator resends its window at its 300-s gap check. The pool switches sealed_tx_keep and digest_dedupe were off in all four runs; whether they change this cost was not measured in these runs (in the earlier arm-walk run, sealed_tx_keep with three other switches on left no gap resends, a reading confounded with the other switches and an epoch boundary). Every gap-resend second of generators 01-04 and 06-10 fell 280 to 345 s after a lost own header of the generator's own validator (385 of 385 seconds here, 175 of 177 in r20261113-1630). At this disk the adaptive form lost fewer headers with overtaken own transactions (858 against 2,202) but on more validators in more minutes; 69 % of the difference comes from validators that barely lost a header under the always-on form, and re-queued batches executed 5.5 to 14 s after the loss here against up to 172.5 s there. That this breadth sets the totals is an inference. The whole difference in the senders' cost between the forms sits in that reordering loss, a mechanism both forms share.
  • Validator-02 was demoted ('epoch_behind') and sat inactive for 4 min 00.0 s (02:51:51.7-02:55:51.7), 30 s after epoch 8 opened; it proposed no header from 02:51:22 to 02:53:52, and its last epoch-7 outputs reached its engine 118 s late. No validator other than validator-05 left the active set in the other three runs. It happened only under the adaptive form, and validator-02 drew most of this run's refused votes as a voter (447). But validator-02 had the dearest engine of the nine (137.8 µs per transaction; validator-01, r20261113-1630's dearest, 136.9), the spread grew through the blast in both runs as the dearest engine fell behind, and the idle moved this run's last loaded boundary to 2,288 s into the mixed phase, where validator-02 was 36.92 to 56.02 outputs behind; r20261113-1630's last one (1,607 s in) met its slowest validator 19.92 to 28.69 outputs behind. Under the always-on form validator-05 was demoted twice at one close. No analysis tested a boundary at the other run's point, so the data does not tie the demotion to the form.
  • Validator-05 took 30 min 00.0 s from the kill to active against 21 min 30.2 s, longer by the schedule, because the kill came 8 min 39 s earlier in the mixed phase; from the mixed phase's end to its promotion it took 6 min 30.2 s against 6 min 31.2 s. Restarted, it refused nearly every vote request while it caught up, under both forms. All nine epoch closes, three of them idle, certified on one record hash and one closing block on all eleven nodes. At the kill close, the one close at the same place in both runs, 8 of 9 voters signed against 7, the first quorum came at 9.87 s against 12.82 s, the commit gap was 8.73 s against 10.31 s and the highest execution lag 36 rounds against 62. 'failed to reach quorum on epoch close' fired once, on validator-04, on a certified record, as in r20261113-1630. The observer followed again 5.1 s after its restart (8.0 s). No node logged a halt, fork, unexpected execution result, fatal database text or panic.
  • In blast 1 generators 02, 06 and 10 started their mixed phase 76, 29 and 33 s late, after losing setup transactions in the preamble; there was no late start in the floor blast. Late starts like these appeared in both adaptive runs and in neither always-on run. No analysis traced the cause, and the analyses disagree on whether the preamble losses are the same kind of loss as the gap resends.
  • A forecast from r20261113-1630's data, made in an earlier pass of the analysis, that the adaptive form would name what the always-on form names and lose what it lost there (40.5 % for validator-06 before the kill), held for the naming but not for the loss: validator-06 certified 69.1 to 69.2 % before the kill by two counts, and the floor-8 arithmetic on this run's own tips matched its fleet.
  • The teardown's provenance check warns 'node startup provenance mismatch' in all four runs: each node's start during staging ran before the arm file was applied. It is known and not a finding.
  • At 50,000 IOPS the adaptive form is ahead on certification and behind on the senders' cost, and its own-tip naming on the three slowest validators gained nothing over the floor. Whether to repeat this pair of runs is open, since one run per form cannot tell the form from which validators lag.
  • Not measured: run-to-run variance at this disk, when a vote reaches its author (not logged), what the age of the named block costs, a floor of 16 or 32 in a real run, the senders' cost with sealed_tx_keep or digest_dedupe on, the outage window per header, and why this run's seen p50 was lower. The probe's parameters (the 128-round bound and the 64-header streak) were not varied.
  • Each configuration ran once (one run per form and disk), so run-to-run variation is not measured. The form of the vote fix is not decided: at 20,000 IOPS the always-on form certified a little more and cost the senders less, at 50,000 IOPS neither form held certification, and the floor's depth has been tested only as arithmetic. A further run of both forms is planned on a build of the newer upstream main branch, which uses a different execution storage engine (tn-lmdb) than these runs (MDBX), with a candidate fix for issue #1807 added; its result is not known.
  • Per-validator rows cover 9 validators; bench-validator-05 excluded as chaos target(s).
  • Commit_latency_p95_s: no tn_primary_commit_latency_seconds.p95 points for bench-validator-04, bench-validator-06.
  • Execution_time_per_output_s: no tn_engine_execution_duration_seconds.mean points for bench-validator-04.
  • Recovery: demoted / rejoined are the first 15-s Prometheus samples showing the change, so they trail the node's own log times by up to a scrape interval plus one step.
  • Recovery: no tn_batch_builder_seal_failures_total count for bench-validator-05; failed_reports unavailable.
  • The harness working tree had uncommitted changes when the run was deployed.
  • Validators bench-validator-06 and bench-validator-10 certified fewer than 90 % of their own headers in blast 2 mixed; their logs are preserved under results/r20261114-0000/logs/ (see INDEX.md).
  • Before the coordinated start each node ran once during staging with every node switch at its default (11 start(s) on 11 node(s)); the run's switches were on from the coordinated start at 2026-11-14T00:19:46Z.

Full report

Full report HTML · 1.4 MB

Changes from the original report:

  • The site adds a banner at the top of the report that links back to this page.

SHA-256 of the committed file, before the banner is added: 98581e407760e9a8a4c6b540daaf82224ce7c959c0ccd7a0617e26ab3158ac23