The always-on vote fix and the order hold on tn-lmdb at 20,000 IOPS

2026-11-15 · started 11:31 UTC · ended 14:41 UTC · merged mixed window 40 min 25 s

telcoin-network 91a15e223 · bench/bench7-2026-11 · commit verified from node startup logs

24 node starts across 11 instances logged telcoin-network 0.1.0 (91a15e22376fa1277c107824e738dce689792c56) The node image probe recorded the same commit before deploy Node logs: fork schedule: seed_signature, multi_workers, prevrandao, leader_seeded_ordering, and subsecond_timestamp active from genesis Each of the 24 node starts logged its node switches line, and each matched the switch state the manifest records for that start Commit 91a15e223 is the head of the branch bench/bench7-2026-11, built for this benchmark: upstream main 45dce8947 plus 114 commits. Main at 45dce8947 runs the tn-lmdb storage engine with storage v2 (#1834) and carries the upstream rework of the vote handler. The arm of both bench 7 runs set seven node switches: quorum_exec_block (the vote fix) and quorum_exec_block_starved_rounds (which picks its form), vote_queue_fix (the voter's lock wait), digest_order_hold (the order hold, for issue #1807) and the three epoch-record switches epoch_record_collect_past_uncertified, epoch_record_conflict_halt and epoch_record_recertify. The closing-block check has no switch. The image was named by registry digest (adiri@sha256:15acdccafff7…68822c) and ran both runs unchanged.

We ran the always-on vote fix again on bench 6's fleet and disk with a new node build. The zones, the machines (ten c3-highcpu-8 validators in ten regions and an observer), the disk (Hyperdisk Balanced at 20,000 provisioned IOPS), the generator and the consensus parameters are those of bench 6's always-on run, r20261113-1351. The build is upstream main 45dce8947 plus 114 commits. Main brought a new execution storage engine, tn-lmdb with storage v2 (#1834), and an upstream rework of the vote handler; the commits carry the vote fix, the voter's lock-wait fix, an order hold for issue #1807, three epoch-record switches and a closing-block check. In the load blast, with all ten generators running and validator-05 killed by design inside the window, average chain throughput was 9,288.6 tx/s against 5,322.1 (+74.5 %), and the median time until a generator saw its receipt was 9.40 s against 12.99 s. Own-header certification fell: 94.0 % for the fleet before the kill against 99.94 %, and 83.6 % for the lowest validator over the load blast against 99.7 %. Outside generator 05 the load blast had no gap resend, no unlanded transaction and no 'nonce too high' skip, against 29,449, 0 and 13,036 on bench 6. Own headers carrying batches were still lost, on all nine validators other than 05, and no resend followed them. The hold's own count of out-of-order re-queues read 0, but it has no bench 6 twin, so this run is consistent with the hold doing its job and does not prove it. The execution engine and the disk both sat below their limits, the engine nearer to its own (busy 978.1 ms a second; the disk at 0.859 of its write cap). All 8 epoch closes certified on one record hash and one closing block, and no node logged a fail-stop line. Validator-09's Prometheus series are partial for the whole run (the methodology says which). A first attempt at the adaptive run was lost to an expired cloud login and is not published. The adaptive form ran on the same build and disk as r20261115-1653: on the load blast's throughput, fleet certification and round rate the two forms are not told apart from this run's own spread. Against r20261113-1351 the storage engine, the vote handler, the order hold, the epoch-record switches and the run's shape changed together and no counter separates them, so neither the rise in throughput nor the fall in certification is put down to one change; each form ran once.

Compared against r20261113-1351.

Key figures

9,288.6tx/s
Average chain throughput, all load generators running
all generators active
10,968.7tx/s
Best 60 seconds, all load generators running
all generators active
7.76s
Time to land, p95
whole run
2.09–2.23s
Commit latency, p95
merged mixed window

How it was run

Software

Upstream committelcoin-network 91a15e22376fa1277c107824e738dce689792c56 · bench/bench7-2026-11 · commit verified from node startup logs
Node imageus-docker.pkg.dev/telcoin-network/tn-public/adiri@sha256:15acdccafff786c2aa60ac69cddc3b09069648d5cf22ae1727c060fafa68822c
Node image digestsha256:15acdccafff786c2aa60ac69cddc3b09069648d5cf22ae1727c060fafa68822c
Node version0.1.0
Fork schedulesubsecond-from-genesis
Load generator commit62bbd5f
Load generator image digestsha256:7cefcc505201b2e40361a7d8cfe034778090b76c568cb2400369bf566c605a59
Analysis commit62bbd5f
Harness commitd8483104c39c3412c29ffaeeded55067e85d0dc4
Harness submodulescommon: 1a6955d8a2c6f19e53cf7aaad5e32a9107a21a7d
tn-node-deployment: d9446f720026fa5ce8b64fce9074e56b05b3f4eb

Fleet

Fleetc3-highcpu-8 + Hyperdisk Balanced at 20,000 IOPS (c3-hdb), ten regions, validator-05 in Ashburn
Validators10
Observers1
Machine typec3-highcpu-8
vCPUs per node8
Memory per node16 GB
Diskhyperdisk-balanced, 100 GB, 20,000 IOPS, 800 MiB/s
Regionsasia-east2 (Hong Kong), asia-northeast1 (Tokyo), asia-northeast3 (Seoul), asia-south1 (Mumbai), asia-southeast1 (Singapore), australia-southeast1 (Sydney), europe-west2 (London), europe-west4 (Netherlands), us-east4 (Ashburn), us-west2 (Los Angeles) (10)
Load generators10 generators × e2-standard-2 × 600 senders each
Load placementone blaster per validator, in that validator's zone, sending only to it

Chain

Chain ID32286
Networkbench
Epoch length1,200 s (20 min)

Consensus parameters

Max header delay1,000 ms
Min header delay350 ms
Max batch delay250 ms
Vote timeout5,000 ms
Parameters fileconsensus-parameters.yaml
Confirmed bynode startup logs

Parameter arms

ArmFrom (UTC)To (UTC)Values
bench7-always2026-11-15 11:51:38—all 11 nodes: digest_order_hold=true, epoch_record_collect_past_uncertified=true, epoch_record_conflict_halt=true, epoch_record_recertify=true, quorum_exec_block=true, vote_queue_fix=true

Workload

Warm-up120 s (2 min)
Mixed phase2,400 s (40 min)
Cool-down180 s (3 min)
Transaction typesbatched_self_send, contract_deploy, faucet_drip, native_transfer, stablecoin_approve, stablecoin_transfer, uniswap_v2_add_liquidity, uniswap_v2_remove_liquidity, uniswap_v2_swap, uniswap_v3_swap, uniswap_v4_swap, wtel_unwrap, wtel_wrap

Chaos events

MarkerTime (UTC)NodeActionDetail
C12026-11-15 13:41:35bench-validator-05docker kill -s KILL telcoinsystemctl stop holds it down (unit has Restart=on-failure). bench-blaster-05 targets ONLY this node, so its sends fail for the outage by design; the generator resends on connection errors
C22026-11-15 13:46:35bench-validator-05systemctl start telcoindocker start is impossible: the unit runs docker run --rm
C32026-11-15 13:51:35bench-observer-11systemctl restart telcoinobserver restart

Steps

  1. Both runs of bench 7 used one node image, built from the branch bench/bench7-2026-11 at commit 91a15e223 (upstream main 45dce8947 plus 114 commits), version 0.1.0, named by registry digest (adiri@sha256:15acdccafff7…68822c). The manifest names the storage engine tn-lmdb in both runs, and every node ran it. The load generator was built from tn-transaction-generator commit 62bbd5f, the commit bench 6's r20261113-1351 used.
  2. Between bench 6 and bench 7 the build changed in several ways at once: the tn-lmdb storage engine with storage v2, which came in one change (#1834) and are not separable from each other; the upstream rework of the vote handler; the order hold; and the epoch-record switches. The run's shape changed too: validator-05 was killed earlier into the load, and no transaction was queued in a pool at the load blast's start, where bench 6 had some on two validators in r20261113-1351 and on five in r20261113-2118. No figure on these pages separates these changes.
  3. The arm set seven node switches through the prestart step, before the nodes' coordinated start: quorum_exec_block, vote_queue_fix, digest_order_hold, epoch_record_collect_past_uncertified, epoch_record_conflict_halt and epoch_record_recertify, all true, and quorum_exec_block_starved_rounds, 0 in r20261115-1131 (arm bench7-always) and 4 in r20261115-1653 (arm bench7-adaptive), the only difference between the two arms. The closing-block check has no switch and ran in both runs. The switch gauges read the arm's values on every validator. A later form of the epoch-record code drops epoch_record_conflict_halt; bench 7 ran with it on. Each node's first process, started during the deploy, ran with every switch at its default; no epoch closed and no blast ran before the arm.
  4. The vote fix is the switch quorum_exec_block: a proposer names in each header a block that a quorum of validators should already have executed, and a voter signs only after it finds that block in its own chain. With quorum_exec_block_starved_rounds at 0, the always-on form, every header uses the quorum rule, never more than 8 outputs behind the author's own tip (the floor). With 4, the adaptive form, a header names the author's own tip until one of the author's headers is cancelled uncertified; the author then uses the quorum rule inside a starved window of up to 128 headers and afterwards returns to its own tip. On this build the window is counted in headers built, not rounds. A header is under-held when, at its proposal, at most five other voters held the block it named.
  5. The order hold is the switch digest_order_hold, the stack's candidate fix for issue #1807. Bench 6's pages describe that defect: when an own header is cancelled or committed late, its batches are queued again behind batches a newer own header already carried, so a sender's later transactions are skipped at execution as 'nonce too high' and land only after the generator resends them at its 300-s gap check. The hold counts every re-queue that still goes out of order, the one counter that isolates it, and the headers built under the hold.
  6. Each run created ten validators and one observer on c3-highcpu-8, each node with one 100 GB Hyperdisk Balanced disk at 20,000 IOPS and 800 MiB/s, boot image ubuntu-2404-noble-amd64-v20261105. One validator ran in each of bench 6's ten zones (Los Angeles, London, Sydney, Tokyo, Ashburn, the Netherlands, Singapore, Hong Kong, Seoul and Mumbai), the observer in Los Angeles. The zones, the machine, the disk, the generator, the epoch length (1,200 s) and the consensus parameters equal r20261113-1351's. The host probe was uniform (kernel 7.0.0-1017-gcp, read-ahead 128 KiB), and no node was flagged slow by the fsync probe (median 1,044 µs), where bench 6 flagged two nodes in each of its 20,000-IOPS runs.
  7. Each run had three blasts. Ten generators on e2-standard-2, one per validator in that validator's zone and sending only to it, ran blast 0 (light, 8 senders each with up to 16 transactions in flight, 8 x 16), blast 1 (light, 600 x 4) and blast 2, the load blast (600 x 16, at most 96,000 in flight), with a 120-s warm-up before its mixed phase.
  8. The chaos plan was bench 6's, inside blast 2's mixed phase: validator-05's container was killed with SIGKILL before an epoch boundary and held down for 300 s, then started, and the observer was restarted 300 s after that. Generator 05 sends only to validator-05, so it is cut off for the outage and for as long as its validator follows the chain afterwards. In bench 7 the kill came earlier into the load than in bench 6 and left more of it afterwards: 1,608 s here against 950 s in r20261113-1351.
  9. Headline figures keep bench 5's definitions, as on bench 6's pages. Throughput is the chain's transactions per second over the window in which all ten generators ran their mixed phase, the kill inside it (2,375 s in this run). Time to land runs to the block's timestamp, time to seen to when the generator's scanner saw it. Own-header certification is the share of a validator's own headers that were certified; the fleet figure before the kill is the certification analysis's count in bench 7 and the rounds analysis's in bench 6, over windows of different lengths (817 s here, 1,477 s in r20261113-1351). The distance to the named block is counted by rank among block-bearing outputs.
  10. Each form ran once. A difference between the two bench 7 runs is judged against this run's own spread, the range over its own buckets of the same window (300-s buckets of the load blast, 60-s blocks of blast 0, the buckets before the kill for per-validator rows); inside it the forms are not told apart from noise. That spread is variation inside one run, not run-to-run variance, and no figure carries a significance test.
  11. The analysis read every node's whole log and the node metrics exported from Prometheus (11:46:38-14:27:13, 15-s step, 397 metrics, 22,504 series). Validator-09's metrics agent lost one remote-write batch of every scrape for the whole run while the node was healthy and its logs whole, so up, the scrape series, tn_engine_root_seconds, tn_executor_output_transactions, tn_primary_vote_stage_seconds and ten more series lack it, and a figure over validators from one of them covers nine. The four series this page's node-side figures read (commit latency, execution duration, node mode and batch-seal failures) are not among them. An end-of-run snapshot of validator-09's own metrics endpoint stands in for whole-run totals. On every node the agent's network label replaces the node's own (primary, worker-0), so per-network series cannot be read from Prometheus.
  12. Times are UTC on 15 November. Deployment ran 11:31:38-12:24:39, with the arm at 11:51:38 and the coordinated start at 11:59:06. The idle check stopped the chain at 12:30:02 on a missing up series from validator-09, a healthy node, and the blasts were launched by hand at 12:40:20, 15.7 min after the deploy's end. The mixed phases ran 12:50:21-12:57:06 (blast 0), 13:03:11-13:19:05 (blast 1) and 13:27:58-14:08:23 (blast 2). Validator-05 was killed at 13:41:35, 817 s into blast 2's mixed phase, closing epoch 5, and started at 13:46:35; the observer was restarted at 13:51:35.
  13. In blast 1 generator 06's preamble wait ran its full 600 s after 737 preamble transactions of 67 senders were lost, so blast 1 has no second with all ten generators in their mixed phase; its figures are over the other nine generators' common window (342 s). Every blast ended with rc 0. At the end checks all 11 nodes reported fatal_db=0 and halt=0, every node at block 98,322. The teardown deleted all 12 instances at 14:42:38, and the archive verified 627 files with no problem. A first attempt at the adaptive run was lost when the cloud login expired during its first light blast; its load blast never ran, and it is not published.

The harness scripts live in a private repository; the steps above describe what they do.

Results

MetricValueWindowSourceMethod
Average chain throughput, all load generators running9,288.6 tx/sall generators activecombined.series.per_second[chain_tx]Chain transactions per block-timestamp second, summed over the 2375-s window when every load generator was in its mixed phase and divided by its length.
Best 60 seconds, all load generators running10,968.7 tx/sall generators activecombined.series.per_second[chain_tx]Highest mean chain tx/s over any 60 consecutive seconds inside the all-generators window (aggregate.py best_window_mean).
Transactions landed over the whole blast24,963,413 txwhole runcombined.totals.landed_chainSum of tx_count over every block the load generators saw during the blast, own and foreign transactions alike.
Blocks per second31.86 blocks/smerged mixed windowcombined.blocks.rows[timestamp]Chain blocks whose timestamp falls in the merged mixed window, divided by its length (aggregate.py landed.capacity.blocks_per_sec over the same window).
Time to land, median1.48 swhole runcombined.latency.ts_ms.p50Block timestamp minus submit time over every landed own transaction in the blast, from the load generators' merged latency histograms (aggregate.py merge_latency), in seconds.
Time to land, p957.76 swhole runcombined.latency.ts_ms.p95Block timestamp minus submit time over every landed own transaction in the blast, from the load generators' merged latency histograms (aggregate.py merge_latency), in seconds.
Time to land, median, whole-second bias removed1.98 swhole runcombined.latency.ts_ms.p50 + 0.5 sTime to land, median, plus 0.5 s. The block timestamp is the committing leader's header creation time cut to the whole second, so every sample reads 0 to 999 ms low, 0.5 s on average.
Time to seen, median9.4 swhole runcombined.latency.seen_ms.p50Submit time to the moment the load generator's block scanner had the block holding the transaction (block and receipts fetched from the generator's own node), over every landed own transaction in the blast, from the load generators' merged latency histograms (aggregate.py merge_latency), in seconds.
Time to seen, p9517.33 swhole runcombined.latency.seen_ms.p95Submit time to the moment the load generator's block scanner had the block holding the transaction (block and receipts fetched from the generator's own node), over every landed own transaction in the blast, from the load generators' merged latency histograms (aggregate.py merge_latency), in seconds.
Time to land at light load, median0.25 slight load, 8 senders per generatorblast-0-20261115T125951Z/blasters latency.ts_ms.p50Block timestamp minus submit time over every landed own transaction submitted in the mixed phase of the light-load blast light8 (index 0: 8 senders per load generator at window 16, chaos off), pooled from the generators' latency histograms with the aggregate.py merge_latency method, in seconds; not bias-corrected.
Time to seen at light load, median1.64 slight load, 8 senders per generatorblast-0-20261115T125951Z/blasters latency.seen_ms.p50Submit time to the generator's block scanner having the block over every landed own transaction submitted in the mixed phase of the light-load blast light8 (index 0: 8 senders per load generator at window 16, chaos off), pooled from the generators' latency histograms with the aggregate.py merge_latency method, in seconds; not bias-corrected.
Commit latency, p952.09–2.23 smerged mixed windowmetrics.prometheus.series.tn_primary_commit_latency_seconds.by_node[healthy validators].p95For each healthy validator, the nearest-rank p95 over the merged mixed window of its 15-s histogram_quantile(0.95) commit-latency series; reported as the range over validators.
Execution time per consensus output0.87–0.92 smerged mixed windowmetrics.prometheus.series.tn_engine_execution_duration_seconds.by_node[healthy validators].meanFor each healthy validator, the nearest-rank median over the merged mixed window of its 15-s mean execution-duration series; reported as the range over validators.
Write throughput, p95, busiest validator286.6 MiB/smerged mixed windowmetrics.gcp[bench-validator-04].disk_write_bytes_per_sec p95Nearest-rank p95 of the 60-s mean write-byte rate over the merged mixed window for each healthy validator; the highest (bench-validator-04) is reported.
Write operations, p95, busiest validator19,924 IOPSmerged mixed windowmetrics.gcp[bench-validator-02].disk_write_ops_per_sec p95Nearest-rank p95 of the 60-s mean write-operation rate over the merged mixed window for each healthy validator; the highest (bench-validator-02) is reported.
CPU, p95, busiest validator55.8 %merged mixed windowmetrics.gcp[bench-validator-04].cpu_utilization p95Nearest-rank p95 of the 60-s CPU utilisation over the merged mixed window for each healthy validator; the highest (bench-validator-04) is reported.
Node resident memory, p95, busiest validator7.07 GiBmerged mixed windowmetrics.gcp[bench-validator-07].telcoin_rss_bytes p95Nearest-rank p95 of the 60-s telcoin process resident memory over the merged mixed window for each healthy validator; the highest (bench-validator-07) is reported.
IO pressure, highest reading, range over validators6.7–12.7 %merged mixed windowsampler[healthy validators].psi_io_some_avg10 maxFor each healthy validator, the highest 10-s 'some' IO pressure reading the on-node sampler logged inside the merged mixed window; reported as the range over validators.

Recovery

Nodebench-validator-05
Killed2026-11-15 13:41:35 UTC
Restarted2026-11-15 13:46:35 UTC
Rejoined2026-11-15 14:16:19 UTC
Catch-up time1,755 s (29 min 15 s)
Failed batch reportsnot recorded

Chain TPS per 5-minute interval

Chain TPS per 5-minute interval 9 intervals from 13:27 to 14:08 UTC; average 8,606 TPS, highest 10,545 TPS. The merged mixed window ran from 13:27 to 14:08 UTC. TPS 0 2,000 4,000 6,000 8,000 10,000 12,000 13:27–13:32 UTC: 10,545 TPS, 9.6 generators active 13:32–13:37 UTC: 10,465 TPS, 10.0 generators active 13:37–13:42 UTC: 9,383 TPS, 10.0 generators active 13:42–13:47 UTC: 9,657 TPS, 10.0 generators active 13:47–13:52 UTC: 9,222 TPS, 10.0 generators active 13:52–13:57 UTC: 8,673 TPS, 10.0 generators active 13:57–14:02 UTC: 8,345 TPS, 10.0 generators active 14:02–14:07 UTC: 8,420 TPS, 10.0 generators active 14:07–14:08 UTC: 2,748 TPS, 4.0 generators active avg 8,606 13:27 UTC 13:37 UTC 13:47 UTC 13:57 UTC 14:07 UTC C1 2026-11-15 13:41:35 UTC: docker kill -s KILL telcoin on bench-validator-05 (systemctl stop holds it down (unit has Restart=on-failure). bench-blaster-05 targets ONLY this node, so its sends fail for the outage by design; the generator resends on connection errors) C1 C2 2026-11-15 13:46:35 UTC: systemctl start telcoin on bench-validator-05 (docker start is impossible: the unit runs docker run --rm) C2 C3 2026-11-15 13:51:35 UTC: systemctl restart telcoin on bench-observer-11 (observer restart) C3

Each bar is the average chain TPS over one 5-minute interval of the merged mixed window (13:27–14:08 UTC); fainter bars had fewer generators active than the maximum, the dashed line is the average of all bars and amber lines mark the chaos events listed below.

  • C1 2026-11-15 13:41:35 UTC: docker kill -s KILL telcoin on bench-validator-05 (systemctl stop holds it down (unit has Restart=on-failure). bench-blaster-05 targets ONLY this node, so its sends fail for the outage by design; the generator resends on connection errors)
  • C2 2026-11-15 13:46:35 UTC: systemctl start telcoin on bench-validator-05 (docker start is impossible: the unit runs docker run --rm)
  • C3 2026-11-15 13:51:35 UTC: systemctl restart telcoin on bench-observer-11 (observer restart)
5-minute data (9 rows)
start_utcend_utcgenerators_activechain_tps
2026-11-15T13:27:58Z2026-11-15T13:32:58Z9.610,544.9
2026-11-15T13:32:58Z2026-11-15T13:37:58Z10.010,465.1
2026-11-15T13:37:58Z2026-11-15T13:42:58Z10.09,382.6
2026-11-15T13:42:58Z2026-11-15T13:47:58Z10.09,656.6
2026-11-15T13:47:58Z2026-11-15T13:52:58Z10.09,221.9
2026-11-15T13:52:58Z2026-11-15T13:57:58Z10.08,672.7
2026-11-15T13:57:58Z2026-11-15T14:02:58Z10.08,344.6
2026-11-15T14:02:58Z2026-11-15T14:07:58Z10.08,420.5
2026-11-15T14:07:58Z2026-11-15T14:08:23Z4.02,747.6

The fine-grained chain throughput series (its step_s field gives the seconds per point) is published as series/tps-1s.json.

Other networks

MeasureTelcoin Network (this run)Tempo
Throughput9,288.6 tx/s (Avg chain TPS)16,455 TPS, shown as "Settled TPS" for the nightly run of 5 Oct 2026 (commit de5f5d3). perf.tempo.xyz describes it as "Median settled throughput across 130 measured blocks."
How throughput is measuredChain transactions per block-timestamp second, summed over the 2375-s window when every load generator was in its mixed phase and divided by its length.The methodology page defines TPS as "Transactions per second (derived from gas / avg tx gas)": gas used divided by the average gas of one transaction, reported as the median over the measured blocks.
Run length40.4 min merged mixed window; each generator was set to a 40 min mixed phase2m 39s for the 5 Oct 2026 nightly run (130 measured blocks at a 687 ms average block time), per perf.tempo.xyz.
Validators and placement10 validators and 1 observer across 10 regions: asia-east2 (Hong Kong), asia-northeast1 (Tokyo), asia-northeast3 (Seoul), asia-south1 (Mumbai), asia-southeast1 (Singapore), australia-southeast1 (Sydney), europe-west2 (London), europe-west4 (Netherlands), us-east4 (Ashburn), us-west2 (Los Angeles); load generators: one blaster per validator, in that validator's zone, sending only to itperf.tempo.xyz labels the scenario "Public mix · multi-region". The methodology page says "Benchmarks run two local validators in consensus on a dedicated bare-metal server with 100 GiB pre-bloated state." The public tempoxyz/tempo repo has nightly workflows for both kinds of setup: bench-e2e runs on one bare-metal runner, and bench-e2e-multi-region defaults to 10 validators on cloud VMs (its regions input reads "Empty uses the selected cloud's five defaults."). The run page does not say which setup produced this number.
Hardwarec3-highcpu-8, 8 vCPUs, 16 GB memory; disk: hyperdisk-balanced, 100 GB, 20,000 IOPS, 800 MiB/sThe methodology page lists a dedicated bare-metal server: "AMD EPYC 4585PX (16C/32T, 4.3/5.7 GHz)", "128 GB 3600 MHz" memory and "2×960 GB NVMe + 2×1.92 TB NVMe via schelk". The repo's multi-region workflow uses cloud VMs instead (c4d-standard-48-lssd on GCP, c6id.8xlarge on AWS).
Workloadbatched_self_send, contract_deploy, faucet_drip, native_transfer, stablecoin_approve, stablecoin_transfer, uniswap_v2_add_liquidity, uniswap_v2_remove_liquidity, uniswap_v2_swap, uniswap_v3_swap, uniswap_v4_swap, wtel_unwrap, wtel_wrap; 10 generators × 600 senders eachperf.tempo.xyz labels the scenario "Public mix"; the methodology page says "100% TIP-20 token transfers (ERC-20 equivalent)". The run used 1,000 accounts and 4 TIP-20 tokens.
StateFresh chain from genesis for every runThe methodology page says "100 GiB pre-bloated state".
Latency (as each source defines it)Time to land, median: 1.48 s; Time to land, p95: 7.76 s; Commit latency, p95: 2.09–2.23 sBlock time of 687 ms, which the methodology page defines as "Average wall-clock time between blocks". The run page gives no per-transaction latency. Separately, tempo.xyz says "Blocks finalize in ~0.6 seconds with no re-orgs." (a network claim, not a figure from this benchmark).
Source and fetch dateThis page; every figure names its source field.perf.tempo.xyz nightly run of 5 Oct 2026 (commit de5f5d3), the perf.tempo.xyz methodology page, tempo.xyz, and the benchmark workflows in the tempoxyz/tempo repo. All fetched on 5 Oct 2026.

Sources for Tempo (fetched 2026-10-05):

Tempo's setup may not match ours: its methodology page describes two validators on one bare-metal server, while our runs use ten validators across five regions, so read this table as context, not a ranking.

This table is context, not a ranking.

Caveats

  • Against r20261113-1351 the zones, machines, disk, generator, epoch length and consensus parameters are the same; the node build and the run's shape differ. In blast 2 throughput rose from 5,322 to 9,289 tx/s (+74.5 %), seen p50/p95 went from 12.99/30.89 s to 9.40/17.33 s, device writes from 0.998 to 0.859 of the 20,000 cap and engine busy was 994.3 and 978.1 ms a second, near full busy in both. The storage engine, the vote handler, the order hold, the epoch-record switches and the run's shape changed together, so no row here is attributed to one of them.
  • Throughput fell through the load blast as the store grew: its 300-s blocks ran from 10,544.9 tx/s in the first to 8,420.5 in the last full one, the kill falling in the third. Between the first and last bucket commit per transaction rose from 21.49 to 42.07 µs and the executed rate fell from 10,470 to 8,393 tx/s, while tn-lmdb's used size went from 1.12 to 2.74 GiB. A least-squares fit gives 14.3 µs of commit per transaction per GiB used (r² 0.966); it is a fit, not a mechanism.
  • The engine and the disk both sat below their limits, the engine nearer to its own. The engine was busy 978.1 ms a second by the log timeline and executed 9,293 tx/s per node against the 9,500 tx/s its task cost allows at full busy: 105.26 µs per transaction (execute 54.29, persist 49.42, of which commit 31.93), persist 47 % of it. The cycle is sequential: no execution task started before the previous one ended (0 of 24,447 starts). The device ran at 0.859 of its 20,000 writes a second at 8.50 ms per write, 1.85 writes and 26,132 bytes per transaction. Bench 6 sat at the write cap (0.998). Writes per transaction fell to 0.49 of bench 6's; with writes per output barely moving and outputs holding 2.15 times as many transactions (8,294 against 3,854), that fall is arithmetic, and nothing isolates which change made the outputs larger.
  • Own-header certification fell against bench 6 while throughput rose. Before the kill the fleet certified 94.0 % against 99.94 % (817 s against 1,477 s of the mixed phase), and four validators other than 05 sat under 98 % against none. Voters answered refused-behind 34.8 times a minute against 0 and superseded 1,199 times against 3.5; 11.13 % of rounds had four or more under-held parents against 0.02 %; the median execution lag at the voters fell from 35.65 to 16.96 rounds. Over all of blast 2, 10.31 % of headers were under-held, and those certified 61.25 % of the time against 99.53 % for the rest. These figures divide the loss by mechanism, not by the change that moved it: the vote-handler rework, the voters' execution lag, the hold and the run's shape are not separable.
  • At a floor of 16 outputs instead of 8, arithmetic on this run's measured tips, rounds and cancels leaves no header under-held. That is arithmetic, not a result, and the deeper floor would name an older block (14.71 block-bearing outputs, 13.16 s at validator-01's output rate, against the measured 7.57 outputs and 6.77 s) whose cost bench 7 did not measure. The run kept the floor at 8.
  • Against r20261115-1653, the adaptive form on the same build and disk, these are not told apart from this run's own spread: blast 2's chain rate (9,289 against 9,337 tx/s; this run's 300-s blocks 8,344.6-10,544.9), seen p50, fleet certification over blast 2 (95.59 against 95.00 %; 93.37-97.29), the lowest validator (83.6 against 76.4 %; 55.2-89.3) and committed rounds per second (2.32 against 2.22; 1.99-2.73). Blast 2's seen p95 (17.33 against 24.18 s; 14.27-18.78) and blast 0's rate (698.3 against 680.0 tx/s; 688.2-722.8) lie outside it against the adaptive run. Per-validator rows outside it move both ways: 28 lower in the adaptive run, 13 higher. The form comparison reads the pair as support for keeping the always-on form as the default at 20,000 IOPS, and weak support.
  • In blast 2, outside generator 05, no generator gap-resent, none left a transaction unlanded after the 180-s tail scan, no node counted a 'nonce too high' skip and no landing was seen 300 s or more after submission; r20261113-1351 had 29,449 gap resends and 13,036 skips from its warm-up start to its cool-down end. Lost own headers carrying batches still occurred on all nine validators other than 05 (981 not certified and 293 certified but never committed over blast 2), and no gap resend followed any of them, where in bench 6 every gap-resend second of generators 01-04 and 06-10 fell 280-345 s after a lost own header of the generator's own validator (31 of 31 seconds in r20261113-1351).
  • The hold's out-of-order re-queue counter read 0 in the log, the warnings and the export. It is the one counter that isolates the hold; it has no bench 6 twin, and it says that no re-queue went out of order here, not how many would have without the hold. The fall in resends, unlanded transactions and skips against bench 6 moved with the hold, the storage engine and the vote path (which moved the rate and the round time, and so how long a lost header's batches wait), the kill earlier into the load and bench 6's queued pools at blast 2's start, and is not separable between them. This run is consistent with the hold doing its job and does not prove it.
  • The exception is blast 1: one gap episode of 4 transactions on generator 06, whose preamble wait had run its full length, with 4 landings seen 300 s or more after submission. Its one resend second fell 280-345 s after a lost own header of validator-06, bench 6's pattern. The nodes also counted 22 'nonce too high' skips, 2 per node on all 11 nodes, all in blast 1. The adaptive run had neither.
  • Over blast 2, 1,866 headers were built under the hold, none carrying a batch. The longest held run was 21 headers (validator-06, before the kill) and the longest in time 9.1 s (after validator-05's start), two different runs. Held headers certified 49.8 % of the time against 97.3 % for the rest, an association no counter ties to the hold. 152 held headers, of 1,110 built on the digest trigger, are candidates for a trigger that counted held digests; that figure is inferred from logged fields, not measured. The fleet ran a digest threshold of 5 batches and a minimum header delay of 350 ms. No over-quota report was logged.
  • Generator 05 landed none of its own transactions for 1,584 s (13:41:38-14:08:02) and had 9,812 gap resends; 32 of its transactions, all sent to validator-05, stayed unlanded after the tail scan, transactions accepted in the last seconds before the kill and batched by the restarted validator after the scan. Its 9,780 landings seen 300 s or more after their send are late sightings of blocks made around the kill, not late landings: a landing's time is its block's timestamp, a sighting's is when the generator's scanner saw it.
  • The far validators, 02 (London) and 06 (the Netherlands), landed a small share of what the others landed in the light blasts: generators 02 and 06 landed 8.8 and 5.5 tx/s in blast 0 against a median of 88.9 for the other eight, and 161.8 and 172.0 against 1,198.8 in blast 1. Every batch-carrying header of the two was certified, yet 47 % and 55 % of them in blast 0, 51 % and 60 % in blast 1, were never committed; their batches waited a median 50 to 58 rounds for the commit-path re-queue, and a batch took 16.1 and 20.3 s in blast 0, 15.8 and 15.2 s in blast 1, at the median from its first own proposal to execution, against 0.93 and 1.35 s on the near validators. The order hold held none of their headers, and bench 6 shows the same pattern in blast 0 without a hold. The cause is open; its last step, near headers not naming the late certificate, is inferred, since which parents a header names is not logged.
  • The run closed 8 epochs, each on one record hash and one closing block and published by all eleven nodes. The closing-block check returned ok 179 times fleet-wide and nothing else, and the close identity check read ok 8 times on every node, with no mismatch. At the kill's close the fewest signers on a quorum line were 7, with the first quorum after 8.86 s; at the other loaded close 7 and 16.0 s (r20261113-1351: 9 and 14.75 s, 7 and 16.44 s). The conflict line, the conflict halt, the walk past an uncertified epoch and the re-certify round never fired on any node: the switches were on and nothing triggered them, so this run does not exercise those paths. The error 'failed to reach quorum on epoch close', a false alarm on bench 6, fell from 3 to 0 because this build logs it only when no certificate is held.
  • Validator-05 was active (CvvActive) again 2,068.6 s after the kill and 450.7 s after the cool-down ended, after replaying the kill's epoch (95.4 s) and one rejoin attempt; no other validator left the active state. In r20261113-1351 it took 1,300.7 s, but there the kill came 1,477 s into the load against 817.1 s here, with 950 s of load left against 1,607.9 s, so the two are not one comparable number. Measured and not explained: after its promotion it proposed its first header within a second, and its first own certificate committed after 1.65 s (59.6 s in the adaptive run). The observer had its first line at distance 1 3.7 s after its restart.
  • No node logged a fail-stop line the census counts (a stored final state that does not match, a refused closing block, a conflict halt, a fork, a fatal database failure, an output gap), and all 11 nodes reported fatal_db=0 at the end checks. Error templates new against r20261113-1351: 'processed certificate that is too new' on validator-05 after its start (19,307 lines), 'error while sealing batch | AntiQuorum' (3 lines) and 'ConsensusOutput channel closed' once on the observer.
  • The idle check before the blasts stopped the chain on a missing up series from validator-09, a healthy node, lost to the metrics agent's defect, and the blasts were launched by hand 15.7 min after the deploy's end; restarting the agent changed nothing. That schedule, not the form, put the kill in epoch 5's close here and in epoch 4's in r20261115-1653, and it is why the two runs' windows before the kill differ (817 s and 1,121 s).
  • Each form ran once on one build and one disk, so run-to-run variation is not measured and no figure carries a significance test. Not measured or open: what the named block's age costs; a run at a floor of 16 outputs; a run at 50,000 IOPS, where bench 6 found the forms differed (bench 7 made none); what lowers the far validators' committed share; what separates the upstream vote-handler rework from the hold's cadence (one run with the hold off and one without the rework would); and the hold's early-trigger candidates, which a logged count of un-held digests would measure.
  • Per-validator rows cover 9 validators; bench-validator-05 excluded as chaos target(s).
  • Recovery: demoted / rejoined are the first 15-s Prometheus samples showing the change, so they trail the node's own log times by up to a scrape interval plus one step.
  • Recovery: no tn_batch_builder_seal_failures_total count for bench-validator-05; failed_reports unavailable.
  • The harness working tree had uncommitted changes when the run was deployed.
  • Validators bench-validator-06 and bench-validator-08 certified fewer than 90 % of their own headers in blast 2 mixed; their logs are preserved under results/r20261115-1131/logs/ (see INDEX.md).
  • Before the coordinated start each node ran once during staging with every node switch at its default (11 start(s) on 11 node(s)); the run's switches were on from the coordinated start at 2026-11-15T11:51:38Z.

Full report

Full report HTML · 1.4 MB

Changes from the original report:

  • The site adds a banner at the top of the report that links back to this page.

SHA-256 of the committed file, before the banner is added: 07acc45abf7ab72b63be8fb0aa1bab7a9143429f0edfc01c5338dd0546c542af