The adaptive vote fix on Hyperdisk Balanced at 20,000 IOPS

2026-11-13 · started 21:18 UTC · ended 23:52 UTC · merged mixed window 40 min 10 s

telcoin-network 2df831e3c · bench/bench6-2026-11 · commit verified from node startup logs

24 node starts across 11 instances logged telcoin-network 0.1.0 (2df831e3ca34c06a361a7acca10397787e7b332e) The node image probe recorded the same commit before deploy Node logs: fork schedule: seed_signature, multi_workers, prevrandao, leader_seeded_ordering, and subsecond_timestamp active from genesis Each of the 24 node starts logged its node switches line, and each matched the switch state the manifest records for that start Commit 2df831e3c is the head of the branch bench/bench6-2026-11, built for this benchmark: a base commit 165e3fe01 plus 98 commits, in which every fix for bench 5's concerns is a node switch, off by default. The twelve switches are quorum_exec_block (the vote fix, whose form the setting quorum_exec_block_starved_rounds picks), vote_queue_fix, epoch_record_state_check, mdbx_random_advice, mdbx_no_writemap, sealed_tx_keep, digest_dedupe, recovery_threads, epoch_close_task_drain, pipelined_persist, rpc_admission_backlog_outputs and parent_grace. A node logs its switches at every start ('node switches') and exports them as tn_node_switch. The image was named by registry digest (adiri@sha256:874537ff…852745) and ran all four runs of this program unchanged.

We repeated r20261113-1351 with one change: the vote fix ran in its adaptive form instead of the always-on form. In the adaptive form a header names the author's own executed tip until one of the author's headers is cancelled uncertified; the author then uses the always-on quorum rule, never more than 8 outputs behind its tip, inside a starved window of up to 128 rounds, and afterwards tries its own tip again. The build (2df831e3c), the other three switches, bench 5's fleet and disk (Hyperdisk Balanced at 20,000 provisioned IOPS), the load and the chaos plan were the same. Throughput and what binds did not change: average chain throughput with all ten generators running was 5,394.6 tx/s against 5,322.1 (+1.4 %, inside the always-on run's own five-minute range), the disks again sat at their limit of 20,000 writes a second (0.999) with every engine busy, and the median time to seen was 12.89 s against 12.99 s. The epoch closes and the killed validator's rejoin matched too. Own-header certification was a little lower: 99.64 % over the load blast against 99.94 %, and before the kill validators 02, 06 and 10 (London, the Netherlands and Mumbai) sat at 98.8 to 98.9 %, about 1 point under and outside the always-on run's range for them. The loss is the adaptive form's probe: those three spent 95.7 to 97.4 % of their headers in starved windows, and every return to the own tip lost a header. In exchange, headers named a block about 0.9 s old on average, against 5.5 s. The senders paid more: 104,280 transactions were resent after the generators' 300-s gap check against 29,546, and 10,767 never landed against 183, through the same reordering of re-queued batches as under the always-on form (issue #1807), exposed on more validators in more minutes. No node logged a halt, fork, unexpected execution result, fatal database error or panic. This run is one of four in one program, one node build with two forms of the vote fix on two disks: the always-on form as r20261113-1351 (20,000 IOPS) and r20261113-1630 (50,000 IOPS), the adaptive form as r20261113-2118 (20,000 IOPS) and r20261114-0000 (50,000 IOPS). Across them the form changed neither throughput nor what binds; the always-on form certified slightly more own headers at 20,000 IOPS, neither form held certification at 50,000 IOPS (the adaptive run certified more), the adaptive form's senders paid more on both disks through the same reordering (issue #1807), and which form goes upstream is not decided.

Compared against r20261113-1351.

Key figures

5,394.6tx/s
Average chain throughput, all load generators running
all generators active
7,113.8tx/s
Best 60 seconds, all load generators running
all generators active
6.79s
Time to land, p95
whole run
0.98–0.98s
Commit latency, p95
merged mixed window

How it was run

Software

Upstream committelcoin-network 2df831e3ca34c06a361a7acca10397787e7b332e · bench/bench6-2026-11 · commit verified from node startup logs
Node imageus-docker.pkg.dev/telcoin-network/tn-public/adiri@sha256:874537ff1cd8d496140154837e7a36524978bdbf3b0440e48972b073d9852745
Node image digestsha256:874537ff1cd8d496140154837e7a36524978bdbf3b0440e48972b073d9852745
Node version0.1.0
Fork schedulesubsecond-from-genesis
Load generator commit62bbd5f
Load generator image digestsha256:7cefcc505201b2e40361a7d8cfe034778090b76c568cb2400369bf566c605a59
Analysis commit62bbd5f
Harness commit4171b897c930d80ba5ce170c79dc21193dc9c213
Harness submodulescommon: 1a6955d8a2c6f19e53cf7aaad5e32a9107a21a7d
tn-node-deployment: d9446f720026fa5ce8b64fce9074e56b05b3f4eb

Fleet

Fleetc3-highcpu-8 + Hyperdisk Balanced at 20,000 IOPS (c3-hdb), ten regions, validator-05 in Ashburn
Validators10
Observers1
Machine typec3-highcpu-8
vCPUs per node8
Memory per node16 GB
Diskhyperdisk-balanced, 100 GB, 20,000 IOPS, 800 MiB/s
Regionsasia-east2 (Hong Kong), asia-northeast1 (Tokyo), asia-northeast3 (Seoul), asia-south1 (Mumbai), asia-southeast1 (Singapore), australia-southeast1 (Sydney), europe-west2 (London), europe-west4 (Netherlands), us-east4 (Ashburn), us-west2 (Los Angeles) (10)
Load generators10 generators × e2-standard-2 × 600 senders each
Load placementone blaster per validator, in that validator's zone, sending only to it

Chain

Chain ID32286
Networkbench
Epoch length1,200 s (20 min)

Consensus parameters

Max header delay1,000 ms
Min header delay350 ms
Max batch delay250 ms
Vote timeout5,000 ms
Parameters fileconsensus-parameters.yaml
Confirmed bynode startup logs

Parameter arms

ArmFrom (UTC)To (UTC)Values
final-adaptive2026-11-13 21:38:28—all 11 nodes: epoch_record_state_check=true, mdbx_random_advice=true, quorum_exec_block=true, quorum_exec_block_starved_rounds=4, vote_queue_fix=true

Workload

Warm-up120 s (2 min)
Mixed phase2,400 s (40 min)
Cool-down180 s (3 min)
Transaction typesbatched_self_send, contract_deploy, faucet_drip, native_transfer, stablecoin_approve, stablecoin_transfer, uniswap_v2_add_liquidity, uniswap_v2_remove_liquidity, uniswap_v2_swap, uniswap_v3_swap, uniswap_v4_swap, wtel_unwrap, wtel_wrap

Chaos events

MarkerTime (UTC)NodeActionDetail
C12026-11-13 23:08:59bench-validator-05docker kill -s KILL telcoinsystemctl stop holds it down (unit has Restart=on-failure). bench-blaster-05 targets ONLY this node, so its sends fail for the outage by design; the generator resends on connection errors
C22026-11-13 23:13:59bench-validator-05systemctl start telcoindocker start is impossible: the unit runs docker run --rm
C32026-11-13 23:18:59bench-observer-11systemctl restart telcoinobserver restart

Steps

  1. All four runs of this program used one node image, built from the branch bench/bench6-2026-11 at commit 2df831e3c (a base commit 165e3fe01 plus 98 commits), with sub-second timestamps from genesis, and named by registry digest (adiri@sha256:874537ff…852745). Nothing was rebuilt between the runs. Every fix in that build is a node switch, off by default; a node logs its switches at every start ('node switches') and exports them as the metric tn_node_switch. The load generator was built from tn-transaction-generator commit 62bbd5f.
  2. The vote fix is the switch quorum_exec_block. Instead of its own newest executed block, a proposer names in each header a block that a quorum of validators should already have executed; a voter signs only after it finds the named block in its own chain. The setting quorum_exec_block_starved_rounds picks the form. With 0, the always-on form, every header takes the quorum value over the author's own executed tip and the blocks its parent headers named, capped at the own tip and never more than 8 outputs behind it (the floor); in practice nearly every header named the author's tip minus 8 outputs. With 4, the adaptive form, a header names the author's own tip until one of the author's headers is cancelled uncertified; the author then uses the always-on rule inside a starved window of up to 128 rounds, and afterwards tries its own tip again.
  3. Three more switches were on in all four runs: vote_queue_fix (the voter's lock wait), epoch_record_state_check (a check of the closing state when an epoch record is built) and mdbx_random_advice (the MDBX database's random-access hint over the whole file). The other eight stayed at their defaults: mdbx_no_writemap, sealed_tx_keep, digest_dedupe, recovery_threads, epoch_close_task_drain, pipelined_persist, rpc_admission_backlog_outputs and parent_grace. The switch set came from an arm file applied at deployment, before the nodes' coordinated start: no arm walk, no cache eviction and no restart before the load. Each node first ran once, alone, during staging (17 to 90 s, every switch at its default, no peer reached, no output executed); the switches of the run were on from the coordinated start, the first process that executed blocks.
  4. Each run created ten validators and one observer on c3-highcpu-8 (8 vCPU, 16 GiB), each node with one 100 GB Hyperdisk Balanced disk for the root and the data directory at 800 MiB/s, provisioned at 20,000 IOPS (bench 5's disk) or 50,000 IOPS. One validator ran in each of bench 5's ten regions (Los Angeles, London, Sydney, Tokyo, Ashburn, the Netherlands, Singapore, Hong Kong, Seoul and Mumbai), the observer in Los Angeles. Epochs last 1,200 s.
  5. Each run had three blasts. Ten generators on e2-standard-2, one per validator in that validator's zone and sending only to it, ran blast 0, the floor, with 8 senders each and up to 16 transactions in flight per sender (8 x 16); blast 1 with 600 senders and a window of 4 (600 x 4, at most 24,000 in flight); and blast 2, the load blast, with 600 senders and a window of 16 (600 x 16, at most 96,000 in flight) and phases of 120 s warm-up, 2,400 s mixed load and 180 s cool-down, the shape of bench 5's load blast.
  6. The chaos plan was bench 5's, inside blast 2: validator-05's container was killed with SIGKILL shortly before an epoch boundary and held down for 300 s, so the outage crossed the boundary; then it was started, and the observer was restarted 300 s after that. Generator 05 sends only to validator-05 and does not fail over to another endpoint, so it is cut off for the outage and for as long as its validator follows the chain afterwards.
  7. Headline figures keep bench 5's definitions and code; bench 5's throughput, latency and landing figures quoted on these pages were recomputed exactly from bench 5's own generator documents with the same code, while its certification, disk, stage and rejoin figures come from bench 5's report or from this program's reading of bench 5's logs and model. Throughput is the chain's transactions per second over the window in which all ten generators ran, with the outage and generator 05 inside it. Time to seen runs from submission until the generator saw the receipt, from the generators' summed histograms (the bundle method); time to land is to the block's timestamp, in whole seconds, so about 0.5 s low. Own-header certification is the share of a validator's own headers that were certified, over blast 2's mixed phase unless another window is named.
  8. The analysis read every node's whole main log (the node's stderr file was not pulled) and the node metrics exported from Prometheus before each teardown. The comparison of the two forms ran the same scripts on both runs of a disk and reproduced the always-on runs' figures.
  9. Against r20261113-1351 only the form of the vote fix changed: the arm file final-adaptive set quorum_exec_block_starved_rounds to 4 instead of 0, with the same other switches. The image, the generator, the fleet, the disk, the load and the chaos plan are the same.
  10. A first deployment, r20261113-2044, stopped before the coordinated start and before the arm was applied: validator-08's first start, during staging, failed with 'Address already in use' on its worker listener. It was torn down and is not part of the run.
  11. Times are UTC on 13 November. Deployment ran from 21:18:25 to 22:02:14 (the second deployment); all eleven nodes started at one due time, with 0 s spread. The fsync probe median was 1,043 µs, with validators 04 and 07 flagged slow. The mixed phases ran 22:12:55-22:19:56 (blast 0), 22:25:37-22:33:06 (blast 1) and 22:45:46-23:25:56 (blast 2). Validator-05 was killed at 23:08:59 and started at 23:13:59, epoch 5 opened during its outage at 23:09:58, and the observer was restarted at 23:18:59.
  12. The teardown ended with rc 0 at 23:52:55, 12 instances deleted and none left. The archive holds 625 files, 2.48 GB. The metric export before the teardown holds 382 metrics (21,023 series) but lacks validator-07's vote-stage series, so the voter's lock wait here comes from the log, and no comparison figure rests on that series alone. Before the kill the mixed phase ran 1,393 s here against 1,477 s in r20261113-1351; where windows differ, only rates and shares are compared.

The harness scripts live in a private repository; the steps above describe what they do.

Results

MetricValueWindowSourceMethod
Average chain throughput, all load generators running5,394.6 tx/sall generators activecombined.series.per_second[chain_tx]Chain transactions per block-timestamp second, summed over the 2390-s window when every load generator was in its mixed phase and divided by its length.
Best 60 seconds, all load generators running7,113.8 tx/sall generators activecombined.series.per_second[chain_tx]Highest mean chain tx/s over any 60 consecutive seconds inside the all-generators window (aggregate.py best_window_mean).
Transactions landed over the whole blast14,736,496 txwhole runcombined.totals.landed_chainSum of tx_count over every block the load generators saw during the blast, own and foreign transactions alike.
Blocks per second23.85 blocks/smerged mixed windowcombined.blocks.rows[timestamp]Chain blocks whose timestamp falls in the merged mixed window, divided by its length (aggregate.py landed.capacity.blocks_per_sec over the same window).
Time to land, median0.75 swhole runcombined.latency.ts_ms.p50Block timestamp minus submit time over every landed own transaction in the blast, from the load generators' merged latency histograms (aggregate.py merge_latency), in seconds.
Time to land, p956.79 swhole runcombined.latency.ts_ms.p95Block timestamp minus submit time over every landed own transaction in the blast, from the load generators' merged latency histograms (aggregate.py merge_latency), in seconds.
Time to land, median, whole-second bias removed1.25 swhole runcombined.latency.ts_ms.p50 + 0.5 sTime to land, median, plus 0.5 s. The block timestamp is the committing leader's header creation time cut to the whole second, so every sample reads 0 to 999 ms low, 0.5 s on average.
Time to seen, median12.89 swhole runcombined.latency.seen_ms.p50Submit time to the moment the load generator's block scanner had the block holding the transaction (block and receipts fetched from the generator's own node), over every landed own transaction in the blast, from the load generators' merged latency histograms (aggregate.py merge_latency), in seconds.
Time to seen, p9528.76 swhole runcombined.latency.seen_ms.p95Submit time to the moment the load generator's block scanner had the block holding the transaction (block and receipts fetched from the generator's own node), over every landed own transaction in the blast, from the load generators' merged latency histograms (aggregate.py merge_latency), in seconds.
Time to land at light load, median0.25 slight load, 8 senders per generatorblast-0-20261113T222217Z/blasters latency.ts_ms.p50Block timestamp minus submit time over every landed own transaction submitted in the mixed phase of the light-load blast light8 (index 0: 8 senders per load generator at window 16, chaos off), pooled from the generators' latency histograms with the aggregate.py merge_latency method, in seconds; not bias-corrected.
Time to seen at light load, median1.63 slight load, 8 senders per generatorblast-0-20261113T222217Z/blasters latency.seen_ms.p50Submit time to the generator's block scanner having the block over every landed own transaction submitted in the mixed phase of the light-load blast light8 (index 0: 8 senders per load generator at window 16, chaos off), pooled from the generators' latency histograms with the aggregate.py merge_latency method, in seconds; not bias-corrected.
Commit latency, p950.98–0.98 smerged mixed windowmetrics.prometheus.series.tn_primary_commit_latency_seconds.by_node[healthy validators].p95For each healthy validator, the nearest-rank p95 over the merged mixed window of its 15-s histogram_quantile(0.95) commit-latency series; reported as the range over validators.
Execution time per consensus output0.74–0.74 smerged mixed windowmetrics.prometheus.series.tn_engine_execution_duration_seconds.by_node[healthy validators].meanFor each healthy validator, the nearest-rank median over the merged mixed window of its 15-s mean execution-duration series; reported as the range over validators.
Write throughput, p95, busiest validator105 MiB/smerged mixed windowmetrics.gcp[bench-validator-02].disk_write_bytes_per_sec p95Nearest-rank p95 of the 60-s mean write-byte rate over the merged mixed window for each healthy validator; the highest (bench-validator-02) is reported.
Write operations, p95, busiest validator20,098 IOPSmerged mixed windowmetrics.gcp[bench-validator-04].disk_write_ops_per_sec p95Nearest-rank p95 of the 60-s mean write-operation rate over the merged mixed window for each healthy validator; the highest (bench-validator-04) is reported.
CPU, p95, busiest validator35.2 %merged mixed windowmetrics.gcp[bench-validator-03].cpu_utilization p95Nearest-rank p95 of the 60-s CPU utilisation over the merged mixed window for each healthy validator; the highest (bench-validator-03) is reported.
Node resident memory, p95, busiest validator6.14 GiBmerged mixed windowmetrics.gcp[bench-validator-02].telcoin_rss_bytes p95Nearest-rank p95 of the 60-s telcoin process resident memory over the merged mixed window for each healthy validator; the highest (bench-validator-02) is reported.
IO pressure, highest reading, range over validators12.0–14.1 %merged mixed windowsampler[healthy validators].psi_io_some_avg10 maxFor each healthy validator, the highest 10-s 'some' IO pressure reading the on-node sampler logged inside the merged mixed window; reported as the range over validators.

Recovery

Nodebench-validator-05
Killed2026-11-13 23:08:59 UTC
Restarted2026-11-13 23:13:59 UTC
Rejoined2026-11-13 23:31:49 UTC
Catch-up time1,050 s (17 min 30 s)
Failed batch reportsnot recorded

Chain TPS per 5-minute interval

Chain TPS per 5-minute interval 9 intervals from 22:45 to 23:25 UTC; average 5,080 TPS, highest 6,235 TPS. The merged mixed window ran from 22:45 to 23:25 UTC. TPS 0 2,000 4,000 6,000 8,000 22:45–22:50 UTC: 6,235 TPS, 9.7 generators active 22:50–22:55 UTC: 5,362 TPS, 10.0 generators active 22:55–23:00 UTC: 5,547 TPS, 10.0 generators active 23:00–23:05 UTC: 5,482 TPS, 10.0 generators active 23:05–23:10 UTC: 5,402 TPS, 10.0 generators active 23:10–23:15 UTC: 5,141 TPS, 10.0 generators active 23:15–23:20 UTC: 5,119 TPS, 10.0 generators active 23:20–23:25 UTC: 5,097 TPS, 10.0 generators active 23:25–23:25 UTC: 2,338 TPS, 8.0 generators active avg 5,080 22:45 UTC 22:55 UTC 23:05 UTC 23:15 UTC 23:25 UTC C1 2026-11-13 23:08:59 UTC: docker kill -s KILL telcoin on bench-validator-05 (systemctl stop holds it down (unit has Restart=on-failure). bench-blaster-05 targets ONLY this node, so its sends fail for the outage by design; the generator resends on connection errors) C1 C2 2026-11-13 23:13:59 UTC: systemctl start telcoin on bench-validator-05 (docker start is impossible: the unit runs docker run --rm) C2 C3 2026-11-13 23:18:59 UTC: systemctl restart telcoin on bench-observer-11 (observer restart) C3

Each bar is the average chain TPS over one 5-minute interval of the merged mixed window (22:45–23:25 UTC); fainter bars had fewer generators active than the maximum, the dashed line is the average of all bars and amber lines mark the chaos events listed below.

  • C1 2026-11-13 23:08:59 UTC: docker kill -s KILL telcoin on bench-validator-05 (systemctl stop holds it down (unit has Restart=on-failure). bench-blaster-05 targets ONLY this node, so its sends fail for the outage by design; the generator resends on connection errors)
  • C2 2026-11-13 23:13:59 UTC: systemctl start telcoin on bench-validator-05 (docker start is impossible: the unit runs docker run --rm)
  • C3 2026-11-13 23:18:59 UTC: systemctl restart telcoin on bench-observer-11 (observer restart)
5-minute data (9 rows)
start_utcend_utcgenerators_activechain_tps
2026-11-13T22:45:46Z2026-11-13T22:50:46Z9.76,234.9
2026-11-13T22:50:46Z2026-11-13T22:55:46Z10.05,361.5
2026-11-13T22:55:46Z2026-11-13T23:00:46Z10.05,547.0
2026-11-13T23:00:46Z2026-11-13T23:05:46Z10.05,482.1
2026-11-13T23:05:46Z2026-11-13T23:10:46Z10.05,401.7
2026-11-13T23:10:46Z2026-11-13T23:15:46Z10.05,141.2
2026-11-13T23:15:46Z2026-11-13T23:20:46Z10.05,119.1
2026-11-13T23:20:46Z2026-11-13T23:25:46Z10.05,097.0
2026-11-13T23:25:46Z2026-11-13T23:25:56Z8.02,338.5

The fine-grained chain throughput series (its step_s field gives the seconds per point) is published as series/tps-1s.json.

Other networks

MeasureTelcoin Network (this run)Tempo
Throughput5,394.6 tx/s (Avg chain TPS)16,455 TPS, shown as "Settled TPS" for the nightly run of 5 Oct 2026 (commit de5f5d3). perf.tempo.xyz describes it as "Median settled throughput across 130 measured blocks."
How throughput is measuredChain transactions per block-timestamp second, summed over the 2390-s window when every load generator was in its mixed phase and divided by its length.The methodology page defines TPS as "Transactions per second (derived from gas / avg tx gas)": gas used divided by the average gas of one transaction, reported as the median over the measured blocks.
Run length40.2 min merged mixed window; each generator was set to a 40 min mixed phase2m 39s for the 5 Oct 2026 nightly run (130 measured blocks at a 687 ms average block time), per perf.tempo.xyz.
Validators and placement10 validators and 1 observer across 10 regions: asia-east2 (Hong Kong), asia-northeast1 (Tokyo), asia-northeast3 (Seoul), asia-south1 (Mumbai), asia-southeast1 (Singapore), australia-southeast1 (Sydney), europe-west2 (London), europe-west4 (Netherlands), us-east4 (Ashburn), us-west2 (Los Angeles); load generators: one blaster per validator, in that validator's zone, sending only to itperf.tempo.xyz labels the scenario "Public mix · multi-region". The methodology page says "Benchmarks run two local validators in consensus on a dedicated bare-metal server with 100 GiB pre-bloated state." The public tempoxyz/tempo repo has nightly workflows for both kinds of setup: bench-e2e runs on one bare-metal runner, and bench-e2e-multi-region defaults to 10 validators on cloud VMs (its regions input reads "Empty uses the selected cloud's five defaults."). The run page does not say which setup produced this number.
Hardwarec3-highcpu-8, 8 vCPUs, 16 GB memory; disk: hyperdisk-balanced, 100 GB, 20,000 IOPS, 800 MiB/sThe methodology page lists a dedicated bare-metal server: "AMD EPYC 4585PX (16C/32T, 4.3/5.7 GHz)", "128 GB 3600 MHz" memory and "2×960 GB NVMe + 2×1.92 TB NVMe via schelk". The repo's multi-region workflow uses cloud VMs instead (c4d-standard-48-lssd on GCP, c6id.8xlarge on AWS).
Workloadbatched_self_send, contract_deploy, faucet_drip, native_transfer, stablecoin_approve, stablecoin_transfer, uniswap_v2_add_liquidity, uniswap_v2_remove_liquidity, uniswap_v2_swap, uniswap_v3_swap, uniswap_v4_swap, wtel_unwrap, wtel_wrap; 10 generators × 600 senders eachperf.tempo.xyz labels the scenario "Public mix"; the methodology page says "100% TIP-20 token transfers (ERC-20 equivalent)". The run used 1,000 accounts and 4 TIP-20 tokens.
StateFresh chain from genesis for every runThe methodology page says "100 GiB pre-bloated state".
Latency (as each source defines it)Time to land, median: 0.75 s; Time to land, p95: 6.79 s; Commit latency, p95: 0.98–0.98 sBlock time of 687 ms, which the methodology page defines as "Average wall-clock time between blocks". The run page gives no per-transaction latency. Separately, tempo.xyz says "Blocks finalize in ~0.6 seconds with no re-orgs." (a network claim, not a figure from this benchmark).
Source and fetch dateThis page; every figure names its source field.perf.tempo.xyz nightly run of 5 Oct 2026 (commit de5f5d3), the perf.tempo.xyz methodology page, tempo.xyz, and the benchmark workflows in the tempoxyz/tempo repo. All fetched on 5 Oct 2026.

Sources for Tempo (fetched 2026-10-05):

Tempo's setup may not match ours: its methodology page describes two validators on one bare-metal server, while our runs use ten validators across five regions, so read this table as context, not a ranking.

This table is context, not a ranking.

Caveats

  • Against r20261113-1351 (ranges are that run's own variation): blast 2 throughput 5,394.6 against 5,322.1 tx/s (+1.4 %; its five-minute blocks 4,718.8 to 5,616.9) and 5,030.1 against 5,066.9 tx/s without generator 05; blast 1 (600 x 4) 6,242.5 against 6,380.8 tx/s (-2.2 %; its 60-s blocks 6,025.2 to 6,638.3); time to seen p50/p95 12.89/28.76 s against 12.99/30.89 s; time to land p95/p99 6.79/28.39 s against 14.66/61.91 s; executed rate per node 5,399 against 5,314 tx/s; engine task 184.5 against 187.2 µs per transaction; device writes 0.999 against 0.998 of 20,000 a second, 3.71 against 3.74 per transaction; transactions per output 4,072 against 3,854 (+5.7 %). With one run per form, a difference inside the always-on run's range is not told apart from noise. Why the land tail was shorter here was not measured.
  • The one throughput exception is the floor blast: 684.7 tx/s against 700.3, 9.5 tx/s under the always-on run's lowest block. Generators 02 and 06 landed 3 % of the other eight's median against 11 % and 12 %, with no resends and no skipped transactions in either run. In blasts 0 and 1 the generators' start spread was 59 to 89 s against 23 to 48 s in r20261113-1351, as generators 02 and 06 began their mixed phase late after losing setup transactions in the preamble; the cause was not traced. The floor's seen p50 was 1.63 s against 1.65 s.
  • At 20,000 IOPS both forms ran the device at its write limit with the engines busy 994 to 996 ms a second; one analysis calls this disk-bound and another engine-bound with the device at its limit, and the measurements agree. In minutes 30 to 39 this run executed 5,103 tx/s against 4,839 (+5.5 %), with outputs 14.9 % larger: at the pinned device, larger outputs cost fewer writes per transaction.
  • Over blast 2's mixed phase the fleet certified 99.64 % of own headers against 99.94 %; the lowest validator was validator-02 at 99.0 % against validator-06 at 99.7 %. Before the kill validators 02, 06 and 10 certified 98.9, 98.9 and 98.8 % against 100, 99.8 and 99.9 %, outside the always-on run's five-minute range for each. In blast 2's first 120 s the fleet certified 97.98 % against 100.0 %, with six validators under 99 % (lowest validator-02, 96.3 %). Votes refused as execution-behind: 15 against 0. Vote timeouts: none in either run, and the voter's lock wait was 0 at the 95th percentile in both.
  • Before the kill the adaptive form named the author's own tip on 66.1 % of headers (always-on 0.2 %), on average 1.21 outputs behind the tip (about 0.9 s) against 7.59 (about 5.5 s). Validators 02, 06 and 10 had 95.7 to 97.4 % of their own headers inside a starved window and entered one 78 to 83 times an hour; inside they certified 99.81 to 99.95 %, outside 56.6 to 77.9 %, and every return to the own tip lost a header (1.19 to 1.39 per return). That a one-output lead fails the tip header within a 353-ms round is inferred. The slow voters (05, 04 and 03) answered 'superseded' 3,164 times before the kill against 85, on others' own-tip headers; 2,747 of the 3,148 answers joined to a header fell on headers that certified anyway. Rounds were 1.1 % slower before the kill: the certify stage took 14 ms longer, as the window's quorum headers certified later (p50 0.287 against 0.171 s), and the proposer's minimum delay absorbed most of it. The lead over the seventh validator never reached the floor at this disk (0.01 to 0.02 % of headers), so the floor had nothing to absorb, and the always-on rule on this run's tips predicts 99.8 to 100 % per validator (arithmetic). What the fresher named block is worth downstream was not measured.
  • In blast 2's first 120 s the adaptive form certified less on both disks. No analysis joined that window per header. The window state resets at each epoch and after 64 certified tip headers, which fits every author naming its own tip when the load begins; that was not tested.
  • Blast 2: gap resends 104,280 against 29,546, from eight generators against three (02, 06 and 10); 'nonce too high' skips 52,588 against 13,036; validator-minutes with an exposed lost own header 104 against 23; transactions unlanded after the 180-s tail scan 10,767 against 183 (9,290 against 0 with generator 05 left out); landings seen 300 s or more after submission 111,508 against 40,263; landed over accepted 99.9267 % against 99.9987 %. The adaptive form had 3.54 times the resends (generator 05 left out), 4.03 times the skips and 59 times the unlanded (all generators). Blast 1 (600 x 4) also had 2,016 gap resends and 5,875 unlanded transactions against 0 and 1,665. The mechanism is the same in all four runs of this program and is tracked as issue #1807: when an own header is cancelled or committed late, its batches are queued again behind batches a newer own header already carried, so the sender's later transactions are skipped at execution as 'nonce too high', sit in no pool, and land only after the generator resends its window at its 300-s gap check. The pool switches sealed_tx_keep and digest_dedupe were off in all four runs; whether they change this cost was not measured in these runs (in the earlier arm-walk run, sealed_tx_keep with three other switches on left no gap resends, a reading confounded with the other switches and an epoch boundary). Every gap-resend second of generators 01-04 and 06-10 fell 280 to 345 s after a lost own header of the generator's own validator (124 of 124 seconds here, 31 of 31 in r20261113-1351). The first resend wave opened from the near validators' cancelled own-tip headers in blast 2's warm-up (22:43:59-22:45:10). The adaptive form exposed the loss on more validators in more minutes; that this breadth sets the totals is an inference. The whole difference in the senders' cost between the forms sits in that reordering loss, a mechanism both forms share.
  • Validator-05 rejoined by the same rule: 22 min 39.8 s from the kill to active against 21 min 40.7 s, a difference the schedule explains (the load ran 58 s longer after the kill), and 5 min 02.8 s from the cool-down's end to promotion against 5 min 01.9 s. All six epoch closes certified on one record hash and one closing block on all eleven nodes; the fewest signers per epoch were 10, 10, 9, 7, 8 and 9 (always-on 10, 10, 7, 7, 9 and 9), the first quorum at the two blast-2 closes came at 14.85 and 13.85 s (16.44 and 14.75 s), and the loaded closes' highest execution lag was 20, 38 and 42 rounds (38, 52 and 48). No node logged a halt, fork, unexpected execution result, fatal database text or panic.
  • The restarted observer took 6 min 00.4 s to its first line at distance 1 and 7 min 12 s to follow output by output, against 23.3 s in r20261113-1351. The adaptive run at 50,000 IOPS (r20261114-0000) took 5.1 s, so the data does not tie this to the form; the cause is unknown, nothing was lost, and closing blocks and record hashes matched.
  • The earlier arm-walk run of the same build (r20261112-1520) had read the always-on form at 84.6 % in its warm-up against the adaptive form's 96.3 %, at different cache states. That order did not recur here, where the adaptive form was lower in blast 2's first 120 s on both disks; the windows differ.
  • The teardown's provenance check warns 'node startup provenance mismatch' in all four runs: each node's start during staging ran before the arm file was applied. It is known and not a finding.
  • At 20,000 IOPS the always-on form is ahead on both own-header certification and the senders' cost. The adaptive form's gain is the fresher named block, whose value downstream was not measured; it paid for it with about 1.1 % of headers lost to the probe on each windowed author and 3.5 times the resends.
  • Not measured: run-to-run variance at this disk, when a vote reaches its author (not logged), what the age of the named block costs, the senders' cost with sealed_tx_keep or digest_dedupe on, why the land tail was shorter here, and the cause of the late generator starts. The probe's parameters (the 128-round bound and the 64-header streak) were not varied.
  • Each configuration ran once (one run per form and disk), so run-to-run variation is not measured. The form of the vote fix is not decided: at 20,000 IOPS the always-on form certified a little more and cost the senders less, at 50,000 IOPS neither form held certification, and the floor's depth has been tested only as arithmetic. A further run of both forms is planned on a build of the newer upstream main branch, which uses a different execution storage engine (tn-lmdb) than these runs (MDBX), with a candidate fix for issue #1807 added; its result is not known.
  • Per-validator rows cover 9 validators; bench-validator-05 excluded as chaos target(s).
  • Commit_latency_p95_s: no tn_primary_commit_latency_seconds.p95 points for bench-validator-07.
  • Execution_time_per_output_s: no tn_engine_execution_duration_seconds.mean points for bench-validator-07.
  • Recovery: demoted / rejoined are the first 15-s Prometheus samples showing the change, so they trail the node's own log times by up to a scrape interval plus one step.
  • Recovery: no tn_batch_builder_seal_failures_total count for bench-validator-05; failed_reports unavailable.
  • The harness working tree had uncommitted changes when the run was deployed.
  • Before the coordinated start each node ran once during staging with every node switch at its default (11 start(s) on 11 node(s)); the run's switches were on from the coordinated start at 2026-11-13T21:38:28Z.

Full report

Full report HTML · 1.3 MB

Changes from the original report:

  • The site adds a banner at the top of the report that links back to this page.

SHA-256 of the committed file, before the banner is added: a011a502ba702f6f01348da65ffa109e4e136ca5094c9de2d76a822abf6173e6