The adaptive vote fix on Hyperdisk Balanced at 20,000 IOPS
2026-11-13 · started 21:18 UTC · ended 23:52 UTC · merged mixed window 40 min 10 s
We repeated r20261113-1351 with one change: the vote fix ran in its adaptive form instead of the always-on form. In the adaptive form a header names the author's own executed tip until one of the author's headers is cancelled uncertified; the author then uses the always-on quorum rule, never more than 8 outputs behind its tip, inside a starved window of up to 128 rounds, and afterwards tries its own tip again. The build (2df831e3c), the other three switches, bench 5's fleet and disk (Hyperdisk Balanced at 20,000 provisioned IOPS), the load and the chaos plan were the same. Throughput and what binds did not change: average chain throughput with all ten generators running was 5,394.6 tx/s against 5,322.1 (+1.4 %, inside the always-on run's own five-minute range), the disks again sat at their limit of 20,000 writes a second (0.999) with every engine busy, and the median time to seen was 12.89 s against 12.99 s. The epoch closes and the killed validator's rejoin matched too. Own-header certification was a little lower: 99.64 % over the load blast against 99.94 %, and before the kill validators 02, 06 and 10 (London, the Netherlands and Mumbai) sat at 98.8 to 98.9 %, about 1 point under and outside the always-on run's range for them. The loss is the adaptive form's probe: those three spent 95.7 to 97.4 % of their headers in starved windows, and every return to the own tip lost a header. In exchange, headers named a block about 0.9 s old on average, against 5.5 s. The senders paid more: 104,280 transactions were resent after the generators' 300-s gap check against 29,546, and 10,767 never landed against 183, through the same reordering of re-queued batches as under the always-on form (issue #1807), exposed on more validators in more minutes. No node logged a halt, fork, unexpected execution result, fatal database error or panic. This run is one of four in one program, one node build with two forms of the vote fix on two disks: the always-on form as r20261113-1351 (20,000 IOPS) and r20261113-1630 (50,000 IOPS), the adaptive form as r20261113-2118 (20,000 IOPS) and r20261114-0000 (50,000 IOPS). Across them the form changed neither throughput nor what binds; the always-on form certified slightly more own headers at 20,000 IOPS, neither form held certification at 50,000 IOPS (the adaptive run certified more), the adaptive form's senders paid more on both disks through the same reordering (issue #1807), and which form goes upstream is not decided.
Compared against r20261113-1351.
Key figures
How it was run
Software
| Upstream commit | telcoin-network 2df831e3ca34c06a361a7acca10397787e7b332e · bench/bench6-2026-11 · commit verified from node startup logs |
|---|---|
| Node image | us-docker.pkg.dev/telcoin-network/tn-public/adiri@sha256:874537ff1cd8d496140154837e7a36524978bdbf3b0440e48972b073d9852745 |
| Node image digest | sha256:874537ff1cd8d496140154837e7a36524978bdbf3b0440e48972b073d9852745 |
| Node version | 0.1.0 |
| Fork schedule | subsecond-from-genesis |
| Load generator commit | 62bbd5f |
| Load generator image digest | sha256:7cefcc505201b2e40361a7d8cfe034778090b76c568cb2400369bf566c605a59 |
| Analysis commit | 62bbd5f |
| Harness commit | 4171b897c930d80ba5ce170c79dc21193dc9c213 |
| Harness submodules | common: 1a6955d8a2c6f19e53cf7aaad5e32a9107a21a7d tn-node-deployment: d9446f720026fa5ce8b64fce9074e56b05b3f4eb |
Fleet
| Fleet | c3-highcpu-8 + Hyperdisk Balanced at 20,000 IOPS (c3-hdb), ten regions, validator-05 in Ashburn |
|---|---|
| Validators | 10 |
| Observers | 1 |
| Machine type | c3-highcpu-8 |
| vCPUs per node | 8 |
| Memory per node | 16 GB |
| Disk | hyperdisk-balanced, 100 GB, 20,000 IOPS, 800 MiB/s |
| Regions | asia-east2 (Hong Kong), asia-northeast1 (Tokyo), asia-northeast3 (Seoul), asia-south1 (Mumbai), asia-southeast1 (Singapore), australia-southeast1 (Sydney), europe-west2 (London), europe-west4 (Netherlands), us-east4 (Ashburn), us-west2 (Los Angeles) (10) |
| Load generators | 10 generators × e2-standard-2 × 600 senders each |
| Load placement | one blaster per validator, in that validator's zone, sending only to it |
Chain
| Chain ID | 32286 |
|---|---|
| Network | bench |
| Epoch length | 1,200 s (20 min) |
Consensus parameters
| Max header delay | 1,000 ms |
|---|---|
| Min header delay | 350 ms |
| Max batch delay | 250 ms |
| Vote timeout | 5,000 ms |
| Parameters file | consensus-parameters.yaml |
| Confirmed by | node startup logs |
Parameter arms
| Arm | From (UTC) | To (UTC) | Values |
|---|---|---|---|
| final-adaptive | 2026-11-13 21:38:28 | — | all 11 nodes: epoch_record_state_check=true, mdbx_random_advice=true, quorum_exec_block=true, quorum_exec_block_starved_rounds=4, vote_queue_fix=true |
Workload
| Warm-up | 120 s (2 min) |
|---|---|
| Mixed phase | 2,400 s (40 min) |
| Cool-down | 180 s (3 min) |
| Transaction types | batched_self_send, contract_deploy, faucet_drip, native_transfer, stablecoin_approve, stablecoin_transfer, uniswap_v2_add_liquidity, uniswap_v2_remove_liquidity, uniswap_v2_swap, uniswap_v3_swap, uniswap_v4_swap, wtel_unwrap, wtel_wrap |
Chaos events
| Marker | Time (UTC) | Node | Action | Detail |
|---|---|---|---|---|
| C1 | 2026-11-13 23:08:59 | bench-validator-05 | docker kill -s KILL telcoin | systemctl stop holds it down (unit has Restart=on-failure). bench-blaster-05 targets ONLY this node, so its sends fail for the outage by design; the generator resends on connection errors |
| C2 | 2026-11-13 23:13:59 | bench-validator-05 | systemctl start telcoin | docker start is impossible: the unit runs docker run --rm |
| C3 | 2026-11-13 23:18:59 | bench-observer-11 | systemctl restart telcoin | observer restart |
Steps
- All four runs of this program used one node image, built from the branch bench/bench6-2026-11 at commit 2df831e3c (a base commit 165e3fe01 plus 98 commits), with sub-second timestamps from genesis, and named by registry digest (adiri@sha256:874537ff…852745). Nothing was rebuilt between the runs. Every fix in that build is a node switch, off by default; a node logs its switches at every start ('node switches') and exports them as the metric tn_node_switch. The load generator was built from tn-transaction-generator commit 62bbd5f.
- The vote fix is the switch quorum_exec_block. Instead of its own newest executed block, a proposer names in each header a block that a quorum of validators should already have executed; a voter signs only after it finds the named block in its own chain. The setting quorum_exec_block_starved_rounds picks the form. With 0, the always-on form, every header takes the quorum value over the author's own executed tip and the blocks its parent headers named, capped at the own tip and never more than 8 outputs behind it (the floor); in practice nearly every header named the author's tip minus 8 outputs. With 4, the adaptive form, a header names the author's own tip until one of the author's headers is cancelled uncertified; the author then uses the always-on rule inside a starved window of up to 128 rounds, and afterwards tries its own tip again.
- Three more switches were on in all four runs: vote_queue_fix (the voter's lock wait), epoch_record_state_check (a check of the closing state when an epoch record is built) and mdbx_random_advice (the MDBX database's random-access hint over the whole file). The other eight stayed at their defaults: mdbx_no_writemap, sealed_tx_keep, digest_dedupe, recovery_threads, epoch_close_task_drain, pipelined_persist, rpc_admission_backlog_outputs and parent_grace. The switch set came from an arm file applied at deployment, before the nodes' coordinated start: no arm walk, no cache eviction and no restart before the load. Each node first ran once, alone, during staging (17 to 90 s, every switch at its default, no peer reached, no output executed); the switches of the run were on from the coordinated start, the first process that executed blocks.
- Each run created ten validators and one observer on c3-highcpu-8 (8 vCPU, 16 GiB), each node with one 100 GB Hyperdisk Balanced disk for the root and the data directory at 800 MiB/s, provisioned at 20,000 IOPS (bench 5's disk) or 50,000 IOPS. One validator ran in each of bench 5's ten regions (Los Angeles, London, Sydney, Tokyo, Ashburn, the Netherlands, Singapore, Hong Kong, Seoul and Mumbai), the observer in Los Angeles. Epochs last 1,200 s.
- Each run had three blasts. Ten generators on e2-standard-2, one per validator in that validator's zone and sending only to it, ran blast 0, the floor, with 8 senders each and up to 16 transactions in flight per sender (8 x 16); blast 1 with 600 senders and a window of 4 (600 x 4, at most 24,000 in flight); and blast 2, the load blast, with 600 senders and a window of 16 (600 x 16, at most 96,000 in flight) and phases of 120 s warm-up, 2,400 s mixed load and 180 s cool-down, the shape of bench 5's load blast.
- The chaos plan was bench 5's, inside blast 2: validator-05's container was killed with SIGKILL shortly before an epoch boundary and held down for 300 s, so the outage crossed the boundary; then it was started, and the observer was restarted 300 s after that. Generator 05 sends only to validator-05 and does not fail over to another endpoint, so it is cut off for the outage and for as long as its validator follows the chain afterwards.
- Headline figures keep bench 5's definitions and code; bench 5's throughput, latency and landing figures quoted on these pages were recomputed exactly from bench 5's own generator documents with the same code, while its certification, disk, stage and rejoin figures come from bench 5's report or from this program's reading of bench 5's logs and model. Throughput is the chain's transactions per second over the window in which all ten generators ran, with the outage and generator 05 inside it. Time to seen runs from submission until the generator saw the receipt, from the generators' summed histograms (the bundle method); time to land is to the block's timestamp, in whole seconds, so about 0.5 s low. Own-header certification is the share of a validator's own headers that were certified, over blast 2's mixed phase unless another window is named.
- The analysis read every node's whole main log (the node's stderr file was not pulled) and the node metrics exported from Prometheus before each teardown. The comparison of the two forms ran the same scripts on both runs of a disk and reproduced the always-on runs' figures.
- Against r20261113-1351 only the form of the vote fix changed: the arm file final-adaptive set quorum_exec_block_starved_rounds to 4 instead of 0, with the same other switches. The image, the generator, the fleet, the disk, the load and the chaos plan are the same.
- A first deployment, r20261113-2044, stopped before the coordinated start and before the arm was applied: validator-08's first start, during staging, failed with 'Address already in use' on its worker listener. It was torn down and is not part of the run.
- Times are UTC on 13 November. Deployment ran from 21:18:25 to 22:02:14 (the second deployment); all eleven nodes started at one due time, with 0 s spread. The fsync probe median was 1,043 µs, with validators 04 and 07 flagged slow. The mixed phases ran 22:12:55-22:19:56 (blast 0), 22:25:37-22:33:06 (blast 1) and 22:45:46-23:25:56 (blast 2). Validator-05 was killed at 23:08:59 and started at 23:13:59, epoch 5 opened during its outage at 23:09:58, and the observer was restarted at 23:18:59.
- The teardown ended with rc 0 at 23:52:55, 12 instances deleted and none left. The archive holds 625 files, 2.48 GB. The metric export before the teardown holds 382 metrics (21,023 series) but lacks validator-07's vote-stage series, so the voter's lock wait here comes from the log, and no comparison figure rests on that series alone. Before the kill the mixed phase ran 1,393 s here against 1,477 s in r20261113-1351; where windows differ, only rates and shares are compared.
The harness scripts live in a private repository; the steps above describe what they do.
Results
| Metric | Value | Window | Source | Method |
|---|---|---|---|---|
| Average chain throughput, all load generators running | 5,394.6 tx/s | all generators active | combined.series.per_second[chain_tx] | Chain transactions per block-timestamp second, summed over the 2390-s window when every load generator was in its mixed phase and divided by its length. |
| Best 60 seconds, all load generators running | 7,113.8 tx/s | all generators active | combined.series.per_second[chain_tx] | Highest mean chain tx/s over any 60 consecutive seconds inside the all-generators window (aggregate.py best_window_mean). |
| Transactions landed over the whole blast | 14,736,496 tx | whole run | combined.totals.landed_chain | Sum of tx_count over every block the load generators saw during the blast, own and foreign transactions alike. |
| Blocks per second | 23.85 blocks/s | merged mixed window | combined.blocks.rows[timestamp] | Chain blocks whose timestamp falls in the merged mixed window, divided by its length (aggregate.py landed.capacity.blocks_per_sec over the same window). |
| Time to land, median | 0.75 s | whole run | combined.latency.ts_ms.p50 | Block timestamp minus submit time over every landed own transaction in the blast, from the load generators' merged latency histograms (aggregate.py merge_latency), in seconds. |
| Time to land, p95 | 6.79 s | whole run | combined.latency.ts_ms.p95 | Block timestamp minus submit time over every landed own transaction in the blast, from the load generators' merged latency histograms (aggregate.py merge_latency), in seconds. |
| Time to land, median, whole-second bias removed | 1.25 s | whole run | combined.latency.ts_ms.p50 + 0.5 s | Time to land, median, plus 0.5 s. The block timestamp is the committing leader's header creation time cut to the whole second, so every sample reads 0 to 999 ms low, 0.5 s on average. |
| Time to seen, median | 12.89 s | whole run | combined.latency.seen_ms.p50 | Submit time to the moment the load generator's block scanner had the block holding the transaction (block and receipts fetched from the generator's own node), over every landed own transaction in the blast, from the load generators' merged latency histograms (aggregate.py merge_latency), in seconds. |
| Time to seen, p95 | 28.76 s | whole run | combined.latency.seen_ms.p95 | Submit time to the moment the load generator's block scanner had the block holding the transaction (block and receipts fetched from the generator's own node), over every landed own transaction in the blast, from the load generators' merged latency histograms (aggregate.py merge_latency), in seconds. |
| Time to land at light load, median | 0.25 s | light load, 8 senders per generator | blast-0-20261113T222217Z/blasters latency.ts_ms.p50 | Block timestamp minus submit time over every landed own transaction submitted in the mixed phase of the light-load blast light8 (index 0: 8 senders per load generator at window 16, chaos off), pooled from the generators' latency histograms with the aggregate.py merge_latency method, in seconds; not bias-corrected. |
| Time to seen at light load, median | 1.63 s | light load, 8 senders per generator | blast-0-20261113T222217Z/blasters latency.seen_ms.p50 | Submit time to the generator's block scanner having the block over every landed own transaction submitted in the mixed phase of the light-load blast light8 (index 0: 8 senders per load generator at window 16, chaos off), pooled from the generators' latency histograms with the aggregate.py merge_latency method, in seconds; not bias-corrected. |
| Commit latency, p95 | 0.98–0.98 s | merged mixed window | metrics.prometheus.series.tn_primary_commit_latency_seconds.by_node[healthy validators].p95 | For each healthy validator, the nearest-rank p95 over the merged mixed window of its 15-s histogram_quantile(0.95) commit-latency series; reported as the range over validators. |
| Execution time per consensus output | 0.74–0.74 s | merged mixed window | metrics.prometheus.series.tn_engine_execution_duration_seconds.by_node[healthy validators].mean | For each healthy validator, the nearest-rank median over the merged mixed window of its 15-s mean execution-duration series; reported as the range over validators. |
| Write throughput, p95, busiest validator | 105 MiB/s | merged mixed window | metrics.gcp[bench-validator-02].disk_write_bytes_per_sec p95 | Nearest-rank p95 of the 60-s mean write-byte rate over the merged mixed window for each healthy validator; the highest (bench-validator-02) is reported. |
| Write operations, p95, busiest validator | 20,098 IOPS | merged mixed window | metrics.gcp[bench-validator-04].disk_write_ops_per_sec p95 | Nearest-rank p95 of the 60-s mean write-operation rate over the merged mixed window for each healthy validator; the highest (bench-validator-04) is reported. |
| CPU, p95, busiest validator | 35.2 % | merged mixed window | metrics.gcp[bench-validator-03].cpu_utilization p95 | Nearest-rank p95 of the 60-s CPU utilisation over the merged mixed window for each healthy validator; the highest (bench-validator-03) is reported. |
| Node resident memory, p95, busiest validator | 6.14 GiB | merged mixed window | metrics.gcp[bench-validator-02].telcoin_rss_bytes p95 | Nearest-rank p95 of the 60-s telcoin process resident memory over the merged mixed window for each healthy validator; the highest (bench-validator-02) is reported. |
| IO pressure, highest reading, range over validators | 12.0–14.1 % | merged mixed window | sampler[healthy validators].psi_io_some_avg10 max | For each healthy validator, the highest 10-s 'some' IO pressure reading the on-node sampler logged inside the merged mixed window; reported as the range over validators. |
Recovery
| Node | bench-validator-05 |
|---|---|
| Killed | 2026-11-13 23:08:59 UTC |
| Restarted | 2026-11-13 23:13:59 UTC |
| Rejoined | 2026-11-13 23:31:49 UTC |
| Catch-up time | 1,050 s (17 min 30 s) |
| Failed batch reports | not recorded |
Chain TPS per 5-minute interval
Each bar is the average chain TPS over one 5-minute interval of the merged mixed window (22:45–23:25 UTC); fainter bars had fewer generators active than the maximum, the dashed line is the average of all bars and amber lines mark the chaos events listed below.
- C1 2026-11-13 23:08:59 UTC: docker kill -s KILL telcoin on bench-validator-05 (systemctl stop holds it down (unit has Restart=on-failure). bench-blaster-05 targets ONLY this node, so its sends fail for the outage by design; the generator resends on connection errors)
- C2 2026-11-13 23:13:59 UTC: systemctl start telcoin on bench-validator-05 (docker start is impossible: the unit runs docker run --rm)
- C3 2026-11-13 23:18:59 UTC: systemctl restart telcoin on bench-observer-11 (observer restart)
5-minute data (9 rows)
| start_utc | end_utc | generators_active | chain_tps |
|---|---|---|---|
| 2026-11-13T22:45:46Z | 2026-11-13T22:50:46Z | 9.7 | 6,234.9 |
| 2026-11-13T22:50:46Z | 2026-11-13T22:55:46Z | 10.0 | 5,361.5 |
| 2026-11-13T22:55:46Z | 2026-11-13T23:00:46Z | 10.0 | 5,547.0 |
| 2026-11-13T23:00:46Z | 2026-11-13T23:05:46Z | 10.0 | 5,482.1 |
| 2026-11-13T23:05:46Z | 2026-11-13T23:10:46Z | 10.0 | 5,401.7 |
| 2026-11-13T23:10:46Z | 2026-11-13T23:15:46Z | 10.0 | 5,141.2 |
| 2026-11-13T23:15:46Z | 2026-11-13T23:20:46Z | 10.0 | 5,119.1 |
| 2026-11-13T23:20:46Z | 2026-11-13T23:25:46Z | 10.0 | 5,097.0 |
| 2026-11-13T23:25:46Z | 2026-11-13T23:25:56Z | 8.0 | 2,338.5 |
The fine-grained chain throughput series (its step_s field gives the seconds per point) is published as series/tps-1s.json.
Other networks
| Measure | Telcoin Network (this run) | Tempo |
|---|---|---|
| Throughput | 5,394.6 tx/s (Avg chain TPS) | 16,455 TPS, shown as "Settled TPS" for the nightly run of 5 Oct 2026 (commit de5f5d3). perf.tempo.xyz describes it as "Median settled throughput across 130 measured blocks." |
| How throughput is measured | Chain transactions per block-timestamp second, summed over the 2390-s window when every load generator was in its mixed phase and divided by its length. | The methodology page defines TPS as "Transactions per second (derived from gas / avg tx gas)": gas used divided by the average gas of one transaction, reported as the median over the measured blocks. |
| Run length | 40.2 min merged mixed window; each generator was set to a 40 min mixed phase | 2m 39s for the 5 Oct 2026 nightly run (130 measured blocks at a 687 ms average block time), per perf.tempo.xyz. |
| Validators and placement | 10 validators and 1 observer across 10 regions: asia-east2 (Hong Kong), asia-northeast1 (Tokyo), asia-northeast3 (Seoul), asia-south1 (Mumbai), asia-southeast1 (Singapore), australia-southeast1 (Sydney), europe-west2 (London), europe-west4 (Netherlands), us-east4 (Ashburn), us-west2 (Los Angeles); load generators: one blaster per validator, in that validator's zone, sending only to it | perf.tempo.xyz labels the scenario "Public mix · multi-region". The methodology page says "Benchmarks run two local validators in consensus on a dedicated bare-metal server with 100 GiB pre-bloated state." The public tempoxyz/tempo repo has nightly workflows for both kinds of setup: bench-e2e runs on one bare-metal runner, and bench-e2e-multi-region defaults to 10 validators on cloud VMs (its regions input reads "Empty uses the selected cloud's five defaults."). The run page does not say which setup produced this number. |
| Hardware | c3-highcpu-8, 8 vCPUs, 16 GB memory; disk: hyperdisk-balanced, 100 GB, 20,000 IOPS, 800 MiB/s | The methodology page lists a dedicated bare-metal server: "AMD EPYC 4585PX (16C/32T, 4.3/5.7 GHz)", "128 GB 3600 MHz" memory and "2×960 GB NVMe + 2×1.92 TB NVMe via schelk". The repo's multi-region workflow uses cloud VMs instead (c4d-standard-48-lssd on GCP, c6id.8xlarge on AWS). |
| Workload | batched_self_send, contract_deploy, faucet_drip, native_transfer, stablecoin_approve, stablecoin_transfer, uniswap_v2_add_liquidity, uniswap_v2_remove_liquidity, uniswap_v2_swap, uniswap_v3_swap, uniswap_v4_swap, wtel_unwrap, wtel_wrap; 10 generators × 600 senders each | perf.tempo.xyz labels the scenario "Public mix"; the methodology page says "100% TIP-20 token transfers (ERC-20 equivalent)". The run used 1,000 accounts and 4 TIP-20 tokens. |
| State | Fresh chain from genesis for every run | The methodology page says "100 GiB pre-bloated state". |
| Latency (as each source defines it) | Time to land, median: 0.75 s; Time to land, p95: 6.79 s; Commit latency, p95: 0.98–0.98 s | Block time of 687 ms, which the methodology page defines as "Average wall-clock time between blocks". The run page gives no per-transaction latency. Separately, tempo.xyz says "Blocks finalize in ~0.6 seconds with no re-orgs." (a network claim, not a figure from this benchmark). |
| Source and fetch date | This page; every figure names its source field. | perf.tempo.xyz nightly run of 5 Oct 2026 (commit de5f5d3), the perf.tempo.xyz methodology page, tempo.xyz, and the benchmark workflows in the tempoxyz/tempo repo. All fetched on 5 Oct 2026. |
Sources for Tempo (fetched 2026-10-05):
- perf.tempo.xyz (nightly benchmark results)
- perf.tempo.xyz methodology
- tempo.xyz
- tempoxyz/tempo repository
- bench-e2e workflow (bare-metal)
- bench-e2e-multi-region workflow
Tempo's setup may not match ours: its methodology page describes two validators on one bare-metal server, while our runs use ten validators across five regions, so read this table as context, not a ranking.
This table is context, not a ranking.
Caveats
- Against r20261113-1351 (ranges are that run's own variation): blast 2 throughput 5,394.6 against 5,322.1 tx/s (+1.4 %; its five-minute blocks 4,718.8 to 5,616.9) and 5,030.1 against 5,066.9 tx/s without generator 05; blast 1 (600 x 4) 6,242.5 against 6,380.8 tx/s (-2.2 %; its 60-s blocks 6,025.2 to 6,638.3); time to seen p50/p95 12.89/28.76 s against 12.99/30.89 s; time to land p95/p99 6.79/28.39 s against 14.66/61.91 s; executed rate per node 5,399 against 5,314 tx/s; engine task 184.5 against 187.2 µs per transaction; device writes 0.999 against 0.998 of 20,000 a second, 3.71 against 3.74 per transaction; transactions per output 4,072 against 3,854 (+5.7 %). With one run per form, a difference inside the always-on run's range is not told apart from noise. Why the land tail was shorter here was not measured.
- The one throughput exception is the floor blast: 684.7 tx/s against 700.3, 9.5 tx/s under the always-on run's lowest block. Generators 02 and 06 landed 3 % of the other eight's median against 11 % and 12 %, with no resends and no skipped transactions in either run. In blasts 0 and 1 the generators' start spread was 59 to 89 s against 23 to 48 s in r20261113-1351, as generators 02 and 06 began their mixed phase late after losing setup transactions in the preamble; the cause was not traced. The floor's seen p50 was 1.63 s against 1.65 s.
- At 20,000 IOPS both forms ran the device at its write limit with the engines busy 994 to 996 ms a second; one analysis calls this disk-bound and another engine-bound with the device at its limit, and the measurements agree. In minutes 30 to 39 this run executed 5,103 tx/s against 4,839 (+5.5 %), with outputs 14.9 % larger: at the pinned device, larger outputs cost fewer writes per transaction.
- Over blast 2's mixed phase the fleet certified 99.64 % of own headers against 99.94 %; the lowest validator was validator-02 at 99.0 % against validator-06 at 99.7 %. Before the kill validators 02, 06 and 10 certified 98.9, 98.9 and 98.8 % against 100, 99.8 and 99.9 %, outside the always-on run's five-minute range for each. In blast 2's first 120 s the fleet certified 97.98 % against 100.0 %, with six validators under 99 % (lowest validator-02, 96.3 %). Votes refused as execution-behind: 15 against 0. Vote timeouts: none in either run, and the voter's lock wait was 0 at the 95th percentile in both.
- Before the kill the adaptive form named the author's own tip on 66.1 % of headers (always-on 0.2 %), on average 1.21 outputs behind the tip (about 0.9 s) against 7.59 (about 5.5 s). Validators 02, 06 and 10 had 95.7 to 97.4 % of their own headers inside a starved window and entered one 78 to 83 times an hour; inside they certified 99.81 to 99.95 %, outside 56.6 to 77.9 %, and every return to the own tip lost a header (1.19 to 1.39 per return). That a one-output lead fails the tip header within a 353-ms round is inferred. The slow voters (05, 04 and 03) answered 'superseded' 3,164 times before the kill against 85, on others' own-tip headers; 2,747 of the 3,148 answers joined to a header fell on headers that certified anyway. Rounds were 1.1 % slower before the kill: the certify stage took 14 ms longer, as the window's quorum headers certified later (p50 0.287 against 0.171 s), and the proposer's minimum delay absorbed most of it. The lead over the seventh validator never reached the floor at this disk (0.01 to 0.02 % of headers), so the floor had nothing to absorb, and the always-on rule on this run's tips predicts 99.8 to 100 % per validator (arithmetic). What the fresher named block is worth downstream was not measured.
- In blast 2's first 120 s the adaptive form certified less on both disks. No analysis joined that window per header. The window state resets at each epoch and after 64 certified tip headers, which fits every author naming its own tip when the load begins; that was not tested.
- Blast 2: gap resends 104,280 against 29,546, from eight generators against three (02, 06 and 10); 'nonce too high' skips 52,588 against 13,036; validator-minutes with an exposed lost own header 104 against 23; transactions unlanded after the 180-s tail scan 10,767 against 183 (9,290 against 0 with generator 05 left out); landings seen 300 s or more after submission 111,508 against 40,263; landed over accepted 99.9267 % against 99.9987 %. The adaptive form had 3.54 times the resends (generator 05 left out), 4.03 times the skips and 59 times the unlanded (all generators). Blast 1 (600 x 4) also had 2,016 gap resends and 5,875 unlanded transactions against 0 and 1,665. The mechanism is the same in all four runs of this program and is tracked as issue #1807: when an own header is cancelled or committed late, its batches are queued again behind batches a newer own header already carried, so the sender's later transactions are skipped at execution as 'nonce too high', sit in no pool, and land only after the generator resends its window at its 300-s gap check. The pool switches sealed_tx_keep and digest_dedupe were off in all four runs; whether they change this cost was not measured in these runs (in the earlier arm-walk run, sealed_tx_keep with three other switches on left no gap resends, a reading confounded with the other switches and an epoch boundary). Every gap-resend second of generators 01-04 and 06-10 fell 280 to 345 s after a lost own header of the generator's own validator (124 of 124 seconds here, 31 of 31 in r20261113-1351). The first resend wave opened from the near validators' cancelled own-tip headers in blast 2's warm-up (22:43:59-22:45:10). The adaptive form exposed the loss on more validators in more minutes; that this breadth sets the totals is an inference. The whole difference in the senders' cost between the forms sits in that reordering loss, a mechanism both forms share.
- Validator-05 rejoined by the same rule: 22 min 39.8 s from the kill to active against 21 min 40.7 s, a difference the schedule explains (the load ran 58 s longer after the kill), and 5 min 02.8 s from the cool-down's end to promotion against 5 min 01.9 s. All six epoch closes certified on one record hash and one closing block on all eleven nodes; the fewest signers per epoch were 10, 10, 9, 7, 8 and 9 (always-on 10, 10, 7, 7, 9 and 9), the first quorum at the two blast-2 closes came at 14.85 and 13.85 s (16.44 and 14.75 s), and the loaded closes' highest execution lag was 20, 38 and 42 rounds (38, 52 and 48). No node logged a halt, fork, unexpected execution result, fatal database text or panic.
- The restarted observer took 6 min 00.4 s to its first line at distance 1 and 7 min 12 s to follow output by output, against 23.3 s in r20261113-1351. The adaptive run at 50,000 IOPS (r20261114-0000) took 5.1 s, so the data does not tie this to the form; the cause is unknown, nothing was lost, and closing blocks and record hashes matched.
- The earlier arm-walk run of the same build (r20261112-1520) had read the always-on form at 84.6 % in its warm-up against the adaptive form's 96.3 %, at different cache states. That order did not recur here, where the adaptive form was lower in blast 2's first 120 s on both disks; the windows differ.
- The teardown's provenance check warns 'node startup provenance mismatch' in all four runs: each node's start during staging ran before the arm file was applied. It is known and not a finding.
- At 20,000 IOPS the always-on form is ahead on both own-header certification and the senders' cost. The adaptive form's gain is the fresher named block, whose value downstream was not measured; it paid for it with about 1.1 % of headers lost to the probe on each windowed author and 3.5 times the resends.
- Not measured: run-to-run variance at this disk, when a vote reaches its author (not logged), what the age of the named block costs, the senders' cost with sealed_tx_keep or digest_dedupe on, why the land tail was shorter here, and the cause of the late generator starts. The probe's parameters (the 128-round bound and the 64-header streak) were not varied.
- Each configuration ran once (one run per form and disk), so run-to-run variation is not measured. The form of the vote fix is not decided: at 20,000 IOPS the always-on form certified a little more and cost the senders less, at 50,000 IOPS neither form held certification, and the floor's depth has been tested only as arithmetic. A further run of both forms is planned on a build of the newer upstream main branch, which uses a different execution storage engine (tn-lmdb) than these runs (MDBX), with a candidate fix for issue #1807 added; its result is not known.
- Per-validator rows cover 9 validators; bench-validator-05 excluded as chaos target(s).
- Commit_latency_p95_s: no tn_primary_commit_latency_seconds.p95 points for bench-validator-07.
- Execution_time_per_output_s: no tn_engine_execution_duration_seconds.mean points for bench-validator-07.
- Recovery: demoted / rejoined are the first 15-s Prometheus samples showing the change, so they trail the node's own log times by up to a scrape interval plus one step.
- Recovery: no tn_batch_builder_seal_failures_total count for bench-validator-05; failed_reports unavailable.
- The harness working tree had uncommitted changes when the run was deployed.
- Before the coordinated start each node ran once during staging with every node switch at its default (11 start(s) on 11 node(s)); the run's switches were on from the coordinated start at 2026-11-13T21:38:28Z.
Full report
Changes from the original report:
- The site adds a banner at the top of the report that links back to this page.
SHA-256 of the committed file, before the banner is added: a011a502ba702f6f01348da65ffa109e4e136ca5094c9de2d76a822abf6173e6