Stage 2 of the latency programme: one-RTT votes, a 350 ms minimum header delay and the 40-minute chaos run
2026-10-07 · started 03:17 UTC · ended 19:56 UTC · merged mixed window 41 min 22 s
We kept the ten-region fleet of stage 1 (one c3-highcpu-8 validator in each of Los Angeles, Montréal, London, the Netherlands, Mumbai, Singapore, Hong Kong, Tokyo, Seoul and Sydney, reaching each other over their public addresses) and changed the node build in one place: an outbound request stream now negotiates its protocol lazily, so a vote request goes out with the protocol proposal instead of after a confirmation round trip. Two experiments ran on one deployment. First, a short re-check of stage 1's A3 timing (maximum header delay 1,000 ms, minimum 250 ms): each validator's time from its own proposal to its own quorum roughly halved (Seoul 264 to 138 ms, London 560 to 286 ms, the Netherlands 582 to 296 ms), the idle round fell from 530 to 313 ms, and an idle vote became wire time alone, the voter spending 1.5 to 1.6 ms on it where stage 1 had seen waits of 155 to 416 ms on the Europe-to-Asia links. Nine validators certified at least 99.9 % of their own headers while idle; the Netherlands certified 83.4 %, because its quorum arrives at 296 to 299 ms and the seventh parent certificate triggers its next proposal at about 304 ms. The programme's rule for a failed check set the minimum header delay to the slowest validator's quorum time plus 50 ms, 350 ms. Second, with every node at 1,000/350 ms, the 40-minute chaos run of 6 October (r20261006-0458) was repeated: ten load generators, 2,400 s of mixed load, and bench-validator-05 killed before an epoch close and held down for 300 s. Idle, every validator reached quorum on at least 99.9 % of its headers and the round was 353 ms on every node, about 2.8 rounds a second. Under load, with all ten generators running, average chain throughput was 5,434 tx/s against 3,890 tx/s on r20261006-0458 and 3,320 tx/s on the run that stalled (r20261006-0053); by the node logs the chain committed 1.39 rounds and 18.99 blocks a second over the mixed phase (0.84 and 13.56 on r20261006-0458); the median time for a transaction to land was 2.44 s and the 95th percentile 12.56 s (3.38 and 17.98 s; 3.17 and 26.07 s), and the generators landed 99.7 % of the 14.36 million transactions they submitted. Every validator certified 91.8 to 97.6 % of its own headers over the loaded 40 minutes, 94.9 % fleet-wide. The chain kept committing through the outage and the epoch closed on time, but the kill fired 41.8 s late, 17 s before the boundary instead of 59 s, and the restarted validator could not catch up while 5,400 transactions a second were landing: it followed about 6,000 blocks behind for 17 minutes and rejoined 24 minutes after its restart, once the load had stopped.
Compared against r20261006-0458.
Key figures
How it was run
Software
| Upstream commit | telcoin-network f8ff81bf472a47890ce0fff7065907dd4913be60 · devnet-bench-10-5-2026 · commit verified from node startup logs |
|---|---|
| Node image | us-docker.pkg.dev/telcoin-network/tn-public/adiri:mmap-subsecond-v1lazy-10-7-2026 |
| Node image digest | sha256:71c5b03bf050762a845fbe87ce201460e675a851b74c138a5a22d4fdfdf3d7cb |
| Node version | 0.1.0 |
| Fork schedule | subsecond-from-genesis |
| Load generator commit | 9f61303 |
| Load generator image digest | sha256:515311a6a8ffdde56ed190f85c11cee6abdec1434e2fff5e8ffd57d4a554eee2 |
| Analysis commit | 9f61303 |
| Harness commit | 427ecd7576a0a39804faf294a793da4e25842cc2 |
| Harness submodules | common: b498b71e107f09c7e6a7a68ae5ba71a514b98609 tn-node-deployment: 1f0385d79a99e008cadf66f0522f4417a09e7543 |
Fleet
| Fleet | c3-highcpu-8 + Hyperdisk Balanced (c3-hdb), one validator per region |
|---|---|
| Validators | 10 |
| Observers | 1 |
| Machine type | c3-highcpu-8 |
| vCPUs per node | 8 |
| Memory per node | 16 GB |
| Disk | hyperdisk-balanced, 100 GB, 20,000 IOPS, 800 MiB/s |
| Regions | asia-east2 (Hong Kong), asia-northeast1 (Tokyo), asia-northeast3 (Seoul), asia-south1 (Mumbai), asia-southeast1 (Singapore), australia-southeast1 (Sydney), europe-west2 (London), europe-west4 (Netherlands), northamerica-northeast1 (Montréal), us-west2 (Los Angeles) (10) |
| Load generators | 10 generators × e2-standard-2 × 600 senders each |
| Load placement | one blaster per validator, in that validator's zone, sending only to it |
Chain
| Chain ID | 32286 |
|---|---|
| Network | bench |
| Epoch length | 1,200 s (20 min) |
Consensus parameters
| Max header delay | 1,000 ms |
|---|---|
| Min header delay | 250 ms |
| Max batch delay | 250 ms |
| Vote timeout | 5,000 ms |
| Parameters file | consensus-parameters.yaml |
| Confirmed by | node startup logs |
Parameter arms
| Arm | From (UTC) | To (UTC) | Values |
|---|---|---|---|
| a3check | 2026-10-07 04:07:06 | 2026-10-07 04:45:36 | all 11 nodes: max_header_delay=1000ms, min_header_delay=250ms |
| stage2 | 2026-10-07 05:11:58 | 2026-10-07 06:30:25 | all 11 nodes: max_header_delay=1000ms, min_header_delay=350ms |
Workload
| Warm-up | 120 s (2 min) |
|---|---|
| Mixed phase | 2,400 s (40 min) |
| Cool-down | 180 s (3 min) |
| Transaction types | batched_self_send, contract_deploy, faucet_drip, native_transfer, stablecoin_approve, stablecoin_transfer, uniswap_v2_add_liquidity, uniswap_v2_remove_liquidity, uniswap_v2_swap, uniswap_v3_swap, uniswap_v4_swap, wtel_unwrap, wtel_wrap |
Chaos events
| Marker | Time (UTC) | Node | Action | Detail |
|---|---|---|---|---|
| C1 | 2026-10-07 05:58:56 | bench-validator-05 | docker kill -s KILL telcoin | systemctl stop holds it down (unit has Restart=on-failure). bench-blaster-05 targets ONLY this node, so its sends fail for the outage by design; the generator resends on connection errors |
| C2 | 2026-10-07 06:03:56 | bench-validator-05 | systemctl start telcoin | docker start is impossible: the unit runs docker run --rm |
| C3 | 2026-10-07 06:08:56 | bench-observer-11 | systemctl restart telcoin | observer restart |
Steps
- Before creating anything, the harness resolved the node image tag adiri:mmap-subsecond-v1lazy-10-7-2026 to its registry digest and pinned the run to that digest (sha256:71c5b03b…). It ran the pinned image to confirm that it reports telcoin-network 0.1.0 built from commit f8ff81bf4, has the sub-second timestamp fork active from genesis and contains the log messages later checks rely on, and it recorded the committed consensus parameters file and its three delays: maximum header delay 1,000 ms, minimum header delay 250 ms and maximum batch delay 250 ms, stage 1's arm A3.
- Preflight checked access, tools, quotas and a clean genesis directory, and confirmed the load-generator image built from tn-transaction-generator commit 9f61303, the image of the stage-1 run (r20261006-2058).
- The harness created ten validators and one observer on c3-highcpu-8 (8 vCPU, 16 GB), each with a 100 GB Hyperdisk Balanced boot disk provisioned at 20,000 IOPS and 800 MiB/s, one validator in each of ten regions (Los Angeles, Montréal, London, the Netherlands, Mumbai, Singapore, Hong Kong, Tokyo, Seoul and Sydney) with the observer in Los Angeles, and opened the consensus ports between the nodes only. The nodes reach each other over their public addresses, as production validators do; a probe of every pair on 7 October had found the public path as fast as the private one.
- It generated validator keys and built genesis for a separate benchmark chain (chain id 32286, 20-minute epochs).
- It finalized each node's configuration with the committed benchmark consensus parameters (maximum header delay 1,000 ms, minimum header delay 250 ms, maximum batch delay 250 ms), with every other key as in the devnet defaults. The file sets no vote timeout, so the node's built-in 5 s applied. The node log filter was extended beyond stage 1's: debug for the certifier, the proposer, the primary's vote-request handler, protocol negotiation, the connection pool, the peer manager and the network layer, info elsewhere, so that both ends of every vote request are logged with timestamps. It then started the chain.
- It relabelled each node's metrics agent so that every sample from these nodes is recorded under the benchmark network, never the live devnet.
- It restarted the nodes one at a time with RPC open on the internal network, waiting after each until the node answered with the benchmark chain id and consensus rounds were advancing. Each node's boot lines had to show the pinned commit, the three consensus delays, the 5,000 ms vote timeout and the fork schedule recorded at the gate, and all eleven did. An on-node sampler was then started on each node. The fleet was healthy at 03:53 UTC.
- A health gate confirmed that all eleven nodes reported to Prometheus under the benchmark network with consensus rounds rising, and a read of the ConsensusRegistry through an RPC tunnel confirmed the 1,200 s epoch duration.
- It deployed the devnet contract set that the mixed workload calls; the deployment finished at 04:06 UTC.
- The A3 check came first. At 04:07 UTC the harness recorded the fleet's deployed values as the arm a3check with the same fleet-wide apply tool as stage 1, which restarted nothing because every node already held them, and left the chain idle until the funding step at 04:17. A one-off funder VM then funded 600 sender accounts for each of ten load generators, one per validator in that validator's zone and sending only to it, and the generators started together at 04:24:39 UTC: 90 s of preamble, 60 s of warm-up, 480 s of mixed load across 13 transaction types (04:27:09 to 04:35:09) and 60 s of cool-down, with chaos off. Nine generators ran on e2-standard-2; the Sydney generator fell back to n2d-standard-2 because e2 machines were out of stock in its zone. The harness collected the generators' results and the node metrics and kept this blast's records apart (blast index 0) when the next began.
- From 04:45 to 04:59 UTC the harness pulled every node's log file to date (672 to 780 MB compressed per validator) without restarting anything and closed the a3check arm at 04:45:36. The certification and vote-path analysis of the check ran on those copies while the chain idled.
- The check failed the programme's criterion that every validator certify at least 90 % of its own headers while idle (the Netherlands 83.4 %), so the recorded rule applied: the minimum header delay becomes the slowest validator's idle proposal-to-quorum time at the 95th percentile (299 ms) plus 50 ms, rounded up to 350 ms, and the maximum header delay stays at 1,000 ms. The harness applied it fleet-wide as the arm stage2, restarting the eleven nodes one at a time (observer first, about 57 s each) from 05:11:58 to 05:22:25 UTC and waiting for each to log the new values and rejoin consensus; all eleven confirmed maximum 1,000 ms, minimum 350 ms and batch 250 ms.
- The chain then idled from 05:22:25 UTC to the funding step at 05:29. The funder funded 600 sender accounts for each of ten load generators, all on e2-standard-2 this time, and the generators started together at 05:36:11 UTC: 90 s of preamble, 120 s of warm-up, 2,400 s of mixed load across 13 transaction types (05:39:41 to 06:19:41) and 180 s of cool-down, the shape of the 6 October chaos runs. No network delay was injected.
- The chaos schedule was timed to the chain's epochs. Before the load generators were created, the harness read the ConsensusRegistry at a pinned block to locate the epoch boundaries and chose the first epoch close at least 600 s into the planned mixed phase, 05:59:14 UTC, the close of epoch 6. It staged three actions on the target nodes, each to fire on the node's own clock: bench-validator-05's container killed with SIGKILL 59 s before that close (05:58:15) and held down, the node restarted 300 s after the kill, and the observer's node restarted 300 s after that. The kill script saves the container's log before it waits for its due time, and with the extended log filter that save overran the 30 s staging margin: the kill began at 05:58:56.8, 41.8 s late and 17 s before the boundary. The restart followed 300 s after the actual kill, at 06:03:56.1, and the observer restart at 06:08:56, each within 0.2 s of its due time. Epoch 7 opened at 05:59:14, so the outage crossed the boundary.
- During the run the harness sampled the fleet's highest block and recorded any period in which it did not advance between samples. The one period it recorded, 06:19:30 to 06:22:30 UTC, is the cool-down after the generators stopped, with rounds still advancing on an idle chain.
- Afterwards, from 06:25 to 06:30 UTC, the harness collected each generator's results and the on-node sampler files, merged the generator results with aggregate.py, derived the sizing figures with derive-sizing.py, and pulled 60-second machine metrics from Cloud Monitoring and 15-second node metrics from Prometheus. The run's top-level results are this stage-2 blast (index 1); the A3-check blast's records sit beside them under blast index 0.
- Teardown saved every node's logs, sampler files and service journal, recorded for every validator the share of its own headers that reached a quorum in each arm, in each phase of the stage-2 blast and over the whole run, took a log snapshot for each validator that fell under 90 % in any window, verified the saved results against a local gate (about 100 GB), and then deleted the fleet. It took four attempts: the first stopped at its gate with nothing deleted when the pull from two nodes failed, the second and third fell to an expired cloud session and to a freshness rule that would have re-pulled every node, and the fourth, allowed to trust the earlier pull records when every file they listed was present at its recorded size, pulled the two missing nodes and passed at 19:51 UTC. The publishing pipeline then read each node's logged commit, consensus delays, vote timeout and fork schedule back from those logs.
The harness scripts live in a private repository; the steps above describe what they do.
Results
| Metric | Value | Window | Source | Method |
|---|---|---|---|---|
| Average chain throughput, all load generators running | 5,433.9 tx/s | all generators active | combined.series.per_second[chain_tx] | Chain transactions per block-timestamp second, summed over the 2318-s window when every load generator was in its mixed phase and divided by its length. |
| Best 60 seconds, all load generators running | 6,909.7 tx/s | all generators active | combined.series.per_second[chain_tx] | Highest mean chain tx/s over any 60 consecutive seconds inside the all-generators window (aggregate.py best_window_mean). |
| Transactions landed over the whole blast | 14,392,567 tx | whole run | combined.totals.landed_chain | Sum of tx_count over every block the load generators saw during the blast, own and foreign transactions alike. |
| Blocks per second | 18.85 blocks/s | merged mixed window | combined.blocks.rows[timestamp] | Chain blocks whose timestamp falls in the merged mixed window, divided by its length (aggregate.py landed.capacity.blocks_per_sec over the same window). |
| Time to land, median | 2.44 s | whole run | combined.latency.ts_ms.p50 | Block timestamp minus submit time over every landed own transaction in the blast, from the load generators' merged latency histograms (aggregate.py merge_latency), in seconds. |
| Time to land, p95 | 12.56 s | whole run | combined.latency.ts_ms.p95 | Block timestamp minus submit time over every landed own transaction in the blast, from the load generators' merged latency histograms (aggregate.py merge_latency), in seconds. |
| Commit latency, p95 | 4.94–28.92 s | merged mixed window | metrics.prometheus.series.tn_primary_commit_latency_seconds.by_node[healthy validators].p95 | For each healthy validator, the nearest-rank p95 over the merged mixed window of its 15-s histogram_quantile(0.95) commit-latency series; reported as the range over validators. |
| Execution time per consensus output | 1.40–1.49 s | merged mixed window | metrics.prometheus.series.tn_engine_execution_duration_seconds.by_node[healthy validators].mean | For each healthy validator, the nearest-rank median over the merged mixed window of its 15-s mean execution-duration series; reported as the range over validators. |
| Write throughput, p95, busiest validator | 425 MiB/s | merged mixed window | metrics.gcp[bench-validator-10].disk_write_bytes_per_sec p95 | Nearest-rank p95 of the 60-s mean write-byte rate over the merged mixed window for each healthy validator; the highest (bench-validator-10) is reported. |
| Write operations, p95, busiest validator | 17,933 IOPS | merged mixed window | metrics.gcp[bench-validator-06].disk_write_ops_per_sec p95 | Nearest-rank p95 of the 60-s mean write-operation rate over the merged mixed window for each healthy validator; the highest (bench-validator-06) is reported. |
| CPU, p95, busiest validator | 45.9 % | merged mixed window | metrics.gcp[bench-validator-01].cpu_utilization p95 | Nearest-rank p95 of the 60-s CPU utilisation over the merged mixed window for each healthy validator; the highest (bench-validator-01) is reported. |
| Node resident memory, p95, busiest validator | 9.16 GiB | merged mixed window | metrics.gcp[bench-validator-02].telcoin_rss_bytes p95 | Nearest-rank p95 of the 60-s telcoin process resident memory over the merged mixed window for each healthy validator; the highest (bench-validator-02) is reported. |
| IO pressure, highest reading, range over validators | 15.1–22.7 % | merged mixed window | sampler[healthy validators].psi_io_some_avg10 max | For each healthy validator, the highest 10-s 'some' IO pressure reading the on-node sampler logged inside the merged mixed window; reported as the range over validators. |
Recovery
| Node | bench-validator-05 |
|---|---|
| Killed | 2026-10-07 05:58:56 UTC |
| Restarted | 2026-10-07 06:03:56 UTC |
| Rejoined | 2026-10-07 06:28:07 UTC |
| Catch-up time | 1,433 s (23 min 53 s) |
| Failed batch reports | 0 |
Chain TPS per 5-minute interval
Each bar is the average chain TPS over one 5-minute interval of the merged mixed window (05:38–06:19 UTC); fainter bars had fewer generators active than the maximum, the dashed line is the average of all bars and amber lines mark the chaos events listed below.
- C1 2026-10-07 05:58:56 UTC: docker kill -s KILL telcoin on bench-validator-05 (systemctl stop holds it down (unit has Restart=on-failure). bench-blaster-05 targets ONLY this node, so its sends fail for the outage by design; the generator resends on connection errors)
- C2 2026-10-07 06:03:56 UTC: systemctl start telcoin on bench-validator-05 (docker start is impossible: the unit runs docker run --rm)
- C3 2026-10-07 06:08:56 UTC: systemctl restart telcoin on bench-observer-11 (observer restart)
5-minute data (9 rows)
| start_utc | end_utc | generators_active | chain_tps |
|---|---|---|---|
| 2026-10-07T05:38:25Z | 2026-10-07T05:43:25Z | 9.0 | 4,781.9 |
| 2026-10-07T05:43:25Z | 2026-10-07T05:48:25Z | 10.0 | 4,845.2 |
| 2026-10-07T05:48:25Z | 2026-10-07T05:53:25Z | 10.0 | 5,645.9 |
| 2026-10-07T05:53:25Z | 2026-10-07T05:58:25Z | 10.0 | 5,630.6 |
| 2026-10-07T05:58:25Z | 2026-10-07T06:03:25Z | 10.0 | 5,390.9 |
| 2026-10-07T06:03:25Z | 2026-10-07T06:08:25Z | 10.0 | 5,906.6 |
| 2026-10-07T06:08:25Z | 2026-10-07T06:13:25Z | 10.0 | 5,608.1 |
| 2026-10-07T06:13:25Z | 2026-10-07T06:18:25Z | 10.0 | 5,597.5 |
| 2026-10-07T06:18:25Z | 2026-10-07T06:19:47Z | 3.2 | 2,732.9 |
The fine-grained chain throughput series (its step_s field gives the seconds per point) is published as series/tps-1s.json.
Other networks
| Measure | Telcoin Network (this run) | Tempo |
|---|---|---|
| Throughput | 5,433.9 tx/s (Avg chain TPS) | 16,455 TPS, shown as "Settled TPS" for the nightly run of 5 Oct 2026 (commit de5f5d3). perf.tempo.xyz describes it as "Median settled throughput across 130 measured blocks." |
| How throughput is measured | Chain transactions per block-timestamp second, summed over the 2318-s window when every load generator was in its mixed phase and divided by its length. | The methodology page defines TPS as "Transactions per second (derived from gas / avg tx gas)": gas used divided by the average gas of one transaction, reported as the median over the measured blocks. |
| Run length | 41.4 min merged mixed window; each generator was set to a 40 min mixed phase | 2m 39s for the 5 Oct 2026 nightly run (130 measured blocks at a 687 ms average block time), per perf.tempo.xyz. |
| Validators and placement | 10 validators and 1 observer across 10 regions: asia-east2 (Hong Kong), asia-northeast1 (Tokyo), asia-northeast3 (Seoul), asia-south1 (Mumbai), asia-southeast1 (Singapore), australia-southeast1 (Sydney), europe-west2 (London), europe-west4 (Netherlands), northamerica-northeast1 (Montréal), us-west2 (Los Angeles); load generators: one blaster per validator, in that validator's zone, sending only to it | perf.tempo.xyz labels the scenario "Public mix · multi-region". The methodology page says "Benchmarks run two local validators in consensus on a dedicated bare-metal server with 100 GiB pre-bloated state." The public tempoxyz/tempo repo has nightly workflows for both kinds of setup: bench-e2e runs on one bare-metal runner, and bench-e2e-multi-region defaults to 10 validators on cloud VMs (its regions input reads "Empty uses the selected cloud's five defaults."). The run page does not say which setup produced this number. |
| Hardware | c3-highcpu-8, 8 vCPUs, 16 GB memory; disk: hyperdisk-balanced, 100 GB, 20,000 IOPS, 800 MiB/s | The methodology page lists a dedicated bare-metal server: "AMD EPYC 4585PX (16C/32T, 4.3/5.7 GHz)", "128 GB 3600 MHz" memory and "2×960 GB NVMe + 2×1.92 TB NVMe via schelk". The repo's multi-region workflow uses cloud VMs instead (c4d-standard-48-lssd on GCP, c6id.8xlarge on AWS). |
| Workload | batched_self_send, contract_deploy, faucet_drip, native_transfer, stablecoin_approve, stablecoin_transfer, uniswap_v2_add_liquidity, uniswap_v2_remove_liquidity, uniswap_v2_swap, uniswap_v3_swap, uniswap_v4_swap, wtel_unwrap, wtel_wrap; 10 generators × 600 senders each | perf.tempo.xyz labels the scenario "Public mix"; the methodology page says "100% TIP-20 token transfers (ERC-20 equivalent)". The run used 1,000 accounts and 4 TIP-20 tokens. |
| State | Fresh chain from genesis for every run | The methodology page says "100 GiB pre-bloated state". |
| Latency (as each source defines it) | Time to land, median: 2.44 s; Time to land, p95: 12.56 s; Commit latency, p95: 4.94–28.92 s | Block time of 687 ms, which the methodology page defines as "Average wall-clock time between blocks". The run page gives no per-transaction latency. Separately, tempo.xyz says "Blocks finalize in ~0.6 seconds with no re-orgs." (a network claim, not a figure from this benchmark). |
| Source and fetch date | This page; every figure names its source field. | perf.tempo.xyz nightly run of 5 Oct 2026 (commit de5f5d3), the perf.tempo.xyz methodology page, tempo.xyz, and the benchmark workflows in the tempoxyz/tempo repo. All fetched on 5 Oct 2026. |
Sources for Tempo (fetched 2026-10-05):
- perf.tempo.xyz (nightly benchmark results)
- perf.tempo.xyz methodology
- tempo.xyz
- tempoxyz/tempo repository
- bench-e2e workflow (bare-metal)
- bench-e2e-multi-region workflow
Tempo's setup may not match ours: its methodology page describes two validators on one bare-metal server, while our runs use ten validators across five regions, so read this table as context, not a ranking.
This table is context, not a ranking.
Caveats
- The A3 check (maximum header delay 1,000 ms, minimum 250 ms; blast index 0, not the blast published here) is why the published blast ran with a 350 ms minimum. Idle (04:07 to 04:17 UTC, 1,970 proposals per validator) the round was 313 ms on every node, nine validators certified 99.9 to 100 % of their own headers and the Netherlands (bench-validator-06) 83.4 %, with 131 cancelled vote collections: its own quorum arrives 296 to 299 ms after its proposal and the seventh parent certificate, which triggers its next proposal, at about 304 ms, so 8 % of its headers were replaced first, the same mechanism as in stage 1 but 290 ms faster. London (bench-validator-02), at 286 to 289 ms, certified 99.9 % with 24 ms to spare. Under the check's 480 s of load Hong Kong was lowest at 83.6 % (48 cancelled), London 90.8 % and the Netherlands 90.6 %. The programme's rule for a failed check sets the minimum header delay to the slowest validator's idle proposal-to-quorum time at the 95th percentile (299 ms) plus 50 ms, rounded up to 350 ms.
- Against stage 1's A3 arm on the previous build (r20261006-2058), each validator's median time from its own proposal to its own quorum roughly halved: Seoul 264 to 138 ms, Tokyo 272 to 142, Los Angeles 297 to 155, Hong Kong 297 to 154, Singapore 344 to 178, Montréal 406 to 210, Sydney 427 to 218, Mumbai 455 to 237, London 560 to 286 and the Netherlands 582 to 296; the idle round fell from 530 to 313 ms. The vote path was decomposed over 177,232 idle vote requests on 90 proposer-voter pairs with both ends' timestamps: the voter spends 1.5 to 1.6 ms between a request's arrival and its vote (none of the 177,232 waited more than 30 ms), no request waited for a protocol negotiation, and the stage-1 voter-side wait of 155 to 416 ms on the Europe-to-Asia links is gone. An idle vote is now wire time alone. Under the check's 480 s of load the chain committed 1.72 rounds and 15.96 blocks a second carrying 4,381 tx/s, with commit latency p50 1.26 to 1.49 s and p95 2.39 to 2.40 s (stage 1's A3: 1.41 rounds, 15.84 blocks, 5,005 tx/s, 1.72 and 2.42 s): the loaded regime is still paced by execution and by the 1,000 ms maximum timer, which binds at the 95th percentile of the loaded round.
- What remains in the vote path is per-direction transit that depends on where the proposer is. From Los Angeles every leg matches the latency probe (Tokyo 51 ms one way, Hong Kong 72, Singapore 84, Mumbai 114); from London, Singapore is 147 to 154 ms and Mumbai 179 to 180 ms per direction against about 45 to 57 ms implied by the probe. On all eight Europe-to-Asia links the one-way time equals Europe-to-Los Angeles plus Los Angeles-to-Asia within 7 ms, symmetric in both directions, and it sits between two network-layer timestamps, not in the vote handler. The 7 October probe of every pair on public and private addresses did not reproduce it; the next step is a probe on the UDP port the nodes' QUIC traffic uses, between live node addresses, not a node change. This is why London's and the Netherlands' own quorum takes 283 to 296 ms against 137 to 155 ms for Seoul, Tokyo, Los Angeles and Hong Kong.
- At the 350 ms minimum (idle 05:22:25 to 05:29:16 UTC, 11,719 proposals) every validator reached quorum on at least 99.9 % of its own headers (the Netherlands 99.9 %, no cancellation) and the round was 353 ms on every node, 13 % slower than the 313 ms at 250 ms: the minimum timer binds fleet-wide, each node's proposal-to-quorum time is unchanged (137 to 296 ms) and the rest of the round is waiting (57 ms on the Netherlands, 216 ms on Seoul). The cost is in what the certificates are used for. London's and the Netherlands' idle certificates are formed, but 90.2 % and 90.9 % of them were never committed and the two led 0.0 % of the committed rounds (fair share 10 %), where at 250 ms 5.0 and 10.4 % were uncommitted. With every node proposing at exactly 353 ms, a far node's certificate for round r (its own quorum at 283 to 296 ms plus 70 to 300 ms of transit) reaches the eight nearer nodes after their round r+1 headers are out, so it is referenced only by the other European node and never ordered; idle consensus effectively runs on seven nodes, and the two European nodes skip about one round in 100 as their timers drift against the fleet's cadence. Idle orphaning costs nothing in throughput: the minimum delay bought quorum, not inclusion.
- Under load the picture inverts and is better than at 250 ms. Over the 2,400 s mixed phase, including the outage, the fleet certified 94.9 % of 31,807 own headers (95.3 % over the 480 s A3-check blast), and every validator was between 91.8 % (the Netherlands) and 97.6 % (Los Angeles); the three lowest were the Netherlands, Hong Kong 92.5 % and Mumbai 92.6 % (Hong Kong 83.6 % in the A3 check). London's and the Netherlands' loaded certificates are committed (3.9 and 5.6 % not committed, against 14.5 and 17.5 % at 250 ms) and they led 7.5 and 5.5 % of the rounds (1.0 and 1.5 %): the loaded round, 505 to 543 ms median and 1.7 to 1.8 s at the 95th percentile, is paced by execution, not by the timer, and the far certificates arrive in time. Re-proposals ran at 671 to 755 per node over the 40 minutes (about 18 a minute) and cancelled headers at 22 to 104. bench-validator-05 proposed 1,722 headers, half the others', because of its outage and catch-up, and certified 95.2 % of them.
- Under load a voter-side wait returns, of a different kind from stage 1's: it tracks the voter's own execution lag minute by minute (Tokyo 91 to 284 blocks behind the chain head for the whole measured slice, Mumbai 201 to 294 in two bursts, Singapore, Los Angeles and Sydney in bursts), looks capped at about 10 s after which most requests go unvoted (in the 05:44 to 05:54 UTC slice Tokyo cast 17 % of its votes, Mumbai 52 %, Singapore 76 %), and coincided with Seoul, the Netherlands and Hong Kong failing to certify 13 to 23 % of their headers in that slice; 25.6 % of all requests waited more than 30 ms (11.2 % in the A3 check). It depends on the proposer only through its execution height, not its location. The lever under load is execution throughput on the lagging voters, not the vote path. A second tail, a vote response arriving 200 to 400 ms late between the voter's vote and the proposer's receipt, affected 6.9 % of responses (15.5 % in the A3 check); which end holds it is undetermined, because the proposer logs no network-level receipt line.
- The kill of bench-validator-05 fired 41.8 s late. The staged kill script saves the container's log before it waits for its due time, and with the extended log filter that log was about 17 GB uncompressed (1.26 GB compressed), so the save ran about 68 s and overran the 30 s staging margin. The kill began at 05:58:56.8 UTC instead of 05:58:15, 17 s before epoch 7 opened at 05:59:14 instead of 59 s. The outage still crossed the boundary: the validator's last committed certificate was at 05:58:53.8 and its last own header at 05:58:56.6, both in epoch 6, and its first line after the restart is at 06:03:56.4. The restart and the observer restart save nothing and were 0.13 and 0.09 s late. The chain kept committing through the outage: every healthy validator committed 1.36 rounds and 19.1 to 19.4 blocks a second over the mixed phase, and the harness's commit-stall sampler recorded no stall (its one entry, 06:19:30 to 06:22:30, is the cool-down with the generators stopped and rounds still rising).
- The restarted validator did not catch up while the load ran. After its restart at 06:03:56 UTC it ran as an active validator for 0.6 s, saw an epoch-7 certificate, went to catch-up mode and replayed epoch 7 through state sync from 06:06:13 to 06:27:55 (blocks 35,939 to 59,558, about 1,080 blocks a minute against the loaded fleet's production of about 1,190 a minute), so it followed about 6,000 blocks behind for 17 minutes and the gap had widened to 8,300 blocks when the generators stopped at 06:19:41; from 06:18:05 to 06:19:04 it logged fifty database errors (environment map size limit reached, on its consensus-output and batch caches) that slowed the replay to about 500 blocks a minute. It missed the epoch-8 boundary at about 06:18:45 because it was still in epoch 7. It became active again at 06:28:07, 24 min 11 s after the restart (29 min 12 s from the kill to its first own committed certificate), reached the chain head at 06:28:17, and from the restart to 06:40 proposed 2,020 headers and certified 2,017. Nothing was lost: it resumed at its last committed round. On r20261006-0458 the same node's catch-up took 750 s against a 3,900 tx/s load; here, at 5,400 tx/s, it rejoined only once the chain went quiet. The load generator pinned to it was the only one below 99.5 % landed (99.0 %, 9,863 transactions unlanded, resends of up to 306 s). Whether the sync path is execution-bound or fetch-bound was not measured. The recovery record on this page takes its rejoin time and catch-up duration from the validator's own log, because the 15-second node-mode metric was pulled before it rejoined.
- The comparison run, r20261006-0458, ran five regions with two validators each (Los Angeles, Montréal, London, the Netherlands and Sydney), the certifier-fix build ba43a1f0e, a maximum header delay of 500 ms and a minimum of 250 ms, the generator build 3644adc and n2-standard-2 generators; here the fleet is ten regions with one validator each, the build adds lazy protocol negotiation, the timing is 1,000/350 ms, the generator build is 9f61303 and the generators ran on e2-standard-2. The chaos plan, the load shape (120 s warm-up, 2,400 s mixed, 180 s cool-down, 600 senders per generator) and the machine and disk types are the same. r20261006-0053 is r20261006-0458 without the certifier fix, and it stalled for 301 s. By the node logs over the mixed phase this run committed 1.39 rounds and 18.99 blocks a second carrying 5,352 tx/s (r20261006-0458: 0.84, 13.56 and 3,890; r20261006-0053: 1.05, 10.99 and 3,414), with per-node commit latency p50 0.91 to 1.29 s and p95 2.40 to 4.66 s (1.91 to 2.00 and 4.55 to 4.65 s; 1.49 to 1.50 and 2.48 to 2.49 s). No difference between the pages can be assigned to any one of the changes.
- Two KPIs on this page need reading with care. The commit-latency p95 comes from the nodes' Prometheus histogram, interpolated within coarse upper buckets, and its range over the nine healthy validators reaches 28.92 s; the per-node p95 from the logs over the same mixed phase is 2.40 to 4.66 s and is the better measure. The busiest validator's disk-write IOPS p95 of 17,933 (bench-validator-06), against 5,324 on r20261006-0458, includes the run's own logging: the extended filter wrote about 1 MB per round per node idle and 21 to 23 GB an hour per copy under load, twice (the container's log and the unit's stdout file), about 46 GB an hour on each validator's boot disk during the mixed phase; the validators' disks were 41 to 45 % full at collect.
- The consensus values at the top of this page (maximum header delay 1,000 ms, minimum 250 ms, maximum batch delay 250 ms) are those of the committed parameters file, which the nodes started with and ran for the A3 check. The blast published here ran with a minimum header delay of 350 ms, applied fleet-wide from 05:11:58 to 05:22:25 UTC, as the arms table under Consensus shows: arm a3check (04:07:06 to 04:45:36, the deployed values, no restart) and arm stage2 (05:11:58 to 06:30:25). The A3-check blast's own records (blast index 0, generators running 04:24:39 to 04:36:09) are not published as a page of their own; its figures here come from the node logs and the generators' results.
- The Certification-by-node table flags three validators under the 90 % threshold. bench-validator-06 is the Netherlands shortfall of the A3 check (82.2 % of 6,390 headers over the whole a3check arm, which also drags its whole-run figure under 90 %); bench-validator-02 (London, 89.1 % of 137 headers in the 90 s preamble) and bench-validator-08 (Hong Kong, in the preamble and the 120 s warm-up, 86.9 % of 175 headers in the warm-up) fall short only in those short windows, just above the 50-header minimum for a judgement, and both pass the 2,400 s mixed window at 94.4 % and 92.5 %. The run window covers both arms and the restarts between them.
- This was the first fleet run with the extended debug filter, and its cost was the logs: about 3 GB an hour uncompressed per validator idle, 21 to 23 GB an hour under load, written twice. Teardown pulled 87 GB of compressed logs (about 100 GB with the mid-run copies, the archived blast and the analysis) and took four attempts over 13 hours. The first pulled nine nodes in two hours and stopped at its gate, with nothing deleted, when one node's pull failed over ssh and a second's fell to an expiring cloud session; the second attempt died on that session; the third, after re-authentication, rejected the nine earlier pulls under a 15-minute freshness rule and began re-pulling the whole fleet (70 to 85 GB against 57 GB free) until it was stopped by hand; the fourth, allowed to trust an earlier pull when every file it recorded is present at its recorded size and refusing to start under a 20 GB free-space floor, pulled the two missing nodes and passed. Those two nodes' saved logs were copied beside their earlier copies, about 10 GB of byte-identical duplicates. The fleet stood idle from 06:30 to 19:51 UTC meanwhile, after the measurements had ended. Two harness fixes came out of it (trusted pull records with the free-space floor, and the publishing pipeline reading node logs in place instead of copying them); a lighter default filter and a kill script that does not save logs between staging and its due time are follow-ups.
- The offered load was less uneven than in stage 1, but still uneven: the generators submitted 2.21 million transactions from Los Angeles, 1.82 from Singapore, 1.69 from Seoul, 1.58 from Sydney, 1.57 from Hong Kong, 1.52 from Tokyo, 1.02 from London, 0.99 from the Netherlands, 0.99 from Montréal and 0.97 from Mumbai, the far generators at about half of Los Angeles' against a fifth to a tenth in stage 1, and every generator landed 99.0 to 100 % of its own transactions (51 to 95 % in stage 1); the lowest is Montréal's, across its validator's outage.
- Each configuration was run once, so run-to-run variation is not measured.
- The tag adiri:mmap-subsecond-v1lazy-10-7-2026 is mutable. The harness resolved it to digest sha256:71c5b03b… before any node was created, and every node ran that digest.
- Per-validator rows cover 9 validators; bench-validator-05 excluded as chaos target(s).
- Recovery: bench-validator-05 did not leave catch-up mode inside the metrics window.
- Recovery: demoted / rejoined are the first 15-s Prometheus samples showing the change, so they trail the node's own log times by up to a scrape interval plus one step.
- Recovery: no tn_batch_builder_seal_failures_total rate series for bench-validator-05; failed_reports unavailable.
- Recovery.rejoined_utc, recovery.catch_up_secs, recovery.failed_reports are the published values, not recomputed from the run data.
- The nodes ran more than one min_header_delay_ms (250, 350) by design: the fleet value is published here, and the per-node values are listed under consensus.param_arms.
- Validators bench-validator-02 (in phase preamble), bench-validator-06 (in arm a3check and run) and bench-validator-08 (in phase preamble and phase warmup) certified fewer than 90 % of their own headers; their logs are preserved under results/r20261007-0316/logs/ (see INDEX.md).
Full report
Changes from the original report:
- The site adds a banner at the top of the report that links back to this page.
SHA-256 of the committed file, before the banner is added: ab5c9dae0153839f2ef73780528d016707f211084f96d94d7ea6074f40cbed05