Sub-second consensus with the certifier fix: the epoch-boundary kill re-run

2026-10-06 · started 05:01 UTC · ended 07:13 UTC · merged mixed window 49 min 52 s

telcoin-network ba43a1f0e · devnet-bench-10-5-2026 · commit verified from node startup logs

35 node starts across 11 instances logged telcoin-network 0.1.0 (ba43a1f0e270ca1c572d1b8378b39d2fc6360b57) The node image probe recorded the same commit before deploy Node logs: fork schedule: seed_signature, multi_workers, prevrandao, leader_seeded_ordering, and subsecond_timestamp active from genesis Commit ba43a1f0e is the head of the public integration branch devnet-bench-10-5-2026 after the merge of fix/certifier-keep-inflight-on-repropose (two commits) into the 6 October benchmark build 964087e02. Against 964087e02 the diff touches only the primary's certifier, its proposer and the certifier tests: the certifier keeps its in-flight vote collection when the proposer hands it an identical, re-proposed header, instead of cancelling the outstanding vote requests and starting again, and the primary logs the maximum header delay in force.

We re-ran the 6 October benchmark (r20261006-0053) on the same fleet shape, ten c3-highcpu-8 validators with Hyperdisk Balanced disks, with the same load generator and the same consensus timing (500 ms maximum header delay, 250 ms minimum header delay, 250 ms maximum batch delay, 5 s vote timeout). The node build changed in one place: a validator that re-proposes an identical header no longer restarts the collection of votes for it. That was the cause of the 6 October stall: a validator re-sends its unchanged header every 500 ms while it waits for parent certificates, and each re-send threw away the votes already received, so a validator whose votes took longer than one window to return never certified a header while the chain was idle; with validator-05 down, only six certificates existed for the round where seven were needed to open the next one. The harness again killed validator-05 59 seconds before an epoch was due to close, this time on the node's own clock, and held it down for 300 seconds. The chain did not stop: the other nine validators kept committing through the outage, the epoch closed on time at 06:37:59 UTC, and chain throughput stayed between 3,085 and 4,355 tx/s in every minute from 06:34 to 06:46. With all ten load generators running, average chain throughput was 3,890 tx/s, against 3,320 tx/s for the run that stalled and 4,177 tx/s for 29 September (r20260929-1647). The median time for a transaction to land was 3.4 s and the 95th percentile 18.0 s (26.1 s on the stalled run, 14.4 s on 29 September). The restarted validator spent 750 seconds in catch-up mode, half the 25 minutes seen on 29 September; it executed blocks at about the rate the loaded chain produced them, so it reached the head only after the load ended. After the eight on-time load generators stopped, the transactions still being sent to the two validators in Sydney did not land for 372 seconds, from 06:48:21 to 06:54:33 UTC, although consensus rounds and commits continued; those two validators had certified none of their own headers whenever the chain was idle and rounds were fast, a latency effect that the certifier fix does not reach and the question the later runs on the consensus delays took up.

Compared against r20261006-0053.

Key figures

3,889.6tx/s
Average chain throughput, all load generators running
all generators active
4,737.4tx/s
Best 60 seconds, all load generators running
all generators active
17.98s
Time to land, p95
whole run
5.75–6.25s
Commit latency, p95
merged mixed window

How it was run

Software

Upstream committelcoin-network ba43a1f0e270ca1c572d1b8378b39d2fc6360b57 · devnet-bench-10-5-2026 · commit verified from node startup logs
Node imageus-docker.pkg.dev/telcoin-network/tn-public/adiri:mmap-subsecond-certfix-10-6-2026
Node image digestsha256:caeb6be2f2732658841fa760b75e1adcec37d3a966a413fb48be8d1b9c32313e
Node version0.1.0
Fork schedulesubsecond-from-genesis
Load generator commit3644adc
Load generator image digestsha256:811ab3a78637b436d69868ab10bdfeab5e50780bd5906d4643f1acc9de1425ca
Analysis commit9f61303
Harness commit3a31603ba4bf172950002c3f4a01e8da350abbc5
Harness submodulescommon: b498b71e107f09c7e6a7a68ae5ba71a514b98609
tn-node-deployment: 1f0385d79a99e008cadf66f0522f4417a09e7543

Fleet

Fleetc3-highcpu-8 + Hyperdisk Balanced (c3-hdb)
Validators10
Observers1
Machine typec3-highcpu-8
vCPUs per node8
Memory per node16 GB
Diskhyperdisk-balanced, 100 GB, 20,000 IOPS, 800 MiB/s
Regionsaustralia-southeast1 (Sydney), europe-west2 (London), europe-west4 (Netherlands), northamerica-northeast1 (Montréal), us-west2 (Los Angeles) (5)
Load generators10 generators × n2-standard-2 × 600 senders each
Load placementone blaster per validator, in that validator's zone, sending only to it

Chain

Chain ID32286
Networkbench
Epoch length1,200 s (20 min)

Consensus parameters

Max header delay500 ms
Min header delay250 ms
Max batch delay250 ms
Vote timeout5,000 ms
Parameters fileconsensus-parameters.yaml
Confirmed bynode startup logs

Workload

Warm-up120 s (2 min)
Mixed phase2,400 s (40 min)
Cool-down180 s (3 min)
Transaction typesbatched_self_send, contract_deploy, faucet_drip, native_transfer, stablecoin_approve, stablecoin_transfer, uniswap_v2_add_liquidity, uniswap_v2_remove_liquidity, uniswap_v2_swap, uniswap_v3_swap, uniswap_v4_swap, wtel_unwrap, wtel_wrap

Chaos events

MarkerTime (UTC)NodeActionDetail
C12026-10-06 06:36:58bench-validator-05docker kill -s KILL telcoinsystemctl stop holds it down (unit has Restart=on-failure). bench-blaster-05 targets ONLY this node, so its sends fail for the outage by design; the generator resends on connection errors
C22026-10-06 06:41:58bench-validator-05systemctl start telcoindocker start is impossible: the unit runs docker run --rm
C32026-10-06 06:46:58bench-observer-11systemctl restart telcoinobserver restart

Steps

  1. Before creating anything, the harness resolved the node image tag adiri:mmap-subsecond-certfix-10-6-2026 to its registry digest and pinned the run to that digest (sha256:caeb6be2…). It ran the pinned image to confirm that it reports telcoin-network 0.1.0 built from commit ba43a1f0e, has the sub-second timestamp fork active from genesis and contains the log messages later checks rely on, and it recorded the committed consensus parameters file and its three delays.
  2. Preflight checked access, tools, quotas and a clean genesis directory, and confirmed the load-generator image built from tn-transaction-generator commit 3644adc, the same image as the 6 October run.
  3. The harness created ten validators and one observer on c3-highcpu-8 (8 vCPU, 16 GB), each with a 100 GB Hyperdisk Balanced boot disk provisioned at 20,000 IOPS and 800 MiB/s, two validators in each of five regions, and opened the consensus ports between the nodes only.
  4. It generated validator keys and built genesis for a separate benchmark chain (chain id 32286, 20-minute epochs).
  5. It finalized each node's configuration with the committed benchmark consensus parameters: maximum header delay 500 ms, minimum header delay 250 ms and maximum batch delay 250 ms, with every other key as in the devnet defaults. The file sets no vote timeout, so the node's built-in 5 s applied. It then started the chain.
  6. It relabelled each node's metrics agent so that every sample from these nodes is recorded under the benchmark network, never the live devnet.
  7. It restarted the nodes one at a time with RPC open on the internal network, waiting after each until the node answered with the benchmark chain id and consensus rounds were advancing. Each node's boot lines had to show the pinned commit, the three consensus delays, the 5,000 ms vote timeout and the fork schedule recorded at the gate, and all eleven did. An on-node sampler was then started on each node.
  8. A health gate confirmed that all eleven nodes reported to Prometheus under the benchmark network with consensus rounds rising, and a read of the ConsensusRegistry through an RPC tunnel confirmed the 1,200 s epoch duration.
  9. It deployed the devnet contract set that the mixed workload calls.
  10. The first two attempts to start the load (blast indexes 0 and 1) stopped before any load was sent, because e2-standard-2 machines were out of stock in australia-southeast1-c and the two load generators for that zone could not be created; the harness deleted the generators it had created and kept their records apart. The third attempt (blast index 2), the one published here, used n2-standard-2 (2 vCPU, 8 GB) for the funder and all ten load generators. Each attempt re-ran the funder.
  11. A one-off funder VM funded 600 sender accounts for each of ten load generators, one per validator in that validator's zone and sending only to it, and the generators then started together at 06:05:38 UTC.
  12. Each generator ran 120 s of warm-up, 2,400 s of mixed load across 13 transaction types and up to 180 s of cool-down, cut short after 30 s in which none of the generator's own transactions landed.
  13. The chaos schedule was timed to the chain's epochs. Before the load generators were created, the harness read the ConsensusRegistry at a pinned block to locate the epoch boundaries and chose the first epoch close at least 600 s into the planned mixed phase, 06:37:57 UTC. It set validator-05's container to be killed with SIGKILL 59 s before that close and held down, the node to be restarted 300 s after the kill, and the observer's node to be restarted 300 s after that. Each action was staged on the target node and fired by the node's own clock: the kill began at 06:36:58.139, the restart at 06:41:58.179 and the observer restart at 06:46:58.067, each less than 0.2 s after its planned second. Epoch 4 opened at 06:37:59, so the outage crossed the boundary. The run had one blast and no injected network delay.
  14. During the run the harness sampled the fleet's highest block and recorded any period in which it did not advance between samples as a commit stall.
  15. Afterwards the harness collected each generator's results and the on-node sampler files, merged the generator results with aggregate.py, derived the sizing figures with derive-sizing.py, and pulled 60-second machine metrics from Cloud Monitoring and 15-second node metrics from Prometheus. The archive copy attempted at collect time failed on a harness defect (a file-exclude pattern the copy tool misread); the teardown gate then copied the run's results to a persistent storage bucket and verified the copy at 07:13:56 UTC, before anything was deleted, and the final sync completed at 07:24:53 UTC.
  16. Teardown began at 07:13 UTC. It saved every node's logs, sampler files and service journal, then deleted the fleet. The publishing pipeline then read each node's logged commit, consensus delays, vote timeout and fork schedule back from those logs.

The harness scripts live in a private repository; the steps above describe what they do.

Results

MetricValueWindowSourceMethod
Average chain throughput, all load generators running3,889.6 tx/sall generators activecombined.series.per_second[chain_tx]Chain transactions per block-timestamp second, summed over the 1808-s window when every load generator was in its mixed phase and divided by its length.
Best 60 seconds, all load generators running4,737.4 tx/sall generators activecombined.series.per_second[chain_tx]Highest mean chain tx/s over any 60 consecutive seconds inside the all-generators window (aggregate.py best_window_mean).
Transactions landed over the whole blast11,504,996 txwhole runcombined.totals.landed_chainSum of tx_count over every block the load generators saw during the blast, own and foreign transactions alike.
Blocks per second11.52 blocks/smerged mixed windowcombined.blocks.rows[timestamp]Chain blocks whose timestamp falls in the merged mixed window, divided by its length (aggregate.py landed.capacity.blocks_per_sec over the same window).
Time to land, median3.38 swhole runcombined.latency.ts_ms.p50Block timestamp minus submit time over every landed own transaction in the blast, from the load generators' merged latency histograms (aggregate.py merge_latency), in seconds.
Time to land, p9517.98 swhole runcombined.latency.ts_ms.p95Block timestamp minus submit time over every landed own transaction in the blast, from the load generators' merged latency histograms (aggregate.py merge_latency), in seconds.
Commit latency, p955.75–6.25 smerged mixed windowmetrics.prometheus.series.tn_primary_commit_latency_seconds.by_node[healthy validators].p95For each healthy validator, the nearest-rank p95 over the merged mixed window of its 15-s histogram_quantile(0.95) commit-latency series; reported as the range over validators.
Execution time per consensus output2.05–2.10 smerged mixed windowmetrics.prometheus.series.tn_engine_execution_duration_seconds.by_node[healthy validators].meanFor each healthy validator, the nearest-rank median over the merged mixed window of its 15-s mean execution-duration series; reported as the range over validators.
Write throughput, p95, busiest validator484.1 MiB/smerged mixed windowmetrics.gcp[bench-validator-09].disk_write_bytes_per_sec p95Nearest-rank p95 of the 60-s mean write-byte rate over the merged mixed window for each healthy validator; the highest (bench-validator-09) is reported.
Write operations, p95, busiest validator5,324 IOPSmerged mixed windowmetrics.gcp[bench-validator-03].disk_write_ops_per_sec p95Nearest-rank p95 of the 60-s mean write-operation rate over the merged mixed window for each healthy validator; the highest (bench-validator-03) is reported.
CPU, p95, busiest validator31.3 %merged mixed windowmetrics.gcp[bench-validator-02].cpu_utilization p95Nearest-rank p95 of the 60-s CPU utilisation over the merged mixed window for each healthy validator; the highest (bench-validator-02) is reported.
Node resident memory, p95, busiest validator9.6 GiBmerged mixed windowmetrics.gcp[bench-validator-03].telcoin_rss_bytes p95Nearest-rank p95 of the 60-s telcoin process resident memory over the merged mixed window for each healthy validator; the highest (bench-validator-03) is reported.
IO pressure, highest reading, range over validators26.4–32.2 %merged mixed windowsampler[healthy validators].psi_io_some_avg10 maxFor each healthy validator, the highest 10-s 'some' IO pressure reading the on-node sampler logged inside the merged mixed window; reported as the range over validators.

Recovery

Nodebench-validator-05
Killed2026-10-06 06:36:58 UTC
Restarted2026-10-06 06:41:58 UTC
Rejoined2026-10-06 06:54:55 UTC
Catch-up time750 s (12 min 30 s)
Failed batch reports0

Chain TPS per 5-minute interval

Chain TPS per 5-minute interval 10 intervals from 06:07 to 06:57 UTC; average 3,296 TPS, highest 4,952 TPS. The merged mixed window ran from 06:07 to 06:57 UTC. TPS 0 1,000 2,000 3,000 4,000 5,000 06:07–06:12 UTC: 4,952 TPS, 8.3 generators active 06:12–06:17 UTC: 4,448 TPS, 9.0 generators active 06:17–06:22 UTC: 4,155 TPS, 10.0 generators active 06:22–06:27 UTC: 3,804 TPS, 10.0 generators active 06:27–06:32 UTC: 4,094 TPS, 10.0 generators active 06:32–06:37 UTC: 3,614 TPS, 10.0 generators active 06:37–06:42 UTC: 3,776 TPS, 10.0 generators active 06:42–06:47 UTC: 3,901 TPS, 10.0 generators active 06:47–06:52 UTC: 160 TPS, 1.7 generators active 06:52–06:57 UTC: 54 TPS, 1.0 generators active avg 3,296 06:07 UTC 06:17 UTC 06:27 UTC 06:37 UTC 06:47 UTC C1 2026-10-06 06:36:58 UTC: docker kill -s KILL telcoin on bench-validator-05 (systemctl stop holds it down (unit has Restart=on-failure). bench-blaster-05 targets ONLY this node, so its sends fail for the outage by design; the generator resends on connection errors) C1 C2 2026-10-06 06:41:58 UTC: systemctl start telcoin on bench-validator-05 (docker start is impossible: the unit runs docker run --rm) C2 C3 2026-10-06 06:46:58 UTC: systemctl restart telcoin on bench-observer-11 (observer restart) C3

Each bar is the average chain TPS over one 5-minute interval of the merged mixed window (06:07–06:57 UTC); fainter bars had fewer generators active than the maximum, the dashed line is the average of all bars and amber lines mark the chaos events listed below.

  • C1 2026-10-06 06:36:58 UTC: docker kill -s KILL telcoin on bench-validator-05 (systemctl stop holds it down (unit has Restart=on-failure). bench-blaster-05 targets ONLY this node, so its sends fail for the outage by design; the generator resends on connection errors)
  • C2 2026-10-06 06:41:58 UTC: systemctl start telcoin on bench-validator-05 (docker start is impossible: the unit runs docker run --rm)
  • C3 2026-10-06 06:46:58 UTC: systemctl restart telcoin on bench-observer-11 (observer restart)
5-minute data (10 rows)
start_utcend_utcgenerators_activechain_tps
2026-10-06T06:07:48Z2026-10-06T06:12:48Z8.34,951.5
2026-10-06T06:12:48Z2026-10-06T06:17:48Z9.04,448.4
2026-10-06T06:17:48Z2026-10-06T06:22:48Z10.04,155.3
2026-10-06T06:22:48Z2026-10-06T06:27:48Z10.03,804.0
2026-10-06T06:27:48Z2026-10-06T06:32:48Z10.04,093.9
2026-10-06T06:32:48Z2026-10-06T06:37:48Z10.03,614.2
2026-10-06T06:37:48Z2026-10-06T06:42:48Z10.03,776.5
2026-10-06T06:42:48Z2026-10-06T06:47:48Z10.03,901.0
2026-10-06T06:47:48Z2026-10-06T06:52:48Z1.7159.6
2026-10-06T06:52:48Z2026-10-06T06:57:40Z1.054.0

The fine-grained chain throughput series (its step_s field gives the seconds per point) is published as series/tps-1s.json.

Other networks

MeasureTelcoin Network (this run)Tempo
Throughput3,889.6 tx/s (Avg chain TPS)16,455 TPS, shown as "Settled TPS" for the nightly run of 5 Oct 2026 (commit de5f5d3). perf.tempo.xyz describes it as "Median settled throughput across 130 measured blocks."
How throughput is measuredChain transactions per block-timestamp second, summed over the 1808-s window when every load generator was in its mixed phase and divided by its length.The methodology page defines TPS as "Transactions per second (derived from gas / avg tx gas)": gas used divided by the average gas of one transaction, reported as the median over the measured blocks.
Run length49.9 min merged mixed window; each generator was set to a 40 min mixed phase2m 39s for the 5 Oct 2026 nightly run (130 measured blocks at a 687 ms average block time), per perf.tempo.xyz.
Validators and placement10 validators and 1 observer across 5 regions: australia-southeast1 (Sydney), europe-west2 (London), europe-west4 (Netherlands), northamerica-northeast1 (Montréal), us-west2 (Los Angeles); load generators: one blaster per validator, in that validator's zone, sending only to itperf.tempo.xyz labels the scenario "Public mix · multi-region". The methodology page says "Benchmarks run two local validators in consensus on a dedicated bare-metal server with 100 GiB pre-bloated state." The public tempoxyz/tempo repo has nightly workflows for both kinds of setup: bench-e2e runs on one bare-metal runner, and bench-e2e-multi-region defaults to 10 validators on cloud VMs (its regions input reads "Empty uses the selected cloud's five defaults."). The run page does not say which setup produced this number.
Hardwarec3-highcpu-8, 8 vCPUs, 16 GB memory; disk: hyperdisk-balanced, 100 GB, 20,000 IOPS, 800 MiB/sThe methodology page lists a dedicated bare-metal server: "AMD EPYC 4585PX (16C/32T, 4.3/5.7 GHz)", "128 GB 3600 MHz" memory and "2×960 GB NVMe + 2×1.92 TB NVMe via schelk". The repo's multi-region workflow uses cloud VMs instead (c4d-standard-48-lssd on GCP, c6id.8xlarge on AWS).
Workloadbatched_self_send, contract_deploy, faucet_drip, native_transfer, stablecoin_approve, stablecoin_transfer, uniswap_v2_add_liquidity, uniswap_v2_remove_liquidity, uniswap_v2_swap, uniswap_v3_swap, uniswap_v4_swap, wtel_unwrap, wtel_wrap; 10 generators × 600 senders eachperf.tempo.xyz labels the scenario "Public mix"; the methodology page says "100% TIP-20 token transfers (ERC-20 equivalent)". The run used 1,000 accounts and 4 TIP-20 tokens.
StateFresh chain from genesis for every runThe methodology page says "100 GiB pre-bloated state".
Latency (as each source defines it)Time to land, median: 3.38 s; Time to land, p95: 17.98 s; Commit latency, p95: 5.75–6.25 sBlock time of 687 ms, which the methodology page defines as "Average wall-clock time between blocks". The run page gives no per-transaction latency. Separately, tempo.xyz says "Blocks finalize in ~0.6 seconds with no re-orgs." (a network claim, not a figure from this benchmark).
Source and fetch dateThis page; every figure names its source field.perf.tempo.xyz nightly run of 5 Oct 2026 (commit de5f5d3), the perf.tempo.xyz methodology page, tempo.xyz, and the benchmark workflows in the tempoxyz/tempo repo. All fetched on 5 Oct 2026.

Sources for Tempo (fetched 2026-10-05):

Tempo's setup may not match ours: its methodology page describes two validators on one bare-metal server, while our runs use ten validators across five regions, so read this table as context, not a ranking.

This table is context, not a ranking.

Caveats

  • The chain did not stall this time. Validator-05 was killed 59 s before the epoch close, as planned, and the other nine validators kept committing: chain throughput in the one-minute buckets from 06:34 to 06:46 UTC ranged from 3,085 to 4,355 tx/s, the longest run of seconds without a landed transaction was 9 s (from 06:38:00, at the epoch boundary), and epoch 4 opened on time at 06:37:59. In the all-generators window the average was 3,921 tx/s before the kill and 3,833 tx/s after it. On the 6 October run (r20261006-0053) the same kill, 42 s before the close, was followed by 301 s with no landed transaction.
  • Validator-05 restarted at 06:41:58 UTC and was in catch-up mode from the first Prometheus sample at 06:42:25 to 06:54:55 (750 s). Its sampler shows its executed block held at 27,969 until about 06:44:30, then rising by about 800 to 870 blocks a minute, close to the 780 a minute the loaded chain produced, so the gap of about 5,700 blocks stayed level while the load ran. It reached the chain head, block 36,577, at 06:54:31, after block production had stopped at 06:48:21. Load generator 05, which sends only to validator-05, landed no transactions from 06:38 to the end of its mixed phase at 06:47:48; the 8,294 transactions it sent after the restart waited in validator-05's pool until the validator rejoined. On 29 September the same node spent 1,500 s in catch-up mode; on the 6 October run the chain was stalled through the outage, so there was almost nothing to catch up on.
  • Eight load generators finished their setup transactions within 12 s and began their mixed phase between 06:07:48 and 06:07:52 UTC. Generator 04 took 161 s and began at 06:10:21. Generator 03 stopped waiting at the 600-second preamble limit with 62 of its 600 senders still missing a setup step (134 setup transactions unlanded) and began at 06:17:40. Both are the australia-southeast1 generators. The all-generators window is therefore 06:17:40 to 06:47:48 UTC (1,808 s), and the merged mixed window, 06:07:48 to 06:57:40 UTC (2,992 s, average 3,305 tx/s), ends with about ten minutes in which only generators 03 and 04, then only 03, were running. On the 6 October run eight generators hit the preamble limit and only two started on time.
  • After the eight on-time generators stopped at 06:47:48 to 06:47:52 UTC, no block carrying transactions was produced from 06:48:21 (block 36,577) to 06:54:33 (block 36,578, empty), although consensus rounds kept advancing on all ten validators at about 2.6 a second and sub-DAGs were committed at about 1.3 a second, each made of certificates from the seven validators outside Australia only, and generators 03 (mixed phase until 06:57:40) and 04 (until 06:50:21) were still sending. Validators 03 and 04 kept sealing batches and re-proposing them but certified none of their headers in that period: with the chain idle, each of their headers was superseded by the next round's before six remote votes could arrive, the effect described in the next caveat. The gap closed at 06:54:31, when validator-05 rejoined and proposed batches of its own; rounds slowed for about 20 s, validator-03 certified ten headers, about 15,800 transactions landed in the minute from 06:54, and none after that before the empty block that closed epoch 4 at 06:57:59. Generators 03, 04 and 05 ended with 9,888, 9,788 and 9,827 accepted transactions that never landed. The harness recorded 06:48:31 to 06:52:04 as a commit stall (its watch ended while the height was still flat); consensus did not stall. A transaction sent to a far validator lands only when the chain is slow enough for that validator to certify a header. How the node should handle this is an open question.
  • In the idle window before the load (05:30 to 05:59 UTC) every validator proposed 3,205 headers. The two in Sydney (03 and 04) had none of theirs certified; the other eight had all of theirs certified, reaching their own quorum 180 to 184 ms after proposing from Montréal and 282 to 309 ms from Los Angeles, London and the Netherlands, while the round period was about 349 ms (median) and 395 ms (95th percentile) on every node. Each node's certification times sit within 3 ms of their median: a fixed path, not jitter. Under load (06:10 to 06:35) the median round period stretched to about 900 ms and the Sydney validators certified 66.8 % of their headers, against 95.4 % for Los Angeles and 90.4 % for the Netherlands. Over the whole mixed phase they certified 60.4 % and 67.8 % (23.6 % and 24.2 % on the 6 October run), led 0.1 % and 0.8 % of the committed sub-DAGs against a fair share of 10 %, and about a third of the certificates they did form (526 of 1,545 and 533 of 1,730) were never committed because the other seven validators already held seven parents when they arrived. The certifier fix removed the cancelled vote collections (fleet-wide 'failed to get vote: Canceled' fell from 7,005 to 1,326, 81 %) but cannot shorten the round trip, so a validator whose votes take longer than one round to return still certifies nothing while the chain is idle. This finding is what the later runs on the consensus delays were set up to measure.
  • The offered load was uneven. Generators 03 and 04 sent about 239,000 and 401,000 transactions over the blast, against about 820,000 to 1.9 million for the other eight. Their transactions also waited longest: median time to land 14.6 s for generator 03 and 6.8 s for generator 04 (95th percentile 378.5 s and 325.7 s), against 7 to 26 s at the 95th percentile for the other generators except 07 (303.5 s). These waits are part of the whole-blast p95 of 18.0 s.
  • Every generator's 99th-percentile time to land is between 301 s and 635 s. Each generator logged that its validator had dropped transactions it had already accepted, about 23,600 times across the ten generators (about 23,700 on the 6 October run), and resent them; those transactions make up most of the slow tail.
  • Consensus committed less often under load than on the 6 October run: about 0.42 sub-DAGs and 0.90 rounds a second before the kill, against 0.59 and 1.21, and 0.32 and 0.69 while validator-05 was down. The commit latency p95 rose from 4.77 to 4.88 s to 5.75 to 6.25 s. The cause has not been looked at.
  • The load generators ran on n2-standard-2, not the e2-standard-2 used on 6 October and 29 September, because e2 capacity was out of stock in australia-southeast1-c. Both are 2-vCPU machines, but the load-generator machine type differs between this run and both comparison runs.
  • Against the 6 October run the node build differs by the certifier change and the load-generator machine type differs; the generator image, fleet shape, consensus timing and chaos plan are the same. The 6 October run stalled for 301 s inside its all-generators window and this one did not, so the difference in averages mostly reflects the stall. The comparison with 29 September (r20260929-1647) also spans a different node build and the 3 s, 1 s, 1 s devnet consensus timing, so no difference can be assigned to any one change.
  • The harness working tree had one uncommitted change when the run was deployed (harness 3a31603, recorded as dirty): a five-line fix to the pattern that excludes files from the archive copy, committed as f89823d while the run was in progress. It touched the archive step only, not the deploy, chaos or measurement path. The collect-time archive copy failed on the uncorrected pattern and the teardown gate's copy succeeded with the fix.
  • This configuration has been run once, so run-to-run variation is not measured.
  • The tag adiri:mmap-subsecond-certfix-10-6-2026 is mutable. The harness resolved it to digest sha256:caeb6be2… before any node was created, and every node ran that digest.
  • Per-validator rows cover 9 validators; bench-validator-05 excluded as chaos target(s).
  • Recovery: demoted / rejoined are the first 15-s Prometheus samples showing the change, so they trail the node's own log times by up to a scrape interval plus one step.
  • Recovery: no tn_batch_builder_seal_failures_total rate series for bench-validator-05; failed_reports unavailable.
  • Recovery.failed_reports is the published value, not recomputed from the run data.
  • The harness working tree had uncommitted changes when the run was deployed.

Full report

Full report HTML · 1.1 MB

Changes from the original report:

  • The site adds a banner at the top of the report that links back to this page.

SHA-256 of the committed file, before the banner is added: 5a070e8ae30cd4996ff42a345b2a006ce9a97bad3d8cce6dd894d109ae25ff7d