Sub-second consensus with the certifier fix: the epoch-boundary kill re-run
2026-10-06 · started 05:01 UTC · ended 07:13 UTC · merged mixed window 49 min 52 s
We re-ran the 6 October benchmark (r20261006-0053) on the same fleet shape, ten c3-highcpu-8 validators with Hyperdisk Balanced disks, with the same load generator and the same consensus timing (500 ms maximum header delay, 250 ms minimum header delay, 250 ms maximum batch delay, 5 s vote timeout). The node build changed in one place: a validator that re-proposes an identical header no longer restarts the collection of votes for it. That was the cause of the 6 October stall: a validator re-sends its unchanged header every 500 ms while it waits for parent certificates, and each re-send threw away the votes already received, so a validator whose votes took longer than one window to return never certified a header while the chain was idle; with validator-05 down, only six certificates existed for the round where seven were needed to open the next one. The harness again killed validator-05 59 seconds before an epoch was due to close, this time on the node's own clock, and held it down for 300 seconds. The chain did not stop: the other nine validators kept committing through the outage, the epoch closed on time at 06:37:59 UTC, and chain throughput stayed between 3,085 and 4,355 tx/s in every minute from 06:34 to 06:46. With all ten load generators running, average chain throughput was 3,890 tx/s, against 3,320 tx/s for the run that stalled and 4,177 tx/s for 29 September (r20260929-1647). The median time for a transaction to land was 3.4 s and the 95th percentile 18.0 s (26.1 s on the stalled run, 14.4 s on 29 September). The restarted validator spent 750 seconds in catch-up mode, half the 25 minutes seen on 29 September; it executed blocks at about the rate the loaded chain produced them, so it reached the head only after the load ended. After the eight on-time load generators stopped, the transactions still being sent to the two validators in Sydney did not land for 372 seconds, from 06:48:21 to 06:54:33 UTC, although consensus rounds and commits continued; those two validators had certified none of their own headers whenever the chain was idle and rounds were fast, a latency effect that the certifier fix does not reach and the question the later runs on the consensus delays took up.
Compared against r20261006-0053.
Key figures
How it was run
Software
| Upstream commit | telcoin-network ba43a1f0e270ca1c572d1b8378b39d2fc6360b57 · devnet-bench-10-5-2026 · commit verified from node startup logs |
|---|---|
| Node image | us-docker.pkg.dev/telcoin-network/tn-public/adiri:mmap-subsecond-certfix-10-6-2026 |
| Node image digest | sha256:caeb6be2f2732658841fa760b75e1adcec37d3a966a413fb48be8d1b9c32313e |
| Node version | 0.1.0 |
| Fork schedule | subsecond-from-genesis |
| Load generator commit | 3644adc |
| Load generator image digest | sha256:811ab3a78637b436d69868ab10bdfeab5e50780bd5906d4643f1acc9de1425ca |
| Analysis commit | 9f61303 |
| Harness commit | 3a31603ba4bf172950002c3f4a01e8da350abbc5 |
| Harness submodules | common: b498b71e107f09c7e6a7a68ae5ba71a514b98609 tn-node-deployment: 1f0385d79a99e008cadf66f0522f4417a09e7543 |
Fleet
| Fleet | c3-highcpu-8 + Hyperdisk Balanced (c3-hdb) |
|---|---|
| Validators | 10 |
| Observers | 1 |
| Machine type | c3-highcpu-8 |
| vCPUs per node | 8 |
| Memory per node | 16 GB |
| Disk | hyperdisk-balanced, 100 GB, 20,000 IOPS, 800 MiB/s |
| Regions | australia-southeast1 (Sydney), europe-west2 (London), europe-west4 (Netherlands), northamerica-northeast1 (Montréal), us-west2 (Los Angeles) (5) |
| Load generators | 10 generators × n2-standard-2 × 600 senders each |
| Load placement | one blaster per validator, in that validator's zone, sending only to it |
Chain
| Chain ID | 32286 |
|---|---|
| Network | bench |
| Epoch length | 1,200 s (20 min) |
Consensus parameters
| Max header delay | 500 ms |
|---|---|
| Min header delay | 250 ms |
| Max batch delay | 250 ms |
| Vote timeout | 5,000 ms |
| Parameters file | consensus-parameters.yaml |
| Confirmed by | node startup logs |
Workload
| Warm-up | 120 s (2 min) |
|---|---|
| Mixed phase | 2,400 s (40 min) |
| Cool-down | 180 s (3 min) |
| Transaction types | batched_self_send, contract_deploy, faucet_drip, native_transfer, stablecoin_approve, stablecoin_transfer, uniswap_v2_add_liquidity, uniswap_v2_remove_liquidity, uniswap_v2_swap, uniswap_v3_swap, uniswap_v4_swap, wtel_unwrap, wtel_wrap |
Chaos events
| Marker | Time (UTC) | Node | Action | Detail |
|---|---|---|---|---|
| C1 | 2026-10-06 06:36:58 | bench-validator-05 | docker kill -s KILL telcoin | systemctl stop holds it down (unit has Restart=on-failure). bench-blaster-05 targets ONLY this node, so its sends fail for the outage by design; the generator resends on connection errors |
| C2 | 2026-10-06 06:41:58 | bench-validator-05 | systemctl start telcoin | docker start is impossible: the unit runs docker run --rm |
| C3 | 2026-10-06 06:46:58 | bench-observer-11 | systemctl restart telcoin | observer restart |
Steps
- Before creating anything, the harness resolved the node image tag adiri:mmap-subsecond-certfix-10-6-2026 to its registry digest and pinned the run to that digest (sha256:caeb6be2…). It ran the pinned image to confirm that it reports telcoin-network 0.1.0 built from commit ba43a1f0e, has the sub-second timestamp fork active from genesis and contains the log messages later checks rely on, and it recorded the committed consensus parameters file and its three delays.
- Preflight checked access, tools, quotas and a clean genesis directory, and confirmed the load-generator image built from tn-transaction-generator commit 3644adc, the same image as the 6 October run.
- The harness created ten validators and one observer on c3-highcpu-8 (8 vCPU, 16 GB), each with a 100 GB Hyperdisk Balanced boot disk provisioned at 20,000 IOPS and 800 MiB/s, two validators in each of five regions, and opened the consensus ports between the nodes only.
- It generated validator keys and built genesis for a separate benchmark chain (chain id 32286, 20-minute epochs).
- It finalized each node's configuration with the committed benchmark consensus parameters: maximum header delay 500 ms, minimum header delay 250 ms and maximum batch delay 250 ms, with every other key as in the devnet defaults. The file sets no vote timeout, so the node's built-in 5 s applied. It then started the chain.
- It relabelled each node's metrics agent so that every sample from these nodes is recorded under the benchmark network, never the live devnet.
- It restarted the nodes one at a time with RPC open on the internal network, waiting after each until the node answered with the benchmark chain id and consensus rounds were advancing. Each node's boot lines had to show the pinned commit, the three consensus delays, the 5,000 ms vote timeout and the fork schedule recorded at the gate, and all eleven did. An on-node sampler was then started on each node.
- A health gate confirmed that all eleven nodes reported to Prometheus under the benchmark network with consensus rounds rising, and a read of the ConsensusRegistry through an RPC tunnel confirmed the 1,200 s epoch duration.
- It deployed the devnet contract set that the mixed workload calls.
- The first two attempts to start the load (blast indexes 0 and 1) stopped before any load was sent, because e2-standard-2 machines were out of stock in australia-southeast1-c and the two load generators for that zone could not be created; the harness deleted the generators it had created and kept their records apart. The third attempt (blast index 2), the one published here, used n2-standard-2 (2 vCPU, 8 GB) for the funder and all ten load generators. Each attempt re-ran the funder.
- A one-off funder VM funded 600 sender accounts for each of ten load generators, one per validator in that validator's zone and sending only to it, and the generators then started together at 06:05:38 UTC.
- Each generator ran 120 s of warm-up, 2,400 s of mixed load across 13 transaction types and up to 180 s of cool-down, cut short after 30 s in which none of the generator's own transactions landed.
- The chaos schedule was timed to the chain's epochs. Before the load generators were created, the harness read the ConsensusRegistry at a pinned block to locate the epoch boundaries and chose the first epoch close at least 600 s into the planned mixed phase, 06:37:57 UTC. It set validator-05's container to be killed with SIGKILL 59 s before that close and held down, the node to be restarted 300 s after the kill, and the observer's node to be restarted 300 s after that. Each action was staged on the target node and fired by the node's own clock: the kill began at 06:36:58.139, the restart at 06:41:58.179 and the observer restart at 06:46:58.067, each less than 0.2 s after its planned second. Epoch 4 opened at 06:37:59, so the outage crossed the boundary. The run had one blast and no injected network delay.
- During the run the harness sampled the fleet's highest block and recorded any period in which it did not advance between samples as a commit stall.
- Afterwards the harness collected each generator's results and the on-node sampler files, merged the generator results with aggregate.py, derived the sizing figures with derive-sizing.py, and pulled 60-second machine metrics from Cloud Monitoring and 15-second node metrics from Prometheus. The archive copy attempted at collect time failed on a harness defect (a file-exclude pattern the copy tool misread); the teardown gate then copied the run's results to a persistent storage bucket and verified the copy at 07:13:56 UTC, before anything was deleted, and the final sync completed at 07:24:53 UTC.
- Teardown began at 07:13 UTC. It saved every node's logs, sampler files and service journal, then deleted the fleet. The publishing pipeline then read each node's logged commit, consensus delays, vote timeout and fork schedule back from those logs.
The harness scripts live in a private repository; the steps above describe what they do.
Results
| Metric | Value | Window | Source | Method |
|---|---|---|---|---|
| Average chain throughput, all load generators running | 3,889.6 tx/s | all generators active | combined.series.per_second[chain_tx] | Chain transactions per block-timestamp second, summed over the 1808-s window when every load generator was in its mixed phase and divided by its length. |
| Best 60 seconds, all load generators running | 4,737.4 tx/s | all generators active | combined.series.per_second[chain_tx] | Highest mean chain tx/s over any 60 consecutive seconds inside the all-generators window (aggregate.py best_window_mean). |
| Transactions landed over the whole blast | 11,504,996 tx | whole run | combined.totals.landed_chain | Sum of tx_count over every block the load generators saw during the blast, own and foreign transactions alike. |
| Blocks per second | 11.52 blocks/s | merged mixed window | combined.blocks.rows[timestamp] | Chain blocks whose timestamp falls in the merged mixed window, divided by its length (aggregate.py landed.capacity.blocks_per_sec over the same window). |
| Time to land, median | 3.38 s | whole run | combined.latency.ts_ms.p50 | Block timestamp minus submit time over every landed own transaction in the blast, from the load generators' merged latency histograms (aggregate.py merge_latency), in seconds. |
| Time to land, p95 | 17.98 s | whole run | combined.latency.ts_ms.p95 | Block timestamp minus submit time over every landed own transaction in the blast, from the load generators' merged latency histograms (aggregate.py merge_latency), in seconds. |
| Commit latency, p95 | 5.75–6.25 s | merged mixed window | metrics.prometheus.series.tn_primary_commit_latency_seconds.by_node[healthy validators].p95 | For each healthy validator, the nearest-rank p95 over the merged mixed window of its 15-s histogram_quantile(0.95) commit-latency series; reported as the range over validators. |
| Execution time per consensus output | 2.05–2.10 s | merged mixed window | metrics.prometheus.series.tn_engine_execution_duration_seconds.by_node[healthy validators].mean | For each healthy validator, the nearest-rank median over the merged mixed window of its 15-s mean execution-duration series; reported as the range over validators. |
| Write throughput, p95, busiest validator | 484.1 MiB/s | merged mixed window | metrics.gcp[bench-validator-09].disk_write_bytes_per_sec p95 | Nearest-rank p95 of the 60-s mean write-byte rate over the merged mixed window for each healthy validator; the highest (bench-validator-09) is reported. |
| Write operations, p95, busiest validator | 5,324 IOPS | merged mixed window | metrics.gcp[bench-validator-03].disk_write_ops_per_sec p95 | Nearest-rank p95 of the 60-s mean write-operation rate over the merged mixed window for each healthy validator; the highest (bench-validator-03) is reported. |
| CPU, p95, busiest validator | 31.3 % | merged mixed window | metrics.gcp[bench-validator-02].cpu_utilization p95 | Nearest-rank p95 of the 60-s CPU utilisation over the merged mixed window for each healthy validator; the highest (bench-validator-02) is reported. |
| Node resident memory, p95, busiest validator | 9.6 GiB | merged mixed window | metrics.gcp[bench-validator-03].telcoin_rss_bytes p95 | Nearest-rank p95 of the 60-s telcoin process resident memory over the merged mixed window for each healthy validator; the highest (bench-validator-03) is reported. |
| IO pressure, highest reading, range over validators | 26.4–32.2 % | merged mixed window | sampler[healthy validators].psi_io_some_avg10 max | For each healthy validator, the highest 10-s 'some' IO pressure reading the on-node sampler logged inside the merged mixed window; reported as the range over validators. |
Recovery
| Node | bench-validator-05 |
|---|---|
| Killed | 2026-10-06 06:36:58 UTC |
| Restarted | 2026-10-06 06:41:58 UTC |
| Rejoined | 2026-10-06 06:54:55 UTC |
| Catch-up time | 750 s (12 min 30 s) |
| Failed batch reports | 0 |
Chain TPS per 5-minute interval
Each bar is the average chain TPS over one 5-minute interval of the merged mixed window (06:07–06:57 UTC); fainter bars had fewer generators active than the maximum, the dashed line is the average of all bars and amber lines mark the chaos events listed below.
- C1 2026-10-06 06:36:58 UTC: docker kill -s KILL telcoin on bench-validator-05 (systemctl stop holds it down (unit has Restart=on-failure). bench-blaster-05 targets ONLY this node, so its sends fail for the outage by design; the generator resends on connection errors)
- C2 2026-10-06 06:41:58 UTC: systemctl start telcoin on bench-validator-05 (docker start is impossible: the unit runs docker run --rm)
- C3 2026-10-06 06:46:58 UTC: systemctl restart telcoin on bench-observer-11 (observer restart)
5-minute data (10 rows)
| start_utc | end_utc | generators_active | chain_tps |
|---|---|---|---|
| 2026-10-06T06:07:48Z | 2026-10-06T06:12:48Z | 8.3 | 4,951.5 |
| 2026-10-06T06:12:48Z | 2026-10-06T06:17:48Z | 9.0 | 4,448.4 |
| 2026-10-06T06:17:48Z | 2026-10-06T06:22:48Z | 10.0 | 4,155.3 |
| 2026-10-06T06:22:48Z | 2026-10-06T06:27:48Z | 10.0 | 3,804.0 |
| 2026-10-06T06:27:48Z | 2026-10-06T06:32:48Z | 10.0 | 4,093.9 |
| 2026-10-06T06:32:48Z | 2026-10-06T06:37:48Z | 10.0 | 3,614.2 |
| 2026-10-06T06:37:48Z | 2026-10-06T06:42:48Z | 10.0 | 3,776.5 |
| 2026-10-06T06:42:48Z | 2026-10-06T06:47:48Z | 10.0 | 3,901.0 |
| 2026-10-06T06:47:48Z | 2026-10-06T06:52:48Z | 1.7 | 159.6 |
| 2026-10-06T06:52:48Z | 2026-10-06T06:57:40Z | 1.0 | 54.0 |
The fine-grained chain throughput series (its step_s field gives the seconds per point) is published as series/tps-1s.json.
Other networks
| Measure | Telcoin Network (this run) | Tempo |
|---|---|---|
| Throughput | 3,889.6 tx/s (Avg chain TPS) | 16,455 TPS, shown as "Settled TPS" for the nightly run of 5 Oct 2026 (commit de5f5d3). perf.tempo.xyz describes it as "Median settled throughput across 130 measured blocks." |
| How throughput is measured | Chain transactions per block-timestamp second, summed over the 1808-s window when every load generator was in its mixed phase and divided by its length. | The methodology page defines TPS as "Transactions per second (derived from gas / avg tx gas)": gas used divided by the average gas of one transaction, reported as the median over the measured blocks. |
| Run length | 49.9 min merged mixed window; each generator was set to a 40 min mixed phase | 2m 39s for the 5 Oct 2026 nightly run (130 measured blocks at a 687 ms average block time), per perf.tempo.xyz. |
| Validators and placement | 10 validators and 1 observer across 5 regions: australia-southeast1 (Sydney), europe-west2 (London), europe-west4 (Netherlands), northamerica-northeast1 (Montréal), us-west2 (Los Angeles); load generators: one blaster per validator, in that validator's zone, sending only to it | perf.tempo.xyz labels the scenario "Public mix · multi-region". The methodology page says "Benchmarks run two local validators in consensus on a dedicated bare-metal server with 100 GiB pre-bloated state." The public tempoxyz/tempo repo has nightly workflows for both kinds of setup: bench-e2e runs on one bare-metal runner, and bench-e2e-multi-region defaults to 10 validators on cloud VMs (its regions input reads "Empty uses the selected cloud's five defaults."). The run page does not say which setup produced this number. |
| Hardware | c3-highcpu-8, 8 vCPUs, 16 GB memory; disk: hyperdisk-balanced, 100 GB, 20,000 IOPS, 800 MiB/s | The methodology page lists a dedicated bare-metal server: "AMD EPYC 4585PX (16C/32T, 4.3/5.7 GHz)", "128 GB 3600 MHz" memory and "2×960 GB NVMe + 2×1.92 TB NVMe via schelk". The repo's multi-region workflow uses cloud VMs instead (c4d-standard-48-lssd on GCP, c6id.8xlarge on AWS). |
| Workload | batched_self_send, contract_deploy, faucet_drip, native_transfer, stablecoin_approve, stablecoin_transfer, uniswap_v2_add_liquidity, uniswap_v2_remove_liquidity, uniswap_v2_swap, uniswap_v3_swap, uniswap_v4_swap, wtel_unwrap, wtel_wrap; 10 generators × 600 senders each | perf.tempo.xyz labels the scenario "Public mix"; the methodology page says "100% TIP-20 token transfers (ERC-20 equivalent)". The run used 1,000 accounts and 4 TIP-20 tokens. |
| State | Fresh chain from genesis for every run | The methodology page says "100 GiB pre-bloated state". |
| Latency (as each source defines it) | Time to land, median: 3.38 s; Time to land, p95: 17.98 s; Commit latency, p95: 5.75–6.25 s | Block time of 687 ms, which the methodology page defines as "Average wall-clock time between blocks". The run page gives no per-transaction latency. Separately, tempo.xyz says "Blocks finalize in ~0.6 seconds with no re-orgs." (a network claim, not a figure from this benchmark). |
| Source and fetch date | This page; every figure names its source field. | perf.tempo.xyz nightly run of 5 Oct 2026 (commit de5f5d3), the perf.tempo.xyz methodology page, tempo.xyz, and the benchmark workflows in the tempoxyz/tempo repo. All fetched on 5 Oct 2026. |
Sources for Tempo (fetched 2026-10-05):
- perf.tempo.xyz (nightly benchmark results)
- perf.tempo.xyz methodology
- tempo.xyz
- tempoxyz/tempo repository
- bench-e2e workflow (bare-metal)
- bench-e2e-multi-region workflow
Tempo's setup may not match ours: its methodology page describes two validators on one bare-metal server, while our runs use ten validators across five regions, so read this table as context, not a ranking.
This table is context, not a ranking.
Caveats
- The chain did not stall this time. Validator-05 was killed 59 s before the epoch close, as planned, and the other nine validators kept committing: chain throughput in the one-minute buckets from 06:34 to 06:46 UTC ranged from 3,085 to 4,355 tx/s, the longest run of seconds without a landed transaction was 9 s (from 06:38:00, at the epoch boundary), and epoch 4 opened on time at 06:37:59. In the all-generators window the average was 3,921 tx/s before the kill and 3,833 tx/s after it. On the 6 October run (r20261006-0053) the same kill, 42 s before the close, was followed by 301 s with no landed transaction.
- Validator-05 restarted at 06:41:58 UTC and was in catch-up mode from the first Prometheus sample at 06:42:25 to 06:54:55 (750 s). Its sampler shows its executed block held at 27,969 until about 06:44:30, then rising by about 800 to 870 blocks a minute, close to the 780 a minute the loaded chain produced, so the gap of about 5,700 blocks stayed level while the load ran. It reached the chain head, block 36,577, at 06:54:31, after block production had stopped at 06:48:21. Load generator 05, which sends only to validator-05, landed no transactions from 06:38 to the end of its mixed phase at 06:47:48; the 8,294 transactions it sent after the restart waited in validator-05's pool until the validator rejoined. On 29 September the same node spent 1,500 s in catch-up mode; on the 6 October run the chain was stalled through the outage, so there was almost nothing to catch up on.
- Eight load generators finished their setup transactions within 12 s and began their mixed phase between 06:07:48 and 06:07:52 UTC. Generator 04 took 161 s and began at 06:10:21. Generator 03 stopped waiting at the 600-second preamble limit with 62 of its 600 senders still missing a setup step (134 setup transactions unlanded) and began at 06:17:40. Both are the australia-southeast1 generators. The all-generators window is therefore 06:17:40 to 06:47:48 UTC (1,808 s), and the merged mixed window, 06:07:48 to 06:57:40 UTC (2,992 s, average 3,305 tx/s), ends with about ten minutes in which only generators 03 and 04, then only 03, were running. On the 6 October run eight generators hit the preamble limit and only two started on time.
- After the eight on-time generators stopped at 06:47:48 to 06:47:52 UTC, no block carrying transactions was produced from 06:48:21 (block 36,577) to 06:54:33 (block 36,578, empty), although consensus rounds kept advancing on all ten validators at about 2.6 a second and sub-DAGs were committed at about 1.3 a second, each made of certificates from the seven validators outside Australia only, and generators 03 (mixed phase until 06:57:40) and 04 (until 06:50:21) were still sending. Validators 03 and 04 kept sealing batches and re-proposing them but certified none of their headers in that period: with the chain idle, each of their headers was superseded by the next round's before six remote votes could arrive, the effect described in the next caveat. The gap closed at 06:54:31, when validator-05 rejoined and proposed batches of its own; rounds slowed for about 20 s, validator-03 certified ten headers, about 15,800 transactions landed in the minute from 06:54, and none after that before the empty block that closed epoch 4 at 06:57:59. Generators 03, 04 and 05 ended with 9,888, 9,788 and 9,827 accepted transactions that never landed. The harness recorded 06:48:31 to 06:52:04 as a commit stall (its watch ended while the height was still flat); consensus did not stall. A transaction sent to a far validator lands only when the chain is slow enough for that validator to certify a header. How the node should handle this is an open question.
- In the idle window before the load (05:30 to 05:59 UTC) every validator proposed 3,205 headers. The two in Sydney (03 and 04) had none of theirs certified; the other eight had all of theirs certified, reaching their own quorum 180 to 184 ms after proposing from Montréal and 282 to 309 ms from Los Angeles, London and the Netherlands, while the round period was about 349 ms (median) and 395 ms (95th percentile) on every node. Each node's certification times sit within 3 ms of their median: a fixed path, not jitter. Under load (06:10 to 06:35) the median round period stretched to about 900 ms and the Sydney validators certified 66.8 % of their headers, against 95.4 % for Los Angeles and 90.4 % for the Netherlands. Over the whole mixed phase they certified 60.4 % and 67.8 % (23.6 % and 24.2 % on the 6 October run), led 0.1 % and 0.8 % of the committed sub-DAGs against a fair share of 10 %, and about a third of the certificates they did form (526 of 1,545 and 533 of 1,730) were never committed because the other seven validators already held seven parents when they arrived. The certifier fix removed the cancelled vote collections (fleet-wide 'failed to get vote: Canceled' fell from 7,005 to 1,326, 81 %) but cannot shorten the round trip, so a validator whose votes take longer than one round to return still certifies nothing while the chain is idle. This finding is what the later runs on the consensus delays were set up to measure.
- The offered load was uneven. Generators 03 and 04 sent about 239,000 and 401,000 transactions over the blast, against about 820,000 to 1.9 million for the other eight. Their transactions also waited longest: median time to land 14.6 s for generator 03 and 6.8 s for generator 04 (95th percentile 378.5 s and 325.7 s), against 7 to 26 s at the 95th percentile for the other generators except 07 (303.5 s). These waits are part of the whole-blast p95 of 18.0 s.
- Every generator's 99th-percentile time to land is between 301 s and 635 s. Each generator logged that its validator had dropped transactions it had already accepted, about 23,600 times across the ten generators (about 23,700 on the 6 October run), and resent them; those transactions make up most of the slow tail.
- Consensus committed less often under load than on the 6 October run: about 0.42 sub-DAGs and 0.90 rounds a second before the kill, against 0.59 and 1.21, and 0.32 and 0.69 while validator-05 was down. The commit latency p95 rose from 4.77 to 4.88 s to 5.75 to 6.25 s. The cause has not been looked at.
- The load generators ran on n2-standard-2, not the e2-standard-2 used on 6 October and 29 September, because e2 capacity was out of stock in australia-southeast1-c. Both are 2-vCPU machines, but the load-generator machine type differs between this run and both comparison runs.
- Against the 6 October run the node build differs by the certifier change and the load-generator machine type differs; the generator image, fleet shape, consensus timing and chaos plan are the same. The 6 October run stalled for 301 s inside its all-generators window and this one did not, so the difference in averages mostly reflects the stall. The comparison with 29 September (r20260929-1647) also spans a different node build and the 3 s, 1 s, 1 s devnet consensus timing, so no difference can be assigned to any one change.
- The harness working tree had one uncommitted change when the run was deployed (harness 3a31603, recorded as dirty): a five-line fix to the pattern that excludes files from the archive copy, committed as f89823d while the run was in progress. It touched the archive step only, not the deploy, chaos or measurement path. The collect-time archive copy failed on the uncorrected pattern and the teardown gate's copy succeeded with the fix.
- This configuration has been run once, so run-to-run variation is not measured.
- The tag adiri:mmap-subsecond-certfix-10-6-2026 is mutable. The harness resolved it to digest sha256:caeb6be2… before any node was created, and every node ran that digest.
- Per-validator rows cover 9 validators; bench-validator-05 excluded as chaos target(s).
- Recovery: demoted / rejoined are the first 15-s Prometheus samples showing the change, so they trail the node's own log times by up to a scrape interval plus one step.
- Recovery: no tn_batch_builder_seal_failures_total rate series for bench-validator-05; failed_reports unavailable.
- Recovery.failed_reports is the published value, not recomputed from the run data.
- The harness working tree had uncommitted changes when the run was deployed.
Full report
Changes from the original report:
- The site adds a banner at the top of the report that links back to this page.
SHA-256 of the committed file, before the banner is added: 5a070e8ae30cd4996ff42a345b2a006ce9a97bad3d8cce6dd894d109ae25ff7d