The v0.2.0-devnet release on operator-deployed nodes with closed admission

2026-11-03 · started 19:22 UTC · ended 2026-11-04 00:39 UTC · merged mixed window 42 min 55 s

telcoin-network 3f9b0709c · devnet/v0.2.0-devnet · commit verified from node startup logs

16 node starts across 7 instances logged telcoin-network 0.1.0 (3f9b0709cd52bfbd384221be5c4932566c9be45e) The node image probe recorded the same commit before deploy Node logs: fork schedule: seed_signature, multi_workers, prevrandao, leader_seeded_ordering, and subsecond_timestamp active from genesis Commit 3f9b0709 is the head of the public branch devnet/v0.2.0-devnet, the build of the v0.2.0-devnet release that the devnet runs. The run did not resolve a tag: the harness took the devnet's node image reference, which names the image by registry digest (adiri@sha256:5257e768…, the image tagged v0.2.0-devnet), the digest recorded in the devnet's signed release manifest for v0.2.0-devnet (releases/v0.2.0-devnet). The build contains the certifier fix of r20261006-0458 (squashed as cfc9e34f, #1579: a re-proposed identical header keeps its in-flight vote collection) and not the lazy protocol negotiation of r20261007-0316 (b0838dd38): outbound substreams are negotiated with strict multistream-select V1, so a vote request waits for the protocol confirmation before it is sent. Against r20261007-0316's build f8ff81bf4 it has 51 commits that build lacks, among them the consensus-storage rework (#1558) and the October staging merges, and it lacks 36, among them b0838dd38; the two differ in 556 files.

We repeated the 40-minute chaos run of 7 October (r20261007-0316) on the v0.2.0-devnet release, the build the devnet runs, deployed the way an operator deploys it. The fleet kept its shape, ten c3-highcpu-8 validators with Hyperdisk Balanced disks, one in each of ten regions, and an observer, except that validator-05 ran in Ashburn instead of Montréal, where no c3-highcpu-8 capacity was left. The node build, telcoin-network 3f9b0709, has the certifier fix of 6 October but not the lazy protocol negotiation that halved each validator's time to quorum on 7 October, and it differs from that run's build in much else. The nodes were set up with the release's operator scripts, accepted connections only from the peers named in their network configuration (closed admission) from their first start, and started together in the same second; the consensus timing was the devnet's, 1,000 ms maximum and 350 ms minimum header delay and 250 ms maximum batch delay, the values the comparison blast ran with. With all ten load generators running, average chain throughput was 4,037 tx/s, against 5,434 tx/s on r20261007-0316 and 3,890 tx/s on r20261006-0458; the chain produced 13.5 blocks a second over the mixed phase (18.9 and 11.5), and the median time for a transaction to land was 3.49 s and the 95th percentile 14.17 s (2.44 and 12.56 s; 3.38 and 17.98 s). London and the Netherlands certified only 5.2 % and 5.0 % of their own headers while the chain was idle (99.9 % on r20261007-0316), 70.0 % and 89.1 % under load and 41.6 % and 26.6 % in the cool-down: every vote took two network round trips instead of one, and their quorum, at 521 and 537 ms, arrived after the 451 ms idle round had moved on. Every other validator apart from the restarted validator-05 certified at least 95 % in each phase its log covers, except Hong Kong at 90.5 % late in the mixed phase. Validator-05 was killed 60 s before an epoch close and held down for 300 s; the chain did not stall and the epoch closed on time. The restarted validator then could not stay in the committee: it came within the rejoin window 20 minutes after its restart, while the load still ran, and was promoted and demoted again eleven times between 00:10:48 and 00:25:11 UTC, each time starting from an empty DAG with a stale header that its peers refused as too old. It rejoined only when epoch 7 began, 36 minutes after its restart and 41 minutes after the kill, and its load generator, which sends only to it, landed nothing in the last 27 minutes of its mixed phase.

Compared against r20261007-0316.

Key figures

4,037.2tx/s
Average chain throughput, all load generators running
all generators active
5,957.1tx/s
Best 60 seconds, all load generators running
all generators active
14.17s
Time to land, p95
whole run
4.68–29.00s
Commit latency, p95
merged mixed window

How it was run

Software

Upstream committelcoin-network 3f9b0709cd52bfbd384221be5c4932566c9be45e · devnet/v0.2.0-devnet · commit verified from node startup logs
Node imageus-docker.pkg.dev/telcoin-network/tn-public/adiri@sha256:5257e768e657586a19e31308906bfacac0df95e5ca3da29a68f818f159d4a610
Node image digestsha256:5257e768e657586a19e31308906bfacac0df95e5ca3da29a68f818f159d4a610
Node version0.1.0
Fork schedulesubsecond-from-genesis
Load generator commit3911723
Load generator image digestsha256:75d384428a8d94520c28903111c509985009d01815e2e62d26ff3bb066c0f157
Analysis commit3911723
Harness commit5269ae2390d938fe1242d0f88f5a052e1cda2aa2
Harness submodulescommon: 0af76982fe3eb5423241b397e150a6abc8d17b85
tn-node-deployment: d9446f720026fa5ce8b64fce9074e56b05b3f4eb

Fleet

Fleetc3-highcpu-8 + Hyperdisk Balanced (c3-hdb), ten regions, validator-05 in Ashburn
Validators10
Observers1
Machine typec3-highcpu-8
vCPUs per node8
Memory per node16 GB
Diskhyperdisk-balanced, 100 GB, 20,000 IOPS, 800 MiB/s
Regionsasia-east2 (Hong Kong), asia-northeast1 (Tokyo), asia-northeast3 (Seoul), asia-south1 (Mumbai), asia-southeast1 (Singapore), australia-southeast1 (Sydney), europe-west2 (London), europe-west4 (Netherlands), us-east4 (Ashburn), us-west2 (Los Angeles) (10)
Load generators10 generators × e2-standard-2 × 600 senders each
Load placementone blaster per validator, in that validator's zone, sending only to it

Chain

Chain ID32286
Networkbench
Epoch length1,200 s (20 min)

Consensus parameters

Max header delay1,000 ms
Min header delay350 ms
Max batch delay250 ms
Vote timeout5,000 ms
Parameters fileconsensus-parameters.yaml
Confirmed bynode startup logs

Workload

Warm-up120 s (2 min)
Mixed phase2,400 s (40 min)
Cool-down180 s (3 min)
Transaction typesbatched_self_send, contract_deploy, faucet_drip, native_transfer, stablecoin_approve, stablecoin_transfer, uniswap_v2_add_liquidity, uniswap_v2_remove_liquidity, uniswap_v2_swap, uniswap_v3_swap, uniswap_v4_swap, wtel_unwrap, wtel_wrap

Chaos events

MarkerTime (UTC)NodeActionDetail
C12026-11-03 23:45:15bench-validator-05docker kill -s KILL telcoinsystemctl stop holds it down (unit has Restart=on-failure). bench-blaster-05 targets ONLY this node, so its sends fail for the outage by design; the generator resends on connection errors
C22026-11-03 23:50:15bench-validator-05systemctl start telcoindocker start is impossible: the unit runs docker run --rm
C32026-11-03 23:55:15bench-observer-11systemctl restart telcoinobserver restart

Steps

  1. The node image was the devnet's v0.2.0-devnet release. The harness takes the devnet's image reference, which names the image by registry digest (sha256:5257e768…, the image tagged v0.2.0-devnet and the digest in the devnet's signed release manifest), so there was no tag to resolve. Before creating anything it ran that image to confirm that it reports telcoin-network 0.1.0 built from commit 3f9b0709, has the sub-second timestamp fork active from genesis and contains the log messages later checks rely on, and it recorded the committed consensus parameters file, which holds the devnet's values: maximum header delay 1,000 ms, minimum header delay 350 ms and maximum batch delay 250 ms.
  2. Preflight checked access, tools, quotas and a clean genesis directory, and confirmed the load-generator image built from tn-transaction-generator commit 3911723. Its source tree is identical to that of 9f61303, the generator of r20261007-0316; the image was rebuilt for this run.
  3. The harness created ten validators and one observer on c3-highcpu-8 (8 vCPU, 16 GB), each with a 100 GB Hyperdisk Balanced boot disk provisioned at 20,000 IOPS and 800 MiB/s, one validator in each of ten regions (Los Angeles, London, Sydney, Tokyo, Ashburn, the Netherlands, Singapore, Hong Kong, Seoul and Mumbai) with the observer in Los Angeles, and created the run's firewall rules. Validator-05 was planned for Montréal, as on 7 October, but c3-highcpu-8 was out of stock in both Montréal zones that offer it (three attempts from 19:24 to 20:25 UTC), so it was created in us-east4-b (Ashburn) at 20:27 UTC; its load generator later ran in the same zone.
  4. Each node's keys were generated on the node, one node at a time, by the operator scripts of the v0.2.0-devnet release (tn-node-deployment d9446f7, the v2 scripts). One node's session dropped after its key had been made; its public node information was copied from the node by hand. The harness then built genesis for a separate benchmark chain (chain id 32286, 20-minute epochs) with the devnet's consensus parameters.
  5. Before any node started, the harness installed on every node a network configuration that puts peer admission in Closed mode, so that a node accepts connections only from the peers the file names. Each node was then finalized by the release's scripts, run on its own just long enough to show that the service started and answered RPC, and stopped again, enabled but inactive; no node joined a network at this stage.
  6. While every node was stopped, one session per node wrote the benchmark settings into its start script: RPC and websocket listening on all interfaces, a local metrics endpoint, and the log filter (debug for the certifier, the proposer, the primary and its network handler, except the primary's certificate validator, manager and fetcher at info, and info elsewhere). No per-node consensus override was set. All eleven confirmed.
  7. All eleven nodes were then started at one due time, 22:26:08 UTC: every node began in that second (spread 0 s, none late) on the first attempt. The harness then checked each node in turn: 60 s of stable running with no restart and the unit enabled, the Closed admission file still in place, no configuration error or panic in its logs, and the container running the pinned image with a read-only root filesystem. All eleven passed. No node was restarted after the start except by the chaos plan below.
  8. With every validator active, the harness relabelled each node's metrics agent so that every sample from these nodes is recorded under the benchmark network, never the live devnet. The node already served the metrics endpoint set before the start, so nothing was restarted: each node's container start time and restart count were unchanged.
  9. A verify-only pass then read back from each running node, one session each, its chain id, its consensus progress since the start, its start-script settings, its parameters file and its boot lines, and would have failed any node whose unit had restarted. All eleven had logged commit 3f9b0709, the three consensus delays, the 5,000 ms vote timeout and the sub-second fork from genesis, and all passed by 23:00:09 UTC; an on-node sampler was then started on each node. The health and admission gate confirmed that all eleven nodes reported to Prometheus under the benchmark network with consensus rounds rising, read each node's metrics from 23:01:51 to 23:02:46 UTC and found admission mode 2 (Closed) on the primary and the worker network with no fallback on all eleven, and a read of the ConsensusRegistry through an RPC tunnel confirmed the 1,200 s epoch duration.
  10. It deployed the devnet contract set that the mixed workload calls, from 23:02:46 to 23:21:13 UTC. Three resets of the RPC tunnel stopped the deployment; each relaunch skipped the contracts already deployed, and the last ran over a single ssh port-forward that replaced the per-connection tunnel. Apart from the contract deployment, the chain was idle from 23:00:09 UTC to the funding step at 23:23:14.
  11. A one-off funder VM funded 600 sender accounts for each of ten load generators from 23:23:14 to 23:24:33 UTC. The generators, on e2-standard-2, one per validator in that validator's zone and sending only to it, started together at 23:29:59 UTC: a preamble of setup transactions (11 to 186 s per generator), 120 s of warm-up, 2,400 s of mixed load across 13 transaction types and up to 180 s of cool-down, cut short after 30 s in which none of the generator's own transactions landed. No network delay was injected.
  12. The chaos schedule was timed to the chain's epochs. Before the load generators started, the harness read the ConsensusRegistry at a pinned block (block 7, epoch 3) and chose the close of epoch 4, due at 23:46:14 UTC. It staged three actions on the target nodes, each to fire on the node's own clock: bench-validator-05's container killed with SIGKILL 59 s before that close and held down, the node restarted 300 s after the kill, and the observer's node restarted 300 s after that. The kill began at 23:45:15.175, the restart at 23:50:15.199 and the observer restart at 23:55:15.144, each less than 0.2 s after its planned second. Epoch 5 opened at 23:46:15, so the outage crossed the boundary.
  13. During the run the harness sampled the fleet's highest block and recorded any period in which it did not advance between samples; it recorded none.
  14. Afterwards, from 00:23:10 to 00:26:36 UTC, the harness collected each generator's results and the on-node sampler files, merged the generator results with aggregate.py, derived the sizing figures with derive-sizing.py, and pulled 60-second machine metrics from Cloud Monitoring and 15-second node metrics from Prometheus (23:02:46 to 00:23:10 UTC).
  15. Teardown saved every node's container log, its current unit log file, sampler files and service journal, computed each validator's share of its own headers that reached a quorum, and began deleting the fleet. It was stopped by hand at 00:35:25 UTC, after four validators (04, 08, 09 and 10) had been deleted, when the saved validator logs turned out not to reach back to the start: the container log is capped, and the unit log had been rotated at midnight. The rotated unit log was copied by hand from the six validators still running (01, 02, 03, 05, 06 and 07) from 00:36:16 to 00:37:19 UTC. A second teardown run, trusting the first run's pull records, recomputed the certification with the rescued files, verified the saved results at 00:39:47, deleted the rest of the fleet, copied the run's storage bucket into the saved results and finished at 00:47:49 UTC with no cloud resource left; a local tunnel process it had missed was stopped by hand afterwards. The publishing pipeline then read each node's logged commit, consensus delays and fork schedule back from the saved logs.

The harness scripts live in a private repository; the steps above describe what they do.

Results

MetricValueWindowSourceMethod
Average chain throughput, all load generators running4,037.2 tx/sall generators activecombined.series.per_second[chain_tx]Chain transactions per block-timestamp second, summed over the 2226-s window when every load generator was in its mixed phase and divided by its length.
Best 60 seconds, all load generators running5,957.1 tx/sall generators activecombined.series.per_second[chain_tx]Highest mean chain tx/s over any 60 consecutive seconds inside the all-generators window (aggregate.py best_window_mean).
Transactions landed over the whole blast12,205,837 txwhole runcombined.totals.landed_chainSum of tx_count over every block the load generators saw during the blast, own and foreign transactions alike.
Blocks per second13.47 blocks/smerged mixed windowcombined.blocks.rows[timestamp]Chain blocks whose timestamp falls in the merged mixed window, divided by its length (aggregate.py landed.capacity.blocks_per_sec over the same window).
Time to land, median3.49 swhole runcombined.latency.ts_ms.p50Block timestamp minus submit time over every landed own transaction in the blast, from the load generators' merged latency histograms (aggregate.py merge_latency), in seconds.
Time to land, p9514.17 swhole runcombined.latency.ts_ms.p95Block timestamp minus submit time over every landed own transaction in the blast, from the load generators' merged latency histograms (aggregate.py merge_latency), in seconds.
Commit latency, p954.68–29.00 smerged mixed windowmetrics.prometheus.series.tn_primary_commit_latency_seconds.by_node[healthy validators].p95For each healthy validator, the nearest-rank p95 over the merged mixed window of its 15-s histogram_quantile(0.95) commit-latency series; reported as the range over validators.
Execution time per consensus output1.98–2.04 smerged mixed windowmetrics.prometheus.series.tn_engine_execution_duration_seconds.by_node[healthy validators].meanFor each healthy validator, the nearest-rank median over the merged mixed window of its 15-s mean execution-duration series; reported as the range over validators.
Write throughput, p95, busiest validator584.6 MiB/smerged mixed windowmetrics.gcp[bench-validator-02].disk_write_bytes_per_sec p95Nearest-rank p95 of the 60-s mean write-byte rate over the merged mixed window for each healthy validator; the highest (bench-validator-02) is reported.
Write operations, p95, busiest validator3,064 IOPSmerged mixed windowmetrics.gcp[bench-validator-07].disk_write_ops_per_sec p95Nearest-rank p95 of the 60-s mean write-operation rate over the merged mixed window for each healthy validator; the highest (bench-validator-07) is reported.
CPU, p95, busiest validator27.4 %merged mixed windowmetrics.gcp[bench-validator-09].cpu_utilization p95Nearest-rank p95 of the 60-s CPU utilisation over the merged mixed window for each healthy validator; the highest (bench-validator-09) is reported.
Node resident memory, p95, busiest validator7.29 GiBmerged mixed windowmetrics.gcp[bench-validator-06].telcoin_rss_bytes p95Nearest-rank p95 of the 60-s telcoin process resident memory over the merged mixed window for each healthy validator; the highest (bench-validator-06) is reported.
IO pressure, highest reading, range over validators31.5–33.2 %merged mixed windowsampler[healthy validators].psi_io_some_avg10 maxFor each healthy validator, the highest 10-s 'some' IO pressure reading the on-node sampler logged inside the merged mixed window; reported as the range over validators.

Recovery

Nodebench-validator-05
Killed2026-11-03 23:45:15 UTC
Restarted2026-11-03 23:50:15 UTC
Rejoined2026-11-04 00:26:24 UTC
Catch-up time2,153 s (35 min 53 s)
Failed batch reports2

Chain TPS per 5-minute interval

Chain TPS per 5-minute interval 9 intervals from 23:32 to 00:15 UTC; average 3,780 TPS, highest 5,986 TPS. The merged mixed window ran from 23:32 to 00:15 UTC. TPS 0 1,000 2,000 3,000 4,000 5,000 6,000 23:32–23:37 UTC: 5,986 TPS, 8.5 generators active 23:37–23:42 UTC: 5,064 TPS, 10.0 generators active 23:42–23:47 UTC: 4,445 TPS, 10.0 generators active 23:47–23:52 UTC: 4,228 TPS, 10.0 generators active 23:52–23:57 UTC: 3,908 TPS, 10.0 generators active 23:57–00:02 UTC: 3,448 TPS, 10.0 generators active 00:02–00:07 UTC: 3,045 TPS, 10.0 generators active 00:07–00:12 UTC: 3,512 TPS, 10.0 generators active 00:12–00:15 UTC: 385 TPS, 2.6 generators active avg 3,780 23:32 UTC 23:42 UTC 23:52 UTC 00:02 UTC 00:12 UTC C1 2026-11-03 23:45:15 UTC: docker kill -s KILL telcoin on bench-validator-05 (systemctl stop holds it down (unit has Restart=on-failure). bench-blaster-05 targets ONLY this node, so its sends fail for the outage by design; the generator resends on connection errors) C1 C2 2026-11-03 23:50:15 UTC: systemctl start telcoin on bench-validator-05 (docker start is impossible: the unit runs docker run --rm) C2 C3 2026-11-03 23:55:15 UTC: systemctl restart telcoin on bench-observer-11 (observer restart) C3

Each bar is the average chain TPS over one 5-minute interval of the merged mixed window (23:32–00:15 UTC); fainter bars had fewer generators active than the maximum, the dashed line is the average of all bars and amber lines mark the chaos events listed below.

  • C1 2026-11-03 23:45:15 UTC: docker kill -s KILL telcoin on bench-validator-05 (systemctl stop holds it down (unit has Restart=on-failure). bench-blaster-05 targets ONLY this node, so its sends fail for the outage by design; the generator resends on connection errors)
  • C2 2026-11-03 23:50:15 UTC: systemctl start telcoin on bench-validator-05 (docker start is impossible: the unit runs docker run --rm)
  • C3 2026-11-03 23:55:15 UTC: systemctl restart telcoin on bench-observer-11 (observer restart)
5-minute data (9 rows)
start_utcend_utcgenerators_activechain_tps
2026-11-03T23:32:12Z2026-11-03T23:37:12Z8.55,985.7
2026-11-03T23:37:12Z2026-11-03T23:42:12Z10.05,064.1
2026-11-03T23:42:12Z2026-11-03T23:47:12Z10.04,445.4
2026-11-03T23:47:12Z2026-11-03T23:52:12Z10.04,228.0
2026-11-03T23:52:12Z2026-11-03T23:57:12Z10.03,908.3
2026-11-03T23:57:12Z2026-11-04T00:02:12Z10.03,447.9
2026-11-04T00:02:12Z2026-11-04T00:07:12Z10.03,045.4
2026-11-04T00:07:12Z2026-11-04T00:12:12Z10.03,512.4
2026-11-04T00:12:12Z2026-11-04T00:15:07Z2.6385.2

The fine-grained chain throughput series (its step_s field gives the seconds per point) is published as series/tps-1s.json.

Other networks

MeasureTelcoin Network (this run)Tempo
Throughput4,037.2 tx/s (Avg chain TPS)16,455 TPS, shown as "Settled TPS" for the nightly run of 5 Oct 2026 (commit de5f5d3). perf.tempo.xyz describes it as "Median settled throughput across 130 measured blocks."
How throughput is measuredChain transactions per block-timestamp second, summed over the 2226-s window when every load generator was in its mixed phase and divided by its length.The methodology page defines TPS as "Transactions per second (derived from gas / avg tx gas)": gas used divided by the average gas of one transaction, reported as the median over the measured blocks.
Run length42.9 min merged mixed window; each generator was set to a 40 min mixed phase2m 39s for the 5 Oct 2026 nightly run (130 measured blocks at a 687 ms average block time), per perf.tempo.xyz.
Validators and placement10 validators and 1 observer across 10 regions: asia-east2 (Hong Kong), asia-northeast1 (Tokyo), asia-northeast3 (Seoul), asia-south1 (Mumbai), asia-southeast1 (Singapore), australia-southeast1 (Sydney), europe-west2 (London), europe-west4 (Netherlands), us-east4 (Ashburn), us-west2 (Los Angeles); load generators: one blaster per validator, in that validator's zone, sending only to itperf.tempo.xyz labels the scenario "Public mix · multi-region". The methodology page says "Benchmarks run two local validators in consensus on a dedicated bare-metal server with 100 GiB pre-bloated state." The public tempoxyz/tempo repo has nightly workflows for both kinds of setup: bench-e2e runs on one bare-metal runner, and bench-e2e-multi-region defaults to 10 validators on cloud VMs (its regions input reads "Empty uses the selected cloud's five defaults."). The run page does not say which setup produced this number.
Hardwarec3-highcpu-8, 8 vCPUs, 16 GB memory; disk: hyperdisk-balanced, 100 GB, 20,000 IOPS, 800 MiB/sThe methodology page lists a dedicated bare-metal server: "AMD EPYC 4585PX (16C/32T, 4.3/5.7 GHz)", "128 GB 3600 MHz" memory and "2×960 GB NVMe + 2×1.92 TB NVMe via schelk". The repo's multi-region workflow uses cloud VMs instead (c4d-standard-48-lssd on GCP, c6id.8xlarge on AWS).
Workloadbatched_self_send, contract_deploy, faucet_drip, native_transfer, stablecoin_approve, stablecoin_transfer, uniswap_v2_add_liquidity, uniswap_v2_remove_liquidity, uniswap_v2_swap, uniswap_v3_swap, uniswap_v4_swap, wtel_unwrap, wtel_wrap; 10 generators × 600 senders eachperf.tempo.xyz labels the scenario "Public mix"; the methodology page says "100% TIP-20 token transfers (ERC-20 equivalent)". The run used 1,000 accounts and 4 TIP-20 tokens.
StateFresh chain from genesis for every runThe methodology page says "100 GiB pre-bloated state".
Latency (as each source defines it)Time to land, median: 3.49 s; Time to land, p95: 14.17 s; Commit latency, p95: 4.68–29.00 sBlock time of 687 ms, which the methodology page defines as "Average wall-clock time between blocks". The run page gives no per-transaction latency. Separately, tempo.xyz says "Blocks finalize in ~0.6 seconds with no re-orgs." (a network claim, not a figure from this benchmark).
Source and fetch dateThis page; every figure names its source field.perf.tempo.xyz nightly run of 5 Oct 2026 (commit de5f5d3), the perf.tempo.xyz methodology page, tempo.xyz, and the benchmark workflows in the tempoxyz/tempo repo. All fetched on 5 Oct 2026.

Sources for Tempo (fetched 2026-10-05):

Tempo's setup may not match ours: its methodology page describes two validators on one bare-metal server, while our runs use ten validators across five regions, so read this table as context, not a ranking.

This table is context, not a ranking.

Caveats

  • The comparison run, r20261007-0316, had the same fleet shape and load shape and the same effective consensus timing: ten c3-highcpu-8 validators with the same disks in the same zones except validator-05 (next caveat), ten e2-standard-2 load generators with 600 senders each, a generator built from the same source tree (9f61303 there, 3911723 here), 120 s of warm-up, 2,400 s of mixed load and 180 s of cool-down, the same chaos plan, and 1,000/350/250 ms, which r20261007-0316 ran inside its stage2 arm. The node build differs. This run's commit 3f9b0709 has the certifier fix but not the lazy protocol negotiation (b0838dd38) that r20261007-0316's build f8ff81bf4 added, so each vote request again waits for a protocol confirmation; r20261007-0316 measured that change as roughly halving each validator's time from its proposal to its own quorum. The builds also differ in 51 commits on this side and 36 on the other, among them the consensus-storage rework (#1558). The nodes were deployed differently as well (the release's operator scripts, closed admission, one coordinated start) and logged with a narrower debug filter. Against r20261007-0316, average chain throughput with all generators running fell from 5,434 to 4,037 tx/s, blocks a second over the mixed phase from 18.85 to 13.47 and committed rounds a second from 1.40 to 1.07, and the time to land rose from 2.44 to 3.49 s at the median and from 12.56 to 14.17 s at the 95th percentile. No difference can be assigned to any one change.
  • Validator-05 and its load generator ran in us-east4-b (Ashburn), not northamerica-northeast1-b (Montréal) as on r20261007-0316. c3-highcpu-8 was out of stock in northamerica-northeast1-b (twice) and northamerica-northeast1-c, the only Montréal zones that offer it; the preflight check that the machine type is offered in every zone does not test stock. The latency matrices measured on 6 and 7 October have no us-east4 row. Validator-05 is the chaos target, so it is excluded from every per-validator figure on this page, as on r20261007-0316.
  • Validator-05 did not get back into the committee until epoch 7 began. It restarted at 23:50:15 UTC, found itself behind 16 s later and followed the chain as an inactive member, executing about 1,000 to 1,100 blocks a minute while the loaded chain produced 600 to 950, so its gap shrank from 4,912 blocks at 23:50:49 to about 500 at 00:08:49 but never closed while the load ran. At 00:10:48 it came within the 20-round rejoin window and was promoted; it was demoted 34 s later, and from 00:14:02 to 00:25:11 it was promoted ten more times and demoted 8 to 10 s after each. Each time it started from a DAG restored empty and re-proposed a stale header that every peer refused as too old (1,203 refusals in all), made no execution progress, and fell more than 40 rounds behind the committee, the activity window past which an active validator is demoted. After each demotion it made no progress for about 61 s, until its sync path re-requested the missing range, and was promoted again about 4 s later: a cycle of about 75 s. It left the loop only at the epoch-7 entry at 00:26:24.96 UTC, when every node starts again from round 1, and assembled its first own certificate at 00:26:26: 36 min 10 s after its restart and 41 min 11 s after the kill (24 min 11 s and 29 min 12 s on r20261007-0316, whose restarted validator never reached the rejoin window under load). From 23:52 to 00:26 it refused every vote request from its peers as too new, so the committee ran on nine voters for about 41 minutes, counting the outage. Neither comparison run logged repeated promotions. With longer epochs or continuous load the same node would keep cycling until the next epoch began.
  • The kill fired 0.175 s after its planned second, 60 s before the epoch-4 close at 23:46:15 UTC (r20261007-0316: 41.8 s late, 17 s before its close). The chain did not stall: with validator-05 down, chain throughput ran 3,219 to 4,477 tx/s in the minutes from 23:46 to 23:49 UTC, epoch 5 opened on time, and the harness recorded no commit stall.
  • The recovery record on this page takes its rejoin time (00:26:24 UTC), catch-up duration (2,153 s, from the validator's switch to catch-up mode at 23:50:32 to its rejoin) and failed-report count (2) from validator-05's own log. The 15-second Prometheus window ends at 00:23:10 UTC, before the rejoin, and shows only the first, transient promotion, at 00:11:01; the node-mode computation therefore leaves the rejoin and catch-up open, and the values here come from the log. The two batch-seal failures, at 00:15:24 and 00:21:35, both came at a demotion, and the counter's rate series begins after the first.
  • Load generator 05 sends only to validator-05, so its lane carried nothing from the kill to the end of its mixed phase: it landed none of its own transactions from 23:45:17 to 00:12:17 UTC (1,620 s, about 27 of the 37 minutes of the all-generators window). It saw connection errors until the restart, and from 23:51:07 to 00:10:47 the follower answered every send with 'validator is inactive; retry another endpoint', which a generator with one endpoint cannot act on. It ended with 9,890 accepted transactions unlanded. The 7,354 transactions in validator-05's pool when it rejoined were proposed in the first rounds of epoch 7 and executed from 00:26:28 UTC, after the generators had stopped. Before the kill this lane landed about 716 tx/s; the other nine lanes together fell about 4 % (from about 4,040 to 3,880 tx/s) across the outage, so nearly all of the chain's drop after the kill is this lane, and every throughput figure on this page includes it.
  • While it followed, validator-05 tried to forward the transactions in its pool to the committee's advertised RPC endpoints and logged 'could not forward transaction to any advertised validator RPC' 79,551 times, and 227 times that it was demoting unreachable endpoints. That is an artefact of the benchmark network: its nodes advertise placeholder RPC addresses under the reserved .invalid domain, which resolve nowhere by design, so no forward could succeed. Whether a follower's forwards succeed on a network with reachable endpoints is not measured here.
  • Both loaded epoch boundaries left the chain without a block for longer than on the comparison runs: 12 s at the epoch-5 close (23:46:15 UTC, validator-05 down) and 26 s at the epoch-6 close (00:06:15 to 00:06:41, 25 s without a landed transaction), against at most 7 s on r20261007-0316 and 10 s on r20261006-0458. Consensus closed both epochs on time; the gap is execution backlog. A node starts the next epoch only after executing the old one up to its closing block, and at 00:06 the healthy validators were 19 to 22 s behind in execution, against 0.4 s on the observer, which serves no load; validator-06 (the Netherlands) started epoch 6 about 20 s after the others. The 26 s is the longest gap without a landed transaction in the all-generators window.
  • London (generator 02) and the Netherlands (generator 06) were slow to start and slow throughout. All of their setup transactions were still unlanded at every resend check from 23:30:32 to 23:31:33 UTC (London) and to 23:32:03 (the Netherlands), so they began their mixed phase at 23:34:59 and 23:35:07, about three minutes after the other eight (23:32:12 to 23:32:51), and the all-generators window starts with them. Their transactions took 330.8 and 305.5 s to land at the 95th percentile (14.3 and 20.6 s on r20261007-0316), and they landed 281,430 and 400,628 of their own transactions, against about 1.0 million each on r20261007-0316. Hong Kong (generator 08) also reached 302.8 s at the 95th percentile (10.9 s), and Mumbai (generator 10) landed nothing for the first 195 s of its mixed phase; Sydney, the farthest region, landed like the near ones (95th percentile 6.85 s, none unlanded). Their validators were the two that could not certify while rounds were fast, and a generator's transactions are sealed into batches by its own validator and land only once one of that validator's headers is certified and committed. Every vote's round trip measured 2.02 times the path's own round trip across 89 proposer-voter pairs, the first votes after the start no different from later ones: this build negotiates every request stream with strict multistream-select V1, which waits one round trip for the listener's confirmation before the request is sent, the round trip that the lazy negotiation of r20261007-0316 removed. A proposer needs six peer votes inside the round. Idle, the round was 451 ms on every node (353 ms on r20261007-0316), set by the arrival of the seventh certificate rather than the 350 ms minimum; London's and the Netherlands' sixth vote took 521 and 537 ms, so 2,696 and 2,702 of their 2,843 idle headers were superseded before reaching quorum, most of the few certificates they did form were never committed, and they led 0.3 % and 0.2 % of the committed sub-DAGs idle and 1.9 % and 2.8 % over the run (fair share 10 %). The fleet logged 4,326 'failed to get vote: Canceled', 2,104 of them on London and 2,154 on the Netherlands, about 1,500 of each before the load began. Under load the round stretched to about 810 to 880 ms and the two certified 70.0 % and 89.1 %; Hong Kong certified 90.5 % and Mumbai, which has exactly six peers inside the idle round, 96.5 % of the parts of the mixed phase their logs cover. From 23:35 to 00:12 UTC the Netherlands also created only 0 to 16 votes a minute, against 1,147 a minute before the load, so from 23:45, with validator-05 down or catching up, every proposer needed six of its seven remaining peers. Why it stopped voting is not settled: its logs place the wait after the vote handler's execution check, most likely in fetching batches (peers refused its batch requests as at capacity 1,331 times), and 99 % of the requests it did answer fell inside its own block-persist window (next caveat).
  • The nodes wrote far more for less work. The telcoin process wrote at a median of 439.3 MiB/s over the mixed phase, 2.8 times r20261007-0316's 157.6 MiB/s, while chain throughput was 26 % lower and each healthy validator's data directory grew 3.22 to 3.28 GiB over the window against 4.29 to 4.33 GiB; storage per landed transaction was unchanged (256 bytes of execution data, 87 bytes of consensus data). The busiest validator's disk writes reached 584.6 MiB/s at the 95th percentile (bench-validator-02), its reads and writes together 73 % of the 800 MiB/s provisioned, and I/O pressure peaked at 31.5 to 33.2 % (15.1 to 22.7 %), while CPU stayed lower (mean 19 to 21 % against 35 to 36 %). Chain throughput also fell from one 5-minute block to the next for most of the run, 5,986, 5,064, 4,445, 4,228, 3,908, 3,448 and 3,045 tx/s from 23:32 to 00:07 UTC, then 3,512, before and after the kill alike; r20261007-0316 held between 4,782 and 5,907. The extra writes come from execution. The execution engine was busy for practically all of the loaded time (busy fraction 0.99 to 1.09 in every 5-minute block of the mixed phase, over the nine healthy validators), and its queue of consensus outputs reached its cap of 8 on all eleven nodes. Each output executes in memory in about 0.5 to 0.65 s and is then persisted in one synchronous database commit before the next output can start; on the two validators measured, that persist step grew from 0.97 s at the median (23:35 to 23:45 UTC) to 1.60 s (00:00 to 00:13), took 60 to 72 % of the engine's busy time and wrote about 1.07 GB per output, rewriting pages of a database file that stayed at 4.30 GB. Disk writes averaged 480 MB/s with per-second peaks of 860 to 880 MB/s from 23:38 on, at or above the provisioned 800 MiB/s (839 MB/s), while only about 2,150 of the 20,000 provisioned IOPS were used: the disk was throughput-bound, CPU was not the limit and memory pressure stayed at zero. With the engine saturated, throughput is the inverse of the engine time per transaction, which rose from about 160 to 310 µs while the bytes written per landed transaction rose from about 80 to 170 KB, and a committed output was executed 6 to 12 s after its commit at the median. A validator waits for its own execution before it votes, which ties the persist step back into consensus. r20261007-0316, on the same disk and with the same transaction mix, wrote about 33 KB per landed transaction, 3.6 times less; what fills the 1 GB per output, and whether the build or the chain state accounts for the difference, has not been measured.
  • The commit-latency p95 on this page (4.68 to 29 s over the healthy validators) comes from the nodes' Prometheus histogram, interpolated within coarse upper buckets. Six validators read 4.68 to 4.70 s, bench-validator-02 9.71 s and bench-validator-06 29.0 s, whose median, 20.0 s, compares with 1.75 to 1.78 s on every other validator; bench-validator-09 has no commit-latency series.
  • The deployment needed twelve launches from 19:22 to 23:21 UTC before it reached the load. The eleven failures: three for the validator-05 stockout; one dropped key-generation session, on a node that had in fact finished generating its key; one genesis-build check that assumed the devnet's post-genesis staker, fixed for the benchmark profile; three transport faults in the finalize step (two cloud-API read timeouts on one node and one dropped ssh session on another), each of which ended the step for every node; and three resets of the RPC tunnel during the contract deployment. Each relaunch resumed where the last had stopped. The first eight failures came before any node had started; the last three came during the contract deployment, after the coordinated start, and touched no node's configuration. No node was restarted outside the chaos plan.
  • The harness is recorded at the commit it was launched from, 5269ae2, and marked dirty: its working tree had an uncommitted edit to the run's planning notes, which no step reads. Five harness commits followed during the run and ran parts of it: two moved validator-05's zone (818c7fa, f4f73a9) before it was created; one made the genesis build's devnet-only funding check skip the benchmark profile (7855f63) before genesis was built; one retried an IAP session after an ssh drop or a cloud-API crash (6ed9429), which reached the tree two minutes into the successful launch, so in that launch only the stage-and-stop pass and the coordinated start used it, and every later launch did; and one replaced the per-connection RPC tunnel with a single ssh port-forward (2c2f95e) for the rest of the contract deployment, the funding, the load and the collection. A sixth (daf809c), committed after the teardown, makes the teardown stop that tunnel's supervisor, which it had missed. Apart from the zone move, none of them changes what the nodes ran or how the measurements were taken.
  • The saved logs of validators 04 (Tokyo), 08 (Hong Kong), 09 (Seoul) and 10 (Mumbai) begin at 23:52:29, 23:54:42, 23:53:19 and 23:37:26 UTC, so they hold nothing of the start, the idle window or the first part of the load, and those of 04, 08 and 09 nothing of the kill or the epoch-5 close. The container log is capped at three 50 MB files, the operator scripts' logrotate rule rotated the node's own log file at 00:00 UTC, and the teardown pulled only the current file of each. The rotated file was copied by hand from the six validators that still existed when the gap was noticed (01, 02, 03, 05, 06 and 07); the teardown had already deleted the other four. The observer's log was never rotated. Every figure on this page drawn from node logs therefore covers seven nodes for the whole run and the other four only from those times. With the rescued files the certification check's verdicts changed: bench-validator-07 no longer falls below 90 %, and the shortfalls of bench-validator-02 and bench-validator-06 begin in the preamble rather than the cool-down. Every node's boot lines had been read from its running container at deploy time, before any rotation: all eleven logged commit 3f9b0709, the 1,000/350/250 ms delays, the 5,000 ms vote timeout and the sub-second fork from genesis.
  • Closed admission was in force on every node: the gate read admission mode 2 (Closed) on the primary and the worker network, with no fallback, on all eleven nodes at 23:01:51 to 23:02:46 UTC. That reading is the only admission record; the metrics saved for this page do not include the admission series. The same reading showed 195 to 211 denied inbound and 186 to 224 denied outbound connections on validators 02 to 10, and none on validator-01 or the observer. The denials were the observer's. Only validator-01's network configuration lists the observer among its trusted nodes, by the configuration renderer's rule for the hub; validators 02 to 10 admit only the committee and their configured operator peers, so each refused the observer's redials, which its peer manager makes every 30 s on each network. On each of the nine, the denied inbound count is the observer's completed dials to that node over both networks, less 2. The outbound denials are most likely the validators' own discovery dials to the observer, whose address they learn from validator-01; nothing logged attributes them. No connection between committee members was denied, and the load generators, the funder and the metrics agents do not use the peer network. The observer therefore synced through validator-01 alone: it logged 1,232 failed sync exchanges, all toward validator-01.
  • This configuration has been run once, so run-to-run variation is not measured.
  • The tag v0.2.0-devnet is mutable. The run named the image by digest sha256:5257e768… from the start, the digest the devnet's signed release manifest records, and every node's boot line logged the commit that image is built from.
  • Per-validator rows cover 9 validators; bench-validator-05 excluded as chaos target(s).
  • Commit_latency_p95_s: no tn_primary_commit_latency_seconds.p95 points for bench-validator-09.
  • Recovery: the metrics window ended while bench-validator-05 was still rejoining (inactive at its last sample, 2026-11-04T00:23:01Z); rejoined and catch-up unavailable.
  • Recovery: the 15-s samples show bench-validator-05 promoted 2 time(s) (2026-11-04T00:11:01Z to 2026-11-04T00:14:16Z) and inactive again at a later sample each time; a rejoin counts only once the node stays active to the end of the metrics window.
  • Recovery: demoted / rejoined are the first 15-s Prometheus samples showing the change, so they trail the node's own log times by up to a scrape interval plus one step.
  • Recovery: failed_reports is the rise of the batch-seal failure counter on bench-validator-05 over the whole metrics window (metrics.prometheus.series.tn_batch_builder_seal_failures_total.count), outage or not.
  • Recovery.rejoined_utc, recovery.catch_up_secs are the published values, not recomputed from the run data.
  • The harness working tree had uncommitted changes when the run was deployed.
  • Validators bench-validator-02 and bench-validator-06 certified fewer than 90 % of their own headers in phase preamble, phase warmup, phase mixed, phase cooldown and run; their logs are preserved under results/r20261103-1922/logs/ (see INDEX.md).

Full report

Full report HTML · 1.2 MB

Changes from the original report:

  • The site adds a banner at the top of the report that links back to this page.

SHA-256 of the committed file, before the banner is added: c1618f55b3aa0a02f73a9aafcc193bc3ac841c4a172a6e92c497ab1e0caf6e7a