The main branch with lazy protocol negotiation and vote-path instrumentation

2026-11-05 · started 08:03 UTC · ended 16:40 UTC · merged mixed window 40 min 6 s

telcoin-network 496ebefb5 · bench/main-v1lazy-obs-2026-11 · commit verified from node startup logs

24 node starts across 11 instances logged telcoin-network 0.1.0 (496ebefb52627fcf20f4074b58d7b3309b8b4bb8) The node image probe recorded the same commit before deploy Node logs: fork schedule: seed_signature, multi_workers, prevrandao, leader_seeded_ordering, and subsecond_timestamp active from genesis Commit 496ebefb is the head of the public branch bench/main-v1lazy-obs-2026-11, built for this benchmark from the telcoin-network main branch at a0d6e6e26 (#1761, 4 November). On top of main it carries the four commits of PR 1783, which make every outbound substream negotiate with multistream-select V1Lazy (a4e55c880: a dialer that proposes exactly one protocol sends the proposal and its request in one flight instead of waiting a round trip for the listener's confirmation) and complete and test the gossipsub fallback (dc1ad5f1d, bbd0880ef, e6d85bd68), then nine instrumentation commits, eight features and one follow-up fix (496ebefb5), that add the log lines and Prometheus series this run's analysis reads: the voter's vote path by stage, the proposer's vote round trips, header storage and the proposal lock, the engine's execute, persist and commit timings and its busy time, rejoin blockers and node-mode transitions with their reasons, the scheduled leader of every round and its outcome, batch-serve denials and denied batch requests, epoch-record signers and missing signers, and outbound request durations by service class. The run named the image by registry digest (adiri@sha256:f6922184…, pushed as adiri:bench-main-v1lazy-obs-20261104). The build contains the certifier fix of r20261006-0458 (cfc9e34f, #1579) and the consensus-storage rework (#1558). Against r20261103-1922's v0.2.0-devnet build 3f9b0709 it has 29 commits that build lacks and lacks 19, and the two differ in 206 files. r20261007-0316's lazy negotiation came from a different commit (b0838dd38), which is not in this build's history.

We repeated the 40-minute chaos run of 3 November (r20261103-1922) on the same fleet, ten c3-highcpu-8 validators with Hyperdisk Balanced disks, one in each of ten regions, and an observer, with a node built for the benchmark from the main branch, the lazy protocol negotiation of PR 1783 and nine commits of vote-path, execution and recovery instrumentation, and with metrics on the run's own Prometheus server. With all ten load generators running, average chain throughput was 4,578 tx/s, against 4,037 tx/s on r20261103-1922 and 5,434 tx/s on r20261007-0316; the chain produced 15.8 blocks a second over the mixed phase, the median time for a transaction to land was 3.3 s and the 95th percentile 12.84 s (3.49 and 14.17 s on r20261103-1922), and throughput stayed flat from one five-minute block to the next instead of falling. Lazy negotiation made every vote one network round trip instead of two: the idle round fell to the 350 ms minimum header delay, and every validator, London and the Netherlands included, certified at least 99.6 % of its own headers while the chain was idle (5.2 % and 5.0 % for those two on r20261103-1922). Throughput was bound by execution: every node's execution engine was busy about 98 % of the loaded time at a flat cost of about 215 µs per transaction, a ceiling of about 4,600 tx/s, and persisting each output to disk was no longer the part that grew. Two validators that were not chaos targets, Hong Kong from 09:20 and Singapore from 09:27 UTC, left the committee under load: their engines, waiting on slower database syncs than the rest of the fleet's, queued longest, their own consensus commits fell 41 rounds behind the certificates they received, one past the limit at which a validator is demoted, and their load generators landed nothing from then on. With validator-05 killed at 09:33:50, 60 s before an epoch close, the committee was left with exactly the seven voters a quorum needs, which cost a 17-second commit pause; the epoch closed on time. The restarted validator-05, like Singapore, then went through the promotion and demotion loop of r20261103-1922 again (13 and 21 promotions, each from an empty DAG with a stale header) and held its place only when epoch 6 began, 2,168 s after its restart; Hong Kong held from the start of epoch 5. Every epoch record reached quorum on every node active at the close, but two of them were signed by exactly seven validators.

Compared against r20261103-1922.

Key figures

4,578tx/s
Average chain throughput, all load generators running
all generators active
5,524.8tx/s
Best 60 seconds, all load generators running
all generators active
12.84s
Time to land, p95
whole run
8–30s
Commit latency, p95
merged mixed window

How it was run

Software

Upstream committelcoin-network 496ebefb52627fcf20f4074b58d7b3309b8b4bb8 · bench/main-v1lazy-obs-2026-11 · commit verified from node startup logs
Node imageus-docker.pkg.dev/telcoin-network/tn-public/adiri@sha256:f6922184c0da9da85ac4c85d4e465f6221ea1f8fe317e43d9ec90fab8f18fd5a
Node image digestsha256:f6922184c0da9da85ac4c85d4e465f6221ea1f8fe317e43d9ec90fab8f18fd5a
Node version0.1.0
Fork schedulesubsecond-from-genesis
Load generator commit3911723
Load generator image digestsha256:75d384428a8d94520c28903111c509985009d01815e2e62d26ff3bb066c0f157
Analysis commit4434bac
Harness commitd6a8d5304772fd41afe4f6741657b5b78ee534b9
Harness submodulescommon: 1a6955d8a2c6f19e53cf7aaad5e32a9107a21a7d
tn-node-deployment: d9446f720026fa5ce8b64fce9074e56b05b3f4eb

Fleet

Fleetc3-highcpu-8 + Hyperdisk Balanced (c3-hdb), ten regions, validator-05 in Ashburn
Validators10
Observers1
Machine typec3-highcpu-8
vCPUs per node8
Memory per node16 GB
Diskhyperdisk-balanced, 100 GB, 20,000 IOPS, 800 MiB/s
Regionsasia-east2 (Hong Kong), asia-northeast1 (Tokyo), asia-northeast3 (Seoul), asia-south1 (Mumbai), asia-southeast1 (Singapore), australia-southeast1 (Sydney), europe-west2 (London), europe-west4 (Netherlands), us-east4 (Ashburn), us-west2 (Los Angeles) (10)
Load generators10 generators × e2-standard-2 × 600 senders each
Load placementone blaster per validator, in that validator's zone, sending only to it

Chain

Chain ID32286
Networkbench
Epoch length1,200 s (20 min)

Consensus parameters

Max header delay1,000 ms
Min header delay350 ms
Max batch delay250 ms
Vote timeout5,000 ms
Parameters fileconsensus-parameters.yaml
Confirmed bynode startup logs

Workload

Warm-up120 s (2 min)
Mixed phase2,400 s (40 min)
Cool-down180 s (3 min)
Transaction typesbatched_self_send, contract_deploy, faucet_drip, native_transfer, stablecoin_approve, stablecoin_transfer, uniswap_v2_add_liquidity, uniswap_v2_remove_liquidity, uniswap_v2_swap, uniswap_v3_swap, uniswap_v4_swap, wtel_unwrap, wtel_wrap

Chaos events

MarkerTime (UTC)NodeActionDetail
C12026-11-05 09:33:50bench-validator-05docker kill -s KILL telcoinsystemctl stop holds it down (unit has Restart=on-failure). bench-blaster-05 targets ONLY this node, so its sends fail for the outage by design; the generator resends on connection errors
C22026-11-05 09:38:50bench-validator-05systemctl start telcoindocker start is impossible: the unit runs docker run --rm
C32026-11-05 09:43:50bench-observer-11systemctl restart telcoinobserver restart

Steps

  1. The node image was built for this benchmark from the branch bench/main-v1lazy-obs-2026-11 (telcoin-network main plus PR 1783 and nine instrumentation commits) and named by registry digest (sha256:f6922184…), so there was no tag to resolve. Before creating anything the harness ran that image to confirm that it reports telcoin-network 0.1.0 built from commit 496ebefb, has the sub-second timestamp fork active from genesis and contains the log messages later checks rely on, and it recorded the committed consensus parameters file, unchanged from r20261103-1922: maximum header delay 1,000 ms, minimum header delay 350 ms and maximum batch delay 250 ms.
  2. The instrumentation commits fix the shape of the log lines and series the analysis reads, so no figure on this page depends on parsing free text: per vote, the voter's time in each stage (vote lock, validation, the wait for its own execution, parents, leader, batches, store, persist) and any vote timeout with its stage; per vote request, the proposer's round trip, attempt and the header's age; per header, its store and send times; per round, the scheduled leader and whether it was swapped or found; per consensus output, the engine's queue wait, execute, persist and commit times; per subscriber decision, why a rejoin was not yet possible and why a node changed mode; per epoch close, who signed the epoch record and who was missing; and per request-response exchange, its service class and duration, with matching Prometheus counters and histograms.
  3. Preflight checked access, tools, quotas and a clean genesis directory, and confirmed the load-generator image built from tn-transaction-generator commit 3911723, the generator of r20261103-1922. The results were merged and sized with that repository's aggregate.py and derive-sizing.py at commit 4434bac, which reproduces every published KPI of r20261103-1922, r20261007-0316 and r20261006-0458 exactly.
  4. The harness created ten validators and one observer on c3-highcpu-8 (8 vCPU, 16 GB), each with a 100 GB Hyperdisk Balanced boot disk provisioned at 20,000 IOPS and 800 MiB/s, one validator in each of ten regions (Los Angeles, London, Sydney, Tokyo, Ashburn, the Netherlands, Singapore, Hong Kong, Seoul and Mumbai) with the observer in Los Angeles, the zones of r20261103-1922; every node was created on the first attempt. It also created a separate metrics VM for the run (bench-obs, e2-standard-4, 50 GB, in Los Angeles) running Prometheus and Grafana; it collects metrics only, no logs.
  5. Each node's keys were generated on the node by the operator scripts of the v0.2.0-devnet release (tn-node-deployment d9446f7, the v2 scripts), four nodes at a time, with each node advertising its RPC endpoint on its internal address (r20261103-1922 advertised unresolvable placeholder names). The harness built genesis for a separate benchmark chain (chain id 32286, 20-minute epochs). Before any node started it installed on every node a network configuration in Closed admission mode that names every validator and lists the observer as trusted by every validator, so that no fleet member refuses another (on r20261103-1922 only validator-01 trusted the observer).
  6. Each node was finalized by the release's scripts, four at a time, run alone just long enough to show that the service started and answered RPC, and stopped again. While every node was stopped, one session per node wrote the benchmark settings into its start script: RPC and websocket listening on all interfaces, a local metrics endpoint, a 20 GB rotation size for the node's log file, and the log filter info,primary=debug,engine=debug,subscriber=debug,worker::network=debug,network::reqres=debug with the primary's certificate validator, manager and fetcher at info. No per-node consensus override was set.
  7. All eleven nodes were then started at one due time, 08:22:35 UTC: every node began in that second (spread 0 s, none late) on the first attempt. Each node's metrics agent was then pointed at bench-obs with the benchmark network label; no node was restarted for it. The devnet's shared observability VM was not used by this run and received none of its samples or logs.
  8. A verify-only pass read back from each running node its chain id, consensus progress, start-script settings, parameters file and boot lines; all eleven had logged commit 496ebefb, the three consensus delays, the 5,000 ms vote timeout and the sub-second fork from genesis, and passed by 08:35:25 UTC; an on-node sampler was then started on each. The health and admission gate, which finished at 08:37:58 UTC, found Prometheus on bench-obs answering with all eleven nodes reporting under the benchmark network and no devnet series, all ten validators active, and on every node admission mode 2 (Closed) on the primary and the worker network, no fallback and no denied connection in either direction; a read of the ConsensusRegistry confirmed the 1,200 s epoch duration.
  9. It deployed the devnet contract set that the mixed workload calls, from 08:37:58 to 08:49:03 UTC, over a single ssh port-forward to validator-01's RPC that held for the whole run. The chain was then idle, apart from the deployment, until the funding step at 09:01:37.
  10. A one-off funder VM funded 600 sender accounts for each of ten load generators from 09:01:37 to 09:03:03 UTC. The generators, on e2-standard-2, one per validator in that validator's zone and sending only to it, started together at 09:08:29 UTC: a preamble of setup transactions (10 to 15 s per generator, none resent), 120 s of warm-up, 2,400 s of mixed load across 13 transaction types and up to 180 s of cool-down, cut short after 30 s in which none of the generator's own transactions landed. All ten mixed phases began within 6 s of each other. No network delay was injected.
  11. The chaos schedule was timed to the chain's epochs. The harness read the ConsensusRegistry at a pinned block (block 180, epoch 2), chose the close of epoch 3, and re-read the schedule once epoch 3 had opened (09:17:29 UTC, drift 0 s). It staged three actions on the target nodes, each to fire on the node's own clock: bench-validator-05's container killed with SIGKILL 59 s before that close, due at 09:34:49 UTC, and held down, the node restarted 300 s after the kill, and the observer's node restarted 300 s after that. The kill began at 09:33:50.164, the restart at 09:38:50.179 and the observer restart at 09:43:50.061, each less than 0.2 s after its planned second; the closing commit came at 09:34:50, so the outage crossed the boundary. Before the kill and the observer restart each node saved its own container log.
  12. During the run the harness sampled the fleet's highest block; the only period without a new block it recorded, 09:51:00 to 09:54:30 UTC, is the end of the load (the last transaction block is at 09:50:54), not a stall.
  13. Afterwards the harness collected each generator's results and the on-node sampler files, then waited until every validator read active again and stayed active for 180 s before closing the metrics window (886 s of waiting, to 10:17:26 UTC). It pulled 60-second machine metrics from Cloud Monitoring and 15-second node metrics from bench-obs's Prometheus over a local forward (08:37:58 to 10:20:29 UTC), with no series missing; the collection finished at 10:21:34 UTC.
  14. The KPIs and windows on this page come from the publishing pipeline's own window and KPI steps, run through the driver that reproduced the published figures of the three comparison runs. Rounds per second come from each node's round counter and from the generators' block streams. Certification, the vote-path breakdown, leader shares, the execution split, rejoin and mode changes and the epoch-record quorum come from the instrumented log lines and series above, read from every node's complete unit log (all eleven nodes, the whole run) in one streaming pass each, with the engine's per-block debug dumps dropped.
  15. Teardown saved every node's container log, its unit log file, sampler files and service journal before deleting anything. The unit logs were about 49 GB each uncompressed, so the pulls, four at a time over IAP at about 2 MB/s each, took from 10:22 to 14:41 UTC; the streams of five nodes were restarted from zero, three after a transport fault and three after a name-resolution failure on the machine running the harness. The post-pull check then refused to delete, because it treated the pull records it had just written as stale; every saved file was checked by hand against its recorded size, and a second teardown at 14:42:31 trusted those records and accepted all eleven. A restart of the operator's session ended that run at the certification step at 14:52 UTC with nothing deleted, and a third run, started detached, continued from there. That third run passed every gate and pulled the Prometheus snapshot, then its first instance delete failed because the installed gcloud rejects --async on instance deletes; after the harness fix a fifth run deleted all 12 instances, the two firewall rules, the bucket and the disks (TEARDOWN_DONE deleted=12 left=0 at 17:19:37Z), and the results directory was re-indexed (251 files, 91.4 GB).

The harness scripts live in a private repository; the steps above describe what they do.

Results

MetricValueWindowSourceMethod
Average chain throughput, all load generators running4,578 tx/sall generators activecombined.series.per_second[chain_tx]Chain transactions per block-timestamp second, summed over the 2394-s window when every load generator was in its mixed phase and divided by its length.
Best 60 seconds, all load generators running5,524.8 tx/sall generators activecombined.series.per_second[chain_tx]Highest mean chain tx/s over any 60 consecutive seconds inside the all-generators window (aggregate.py best_window_mean).
Transactions landed over the whole blast12,306,384 txwhole runcombined.totals.landed_chainSum of tx_count over every block the load generators saw during the blast, own and foreign transactions alike.
Blocks per second15.84 blocks/smerged mixed windowcombined.blocks.rows[timestamp]Chain blocks whose timestamp falls in the merged mixed window, divided by its length (aggregate.py landed.capacity.blocks_per_sec over the same window).
Time to land, median3.3 swhole runcombined.latency.ts_ms.p50Block timestamp minus submit time over every landed own transaction in the blast, from the load generators' merged latency histograms (aggregate.py merge_latency), in seconds.
Time to land, p9512.84 swhole runcombined.latency.ts_ms.p95Block timestamp minus submit time over every landed own transaction in the blast, from the load generators' merged latency histograms (aggregate.py merge_latency), in seconds.
Commit latency, p958–30 smerged mixed windowmetrics.prometheus.series.tn_primary_commit_latency_seconds.by_node[healthy validators].p95For each healthy validator, the nearest-rank p95 over the merged mixed window of its 15-s histogram_quantile(0.95) commit-latency series; reported as the range over validators.
Execution time per consensus output2.20–2.53 smerged mixed windowmetrics.prometheus.series.tn_engine_execution_duration_seconds.by_node[healthy validators].meanFor each healthy validator, the nearest-rank median over the merged mixed window of its 15-s mean execution-duration series; reported as the range over validators.
Write throughput, p95, busiest validator310 MiB/smerged mixed windowmetrics.gcp[bench-validator-08].disk_write_bytes_per_sec p95Nearest-rank p95 of the 60-s mean write-byte rate over the merged mixed window for each healthy validator; the highest (bench-validator-08) is reported.
Write operations, p95, busiest validator9,882 IOPSmerged mixed windowmetrics.gcp[bench-validator-10].disk_write_ops_per_sec p95Nearest-rank p95 of the 60-s mean write-operation rate over the merged mixed window for each healthy validator; the highest (bench-validator-10) is reported.
CPU, p95, busiest validator36.7 %merged mixed windowmetrics.gcp[bench-validator-09].cpu_utilization p95Nearest-rank p95 of the 60-s CPU utilisation over the merged mixed window for each healthy validator; the highest (bench-validator-09) is reported.
Node resident memory, p95, busiest validator8.64 GiBmerged mixed windowmetrics.gcp[bench-validator-10].telcoin_rss_bytes p95Nearest-rank p95 of the 60-s telcoin process resident memory over the merged mixed window for each healthy validator; the highest (bench-validator-10) is reported.
IO pressure, highest reading, range over validators10.2–17.4 %merged mixed windowsampler[healthy validators].psi_io_some_avg10 maxFor each healthy validator, the highest 10-s 'some' IO pressure reading the on-node sampler logged inside the merged mixed window; reported as the range over validators.

Recovery

Nodebench-validator-05
Killed2026-11-05 09:33:50 UTC
Restarted2026-11-05 09:38:50 UTC
Rejoined2026-11-05 10:15:13 UTC
Catch-up time2,160 s (36 min)
Failed batch reports0

Chain TPS per 5-minute interval

Chain TPS per 5-minute interval 9 intervals from 09:10 to 09:50 UTC; average 4,357 TPS, highest 4,829 TPS. The merged mixed window ran from 09:10 to 09:50 UTC. TPS 0 1,000 2,000 3,000 4,000 5,000 09:10–09:15 UTC: 4,751 TPS, 9.7 generators active 09:15–09:20 UTC: 4,179 TPS, 10.0 generators active 09:20–09:25 UTC: 4,605 TPS, 10.0 generators active 09:25–09:30 UTC: 4,575 TPS, 10.0 generators active 09:30–09:35 UTC: 4,439 TPS, 10.0 generators active 09:35–09:40 UTC: 4,829 TPS, 10.0 generators active 09:40–09:45 UTC: 4,737 TPS, 10.0 generators active 09:45–09:50 UTC: 4,641 TPS, 10.0 generators active 09:50–09:50 UTC: 2,456 TPS, 9.0 generators active avg 4,357 09:10 UTC 09:20 UTC 09:30 UTC 09:40 UTC 09:50 UTC C1 2026-11-05 09:33:50 UTC: docker kill -s KILL telcoin on bench-validator-05 (systemctl stop holds it down (unit has Restart=on-failure). bench-blaster-05 targets ONLY this node, so its sends fail for the outage by design; the generator resends on connection errors) C1 C2 2026-11-05 09:38:50 UTC: systemctl start telcoin on bench-validator-05 (docker start is impossible: the unit runs docker run --rm) C2 C3 2026-11-05 09:43:50 UTC: systemctl restart telcoin on bench-observer-11 (observer restart) C3

Each bar is the average chain TPS over one 5-minute interval of the merged mixed window (09:10–09:50 UTC); fainter bars had fewer generators active than the maximum, the dashed line is the average of all bars and amber lines mark the chaos events listed below.

  • C1 2026-11-05 09:33:50 UTC: docker kill -s KILL telcoin on bench-validator-05 (systemctl stop holds it down (unit has Restart=on-failure). bench-blaster-05 targets ONLY this node, so its sends fail for the outage by design; the generator resends on connection errors)
  • C2 2026-11-05 09:38:50 UTC: systemctl start telcoin on bench-validator-05 (docker start is impossible: the unit runs docker run --rm)
  • C3 2026-11-05 09:43:50 UTC: systemctl restart telcoin on bench-observer-11 (observer restart)
5-minute data (9 rows)
start_utcend_utcgenerators_activechain_tps
2026-11-05T09:10:40Z2026-11-05T09:15:40Z9.74,750.8
2026-11-05T09:15:40Z2026-11-05T09:20:40Z10.04,179.1
2026-11-05T09:20:40Z2026-11-05T09:25:40Z10.04,605.0
2026-11-05T09:25:40Z2026-11-05T09:30:40Z10.04,575.1
2026-11-05T09:30:40Z2026-11-05T09:35:40Z10.04,439.2
2026-11-05T09:35:40Z2026-11-05T09:40:40Z10.04,829.2
2026-11-05T09:40:40Z2026-11-05T09:45:40Z10.04,737.2
2026-11-05T09:45:40Z2026-11-05T09:50:40Z10.04,641.4
2026-11-05T09:50:40Z2026-11-05T09:50:46Z9.02,456.2

The fine-grained chain throughput series (its step_s field gives the seconds per point) is published as series/tps-1s.json.

Other networks

MeasureTelcoin Network (this run)Tempo
Throughput4,578 tx/s (Avg chain TPS)16,455 TPS, shown as "Settled TPS" for the nightly run of 5 Oct 2026 (commit de5f5d3). perf.tempo.xyz describes it as "Median settled throughput across 130 measured blocks."
How throughput is measuredChain transactions per block-timestamp second, summed over the 2394-s window when every load generator was in its mixed phase and divided by its length.The methodology page defines TPS as "Transactions per second (derived from gas / avg tx gas)": gas used divided by the average gas of one transaction, reported as the median over the measured blocks.
Run length40.1 min merged mixed window; each generator was set to a 40 min mixed phase2m 39s for the 5 Oct 2026 nightly run (130 measured blocks at a 687 ms average block time), per perf.tempo.xyz.
Validators and placement10 validators and 1 observer across 10 regions: asia-east2 (Hong Kong), asia-northeast1 (Tokyo), asia-northeast3 (Seoul), asia-south1 (Mumbai), asia-southeast1 (Singapore), australia-southeast1 (Sydney), europe-west2 (London), europe-west4 (Netherlands), us-east4 (Ashburn), us-west2 (Los Angeles); load generators: one blaster per validator, in that validator's zone, sending only to itperf.tempo.xyz labels the scenario "Public mix · multi-region". The methodology page says "Benchmarks run two local validators in consensus on a dedicated bare-metal server with 100 GiB pre-bloated state." The public tempoxyz/tempo repo has nightly workflows for both kinds of setup: bench-e2e runs on one bare-metal runner, and bench-e2e-multi-region defaults to 10 validators on cloud VMs (its regions input reads "Empty uses the selected cloud's five defaults."). The run page does not say which setup produced this number.
Hardwarec3-highcpu-8, 8 vCPUs, 16 GB memory; disk: hyperdisk-balanced, 100 GB, 20,000 IOPS, 800 MiB/sThe methodology page lists a dedicated bare-metal server: "AMD EPYC 4585PX (16C/32T, 4.3/5.7 GHz)", "128 GB 3600 MHz" memory and "2×960 GB NVMe + 2×1.92 TB NVMe via schelk". The repo's multi-region workflow uses cloud VMs instead (c4d-standard-48-lssd on GCP, c6id.8xlarge on AWS).
Workloadbatched_self_send, contract_deploy, faucet_drip, native_transfer, stablecoin_approve, stablecoin_transfer, uniswap_v2_add_liquidity, uniswap_v2_remove_liquidity, uniswap_v2_swap, uniswap_v3_swap, uniswap_v4_swap, wtel_unwrap, wtel_wrap; 10 generators × 600 senders eachperf.tempo.xyz labels the scenario "Public mix"; the methodology page says "100% TIP-20 token transfers (ERC-20 equivalent)". The run used 1,000 accounts and 4 TIP-20 tokens.
StateFresh chain from genesis for every runThe methodology page says "100 GiB pre-bloated state".
Latency (as each source defines it)Time to land, median: 3.3 s; Time to land, p95: 12.84 s; Commit latency, p95: 8–30 sBlock time of 687 ms, which the methodology page defines as "Average wall-clock time between blocks". The run page gives no per-transaction latency. Separately, tempo.xyz says "Blocks finalize in ~0.6 seconds with no re-orgs." (a network claim, not a figure from this benchmark).
Source and fetch dateThis page; every figure names its source field.perf.tempo.xyz nightly run of 5 Oct 2026 (commit de5f5d3), the perf.tempo.xyz methodology page, tempo.xyz, and the benchmark workflows in the tempoxyz/tempo repo. All fetched on 5 Oct 2026.

Sources for Tempo (fetched 2026-10-05):

Tempo's setup may not match ours: its methodology page describes two validators on one bare-metal server, while our runs use ten validators across five regions, so read this table as context, not a ranking.

This table is context, not a ranking.

Caveats

  • The comparison run, r20261103-1922, had the same fleet in the same zones, the same load generator, load shape and chaos plan, the same consensus timing (1,000/350/250 ms) and the same operator scripts. The node build differs: this run's 496ebefb is the main branch with lazy protocol negotiation and the instrumentation commits, where r20261103-1922 ran the v0.2.0-devnet release 3f9b0709 with strict negotiation; the two differ in 29 and 19 commits and 206 files. The deployment differs in admission (here every validator trusts the observer, so no connection was denied; there the observer was refused by nine validators), in the RPC addresses the nodes advertise, and in a much heavier log filter (next caveats). Against r20261103-1922, average chain throughput with all generators running rose from 4,037 to 4,578 tx/s, blocks a second over the mixed phase from 13.47 to 15.84, the median time to land fell from 3.49 to 3.3 s and the 95th percentile from 14.17 to 12.84 s, while primary rounds a second fell from 1.07-1.09 to 0.84-0.91. No difference can be assigned to any one change.
  • Two validators that were not chaos targets left the committee under load, mid-epoch: bench-validator-08 (Hong Kong) at 09:20:07 UTC and bench-validator-07 (Singapore) at 09:27:01. This was not a committee rotation: every epoch record names all ten validators. Each node's own consensus commit had drifted behind the committee, Singapore's from the start of the load at about 2 s a minute and Hong Kong's from the epoch-2 close at 09:14:49 at about 6 s a minute, until a certificate arrived 41 rounds above its committed round, one past the 40-round activity window, and the node demoted itself to follower ('we are behind', reason outside_gc_window). Their proposers were still at the frontier, but their votes had become late and then stopped. Hong Kong held active again from the epoch-5 entry at 09:54:55 (two promotions, one failed); Singapore went through the promotion loop of the next caveats 21 times and held only at the epoch-6 entry at 10:14:56. Tokyo's commit drifted the same way, to about 10 rounds, and stayed active; every other validator stayed within 0.2 s.
  • Their load generators, 08 and 07, each send only to their own validator, which refused every send while inactive ('validator is inactive'): generator 08 landed none of its transactions after 09:20:06 and generator 07 none after 09:27:02, and generator 05 none after the kill at 09:33:50. The all-generators window (09:10:46 to 09:50:40 UTC) is therefore the window in which all ten generators ran, not one in which all ten landed: ten lanes landed in its first 9 minutes and seven in its last 17. Chain throughput did not fall as the landing set shrank (4,433 tx/s with ten, 4,515 with nine, 4,672 with eight and 4,646 with seven), because the chain was execution-bound (below).
  • The per-validator ranges on this page cover the nine validators other than the chaos target, so they include bench-validator-07 and -08 although both were inactive for most of the mixed window; their resources are those of nodes that had stopped voting. Their commit latency sets the high end of the commit-latency p95 range (30 s each); Tokyo is next at 23.7 s and the other five read 8.0 to 8.4 s, with medians of 1.12 to 1.19 s. bench-validator-10 has no commit-latency series, so that KPI covers eight nodes. The values are interpolated within the histogram's coarse upper buckets.
  • With Hong Kong and Singapore out, the kill of validator-05 left seven voting validators, exactly the 7-of-10 quorum, from 09:33:50 until Hong Kong returned at 09:54:55, so every certificate in that time needed every remaining vote. The kill paused commits for 17 s (leader round 1,144 at 09:33:50.2, the next at 09:34:07.4), and the chain had its longest gap of the run without a block, 19 s from block 24,434 at 09:33:50 to 24,435 at 09:34:09; r20261103-1922's kill, with nine voters left, paused nothing. Chain throughput across the outage stayed at 4,050 to 5,002 tx/s a minute and epoch 4 opened on time.
  • Validator-05 restarted at 09:38:50 UTC, found itself an epoch behind 14 s later and followed the chain as an inactive member. Under load it did not gain: its engine was busy all the time and executed 700 to 1,000 blocks a minute while the chain made about 920, so it fell from 4,797 blocks behind at 09:38 to 7,092 at 09:49 and stayed 236 to 308 rounds behind the newest header, far outside the 20-round rejoin window. It reached the chain head only about ten minutes after the load ended, entered epoch 5 six minutes after the fleet, and was promoted for the first time at 10:00:59. That promotion and the eleven after it failed as on r20261103-1922: each started from a DAG restored empty, re-proposed a stale stored header that every peer refused as too old (257 refusals), made no execution progress, and was demoted 7 to 9 s later when the committee was 41 rounds ahead; each demotion was followed by about 60 s without progress until the sync path re-requested the missing range, a cycle of 70 to 73 s. It held only at the epoch-6 entry, 10:14:58, when every node starts again from round 1: 2,168 s (36 min 8 s) after its restart and 2,469 s after the kill to its first own certificate (r20261103-1922: 2,170 and 2,471 s). Its load generator landed nothing for the last 1,012 s of its mixed phase and ended with 6,918 of the run's 7,677 unlanded transactions.
  • The recovery record on this page takes its rejoin time (10:15:13 UTC) and catch-up duration (2,160 s) from the 15-second node-mode samples; validator-05's own log puts the switch to catch-up mode at 09:39:05.8 and the rejoin at 10:14:58.2, 2,152 s. The failed-report count (0) is from the log, which has no batch-seal failure after the restart; the node has no seal-failure series in the metrics. The 15-second gauge also misses promotions that last less than a scrape: it shows 7 of validator-05's 13 promotions and 13 of Singapore's 21. The comparison runs' recovery figures were published from other sources (r20261103-1922's from its log), so the catch-up comparison is not exactly like for like.
  • Execution bounded throughput. Over the loaded time every node's engine was busy 976 to 988 ms in each second, its queue of consensus outputs sat at its cap of 8, and on validator-01 an output waited 7.3 s at the median (11.1 s at the 95th percentile) before it began executing. Per output, executing took about 75 % of the engine's time and persisting it 24 %, most of that the database commit's sync; the cost per transaction stayed about flat, 185 to 231 µs and about 215 µs through most of the load (execute 138 to 166, persist 45 to 64), while outputs grew with the rounds, which at 98 % busy is a ceiling of about 4,600 tx/s, the chain's own rate. On r20261103-1922 persisting was 60 to 72 % of the busy time and the cost per transaction rose from 160 to 310 µs, so throughput fell block over block; here the five-minute chain rate stayed between 4,179 and 4,829 tx/s, and the telcoin process wrote 131.6 MiB/s at the median (439.3 MiB/s there). A validator also waits for its own execution before it votes and before consensus emits each committed sub-DAG, which ties the engine back into the round.
  • Hong Kong and Singapore did not execute more slowly than the rest; they persisted more slowly. Before the drops their database commit took 533 and 420 ms at the median against 305 to 364 ms on the other validators, their engine queue waits reached 15.3 and 14.5 s at the 95th percentile (Tokyo 13.6 s, the others 10.0 to 11.3 s), and Hong Kong's disk wrote 28 % more bytes a second at a third of the IOPS, with I/O pressure at 11.4 % against 6.7 to 7.5 %; CPU was no higher on either. The observer, in the same zone as validator-01, whose disk was normal, showed the same slow-sync signature and cannot be demoted, so the cause is most likely variance between disk volumes of the same type, but that has not been measured.
  • Every vote now takes one network round trip. Over 72 proposer-voter pairs, a vote's round trip less the voter's own service time equals the pair's measured UDP round trip at the median (ratio 1.00), and the request reaches the voter without waiting for a protocol confirmation, where r20261103-1922 measured 2.02 round trips. Idle, the round settled at 352 ms, the 350 ms minimum header delay (451 ms on r20261103-1922), each proposer's quorum formed with its sixth-fastest vote plus 4 to 6 ms, and every proposer certified 99.6 to 100 % of its headers. London's and the Netherlands' quorums, at 264 and 273 ms, still arrive after the other proposers have moved to the next round, so 70.6 and 70.2 % of their idle certificates were never committed and they led 0.2 % of the idle rounds. Some paths no longer match the October probe used as the reference: London and the Netherlands to Singapore, Hong Kong and Mumbai now take 1.26 to 2.26 times the probed round trip in both directions, and the probe has no row for Ashburn.
  • Under load the vote is slow because of the voter, not the wire. The voter's wait is its own execution: before voting, a validator waits until it has executed the block the header names (95th percentile 930 ms over the fleet), and the same proposer's later requests queue behind that wait on the vote lock; fetching parents and batches took at most 1 ms at the 95th percentile. Voters whose execution kept up (Sydney, the Netherlands, Seoul, Mumbai) answered within 90 to 183 ms at the 95th percentile; Los Angeles, London and Ashburn waited 0.85 to 1.45 s, and Tokyo, Singapore and Hong Kong rejected 21 to 62 % of their requests after the wait. Rounds lengthened as voters dropped out: the mean committed round took 806 ms with ten voters, 1,047 ms with nine, 1,185 ms with eight and 1,458 ms with seven after validator-05's restart, against 346 ms idle with all ten after the rejoins, and about a third of loaded rounds ended on the 1 s maximum header delay.
  • Both loaded epoch boundaries left the chain 13 s without a block (09:14:49 and 09:34:50 UTC), against 12 and 26 s on r20261103-1922: a node starts the next epoch only after executing the old one through its closing block, and this run's execution backlog at the close was 4 to 10 s (19 to 22 s there). The observer, which serves no load, detected the epoch-3 close 21 s after the validators. Every epoch record reached quorum, with no timeouts, on every node that was active at the close (one timeout on validator-05 at the epoch-2 close); late or inactive nodes fetched the finished record from a peer. The epoch-3 and epoch-4 records were signed by exactly seven of the ten validators, the quorum with no margin. The five 'failed to reach quorum on epoch close' errors in the logs (two inside the metrics window) all come from a node whose collector missed the first broadcast of peer votes and never retried; each of those nodes had the finished record from a peer, so liveness was not affected.
  • The leader schedule kept naming validators that were not taking part. Of 669 scheduled leader rounds under load, 42 (6.3 %) were not committed, 40 of them because the leader's certificate never reached the DAG (Seoul 17, 13 of them before any demotion; Hong Kong 11 and Singapore 5 after theirs); after the load 246 of 1,704 (14.4 %) failed, half for validator-05 and half for Singapore, each costing a leader timeout. The three inactive members also kept receiving every header and refused 53,344 vote requests as too new. London and the Netherlands led 4 and 5 of 1,958 idle rounds (the reputation schedule swaps them out) and more under load.
  • No validator ran short of batch-serving capacity under load: the batch-serve pool never filled and no serve was denied on any node (r20261103-1922: 1,560 denials, sustained, to the far validators). The 1,148 batch-sync streams refused at open fell almost all in 8 s after the epoch-2 close (09:15:03 to 09:15:11 UTC), almost all of them Singapore's, which had asked every peer at once and overran the per-peer limit, plus Tokyo's.
  • The log filter of this run puts the execution engine at debug level, which logs a full dump of every executed block and every consensus output. Under load that wrote about 1.16 GB a minute per validator, 99.7 % of the bytes, and each validator's unit log reached about 49 GB over the run; each node's root disk, which also holds the database, went from 7 to 65 % full. The log appends added about 25 to 45 MB/s, 15 to 25 % of each node's disk writes, and doubled the write IOPS; they did not slow the database commit on the validators that stayed active. The CPU (mean 32.2 to 35.2 % against 19.1 to 20.8 % on r20261103-1922), disk and write figures on this page include this logging, which no production node does, and have not been corrected for it.
  • Resident memory of the telcoin process grew from 1.2 GB idle to 8.0 to 9.7 GB by the end of the load on 16 GB machines and held there, with no memory pressure and no out-of-memory event; a longer load could reach the limit. The disk throttle series were unavailable from Cloud Monitoring, the samplers record I/O pressure as 'some' only, and bench-validator-09's persist histograms are empty in the metrics, so its persist split comes from its log.
  • The 99th-percentile time to land of four generators (Sydney, the Netherlands, Seoul, Mumbai) reads about 300 s, interpolated inside the latency histogram's 256-to-600 s bucket, so it says only that a small tail took minutes; the run-wide 99th percentile is 31.38 s (443.43 s on r20261103-1922).
  • The node metrics on this page come from the run's own Prometheus on bench-obs, scraped every 15 s and read at the end of the load; the snapshot of that Prometheus taken at teardown is not used. Earlier the same day the first launch of this benchmark had stopped at its preflight gate, because the devnet's shared observability VM, where earlier runs sent their metrics, was filling with the live devnet's own debug logs; the run waited 7 h 26 min while those logs were given a 24-hour retention and the devnet nodes were set to info-level file logging, and then ran on its own metrics VM. No sample or log line of this run went to the shared VM.
  • The teardown saved every node's logs before it deleted anything, but its post-pull check refused to proceed because it applied a 15-minute freshness rule to the pull records it had written itself over four hours; every saved file was checked against its recorded size and the re-run trusted the records. During the pulls a name-resolution failure on the machine running the harness restarted three streams from zero, and a later restart of the operator's session ended the re-run at its certification step; neither touched the nodes or the saved files, and all the analysis on this page reads only the saved files.
  • The harness is recorded at commit d6a8d53, marked dirty; the only uncommitted change was a notes file in the operator-scripts submodule's working tree, which no step reads. The coordinated-start record carries the time it was written (08:30:13 UTC), not the start, which its due time puts at 08:22:35. The final wait for every validator to read active lasted until the epoch-6 entry, because Singapore and validator-05 could rejoin only there; it changes the metrics window, not the load. No node was restarted outside the chaos plan.
  • This configuration has been run once, so run-to-run variation is not measured.
  • Per-validator rows cover 9 validators; bench-validator-05 excluded as chaos target(s).
  • Commit_latency_p95_s: no tn_primary_commit_latency_seconds.p95 points for bench-validator-10.
  • Recovery: the 15-s samples show bench-validator-05 promoted 6 time(s) (2026-11-05T10:03:28Z to 2026-11-05T10:12:58Z) and inactive again at a later sample each time; a rejoin counts only once the node stays active to the end of the metrics window.
  • Recovery: demoted / rejoined are the first 15-s Prometheus samples showing the change, so they trail the node's own log times by up to a scrape interval plus one step.
  • Recovery: no tn_batch_builder_seal_failures_total count for bench-validator-05; failed_reports unavailable.
  • Recovery.failed_reports is the published value, not recomputed from the run data.
  • The harness working tree had uncommitted changes when the run was deployed.
  • Validators bench-validator-08 (in phase preamble, phase warmup and run), bench-validator-09 (in phase warmup, phase mixed and run) and bench-validator-10 (in phase preamble) certified fewer than 90 % of their own headers; their logs are preserved under results/r20261105-0803/logs/ (see INDEX.md).

Full report

Full report HTML · 1.1 MB

Changes from the original report:

  • The site adds a banner at the top of the report that links back to this page.

SHA-256 of the committed file, before the banner is added: eebbcbd82638d1a15822489c0044039463814e54227a2a58249636b01168d5b6