The main branch with the commit-lag, rejoin and execution fixes
2026-11-07 · started 15:04 UTC · ended 19:00 UTC · merged mixed window 40 min 46 s
We repeated the chaos run of 5 November (r20261105-0803) on the same fleet: ten c3-highcpu-8 validators with Hyperdisk Balanced disks, one in each of ten regions, and an observer. The node was built from the main branch plus 29 commits written after that run. With them, consensus commits no longer wait for the validator's own execution, a restarted validator rejoins once at the network head, the execution engine logs less and recovers signers off its main thread, and new log lines time every stage of a transaction's path. Before the usual 40-minute load, three short light blasts measured the floor. With 8 senders per generator, the median transaction reached a block timestamp 0.25 s after it was sent, and its generator saw the receipt after 1.63 s. Under full load, with all ten generators running, average chain throughput was 3,449 tx/s, 24.7 % below the 4,578 tx/s of r20261105-0803. The median time to land was 2.81 s (3.3 s there) and the 95th percentile 14.40 s (12.84 s). The median time until a generator saw its receipt was 14.18 s (12.92 s). Executing a transaction got 47 % cheaper, but the database commit after each consensus output took about 1.05 s instead of 0.3 s, with the same logical writes and about four times the bytes on disk, so throughput was bound by persisting. After the run, a test with the same database library on the same kernel series reproduced the extra bytes: when the kernel caches the database file in large multi-page blocks, every 4 KB page the database changes through its memory map is written to disk as the whole block. The trigger on this run's nodes is not identified. In later tests, reopening a database created during staging, the first suspect, did not reproduce the extra bytes by itself. No healthy validator left the committee (two did on r20261105-0803). The killed validator rejoined once, 28 minutes after the kill and a minute or two after the load ended; on r20261105-0803 it went through 13 promotions. One of the new commits backfired. A validator now refuses to vote for a header that names a block it has not executed yet, so the three fastest executors, Tokyo, Hong Kong and Mumbai, had almost none of their headers certified, and their generators landed nothing for 8 to 14 minutes before the kill. At light load, London's and the Netherlands' transactions landed in bursts about 20 s apart.
Compared against r20261105-0803.
Key figures
How it was run
Software
| Upstream commit | telcoin-network 067639def8b9bd4664af05cdcaa3590661a5e0c7 · bench/main-fixes-2026-11 · commit verified from node startup logs |
|---|---|
| Node image | us-docker.pkg.dev/telcoin-network/tn-public/adiri@sha256:a1a82341129cfa82bf0f7f19fbfc26f84888995e96333f428bd726462f23e402 |
| Node image digest | sha256:a1a82341129cfa82bf0f7f19fbfc26f84888995e96333f428bd726462f23e402 |
| Node version | 0.1.0 |
| Fork schedule | subsecond-from-genesis |
| Load generator commit | 6dad96f |
| Load generator image digest | sha256:5766094dec3c5efea367d2881ae53a525e9b97f18aefffcb92b7e5addc0f6227 |
| Analysis commit | 6dad96f |
| Harness commit | 538af51ccbf30743b141fa4924eb078f0a8f3ac2 |
| Harness submodules | common: 1a6955d8a2c6f19e53cf7aaad5e32a9107a21a7d tn-node-deployment: d9446f720026fa5ce8b64fce9074e56b05b3f4eb |
Fleet
| Fleet | c3-highcpu-8 + Hyperdisk Balanced (c3-hdb), ten regions, validator-05 in Ashburn |
|---|---|
| Validators | 10 |
| Observers | 1 |
| Machine type | c3-highcpu-8 |
| vCPUs per node | 8 |
| Memory per node | 16 GB |
| Disk | hyperdisk-balanced, 100 GB, 20,000 IOPS, 800 MiB/s |
| Regions | asia-east2 (Hong Kong), asia-northeast1 (Tokyo), asia-northeast3 (Seoul), asia-south1 (Mumbai), asia-southeast1 (Singapore), australia-southeast1 (Sydney), europe-west2 (London), europe-west4 (Netherlands), us-east4 (Ashburn), us-west2 (Los Angeles) (10) |
| Load generators | 10 generators × e2-standard-2 × 600 senders each |
| Load placement | one blaster per validator, in that validator's zone, sending only to it |
Chain
| Chain ID | 32286 |
|---|---|
| Network | bench |
| Epoch length | 1,200 s (20 min) |
Consensus parameters
| Max header delay | 1,000 ms |
|---|---|
| Min header delay | 350 ms |
| Max batch delay | 250 ms |
| Vote timeout | 5,000 ms |
| Parameters file | consensus-parameters.yaml |
| Confirmed by | node startup logs |
Workload
| Warm-up | 120 s (2 min) |
|---|---|
| Mixed phase | 2,400 s (40 min) |
| Cool-down | 180 s (3 min) |
| Transaction types | batched_self_send, contract_deploy, faucet_drip, native_transfer, stablecoin_approve, stablecoin_transfer, uniswap_v2_add_liquidity, uniswap_v2_remove_liquidity, uniswap_v2_swap, uniswap_v3_swap, uniswap_v4_swap, wtel_unwrap, wtel_wrap |
Chaos events
| Marker | Time (UTC) | Node | Action | Detail |
|---|---|---|---|---|
| C1 | 2026-11-07 17:54:44 | bench-validator-05 | docker kill -s KILL telcoin | systemctl stop holds it down (unit has Restart=on-failure). bench-blaster-05 targets ONLY this node, so its sends fail for the outage by design; the generator resends on connection errors |
| C2 | 2026-11-07 17:59:44 | bench-validator-05 | systemctl start telcoin | docker start is impossible: the unit runs docker run --rm |
| C3 | 2026-11-07 18:04:44 | bench-observer-11 | systemctl restart telcoin | observer restart |
Steps
- The node image was built for this benchmark from the branch bench/main-fixes-2026-11 (telcoin-network main plus the nine instrumentation commits of r20261105-0803 and 29 topic commits) and named by registry digest (sha256:a1a82341…). Before creating anything the harness ran that image to confirm that it reports telcoin-network 0.1.0 built from commit 067639def, has the sub-second timestamp fork active from genesis and contains the log messages later checks rely on. The committed consensus parameters were unchanged from r20261105-0803: maximum header delay 1,000 ms, minimum header delay 350 ms, maximum batch delay 250 ms and a header batch threshold of 5.
- The analysis reads fixed-shape log lines and Prometheus series, not free text. The instrumentation of r20261105-0803 is unchanged: the voter's vote path by stage, the proposer's vote round trips, the engine's execute, persist and commit timings, rejoin blockers and node-mode transitions with their reasons, the scheduled leader of every round, batch-serve denials, epoch-record signers and request durations by service class. This build adds one line per batch (why it was built, how long its oldest transaction waited, its quorum and report times), per header (digests, the wait of its oldest digest, why it was proposed, and why a header was never certified), per committed certificate and sub-DAG (with the leader's creation time and the commit timestamp), and per consensus output (fetch, persist, the engine queue, and execution split into signer recovery, EVM, state root and finish). It also adds the commit-lag and execution-lag gauges, rejoin attempts and holds, vote refusals and their reasons.
- Preflight checked access, tools, quotas and a clean genesis directory, and confirmed the load-generator image built from tn-transaction-generator commit 6dad96f (sha256:5766094d…). This build records, per landed transaction, both the block timestamp (whole seconds, the basis of time to land) and the moment the generator's own block scanner saw the receipt (time to seen, what a user waits for), by submission phase. It also exports a 1-in-100 sample of landings with each block's batch and consensus position. The KPI driver used for this page reproduces every published KPI of r20261105-0803, r20261103-1922, r20261007-0316 and r20261006-0458 exactly.
- The harness created ten validators and one observer on c3-highcpu-8 (8 vCPU, 16 GB), each with a 100 GB Hyperdisk Balanced boot disk provisioned at 20,000 IOPS and 800 MiB/s, one validator in each of ten regions (Los Angeles, London, Sydney, Tokyo, Ashburn, the Netherlands, Singapore, Hong Kong, Seoul and Mumbai) with the observer in Los Angeles, the zones of r20261105-0803; every node was created on the first attempt. It also created the run's own metrics VM (bench-obs, e2-standard-4, 50 GB, in Los Angeles) running Prometheus and Grafana; it collects metrics only, no logs.
- As on r20261105-0803, each node's keys were generated on the node by the operator scripts of the v0.2.0-devnet release (the v2 scripts), four nodes at a time, with each node advertising its RPC endpoint on its internal address. The harness built genesis for a separate benchmark chain (chain id 32286, 20-minute epochs) and, before any node started, installed on every node a network configuration in Closed admission mode that names every validator and lists the observer as trusted by every validator.
- Each node was finalized by the release's scripts, four at a time, and stopped again. While every node was stopped, the harness wrote the benchmark settings into each start script, with the log filter of r20261105-0803 (info, with debug for the primary, the engine, the subscriber, worker networking and request-response), and ran a short fsync probe on each node's disk (fleet median 986 µs; slowest validator-06, 1,253 µs). The harness now records each node's database sync mode: every node ran the default durable mode, which syncs the database on every commit. Because this build logs the engine's dumps at trace, the same filter wrote 8 to 11 MB a minute on validator-01 during the light blasts and about 5 MB a minute under load, against 1.16 GB a minute on r20261105-0803.
- All eleven nodes were started at one due time, 15:23:30 UTC, in the same second, on the first attempt. Each node's metrics agent was then pointed at bench-obs; no node was restarted for it. The devnet's shared observability VM received none of this run's samples or logs.
- A verify-only pass read back from each running node its commit (067639def), fork schedule, start-script settings and parameters, and all eleven passed by 15:37:37 UTC. The health and admission gate, which finished at 15:40:20, found all eleven nodes reporting to bench-obs with no devnet series, all ten validators active, and on every node Closed admission on the primary and worker networks with no fallback and no denied connection.
- The harness deployed the devnet contract set that the mixed workload calls from 15:40:20 to 15:50:47 UTC. The chain was then idle until 16:01:58.
- The light-load arm came first, with chaos off. One-off funder VMs, one per planned blast, funded the senders of all four (16:03 to 16:10:48 UTC). Ten generators, one per validator in that validator's zone and sending only to it, then ran three blasts in turn, each with 60 s of warm-up, 360 s of mixed load and 60 s of cool-down: index 0 with 8 senders per generator and up to 16 transactions in flight per sender (mixed load 16:16:56 to 16:22:56), index 1 with 16 senders and the same window (16:36:54 to 16:42:54), and index 2 with 600 senders and a window of 4 (16:56:55 to 17:02:55). Each blast's results were archived in a directory of their own.
- The fourth blast, index 3, was the full load profile. Creating one of its ten generator VMs failed at about 17:18 UTC when the gcloud client timed out after 300 s, and the harness stopped the blast at 17:20:07. The nine generators already created had sent for at most about 90 s, all inside their warm-up, and the chaos had not fired. Their partial results are kept but used nowhere on this page. Following the harness's printed resume instruction, the operator funded a fresh set of senders (17:30:25 to 17:32:50) and ran the same profile again as index 4, which is the headline blast.
- In index 4 ten generators on e2-standard-2, each with 600 senders and a window of 16 (at most 96,000 transactions in flight), started together at 17:38:02 UTC: a preamble of setup transactions, 120 s of warm-up, 2,400 s of the mixed load of r20261105-0803 and up to 180 s of cool-down. Measured from the generators' own phase records, the merged mixed window ran from 17:40:15 to 18:21:01 and the window with all ten generators running from 17:41:01 to 18:20:15. No network delay was injected.
- The chaos schedule was timed to the chain's epochs. The harness read the ConsensusRegistry at a pinned block (block 15,099, epoch 6), chose the close of epoch 7, and re-read the schedule at 17:39:03 UTC (drift 0 s). bench-validator-05's container was killed with SIGKILL at 17:54:44, 59 s before that close, and held down; the node was restarted at 17:59:44 and the observer at 18:04:44. Before the kill and the observer restart each node saved its own container log.
- The harness sampled the fleet's highest block during the run. The only period it recorded without a new block, 18:21:00 to 18:22:30 UTC, is the end of the load, not a stall. Afterwards it collected each generator's results and the on-node sampler files and waited for every validator to read active (validator-05 already did at the first poll). It pulled 60-second machine metrics from Cloud Monitoring and 15-second node metrics from bench-obs's Prometheus (15:40:20 to 18:39:18 UTC), with no series missing; collection finished at 18:41:19.
- The KPIs and windows on this page come from the publishing pipeline's own window and KPI steps, run through the driver that reproduced the comparison runs. Windows are cut from the generators' own phase records, not the planned phase times. The floor KPIs read index 0. The bias-free time to commit joins the landing sample to each block's millisecond commit timestamp from validator-01's log. The stage breakdown of a transaction's path, certification, the vote path, the F1 and F3 checks, the execution split and the log census come from the log lines above, read from every node's complete unit log in one streaming pass each, and from a local replay of the Prometheus snapshot taken at teardown.
- Teardown pulled every node's logs before deleting anything. With the engine dumps at trace, the unit logs of all eleven nodes came to about 1.87 GB together (about 49 GB each on r20261105-0803), and the pulls took five minutes (18:43:54 to 18:48:54 UTC). The teardown then ran the certification check, took a 20 MB snapshot of the bench-obs Prometheus, checked provenance (commit 067639def and the sub-second fork on every node) and deleted all 12 instances at 19:17:24, in one attempt (TEARDOWN_DONE deleted=12 left=0). The results directory was re-indexed: 804 files, 2.64 GB. The whole run took 4 h 12 min, against 9 h 16 min for r20261105-0803.
The harness scripts live in a private repository; the steps above describe what they do.
Results
| Metric | Value | Window | Source | Method |
|---|---|---|---|---|
| Average chain throughput, all load generators running | 3,448.7 tx/s | all generators active | combined.series.per_second[chain_tx] | Chain transactions per block-timestamp second, summed over the 2354-s window when every load generator was in its mixed phase and divided by its length. |
| Best 60 seconds, all load generators running | 4,939.2 tx/s | all generators active | combined.series.per_second[chain_tx] | Highest mean chain tx/s over any 60 consecutive seconds inside the all-generators window (aggregate.py best_window_mean). |
| Transactions landed over the whole blast | 9,288,504 tx | whole run | combined.totals.landed_chain | Sum of tx_count over every block the load generators saw during the blast, own and foreign transactions alike. |
| Blocks per second | 12.96 blocks/s | merged mixed window | combined.blocks.rows[timestamp] | Chain blocks whose timestamp falls in the merged mixed window, divided by its length (aggregate.py landed.capacity.blocks_per_sec over the same window). |
| Time to land, median | 2.81 s | whole run | combined.latency.ts_ms.p50 | Block timestamp minus submit time over every landed own transaction in the blast, from the load generators' merged latency histograms (aggregate.py merge_latency), in seconds. |
| Time to land, p95 | 14.4 s | whole run | combined.latency.ts_ms.p95 | Block timestamp minus submit time over every landed own transaction in the blast, from the load generators' merged latency histograms (aggregate.py merge_latency), in seconds. |
| Time to land, median, whole-second bias removed | 3.31 s | whole run | combined.latency.ts_ms.p50 + 0.5 s | Time to land, median, plus 0.5 s. The block timestamp is the committing leader's header creation time cut to the whole second, so every sample reads 0 to 999 ms low, 0.5 s on average. |
| Time to seen, median | 14.18 s | whole run | combined.latency.seen_ms.p50 | Submit time to the moment the load generator's block scanner had the block holding the transaction (block and receipts fetched from the generator's own node), over every landed own transaction in the blast, from the load generators' merged latency histograms (aggregate.py merge_latency), in seconds. |
| Time to seen, p95 | 53.39 s | whole run | combined.latency.seen_ms.p95 | Submit time to the moment the load generator's block scanner had the block holding the transaction (block and receipts fetched from the generator's own node), over every landed own transaction in the blast, from the load generators' merged latency histograms (aggregate.py merge_latency), in seconds. |
| Time to land at light load, median | 0.25 s | light load, 8 senders per generator | blast-0-20261107T163112Z/blasters latency.ts_ms.p50 | Block timestamp minus submit time over every landed own transaction submitted in the mixed phase of the light-load blast light8 (index 0: 8 senders per load generator at window 16, chaos off), pooled from the generators' latency histograms with the aggregate.py merge_latency method, in seconds; not bias-corrected. |
| Time to seen at light load, median | 1.63 s | light load, 8 senders per generator | blast-0-20261107T163112Z/blasters latency.seen_ms.p50 | Submit time to the generator's block scanner having the block over every landed own transaction submitted in the mixed phase of the light-load blast light8 (index 0: 8 senders per load generator at window 16, chaos off), pooled from the generators' latency histograms with the aggregate.py merge_latency method, in seconds; not bias-corrected. |
| Commit latency, p95 | 5.00–6.25 s | merged mixed window | metrics.prometheus.series.tn_primary_commit_latency_seconds.by_node[healthy validators].p95 | For each healthy validator, the nearest-rank p95 over the merged mixed window of its 15-s histogram_quantile(0.95) commit-latency series; reported as the range over validators. |
| Execution time per consensus output | 2.02–2.15 s | merged mixed window | metrics.prometheus.series.tn_engine_execution_duration_seconds.by_node[healthy validators].mean | For each healthy validator, the nearest-rank median over the merged mixed window of its 15-s mean execution-duration series; reported as the range over validators. |
| Write throughput, p95, busiest validator | 567.6 MiB/s | merged mixed window | metrics.gcp[bench-validator-10].disk_write_bytes_per_sec p95 | Nearest-rank p95 of the 60-s mean write-byte rate over the merged mixed window for each healthy validator; the highest (bench-validator-10) is reported. |
| Write operations, p95, busiest validator | 5,897 IOPS | merged mixed window | metrics.gcp[bench-validator-08].disk_write_ops_per_sec p95 | Nearest-rank p95 of the 60-s mean write-operation rate over the merged mixed window for each healthy validator; the highest (bench-validator-08) is reported. |
| CPU, p95, busiest validator | 22.2 % | merged mixed window | metrics.gcp[bench-validator-08].cpu_utilization p95 | Nearest-rank p95 of the 60-s CPU utilisation over the merged mixed window for each healthy validator; the highest (bench-validator-08) is reported. |
| Node resident memory, p95, busiest validator | 7.55 GiB | merged mixed window | metrics.gcp[bench-validator-02].telcoin_rss_bytes p95 | Nearest-rank p95 of the 60-s telcoin process resident memory over the merged mixed window for each healthy validator; the highest (bench-validator-02) is reported. |
| IO pressure, highest reading, range over validators | 30.7–34.2 % | merged mixed window | sampler[healthy validators].psi_io_some_avg10 max | For each healthy validator, the highest 10-s 'some' IO pressure reading the on-node sampler logged inside the merged mixed window; reported as the range over validators. |
Recovery
| Node | bench-validator-05 |
|---|---|
| Killed | 2026-11-07 17:54:44 UTC |
| Restarted | 2026-11-07 17:59:44 UTC |
| Rejoined | 2026-11-07 18:23:05 UTC |
| Catch-up time | 1,365 s (22 min 45 s) |
| Failed batch reports | 0 |
Chain TPS per 5-minute interval
Each bar is the average chain TPS over one 5-minute interval of the merged mixed window (17:40–18:21 UTC); fainter bars had fewer generators active than the maximum, the dashed line is the average of all bars and amber lines mark the chaos events listed below.
- C1 2026-11-07 17:54:44 UTC: docker kill -s KILL telcoin on bench-validator-05 (systemctl stop holds it down (unit has Restart=on-failure). bench-blaster-05 targets ONLY this node, so its sends fail for the outage by design; the generator resends on connection errors)
- C2 2026-11-07 17:59:44 UTC: systemctl start telcoin on bench-validator-05 (docker start is impossible: the unit runs docker run --rm)
- C3 2026-11-07 18:04:44 UTC: systemctl restart telcoin on bench-observer-11 (observer restart)
5-minute data (9 rows)
| start_utc | end_utc | generators_active | chain_tps |
|---|---|---|---|
| 2026-11-07T17:40:15Z | 2026-11-07T17:45:15Z | 9.4 | 3,694.8 |
| 2026-11-07T17:45:15Z | 2026-11-07T17:50:15Z | 10.0 | 3,571.9 |
| 2026-11-07T17:50:15Z | 2026-11-07T17:55:15Z | 10.0 | 3,467.1 |
| 2026-11-07T17:55:15Z | 2026-11-07T18:00:15Z | 10.0 | 3,199.5 |
| 2026-11-07T18:00:15Z | 2026-11-07T18:05:15Z | 10.0 | 3,339.7 |
| 2026-11-07T18:05:15Z | 2026-11-07T18:10:15Z | 10.0 | 3,254.7 |
| 2026-11-07T18:10:15Z | 2026-11-07T18:15:15Z | 10.0 | 3,364.0 |
| 2026-11-07T18:15:15Z | 2026-11-07T18:20:15Z | 10.0 | 3,901.3 |
| 2026-11-07T18:20:15Z | 2026-11-07T18:21:01Z | 3.8 | 2,174.0 |
The fine-grained chain throughput series (its step_s field gives the seconds per point) is published as series/tps-1s.json.
Other networks
| Measure | Telcoin Network (this run) | Tempo |
|---|---|---|
| Throughput | 3,448.7 tx/s (Avg chain TPS) | 16,455 TPS, shown as "Settled TPS" for the nightly run of 5 Oct 2026 (commit de5f5d3). perf.tempo.xyz describes it as "Median settled throughput across 130 measured blocks." |
| How throughput is measured | Chain transactions per block-timestamp second, summed over the 2354-s window when every load generator was in its mixed phase and divided by its length. | The methodology page defines TPS as "Transactions per second (derived from gas / avg tx gas)": gas used divided by the average gas of one transaction, reported as the median over the measured blocks. |
| Run length | 40.8 min merged mixed window; each generator was set to a 40 min mixed phase | 2m 39s for the 5 Oct 2026 nightly run (130 measured blocks at a 687 ms average block time), per perf.tempo.xyz. |
| Validators and placement | 10 validators and 1 observer across 10 regions: asia-east2 (Hong Kong), asia-northeast1 (Tokyo), asia-northeast3 (Seoul), asia-south1 (Mumbai), asia-southeast1 (Singapore), australia-southeast1 (Sydney), europe-west2 (London), europe-west4 (Netherlands), us-east4 (Ashburn), us-west2 (Los Angeles); load generators: one blaster per validator, in that validator's zone, sending only to it | perf.tempo.xyz labels the scenario "Public mix · multi-region". The methodology page says "Benchmarks run two local validators in consensus on a dedicated bare-metal server with 100 GiB pre-bloated state." The public tempoxyz/tempo repo has nightly workflows for both kinds of setup: bench-e2e runs on one bare-metal runner, and bench-e2e-multi-region defaults to 10 validators on cloud VMs (its regions input reads "Empty uses the selected cloud's five defaults."). The run page does not say which setup produced this number. |
| Hardware | c3-highcpu-8, 8 vCPUs, 16 GB memory; disk: hyperdisk-balanced, 100 GB, 20,000 IOPS, 800 MiB/s | The methodology page lists a dedicated bare-metal server: "AMD EPYC 4585PX (16C/32T, 4.3/5.7 GHz)", "128 GB 3600 MHz" memory and "2×960 GB NVMe + 2×1.92 TB NVMe via schelk". The repo's multi-region workflow uses cloud VMs instead (c4d-standard-48-lssd on GCP, c6id.8xlarge on AWS). |
| Workload | batched_self_send, contract_deploy, faucet_drip, native_transfer, stablecoin_approve, stablecoin_transfer, uniswap_v2_add_liquidity, uniswap_v2_remove_liquidity, uniswap_v2_swap, uniswap_v3_swap, uniswap_v4_swap, wtel_unwrap, wtel_wrap; 10 generators × 600 senders each | perf.tempo.xyz labels the scenario "Public mix"; the methodology page says "100% TIP-20 token transfers (ERC-20 equivalent)". The run used 1,000 accounts and 4 TIP-20 tokens. |
| State | Fresh chain from genesis for every run | The methodology page says "100 GiB pre-bloated state". |
| Latency (as each source defines it) | Time to land, median: 2.81 s; Time to land, p95: 14.4 s; Commit latency, p95: 5.00–6.25 s | Block time of 687 ms, which the methodology page defines as "Average wall-clock time between blocks". The run page gives no per-transaction latency. Separately, tempo.xyz says "Blocks finalize in ~0.6 seconds with no re-orgs." (a network claim, not a figure from this benchmark). |
| Source and fetch date | This page; every figure names its source field. | perf.tempo.xyz nightly run of 5 Oct 2026 (commit de5f5d3), the perf.tempo.xyz methodology page, tempo.xyz, and the benchmark workflows in the tempoxyz/tempo repo. All fetched on 5 Oct 2026. |
Sources for Tempo (fetched 2026-10-05):
- perf.tempo.xyz (nightly benchmark results)
- perf.tempo.xyz methodology
- tempo.xyz
- tempoxyz/tempo repository
- bench-e2e workflow (bare-metal)
- bench-e2e-multi-region workflow
Tempo's setup may not match ours: its methodology page describes two validators on one bare-metal server, while our runs use ten validators across five regions, so read this table as context, not a ranking.
This table is context, not a ranking.
Caveats
- The comparison run, r20261105-0803, had the same fleet in the same zones, the same load shape and chaos plan, the same consensus timing (1,000/350/250 ms) and the same operator scripts. The node build differs by the 29 topic commits and a newer main underneath. The generator build differs (6dad96f against 3911723; it adds records). The log volume differs about 200-fold, and here the load followed an hour of light blasts. Against r20261105-0803, average chain throughput with all generators running fell from 4,578 to 3,449 tx/s (-24.7 %), blocks a second from 15.84 to 12.96, while primary rounds a second rose from 0.84-0.91 to 0.97-1.00, so each round carried fewer blocks. The median time to land fell from 3.3 to 2.81 s, the 95th percentile rose from 12.84 to 14.40 s, and the median time to seen rose from 12.92 to 14.18 s (95th percentile 29.78 to 53.39 s). No difference can be assigned to any one change.
- The headline blast is index 4, not index 3. Index 3 aborted when one gcloud call to create a generator VM timed out on the client, before the chaos fired and with every generator still in its warm-up. Its partial results are excluded from every figure. The re-run started 20 minutes later than planned (17:38:02 instead of 17:17:36 UTC), with the same profile and a freshly funded set of senders. A single transient client timeout aborting a blast is a harness defect, the third run in a row with this class of fault.
- Throughput fell because persisting got slower, not executing. Per transaction the engine spent 288.6 µs (215 µs on r20261105-0803): 87.4 µs executing and 199.2 µs persisting, about four times the persist cost there. The inverse, about 3,465 tx/s, matches the chain's rate. Most of the persist time (178 of 199 µs) is the database commit and its sync, 1.03 to 1.06 s per consensus output on every node against 0.30 to 0.34 s there. The logical writes per transaction were the same in both runs (table growth 73, 33 and 47 bytes per transaction; 0.44 accounts and 0.62 storage slots touched), but each transaction cost about 4.3 times the device bytes (155 KB against 36 KB). The disks wrote 470 to 530 MB/s sustained, with one-minute peaks at the 800 MiB/s provisioned limit, while CPU stayed at 17 to 19 %. The disk type, provisioning, OS image family and operator scripts were the same. r20261103-1922 showed the same write pattern, so r20261105-0803 looks like the outlier. Those two runs booted the same OS image and kernel and still wrote differently, so an image or kernel change is not the cause. After the run, a test with the same database library on a 7.0 kernel reproduced the extra bytes: when the kernel caches the database file in large multi-page blocks, each 4 KB page a commit changes through the database's memory map is written to disk as the whole block, 1.7 to 14.5 times the bytes per commit in the test; this run's 4.3 times lies inside that range. The trigger on this run's nodes is not identified. The first suspect was a reopen: each node's database was created by a start during staging and reopened for the run, and on reopening the database library gives the kernel its random-access hint only for the pages then in use. Tested alone, a reopen did not reproduce the extra bytes: the disk wrote 1.00 times the bytes the database changed, both on the test machine and on a cloud VM of the validators' machine type and disk. On that VM the extra bytes appeared in only three cases. When the file's pages had been read into the page cache before they were changed, the disk wrote 11.9 times the bytes, in write operations of about 262 KB each; this run's were 222 to 228 KB. When the database library's pre-write of new pages was off, it wrote 9.7 times. When another process had opened the database, it wrote 1.8 times. What caused one of these on the run's nodes is not known. r20261105-0803's 1.16 GB-a-minute debug log on the same volume is not a direct cause, but it filled the page cache: those nodes began re-reading evicted database pages within the first minutes of the load and wrote fewer bytes from then on. This run's nodes first re-read pages near 18:00 UTC, and most of them wrote about a fifth less after that; r20261103-1922's never did. Memory pressure is the likely reason r20261105-0803 wrote less (about 75 %). In a later test on the test machine, memory pressure cut the disk's writes from 11 times the bytes the database changed to 4.3 times, where they stayed, and with the random-access hint applied it removed the extra bytes (5.0 times to 1.00).
- Executing got 47 % cheaper per transaction (87.4 against about 165 µs). The engine dumps are gone from the hot path (0.2 to 0.3 µs per transaction now). Waiting for signer recovery fell to almost nothing, because recovery runs off the execution thread. Of the rest, the EVM took 26.5 µs, the state root 49.2 µs (56 % of execution, so a parallel state root is the next execution step) and finishing a block 11.4 µs. The queue of consensus outputs in front of the engine never held more than two: consensus committed at the rate execution finished (on validator-01 before the kill, 0.573 sub-DAGs committed against 0.559 outputs executed a second).
- Under load, three validators' users landed nothing for 8 to 14 minutes with no RPC error. Tokyo, Hong Kong and Mumbai executed fastest (6.4 to 9.1 rounds of execution lag, against 12.6 to 13.5 for the middle of the fleet and 22 to 31 for Los Angeles, Sydney and Ashburn), so the blocks their headers named were ones the other seven voters had not executed yet. A voter checks that block before voting; a header lives one round (about 0.9 s) before its proposer cancels it for the next one, and the middle of the fleet reached those blocks 3 to 4 s later, so the seven slower voters refused every header of the three and, being a quorum by themselves, moved the rounds on without them: 0.2 %, 0.0 % and 16.3 % of their own headers were certified in the loaded window. The spread of executed tips is a consequence of this build's commit path no longer waiting for execution; the 1 s bound on the voter's wait made the refusals explicit but is not the cause, and the earlier 5 s wait would also have voted too late. The proposed fix is proposer-side: name the block that at least 2f+1 validators have executed instead of the proposer's own newest block.
- Not every generator landed transactions all the time. No generator got the 'validator is inactive' error before the kill (two did on r20261105-0803, after their validators were demoted). But besides the three starved generators of the last caveat, generator 05 landed nothing for the first 193 s of its mixed phase, and its warm-up transactions were seen after 318 s at the median. After the kill it stayed pinned to its down and then inactive validator and landed nothing more. The 99th-percentile time to land, 402.56 s, is not ordinary latency either. Under load 1.85 % of the sampled transactions (1.81 % of all landed transactions, counted from the generators' full histograms), mostly on eight of the ten generators, landed only after their generator resent them 300 s later. The node had accepted them without error, then skipped them at execution because they ran before an earlier transaction from the same sender, or queued them behind such a skip; on three validators the transaction pool also dropped 3,804 accepted transactions from its queue in the first minutes of the load. Until a transaction executes the node cannot find it, so the resend was the only way back. The stalled generators account for about 14 % of them. The same resend cliff was present on r20261103-1922 (2.20 % of landings) and r20261105-0803 (0.87 %); r20261105-0803's stayed under 1 %, below its 99th percentile of 31.38 s, so the jump of the 99th percentile against that run is a threshold, not a new fault.
- The floor KPIs come from the light blasts, not the load: index 0 (8 senders per generator) is the published floor, passed to the pipeline with --floor-blast-dir. Index 1 (16 senders) gives nearly the same: land 0.31 s, seen 1.73 s. The light windows are short (358 s of all ten generators each), came before the load and include no epoch boundary and no chaos. At that load every sender's window is full, so throughput there (694 and 1,291 tx/s) is set by the window and the latency, not by the chain. The block timestamp is whole seconds and rounds down: on the landing sample a transaction reached its leader's commit timestamp after 0.64 s at the median and was seen after 1.50 s; the mean rounding bias there was 440 ms.
- The 600 x 4 point comes from index 2: 600 senders per generator with a window of 4, at most 24,000 in flight, with no chaos and a 324-s window. It carried 3,868.5 tx/s, 12 % more than the load blast, with a median time to land of 0.88 s and to seen of 3.22 s. By Little's law its mean time to seen (4.79 s) implies 18,544 transactions in flight, 77 % of the cap. The load blast, with four times the window, held about 82,000 in flight for less throughput: 3,466 tx/s up to the kill, with both blasts bound by the execution engine. Past about 24,000 in flight, a larger window adds queueing, not throughput, but the difference in throughput does not come from the window: under load the database commit per consensus output grew by about 0.03 s every minute, and index 2 ran 40 minutes earlier. The load blast also includes the chaos, two epoch boundaries and the starved generators.
- At light load London's and the Netherlands' generators landed at a median of 12.8 and 13.6 s (index 0) and 11.3 and 17.0 s (index 1), while the other eight landed in 0.12 to 0.84 s. Their headers were certified (99.5 %), but only 13 to 18 % of their certificates were committed. The rest waited for the periodic re-send of undelivered headers, about every 20 s and within 10 ms of each other on the two nodes, so their transactions landed in bursts; in index 0 both generators landed nothing for the same 41 s. The pooled floor median is not affected, but the floor's 95th and 99th percentiles are theirs. At 600 x 4 they landed in 1.39 and 1.51 s. This is the far-validator certificate inclusion seen on earlier runs, not a change of this build.
- Because the block timestamp rounds down to the second, the page also gives a time to land with 0.5 s added (3.31 s at the median under load). The landing sample, 1 in 100 joined on validator-01's log, gives a bias-free median time to commit of 3.07 s and a 95th percentile of 13.42 s, with a mean rounding bias of exactly 500 ms. The commit timestamp is the leader header's creation time, so it reads earlier than the commit itself; validator-01 logged the commit itself 4.70 s after submission at the median.
- Under load the time to seen sits about ten seconds above the commit. Before the kill a consensus output waited 9.6 s at the median in the engine's queue, 64 % of the time to seen, because executing and persisting one output took 1.66 s, of which the database commit took 1.04 s. Persisting is off the visibility path (blocks reach RPC once executed) and reaches the user only through that queue. Los Angeles, Sydney and Ashburn executed 2 to 3 % slower than the chain committed (94 to 97 µs per transaction against 84 to 88, all in finishing blocks and signer recovery), so their backlog grew. That accounts for 9 to 13 s of the 21 to 35 s by which their generators' receipts trailed the landing. Investigated after the run: on a node that is behind, the job that recovers the next output's signers runs on the same thread pool as finishing the current output's blocks and slows each finish by 10 to 11 µs per transaction, more than the node's margin against the chain, so a node that falls behind stays behind until the load ends. At light load consensus took 82 % of the time to seen; its largest stage, from the leader's header to the commit, took 0.61 s at the median (40 %).
- No healthy validator left the committee. On r20261105-0803 two did, when their own commit fell 41 rounds behind; here commit no longer waits for execution. Execution fell up to 62 rounds behind on Los Angeles, 54 on Sydney and 58 on Seoul (during index 2), and every one of them stayed active. Before the kill the commit lag stayed at or below 5 rounds at the 99th percentile (6.12 at most during the chaos window), and all ten validators were active until the kill. The wait moved to the subscriber: on Los Angeles a consensus output waited 14 s at the median for its execution before it was assembled.
- Validator-05 restarted at 17:59:44 UTC with 212 certificates restored and no gap in its stored DAG. Because it was killed 59 s before an epoch close, it found itself an epoch behind 19 s later and followed the chain as an inactive member. Under load it barely gained: 165 outputs behind at 18:00:37, it replayed about 30.5 outputs a minute while the chain made 23.3. It made one rejoin attempt, at a gap of 20 rounds, at 18:22:49.65, was active 0.3 s later and had its first own certificate committed 1.85 s after that: 28 min 05.8 s after the kill, a minute or two after the mixed load ended, in the middle of epoch 9. r20261105-0803's validator-05 needed 13 promotions and the next epoch boundary. The check 'active within 2 minutes of the kill' is not met, but inconclusive: catch-up was bound by execution under saturation. While inactive, the node re-requested a stuck replay 8 times, and its replay never froze.
- The recovery record on this page takes its rejoin time (18:23:05 UTC) and catch-up duration (1,365 s, from the first inactive sample) from the 15-second node-mode samples. Validator-05's own log puts the demotion at 18:00:03.5, the promotion at 18:22:49.98 and restart to active at 1,385.8 s. The failed-report count (0) is from the log; the node has no seal-failure series in the metrics.
- With validator-05 down, nine validators voted. The kill paused commits for about 13 s, and the epoch-7 close 59 s later left the chain 24 s without a block with transactions (13 s on r20261105-0803); the epoch-8 close left 14 s. Los Angeles and Sydney reached each loaded close 11 to 27 s late and never signed the epoch-7 and epoch-8 records, so both were signed by exactly seven validators, the quorum with no margin. After the kill those two refused 14,785 and 11,953 votes as execution-behind, so every remaining voter was on the critical path and commits paused 4 to 15 s at a time from 18:16 to 18:21. No runbook restart was needed.
- The nodes logged 77,168 warnings and errors (266,118 on r20261105-0803), with no panic and no database error, and the storm of 'too new' vote refusals is gone (12 against 100,852). Two new messages appeared at the loaded epoch-8 boundary, and the investigations after the run explained both. About 1,400 'failed to track spawned task' messages (1,248 inside the chaos window) came from a 1,000-slot task channel that stops being drained once an epoch starts to close, while network messages keep spawning tasks. The 414,513 transactions skipped as 'nonce too low' (about 37,700 per node) were second copies: the starved validators' transactions sat uncertified past the generators' 300-s resend, the resent copies were accepted again, and both copies were committed. The batch-serve pool never filled; 6,675 batch-sync streams were refused at open, all after the epoch-8 boundary.
- The validators did less CPU work than on r20261105-0803 (busiest p95 22.2 % against 36.7 %) and wrote more: the telcoin process wrote 498.7 MiB/s at the median (131.6 MiB/s there) and I/O pressure peaked at 30.7 to 34.2 % (10.2 to 17.4 %). These figures include almost no logging, where r20261105-0803's included 25 to 45 MB/s of debug log. The pre-start fsync probe does not predict persist time: validator-06, the slowest probe, committed in 1,079 ms at the median against 1,069 to 1,081 ms for the fleet.
- The node metrics on this page come from the run's own Prometheus on bench-obs, scraped every 15 s. The analysis behind the caveats also replays the snapshot of that Prometheus taken at teardown. No sample or log line of this run went to the devnet's shared observability VM.
- The harness is recorded at commit 538af51, marked dirty; the dirty flag is a known false positive of the harness's record and was not re-checked for this run. The run's planned phase times differ from the generators' real ones by up to 77 s, so the analysis cuts every window from the generators' own records. No node was restarted outside the chaos plan.
- This configuration has been run once, so run-to-run variation is not measured.
- Per-validator rows cover 9 validators; bench-validator-05 excluded as chaos target(s).
- Recovery: demoted / rejoined are the first 15-s Prometheus samples showing the change, so they trail the node's own log times by up to a scrape interval plus one step.
- Recovery: no tn_batch_builder_seal_failures_total count for bench-validator-05; failed_reports unavailable.
- Recovery.failed_reports is the published value, not recomputed from the run data.
- The harness working tree had uncommitted changes when the run was deployed.
- Validators bench-validator-04 and bench-validator-08 (in phase warmup, phase mixed and run) and bench-validator-10 (in phase mixed and run) certified fewer than 90 % of their own headers; their logs are preserved under results/r20261107-1504/logs/ (see INDEX.md).
Full report
Changes from the original report:
- The site adds a banner at the top of the report that links back to this page.
SHA-256 of the committed file, before the banner is added: 3dcfadbfcd9b07e3e10bb5930e0421e397deb98f0c2d6d2c7876ff2a9afc906c