The adaptive vote fix and the order hold on tn-lmdb at 20,000 IOPS
2026-11-15 · started 16:53 UTC · ended 19:33 UTC · merged mixed window 40 min 11 s
We repeated r20261115-1131 with one change: the vote fix ran in its adaptive form instead of the always-on form. In the adaptive form a header names the author's own executed tip until one of the author's headers is cancelled uncertified; the author then uses the quorum rule, never more than 8 outputs behind its tip, inside a starved window of up to 128 headers, and afterwards returns to its own tip. The build (upstream main 45dce8947 plus 114 commits, with the tn-lmdb storage engine, the order hold for issue #1807 and the epoch-record switches), the fleet of ten c3-highcpu-8 validators and an observer on Hyperdisk Balanced at 20,000 provisioned IOPS, the load and the chaos plan were the same; validator-05 was killed by design inside the load blast. On the load blast's chain rate (9,337.2 against 9,288.6 tx/s), seen p50 (8.62 against 9.40 s), fleet own-header certification (95.00 against 95.59 %), lowest validator (76.4 against 83.6 %) and committed round rate the two forms are not told apart from r20261115-1131's own spread. Outside it, against the adaptive form, lie the load blast's seen p95 (24.18 against 17.33 s) and blast 0's rate (680.0 against 698.3 tx/s), and the per-validator rows outside it moved both ways. Under load 61.5 % of the headers built in the load blast lay inside a starved window, and 38.51 % named the author's tip against 0.26 %, so the named block was fresher (4.83 against 7.57 block-bearing outputs behind the tip); what that is worth was not measured. Outside generator 05 the load blast had no gap resend, no unlanded transaction and no 'nonce too high' skip, against 104,168, 9,290 and 52,588 in bench 6's adaptive run; the hold's out-of-order counter read 0, but it has no bench 6 twin, so this run is consistent with the hold doing its job and does not prove it. All 7 epoch closes certified on one record hash and one closing block, and no node logged a fail-stop line. Prometheus series are partial on validators 01, 05 and 08 and the observer (the methodology says which). A first attempt at this run was lost to an expired cloud login and is not published. The always-on run on the same build and disk is r20261115-1131; the form comparison reads the pair as weak support for keeping the always-on form as the default at 20,000 IOPS. Against bench 6's adaptive run, r20261113-2118, throughput rose (9,337 against 5,395 tx/s) and fleet certification before the kill fell (94.95 against 99.56 %), with the storage engine, the vote handler, the order hold, the epoch-record switches, the change to the adaptive rule and the run's shape changing together, so neither is put down to one change; each form ran once.
Compared against r20261115-1131.
Key figures
How it was run
Software
| Upstream commit | telcoin-network 91a15e22376fa1277c107824e738dce689792c56 · bench/bench7-2026-11 · commit verified from node startup logs |
|---|---|
| Node image | us-docker.pkg.dev/telcoin-network/tn-public/adiri@sha256:15acdccafff786c2aa60ac69cddc3b09069648d5cf22ae1727c060fafa68822c |
| Node image digest | sha256:15acdccafff786c2aa60ac69cddc3b09069648d5cf22ae1727c060fafa68822c |
| Node version | 0.1.0 |
| Fork schedule | subsecond-from-genesis |
| Load generator commit | 62bbd5f |
| Load generator image digest | sha256:7cefcc505201b2e40361a7d8cfe034778090b76c568cb2400369bf566c605a59 |
| Analysis commit | 62bbd5f |
| Harness commit | 2ff39153a9d0b75ce462f1c53ec593b5020d3a85 |
| Harness submodules | common: 1a6955d8a2c6f19e53cf7aaad5e32a9107a21a7d tn-node-deployment: d9446f720026fa5ce8b64fce9074e56b05b3f4eb |
Fleet
| Fleet | c3-highcpu-8 + Hyperdisk Balanced at 20,000 IOPS (c3-hdb), ten regions, validator-05 in Ashburn |
|---|---|
| Validators | 10 |
| Observers | 1 |
| Machine type | c3-highcpu-8 |
| vCPUs per node | 8 |
| Memory per node | 16 GB |
| Disk | hyperdisk-balanced, 100 GB, 20,000 IOPS, 800 MiB/s |
| Regions | asia-east2 (Hong Kong), asia-northeast1 (Tokyo), asia-northeast3 (Seoul), asia-south1 (Mumbai), asia-southeast1 (Singapore), australia-southeast1 (Sydney), europe-west2 (London), europe-west4 (Netherlands), us-east4 (Ashburn), us-west2 (Los Angeles) (10) |
| Load generators | 10 generators × e2-standard-2 × 600 senders each |
| Load placement | one blaster per validator, in that validator's zone, sending only to it |
Chain
| Chain ID | 32286 |
|---|---|
| Network | bench |
| Epoch length | 1,200 s (20 min) |
Consensus parameters
| Max header delay | 1,000 ms |
|---|---|
| Min header delay | 350 ms |
| Max batch delay | 250 ms |
| Vote timeout | 5,000 ms |
| Parameters file | consensus-parameters.yaml |
| Confirmed by | node startup logs |
Parameter arms
| Arm | From (UTC) | To (UTC) | Values |
|---|---|---|---|
| bench7-adaptive | 2026-11-15 17:13:50 | — | all 11 nodes: digest_order_hold=true, epoch_record_collect_past_uncertified=true, epoch_record_conflict_halt=true, epoch_record_recertify=true, quorum_exec_block=true, quorum_exec_block_starved_rounds=4, vote_queue_fix=true |
Workload
| Warm-up | 120 s (2 min) |
|---|---|
| Mixed phase | 2,400 s (40 min) |
| Cool-down | 180 s (3 min) |
| Transaction types | batched_self_send, contract_deploy, faucet_drip, native_transfer, stablecoin_approve, stablecoin_transfer, uniswap_v2_add_liquidity, uniswap_v2_remove_liquidity, uniswap_v2_swap, uniswap_v3_swap, uniswap_v4_swap, wtel_unwrap, wtel_wrap |
Chaos events
| Marker | Time (UTC) | Node | Action | Detail |
|---|---|---|---|---|
| C1 | 2026-11-15 18:44:01 | bench-validator-05 | docker kill -s KILL telcoin | systemctl stop holds it down (unit has Restart=on-failure). bench-blaster-05 targets ONLY this node, so its sends fail for the outage by design; the generator resends on connection errors |
| C2 | 2026-11-15 18:49:01 | bench-validator-05 | systemctl start telcoin | docker start is impossible: the unit runs docker run --rm |
| C3 | 2026-11-15 18:54:01 | bench-observer-11 | systemctl restart telcoin | observer restart |
Steps
- Both runs of bench 7 used one node image, built from the branch bench/bench7-2026-11 at commit 91a15e223 (upstream main 45dce8947 plus 114 commits), version 0.1.0, named by registry digest (adiri@sha256:15acdccafff7…68822c). The manifest names the storage engine tn-lmdb in both runs, and every node ran it. The load generator was built from tn-transaction-generator commit 62bbd5f, the commit bench 6's r20261113-1351 used.
- Between bench 6 and bench 7 the build changed in several ways at once: the tn-lmdb storage engine with storage v2, which came in one change (#1834) and are not separable from each other; the upstream rework of the vote handler; the order hold; and the epoch-record switches. The run's shape changed too: validator-05 was killed earlier into the load, and no transaction was queued in a pool at the load blast's start, where bench 6 had some on two validators in r20261113-1351 and on five in r20261113-2118. No figure on these pages separates these changes.
- The arm set seven node switches through the prestart step, before the nodes' coordinated start: quorum_exec_block, vote_queue_fix, digest_order_hold, epoch_record_collect_past_uncertified, epoch_record_conflict_halt and epoch_record_recertify, all true, and quorum_exec_block_starved_rounds, 0 in r20261115-1131 (arm bench7-always) and 4 in r20261115-1653 (arm bench7-adaptive), the only difference between the two arms. The closing-block check has no switch and ran in both runs. The switch gauges read the arm's values on every validator. A later form of the epoch-record code drops epoch_record_conflict_halt; bench 7 ran with it on. Each node's first process, started during the deploy, ran with every switch at its default; no epoch closed and no blast ran before the arm.
- The vote fix is the switch quorum_exec_block: a proposer names in each header a block that a quorum of validators should already have executed, and a voter signs only after it finds that block in its own chain. With quorum_exec_block_starved_rounds at 0, the always-on form, every header uses the quorum rule, never more than 8 outputs behind the author's own tip (the floor). With 4, the adaptive form, a header names the author's own tip until one of the author's headers is cancelled uncertified; the author then uses the quorum rule inside a starved window of up to 128 headers and afterwards returns to its own tip. On this build the window is counted in headers built, not rounds. A header is under-held when, at its proposal, at most five other voters held the block it named.
- The order hold is the switch digest_order_hold, the stack's candidate fix for issue #1807. Bench 6's pages describe that defect: when an own header is cancelled or committed late, its batches are queued again behind batches a newer own header already carried, so a sender's later transactions are skipped at execution as 'nonce too high' and land only after the generator resends them at its 300-s gap check. The hold counts every re-queue that still goes out of order, the one counter that isolates it, and the headers built under the hold.
- Against r20261115-1131 only the form changed: the arm bench7-adaptive set quorum_exec_block_starved_rounds to 4 instead of 0, with the same other switches. The image, the generator, the fleet, the zones, the disk, the load and the chaos plan are the same; the schedule differed (below), and so did the length of the window before the kill.
- Each run created ten validators and one observer on c3-highcpu-8, each node with one 100 GB Hyperdisk Balanced disk at 20,000 IOPS and 800 MiB/s, boot image ubuntu-2404-noble-amd64-v20261105. One validator ran in each of bench 6's ten zones (Los Angeles, London, Sydney, Tokyo, Ashburn, the Netherlands, Singapore, Hong Kong, Seoul and Mumbai), the observer in Los Angeles. The host probe was uniform (kernel 7.0.0-1017-gcp, read-ahead 128 KiB), and no node was flagged slow by the fsync probe (median 1,074 µs), where bench 6 flagged two nodes in each of its 20,000-IOPS runs.
- Each run had three blasts. Ten generators on e2-standard-2, one per validator in that validator's zone and sending only to it, ran blast 0 (light, 8 senders each with up to 16 transactions in flight, 8 x 16), blast 1 (light, 600 x 4) and blast 2, the load blast (600 x 16, at most 96,000 in flight), with a 120-s warm-up before its mixed phase. The chaos plan was bench 6's, inside blast 2's mixed phase: validator-05's container was killed with SIGKILL before an epoch boundary and held down for 300 s, then started, and the observer was restarted 300 s after that. Generator 05 sends only to validator-05, so it is cut off for the outage and for as long as its validator follows the chain afterwards. The kill left 1,290 s of load after it here, against 1,017 s in r20261113-2118.
- Headline figures keep bench 5's definitions, as on bench 6's pages. Throughput is the chain's transactions per second over the window in which all ten generators ran their mixed phase, the kill inside it (2,389 s in this run). Time to land runs to the block's timestamp, time to seen to when the generator's scanner saw it. Own-header certification is the share of a validator's own headers that were certified; the fleet figure before the kill is the certification analysis's count in bench 7 and the rounds analysis's in bench 6, over windows of different lengths (1,121 s here, 817 s in r20261115-1131, 1,393 s in r20261113-2118). The distance to the named block is counted by rank among block-bearing outputs.
- Each form ran once. A difference between the two bench 7 runs is judged against r20261115-1131's own spread, the range over its own buckets of the same window (300-s buckets of the load blast, 60-s blocks of blast 0, the buckets before the kill for per-validator rows); inside it the forms are not told apart from noise. That spread is variation inside one run, not run-to-run variance, and no figure carries a significance test.
- The analysis read every node's whole log and the node metrics exported from Prometheus (17:08:30-19:19:50, 22,759 series). The metrics agent's defect left series partial on three validators and the observer: validator-01 lacks four for the whole run (among them tn_engine_persist_duration_seconds), validator-05 three until 18:40:00, validator-08 six engine, executor, reth and write-stall series after 18:29:45, and the observer its save, commit, persist-duration and write-stall series after 18:58:45. Validator-01's missing series include tn_engine_execution_duration_seconds, which this page's execution-duration figure reads, so that figure covers the other validators; any other node-side figure that reads a series named here covers fewer nodes or stops early. Per-minute samples of every node's own metrics endpoint stand in for the analysis (within about 1 % over whole windows). On every node the agent's network label replaces the node's own (primary, worker-0), so per-network series cannot be read from Prometheus.
- Times are UTC on 15 November. Deployment ran 16:53:30-17:40:41, with the arm at 17:13:50 and the coordinated start at 17:21:38. The idle check stopped the chain at 17:45:57 on a missing up series from validator-05, a healthy node, and the blasts were launched by hand at 17:46:24, 5.7 min after the deploy's end. The mixed phases ran 17:56:45-18:02:47 (blast 0), 18:09:19-18:15:33 (blast 1) and 18:25:20-19:05:31 (blast 2). Validator-05 was killed at 18:44:01, 1,121 s into blast 2's mixed phase, closing epoch 4, and started at 18:49:01; the observer was restarted at 18:54:01.
- No generator lost preamble steps, so every blast has an all-generators window (blast 1: 346 s). Every blast ended with rc 0. At the end checks all 11 nodes reported fatal_db=0 and halt=0, at blocks 96,259-96,263. The teardown deleted all 12 instances at 19:34:12, and the archive verified 626 files with no problem. This run is the second attempt: the first was lost when the cloud login expired during its first light blast, its load blast never ran, its fleet was torn down, and no figure on this page comes from it. This run used a fresh fleet, the same image digest and an environment identical but for the run id.
The harness scripts live in a private repository; the steps above describe what they do.
Results
| Metric | Value | Window | Source | Method |
|---|---|---|---|---|
| Average chain throughput, all load generators running | 9,337.2 tx/s | all generators active | combined.series.per_second[chain_tx] | Chain transactions per block-timestamp second, summed over the 2389-s window when every load generator was in its mixed phase and divided by its length. |
| Best 60 seconds, all load generators running | 11,357.8 tx/s | all generators active | combined.series.per_second[chain_tx] | Highest mean chain tx/s over any 60 consecutive seconds inside the all-generators window (aggregate.py best_window_mean). |
| Transactions landed over the whole blast | 25,134,603 tx | whole run | combined.totals.landed_chain | Sum of tx_count over every block the load generators saw during the blast, own and foreign transactions alike. |
| Blocks per second | 31.33 blocks/s | merged mixed window | combined.blocks.rows[timestamp] | Chain blocks whose timestamp falls in the merged mixed window, divided by its length (aggregate.py landed.capacity.blocks_per_sec over the same window). |
| Time to land, median | 1.66 s | whole run | combined.latency.ts_ms.p50 | Block timestamp minus submit time over every landed own transaction in the blast, from the load generators' merged latency histograms (aggregate.py merge_latency), in seconds. |
| Time to land, p95 | 8.77 s | whole run | combined.latency.ts_ms.p95 | Block timestamp minus submit time over every landed own transaction in the blast, from the load generators' merged latency histograms (aggregate.py merge_latency), in seconds. |
| Time to land, median, whole-second bias removed | 2.16 s | whole run | combined.latency.ts_ms.p50 + 0.5 s | Time to land, median, plus 0.5 s. The block timestamp is the committing leader's header creation time cut to the whole second, so every sample reads 0 to 999 ms low, 0.5 s on average. |
| Time to seen, median | 8.62 s | whole run | combined.latency.seen_ms.p50 | Submit time to the moment the load generator's block scanner had the block holding the transaction (block and receipts fetched from the generator's own node), over every landed own transaction in the blast, from the load generators' merged latency histograms (aggregate.py merge_latency), in seconds. |
| Time to seen, p95 | 24.18 s | whole run | combined.latency.seen_ms.p95 | Submit time to the moment the load generator's block scanner had the block holding the transaction (block and receipts fetched from the generator's own node), over every landed own transaction in the blast, from the load generators' merged latency histograms (aggregate.py merge_latency), in seconds. |
| Time to land at light load, median | 0.25 s | light load, 8 senders per generator | blast-0-20261115T180602Z/blasters latency.ts_ms.p50 | Block timestamp minus submit time over every landed own transaction submitted in the mixed phase of the light-load blast light8 (index 0: 8 senders per load generator at window 16, chaos off), pooled from the generators' latency histograms with the aggregate.py merge_latency method, in seconds; not bias-corrected. |
| Time to seen at light load, median | 1.64 s | light load, 8 senders per generator | blast-0-20261115T180602Z/blasters latency.seen_ms.p50 | Submit time to the generator's block scanner having the block over every landed own transaction submitted in the mixed phase of the light-load blast light8 (index 0: 8 senders per load generator at window 16, chaos off), pooled from the generators' latency histograms with the aggregate.py merge_latency method, in seconds; not bias-corrected. |
| Commit latency, p95 | 2.19–2.29 s | merged mixed window | metrics.prometheus.series.tn_primary_commit_latency_seconds.by_node[healthy validators].p95 | For each healthy validator, the nearest-rank p95 over the merged mixed window of its 15-s histogram_quantile(0.95) commit-latency series; reported as the range over validators. |
| Execution time per consensus output | 0.89–0.92 s | merged mixed window | metrics.prometheus.series.tn_engine_execution_duration_seconds.by_node[healthy validators].mean | For each healthy validator, the nearest-rank median over the merged mixed window of its 15-s mean execution-duration series; reported as the range over validators. |
| Write throughput, p95, busiest validator | 286 MiB/s | merged mixed window | metrics.gcp[bench-validator-06].disk_write_bytes_per_sec p95 | Nearest-rank p95 of the 60-s mean write-byte rate over the merged mixed window for each healthy validator; the highest (bench-validator-06) is reported. |
| Write operations, p95, busiest validator | 20,039 IOPS | merged mixed window | metrics.gcp[bench-validator-03].disk_write_ops_per_sec p95 | Nearest-rank p95 of the 60-s mean write-operation rate over the merged mixed window for each healthy validator; the highest (bench-validator-03) is reported. |
| CPU, p95, busiest validator | 56.5 % | merged mixed window | metrics.gcp[bench-validator-03].cpu_utilization p95 | Nearest-rank p95 of the 60-s CPU utilisation over the merged mixed window for each healthy validator; the highest (bench-validator-03) is reported. |
| Node resident memory, p95, busiest validator | 7.09 GiB | merged mixed window | metrics.gcp[bench-validator-04].telcoin_rss_bytes p95 | Nearest-rank p95 of the 60-s telcoin process resident memory over the merged mixed window for each healthy validator; the highest (bench-validator-04) is reported. |
| IO pressure, highest reading, range over validators | 7.7–11.6 % | merged mixed window | sampler[healthy validators].psi_io_some_avg10 max | For each healthy validator, the highest 10-s 'some' IO pressure reading the on-node sampler logged inside the merged mixed window; reported as the range over validators. |
Recovery
| Node | bench-validator-05 |
|---|---|
| Killed | 2026-11-15 18:44:01 UTC |
| Restarted | 2026-11-15 18:49:01 UTC |
| Rejoined | 2026-11-15 19:12:46 UTC |
| Catch-up time | 1,395 s (23 min 15 s) |
| Failed batch reports | 1 |
Chain TPS per 5-minute interval
Each bar is the average chain TPS over one 5-minute interval of the merged mixed window (18:25–19:05 UTC); fainter bars had fewer generators active than the maximum, the dashed line is the average of all bars and amber lines mark the chaos events listed below.
- C1 2026-11-15 18:44:01 UTC: docker kill -s KILL telcoin on bench-validator-05 (systemctl stop holds it down (unit has Restart=on-failure). bench-blaster-05 targets ONLY this node, so its sends fail for the outage by design; the generator resends on connection errors)
- C2 2026-11-15 18:49:01 UTC: systemctl start telcoin on bench-validator-05 (docker start is impossible: the unit runs docker run --rm)
- C3 2026-11-15 18:54:01 UTC: systemctl restart telcoin on bench-observer-11 (observer restart)
5-minute data (9 rows)
| start_utc | end_utc | generators_active | chain_tps |
|---|---|---|---|
| 2026-11-15T18:25:20Z | 2026-11-15T18:30:20Z | 9.7 | 10,601.8 |
| 2026-11-15T18:30:20Z | 2026-11-15T18:35:20Z | 10.0 | 10,434.2 |
| 2026-11-15T18:35:20Z | 2026-11-15T18:40:20Z | 10.0 | 10,065.9 |
| 2026-11-15T18:40:20Z | 2026-11-15T18:45:20Z | 10.0 | 9,018.7 |
| 2026-11-15T18:45:20Z | 2026-11-15T18:50:20Z | 10.0 | 9,396.5 |
| 2026-11-15T18:50:20Z | 2026-11-15T18:55:20Z | 10.0 | 8,821.9 |
| 2026-11-15T18:55:20Z | 2026-11-15T19:00:20Z | 10.0 | 8,554.5 |
| 2026-11-15T19:00:20Z | 2026-11-15T19:05:20Z | 10.0 | 8,094.5 |
| 2026-11-15T19:05:20Z | 2026-11-15T19:05:31Z | 4.5 | 6,510.7 |
The fine-grained chain throughput series (its step_s field gives the seconds per point) is published as series/tps-1s.json.
Other networks
| Measure | Telcoin Network (this run) | Tempo |
|---|---|---|
| Throughput | 9,337.2 tx/s (Avg chain TPS) | 16,455 TPS, shown as "Settled TPS" for the nightly run of 5 Oct 2026 (commit de5f5d3). perf.tempo.xyz describes it as "Median settled throughput across 130 measured blocks." |
| How throughput is measured | Chain transactions per block-timestamp second, summed over the 2389-s window when every load generator was in its mixed phase and divided by its length. | The methodology page defines TPS as "Transactions per second (derived from gas / avg tx gas)": gas used divided by the average gas of one transaction, reported as the median over the measured blocks. |
| Run length | 40.2 min merged mixed window; each generator was set to a 40 min mixed phase | 2m 39s for the 5 Oct 2026 nightly run (130 measured blocks at a 687 ms average block time), per perf.tempo.xyz. |
| Validators and placement | 10 validators and 1 observer across 10 regions: asia-east2 (Hong Kong), asia-northeast1 (Tokyo), asia-northeast3 (Seoul), asia-south1 (Mumbai), asia-southeast1 (Singapore), australia-southeast1 (Sydney), europe-west2 (London), europe-west4 (Netherlands), us-east4 (Ashburn), us-west2 (Los Angeles); load generators: one blaster per validator, in that validator's zone, sending only to it | perf.tempo.xyz labels the scenario "Public mix · multi-region". The methodology page says "Benchmarks run two local validators in consensus on a dedicated bare-metal server with 100 GiB pre-bloated state." The public tempoxyz/tempo repo has nightly workflows for both kinds of setup: bench-e2e runs on one bare-metal runner, and bench-e2e-multi-region defaults to 10 validators on cloud VMs (its regions input reads "Empty uses the selected cloud's five defaults."). The run page does not say which setup produced this number. |
| Hardware | c3-highcpu-8, 8 vCPUs, 16 GB memory; disk: hyperdisk-balanced, 100 GB, 20,000 IOPS, 800 MiB/s | The methodology page lists a dedicated bare-metal server: "AMD EPYC 4585PX (16C/32T, 4.3/5.7 GHz)", "128 GB 3600 MHz" memory and "2×960 GB NVMe + 2×1.92 TB NVMe via schelk". The repo's multi-region workflow uses cloud VMs instead (c4d-standard-48-lssd on GCP, c6id.8xlarge on AWS). |
| Workload | batched_self_send, contract_deploy, faucet_drip, native_transfer, stablecoin_approve, stablecoin_transfer, uniswap_v2_add_liquidity, uniswap_v2_remove_liquidity, uniswap_v2_swap, uniswap_v3_swap, uniswap_v4_swap, wtel_unwrap, wtel_wrap; 10 generators × 600 senders each | perf.tempo.xyz labels the scenario "Public mix"; the methodology page says "100% TIP-20 token transfers (ERC-20 equivalent)". The run used 1,000 accounts and 4 TIP-20 tokens. |
| State | Fresh chain from genesis for every run | The methodology page says "100 GiB pre-bloated state". |
| Latency (as each source defines it) | Time to land, median: 1.66 s; Time to land, p95: 8.77 s; Commit latency, p95: 2.19–2.29 s | Block time of 687 ms, which the methodology page defines as "Average wall-clock time between blocks". The run page gives no per-transaction latency. Separately, tempo.xyz says "Blocks finalize in ~0.6 seconds with no re-orgs." (a network claim, not a figure from this benchmark). |
| Source and fetch date | This page; every figure names its source field. | perf.tempo.xyz nightly run of 5 Oct 2026 (commit de5f5d3), the perf.tempo.xyz methodology page, tempo.xyz, and the benchmark workflows in the tempoxyz/tempo repo. All fetched on 5 Oct 2026. |
Sources for Tempo (fetched 2026-10-05):
- perf.tempo.xyz (nightly benchmark results)
- perf.tempo.xyz methodology
- tempo.xyz
- tempoxyz/tempo repository
- bench-e2e workflow (bare-metal)
- bench-e2e-multi-region workflow
Tempo's setup may not match ours: its methodology page describes two validators on one bare-metal server, while our runs use ten validators across five regions, so read this table as context, not a ranking.
This table is context, not a ranking.
Caveats
- Against r20261115-1131 (ranges are that run's own spread over its 300-s blocks of blast 2): chain rate 9,337.2 against 9,288.6 tx/s (8,344.6-10,544.9), seen p50 8.62 against 9.40 s (8.28-10.90), fleet certification over blast 2 95.00 against 95.59 % (93.37-97.29), the lowest validator 76.4 against 83.6 % (55.2-89.3) and committed rounds per second 2.22 against 2.32 (1.99-2.73): none is told apart from noise. Blast 1, over the nine generators' common window, landed 9,412.2 against 9,399.4 tx/s, not told apart either.
- Outside that spread, against the adaptive form: blast 2's seen p95, 24.18 against 17.33 s (14.27-18.78 s; the values from the generators' combined documents, the spread from 1-in-100 sampled landings), and blast 0's rate, 680.0 against 698.3 tx/s (688.2-722.8 over 60-s blocks). The land tail has no spread to judge it against: p95 8.77 against 7.76 s, p99 22.06 against 17.40 s.
- Per-validator certification before the kill moved both ways, outside r20261115-1131's range for each: validator-01 89.2 against 100.0 %, validator-05 90.8 against 100.0 %, validator-04 99.4 against 94.1 %, validator-06 79.0 against 62.7 %, validator-08 99.2 against 86.4 %. Of 41 per-bucket rows outside the validator's own range, 28 are lower here and 13 higher. Under 98 % over blast 2 were validators 01, 02, 05, 06 and 10 here and 04, 06, 08 and 10 there; the slowest executor was the same validator in 0 of 8 buckets. As in bench 6, the pair does not separate the form from which validators lagged.
- By the logged policy, 61.5 % of the headers built over blast 2 lay inside a starved window (59.7 % before the kill). Replayed from the logs, with the replay matching the logged policy on 99.875 % of headers, 56.3 % of the windows before the kill and 89.1 % of those after validator-05's start opened at the 128-header cap; those two figures are replayed, not logged. A return to the own tip lost 1.66 own-tip headers on average, and 64.0 % of returns lost their first header. Headers that named the tip certified 98.1 % of the time against 96.4 % in r20261115-1131, not told apart.
- What the window bought is a fresher named block: 38.51 % of headers named the author's tip against 0.26 %, and 54.0 % named the floor against 77.0 %. The fleet mean distance from the tip to the named block fell from 7.57 to 4.83 block-bearing outputs (r20261115-1131's spread over its nine proposers 6.96-7.95; the fleet median was 8 in both runs), from 6.77 to 4.45 s at validator-01's output rate, and a lower bound on the block's age fell from 7.36 to 4.43 s. What that age costs was not measured; if it costs something, the fresher block is a reason to prefer the adaptive form.
- Against r20261113-2118, bench 6's adaptive run on the older build: blast 2 throughput 9,337 against 5,395 tx/s (+73.1 %), seen p50/p95 8.62/24.18 against 12.89/28.76 s. Fleet certification before the kill was 94.95 against 99.56 % (1,121 s against 1,393 s), with four validators other than 05 under 98 % against none; refused-behind answers ran 39.3 a minute against 0.6 and superseded answers 2,089 against 136; 10.34 % of rounds had four or more under-held parents against 2.98 %; the median execution lag at the voters fell from 34.0 to 21.36 rounds. Over blast 2, 12.55 % of headers were under-held and certified 66.70 % of the time against 99.06 % for the rest. No counter divides the fall between the vote-handler rework, the voters' execution lag, the hold, the change to the adaptive rule and the run's shape.
- As in r20261115-1131, the engine and the disk both sat below their limits, the engine nearer to its own: busy 978.4 ms a second by the log timeline, 9,376 tx/s per node executed against the 9,582 its task cost allows at full busy (104.36 µs per transaction, persist 47 % of it; no task started before the previous one ended, 0 of 23,554 starts), and the device at 0.863 of its write cap at 8.67 ms a write, 1.85 writes and 25,498 bytes per transaction. Through blast 2 commit per transaction rose from 20.32 to 42.34 µs and the executed rate fell from 10,497 to 8,164 tx/s while tn-lmdb's used size went from 1.11 to 2.72 GiB; a least-squares fit gives 14.9 µs per GiB (r² 0.968), a fit, not a mechanism. Bench 6's r20261113-2118 sat at the write cap (0.999).
- In blast 2, outside generator 05, no generator gap-resent, none left a transaction unlanded after the 180-s tail scan, no node counted a 'nonce too high' skip and no landing was seen 300 s or more after submission, against 104,168 gap resends, 9,290 unlanded and 52,588 skips in r20261113-2118 (its resends and skips from its warm-up start to its cool-down end); the plan's fleet checks found 0 gap resends outside generator 05 and 0 skips for the run. Lost own headers carrying batches still occurred on all nine validators other than 05 (988 not certified and 338 certified but never committed over blast 2, validator-05's own left out), and no gap resend followed them, where in bench 6 every gap-resend second of those generators fell 280-345 s after one (124 of 124 seconds in r20261113-2118).
- The hold's out-of-order re-queue counter read 0 in the log, the warnings and the export. It is the one counter that isolates the hold; it has no bench 6 twin, and it says that no re-queue went out of order here, not how many would have without the hold. The fall against bench 6 moved with the hold, the storage engine, the vote path, the kill earlier into the load and bench 6's queued pools at blast 2's start, and is not separable between them. This run is consistent with the hold doing its job and does not prove it.
- Over blast 2, 3,259 headers were built under the hold against 1,866 in r20261115-1131, none carrying a batch. The longest held run was 113 headers and 55.3 s (validator-06, after validator-05's start), against 21 headers (before the kill) and 9.1 s (after validator-05's start) there, two different runs. Held headers certified 60.5 % of the time against 97.4 % for the rest, an association no counter ties to the hold. 183 held headers, of 2,056 built on the digest trigger, are candidates for a trigger that counted held digests; that figure is inferred from logged fields, not measured. The fleet ran a digest threshold of 5 batches and a minimum header delay of 350 ms. No over-quota report was logged.
- Generator 05 landed none of its own transactions for 1,281 s (18:44:02-19:05:23) and had 8,129 gap resends. 2,137 of its transactions, from 187 senders, stayed unlanded after the tail scan: all were sent to validator-05 and accepted in the last 2 s before the kill, and the restarted validator built batches holding 1,387 of them at 19:12:33, after the tail scan had ended. Its 7,724 landings seen 300 s or more after their send are late sightings of blocks stamped just before the kill, not late landings.
- In the light blasts the far validators, 02 (London) and 06 (the Netherlands), landed a small share of what the others landed: generators 02 and 06 landed 2.2 and 1.8 tx/s in blast 0 against a median of 88.3 for the other eight, with seen p50 56.52 and 62.09 s (r20261115-1131: 8.8 and 5.5 tx/s; r20261113-2118: 2.91 and 2.54), and 169.6 and 157.0 against 1,186.6 in blast 1. Every batch-carrying header of the two was certified, yet 75 % and 76 % of them in blast 0, 46 % and 50 % in blast 1, were never committed; the commit-path re-queue came a median 60 to 61 rounds later (21.0 to 21.4 s), and a batch took 44.9 and 61.4 s in blast 0, 12.9 and 14.7 s in blast 1, at the median from its first own proposal to execution, against 0.93 and 1.37 s on the near validators. The order hold held none of their headers, and bench 6 shows the same pattern in blast 0 without a hold. The last step of the explanation, near headers not naming the late certificate, is inferred, since which parents a header names is not logged.
- The run closed 7 epochs, each on one record hash and one closing block, but only 6 lie inside the export and were published by all eleven nodes: the last (epoch 6, 19:25:01) falls after the export's last sample and after three validators' logs were pulled, so eight nodes show it, and 6 of 7 is not like for like with r20261115-1131's 8 of 8. The closing-block check returned ok 135 times and nothing else, and the close identity check read ok 6 times on every node, with no mismatch. At the kill's close the fewest signers on a quorum line were 7, with the first quorum after 11.84 s; at the other loaded close 7 and 11.93 s (r20261113-2118: 8 and 13.85 s, 7 and 14.85 s). Validators 04 and 03, which did not restart, published records from the collector as the restarted validator-05 did; both lagged in execution at those closes. The conflict, re-certify and walk paths never fired on any node, so the run does not exercise them.
- Validator-05 was active (CvvActive) again 1,712.4 s after the kill and 400.5 s after the cool-down ended, after replaying the kill's epoch (68.2 s) and one rejoin attempt; no other validator left the active state. The export's mark of its promotion falls 11.5 s after the node's log line. The kill came 1,121.2 s into the load against 817.1 s in r20261115-1131 and 1,393 s in r20261113-2118 (1,359.8 s to active there), so the times are not one comparable number. Measured and not explained: after its promotion its first own certificate committed after 59.6 s, against 1.65 s in r20261115-1131; and validator-04 logged 516 spawn-capacity errors ('no available capacity') in under 5 s at 18:45:24, as epoch 5 began. The observer had its first line at distance 1 4.5 s after its restart.
- No node logged a fail-stop line the census counts, and all 11 nodes reported fatal_db=0 at the end checks. The idle check stopped the chain on a missing up series from validator-05, a healthy node, lost to the metrics agent's defect, and the blasts were launched by hand 5.7 min after the deploy's end, against 15.7 min in r20261115-1131; that schedule, not the form, put the kill in epoch 4's close here and in epoch 5's there.
- Each form ran once on one build and one disk, so run-to-run variation is not measured and no figure carries a significance test. The form comparison reads the pair as support for keeping the always-on form as the default at 20,000 IOPS, and weak support: besides the two rows outside the spread, the adaptive form held more headers and adds code the always-on form does not carry (the starved window, its 128-header cap and the return to the own tip). Not measured or open: what the named block's age costs; a run at 50,000 IOPS, where bench 6 found the adaptive form ahead on certification and behind on the senders' cost (bench 7 made none); a floor of 16 outputs, which arithmetic on these runs says leaves no header under-held; what lowers the far validators' committed share; and what separates the upstream vote-handler rework from the hold's cadence.
- Per-validator rows cover 9 validators; bench-validator-05 excluded as chaos target(s).
- Commit_latency_p95_s: no tn_primary_commit_latency_seconds.p95 points for bench-validator-01.
- Execution_time_per_output_s: no tn_engine_execution_duration_seconds.mean points for bench-validator-01.
- Recovery: demoted / rejoined are the first 15-s Prometheus samples showing the change, so they trail the node's own log times by up to a scrape interval plus one step.
- Recovery: failed_reports is the rise of the batch-seal failure counter on bench-validator-05 over the whole metrics window (metrics.prometheus.series.tn_batch_builder_seal_failures_total.count), outage or not.
- The harness working tree had uncommitted changes when the run was deployed.
- Validator bench-validator-06 certified fewer than 90 % of its own headers in blast 2 mixed and run; its logs are preserved under results/r20261115-1653/logs/ (see INDEX.md).
- Before the coordinated start each node ran once during staging with every node switch at its default (11 start(s) on 11 node(s)); the run's switches were on from the coordinated start at 2026-11-15T17:13:50Z.
Full report
Changes from the original report:
- The site adds a banner at the top of the report that links back to this page.
SHA-256 of the committed file, before the banner is added: 10daca180bbf939bc78474c69fb7eaac8cfce37dbf14365fd44a11d818bf5684