Sub-second consensus and the mmap pack-file backend on the c3 Hyperdisk fleet
2026-10-06 · started 00:53 UTC · ended 02:54 UTC · merged mixed window 49 min 53 s
We re-ran the 29 September benchmark on the same fleet shape, ten c3-highcpu-8 validators with Hyperdisk Balanced disks, with a telcoin-network build that has sub-second consensus timestamps active from genesis and the mmap pack-file storage backend, and with the consensus delays cut from 3 s, 1 s and 1 s to a 500 ms maximum header delay, a 250 ms minimum header delay and a 250 ms maximum batch delay. The harness killed validator-05 at 02:09:55 UTC, 42 seconds before its epoch was due to close, and about 20 seconds later the other nine validators stopped committing: no transaction landed for 301 seconds, from 02:10:15 to 02:15:16 UTC, and the chain moved again only after validator-05 was restarted at 02:15:00 and its first header was certified. Against the 29 September run (r20260929-1647), where a mid-epoch kill of the same validator left the other nine running, average chain throughput with all ten load generators running fell 21%, from 4,177 to 3,320 tx/s; the stall sits inside that window, and with its 301 seconds left out the average is 3,983 tx/s, 5% lower. The median time for a transaction to land over the whole blast fell from 4.0 s to 3.2 s while the 95th percentile rose from 14.4 s to 26.1 s; the stall and the two under-funded generators (03 and 04) sit inside that latency measurement. The restarted validator was proposing again 16 seconds after its restart, but because the chain had stalled it had only about 20 seconds of commits to catch up on, so this run says nothing about the 25-minute catch-up seen on 29 September. The logs show the stall sequence but not why the nine running validators could not proceed without the tenth.
Compared against r20260929-1647.
Key figures
How it was run
Software
| Upstream commit | telcoin-network 964087e021be394a250df4ca918266113cb70fd7 · devnet-bench-10-5-2026 · commit verified from node startup logs |
|---|---|
| Node image | us-docker.pkg.dev/telcoin-network/tn-public/adiri:mmap-subsecond-10-5-2026 |
| Node image digest | sha256:61fe433a9bd351b4cadf076ead04add33a0ba8b791b023eeef85d2f7c20675d8 |
| Node version | 0.1.0 |
| Fork schedule | subsecond-from-genesis |
| Load generator commit | 3644adc |
| Load generator image digest | sha256:811ab3a78637b436d69868ab10bdfeab5e50780bd5906d4643f1acc9de1425ca |
| Analysis commit | 1046acb |
| Harness commit | f186c4c3ff056943410f03514a5967abe887d25d |
| Harness submodules | common: b498b71e107f09c7e6a7a68ae5ba71a514b98609 tn-node-deployment: 1f0385d79a99e008cadf66f0522f4417a09e7543 |
Fleet
| Fleet | c3-highcpu-8 + Hyperdisk Balanced (c3-hdb) |
|---|---|
| Validators | 10 |
| Observers | 1 |
| Machine type | c3-highcpu-8 |
| vCPUs per node | 8 |
| Memory per node | 16 GB |
| Disk | hyperdisk-balanced, 100 GB, 20,000 IOPS, 800 MiB/s |
| Regions | australia-southeast1, europe-west2, europe-west4, northamerica-northeast1, us-west2 (5) |
| Load generators | 10 generators × e2-standard-2 × 600 senders each |
| Load placement | one blaster per validator, in that validator's zone, sending only to it |
Chain
| Chain ID | 32286 |
|---|---|
| Network | bench |
| Epoch length | 1,200 s (20 min) |
Consensus parameters
| Max header delay | 500 ms |
|---|---|
| Min header delay | 250 ms |
| Max batch delay | 250 ms |
| Vote timeout | 5,000 ms |
| Parameters file | consensus-parameters.yaml |
| Confirmed by | node startup logs |
Workload
| Warm-up | 120 s (2 min) |
|---|---|
| Mixed phase | 2,400 s (40 min) |
| Cool-down | 180 s (3 min) |
| Transaction types | batched_self_send, contract_deploy, faucet_drip, native_transfer, stablecoin_approve, stablecoin_transfer, uniswap_v2_add_liquidity, uniswap_v2_remove_liquidity, uniswap_v2_swap, uniswap_v3_swap, uniswap_v4_swap, wtel_unwrap, wtel_wrap |
Chaos events
| Marker | Time (UTC) | Node | Action | Detail |
|---|---|---|---|---|
| C1 | 2026-10-06 02:09:55 | bench-validator-05 | docker kill -s KILL telcoin | restart across the epoch boundary; systemctl stop holds it down (unit has Restart=on-failure). bench-blaster-05 targets ONLY this node, so its sends fail for the outage by design; the generator resends on connection errors |
| C2 | 2026-10-06 02:15:00 | bench-validator-05 | systemctl start telcoin | docker start is impossible: the unit runs docker run --rm |
| C3 | 2026-10-06 02:20:04 | bench-observer-11 | systemctl restart telcoin | observer restart |
Steps
- Before creating anything, the harness resolved the node image tag adiri:mmap-subsecond-10-5-2026 to its registry digest and pinned the run to that digest. It ran the pinned image to confirm that it reports telcoin-network 0.1.0 built from commit 964087e02, has the sub-second timestamp fork active from genesis and contains the log messages later checks rely on, and it recorded the committed consensus parameters file and its three delays.
- Preflight checked access, tools, quotas and a clean genesis directory, and confirmed the load-generator image built from tn-transaction-generator commit 3644adc.
- The harness created ten validators and one observer on c3-highcpu-8 (8 vCPU, 16 GB), each with a 100 GB Hyperdisk Balanced boot disk provisioned at 20,000 IOPS and 800 MiB/s, two validators in each of five regions, and opened the consensus ports between the nodes only.
- It generated validator keys and built genesis for a separate benchmark chain (chain id 32286, 20-minute epochs).
- It finalized each node's configuration with the committed benchmark consensus parameters: maximum header delay 500 ms, minimum header delay 250 ms and maximum batch delay 250 ms, with every other key as in the devnet defaults. The file sets no vote timeout, so the node's built-in 5 s applied. It then started the chain.
- It relabelled each node's metrics agent so that every sample from these nodes is recorded under the benchmark network, never the live devnet.
- It restarted the nodes one at a time with RPC open on the internal network, waiting after each until the node answered with the benchmark chain id and consensus rounds were advancing. Each node's boot lines had to show the pinned commit, the three consensus delays and the fork schedule recorded at the gate, and all eleven did. An on-node sampler was then started on each node.
- A health gate confirmed that all eleven nodes reported to Prometheus under the benchmark network with consensus rounds rising, and a read of the ConsensusRegistry through an RPC tunnel confirmed the 1,200 s epoch duration.
- It deployed the devnet contract set that the mixed workload calls.
- A one-off funder VM funded 600 sender accounts for each of ten e2-standard-2 load generators, one per validator in that validator's zone and sending only to it, and the generators then started together.
- Each generator ran 120 s of warm-up, 2,400 s of mixed load across 13 transaction types and 180 s of cool-down.
- The chaos schedule was timed to the chain's epochs. Before the load generators were created, the harness read the ConsensusRegistry at a pinned block to locate the epoch boundaries and chose the first epoch close at least 600 s into the planned mixed phase, 02:10:37 UTC. It set validator-05's container to be killed with SIGKILL 59 s before that close and held down, the node to be restarted 300 s after the kill, and the observer's node to be restarted 300 s after that. The run had one blast and no injected network delay.
- Afterwards the harness collected each generator's results and the on-node sampler files, merged the generator results with aggregate.py, derived the sizing figures with derive-sizing.py, and pulled 60-second machine metrics from Cloud Monitoring and 15-second node metrics from Prometheus.
- Teardown began at 02:54 UTC. It saved every node's logs, sampler files and service journal, then deleted the fleet, which was gone by 03:06 UTC. The publishing pipeline then read each node's logged commit, consensus delays, vote timeout and fork schedule back from those logs.
The harness scripts live in a private repository; the steps above describe what they do.
Results
| Metric | Value | Window | Source | Method |
|---|---|---|---|---|
| Average chain throughput, all load generators running | 3,319.6 tx/s | all generators active | combined.series.per_second[chain_tx] | Chain transactions per block-timestamp second, summed over the 1807-s window when every load generator was in its mixed phase and divided by its length. |
| Best 60 seconds, all load generators running | 5,214.4 tx/s | all generators active | combined.series.per_second[chain_tx] | Highest mean chain tx/s over any 60 consecutive seconds inside the all-generators window (aggregate.py best_window_mean). |
| Transactions landed over the whole blast | 10,763,764 tx | whole run | combined.totals.landed_chain | Sum of tx_count over every block the load generators saw during the blast, own and foreign transactions alike. |
| Blocks per second | 10.75 blocks/s | merged mixed window | combined.blocks.rows[timestamp] | Chain blocks whose timestamp falls in the merged mixed window, divided by its length (aggregate.py landed.capacity.blocks_per_sec over the same window). |
| Time to land, median | 3.17 s | whole run | combined.latency.ts_ms.p50 | Block timestamp minus submit time over every landed own transaction in the blast, from the load generators' merged latency histograms (aggregate.py merge_latency), in seconds. |
| Time to land, p95 | 26.07 s | whole run | combined.latency.ts_ms.p95 | Block timestamp minus submit time over every landed own transaction in the blast, from the load generators' merged latency histograms (aggregate.py merge_latency), in seconds. |
| Commit latency, p95 | 4.77–4.88 s | merged mixed window | metrics.prometheus.series.tn_primary_commit_latency_seconds.by_node[healthy validators].p95 | For each healthy validator, the nearest-rank p95 over the merged mixed window of its 15-s histogram_quantile(0.95) commit-latency series; reported as the range over validators. |
| Execution time per consensus output | 1.76–2.01 s | merged mixed window | metrics.prometheus.series.tn_engine_execution_duration_seconds.by_node[healthy validators].mean | For each healthy validator, the nearest-rank median over the merged mixed window of its 15-s mean execution-duration series; reported as the range over validators. |
| Write throughput, p95, busiest validator | 529.4 MiB/s | merged mixed window | metrics.gcp[bench-validator-10].disk_write_bytes_per_sec p95 | Nearest-rank p95 of the 60-s mean write-byte rate over the merged mixed window for each healthy validator; the highest (bench-validator-10) is reported. |
| Write operations, p95, busiest validator | 5,253 IOPS | merged mixed window | metrics.gcp[bench-validator-03].disk_write_ops_per_sec p95 | Nearest-rank p95 of the 60-s mean write-operation rate over the merged mixed window for each healthy validator; the highest (bench-validator-03) is reported. |
| CPU, p95, busiest validator | 28.2 % | merged mixed window | metrics.gcp[bench-validator-03].cpu_utilization p95 | Nearest-rank p95 of the 60-s CPU utilisation over the merged mixed window for each healthy validator; the highest (bench-validator-03) is reported. |
| Node resident memory, p95, busiest validator | 9.43 GiB | merged mixed window | metrics.gcp[bench-validator-03].telcoin_rss_bytes p95 | Nearest-rank p95 of the 60-s telcoin process resident memory over the merged mixed window for each healthy validator; the highest (bench-validator-03) is reported. |
| IO pressure, highest reading, range over validators | 26.8–33.6 % | merged mixed window | sampler[healthy validators].psi_io_some_avg10 max | For each healthy validator, the highest 10-s 'some' IO pressure reading the on-node sampler logged inside the merged mixed window; reported as the range over validators. |
Recovery
| Node | bench-validator-05 |
|---|---|
| Killed | 2026-10-06 02:09:55 UTC |
| Restarted | 2026-10-06 02:15:00 UTC |
| Rejoined | 2026-10-06 02:15:28 UTC |
| Catch-up time | none observed |
| Failed batch reports | 0 |
Chain TPS per 5-minute interval
Each bar is the average chain TPS over one 5-minute interval of the merged mixed window (01:50–02:40 UTC); fainter bars had fewer generators active than the maximum, the dashed line is the average of all bars and amber lines mark the chaos events listed below.
- C1 2026-10-06 02:09:55 UTC: docker kill -s KILL telcoin on bench-validator-05 (restart across the epoch boundary; systemctl stop holds it down (unit has Restart=on-failure). bench-blaster-05 targets ONLY this node, so its sends fail for the outage by design; the generator resends on connection errors)
- C2 2026-10-06 02:15:00 UTC: systemctl start telcoin on bench-validator-05 (docker start is impossible: the unit runs docker run --rm)
- C3 2026-10-06 02:20:04 UTC: systemctl restart telcoin on bench-observer-11 (observer restart)
5-minute data (10 rows)
| start_utc | end_utc | generators_active | chain_tps |
|---|---|---|---|
| 2026-10-06T01:50:16Z | 2026-10-06T01:55:16Z | 2 | 2,935.0 |
| 2026-10-06T01:55:16Z | 2026-10-06T02:00:16Z | 2 | 4,573.5 |
| 2026-10-06T02:00:16Z | 2026-10-06T02:05:16Z | 10 | 4,299.4 |
| 2026-10-06T02:05:16Z | 2026-10-06T02:10:16Z | 10 | 3,943.9 |
| 2026-10-06T02:10:16Z | 2026-10-06T02:15:16Z | 10 | 0.0 |
| 2026-10-06T02:15:16Z | 2026-10-06T02:20:16Z | 10 | 3,968.7 |
| 2026-10-06T02:20:16Z | 2026-10-06T02:25:16Z | 10 | 3,754.5 |
| 2026-10-06T02:25:16Z | 2026-10-06T02:30:16Z | 10 | 3,868.8 |
| 2026-10-06T02:30:16Z | 2026-10-06T02:35:16Z | 8 | 2,941.9 |
| 2026-10-06T02:35:16Z | 2026-10-06T02:40:09Z | 8 | 3,000.2 |
The fine-grained chain throughput series (its step_s field gives the seconds per point) is published as series/tps-1s.json.
Other networks
| Measure | Telcoin Network (this run) | Tempo |
|---|---|---|
| Throughput | 3,319.6 tx/s (Avg chain TPS) | 16,455 TPS, shown as "Settled TPS" for the nightly run of 5 Oct 2026 (commit de5f5d3). perf.tempo.xyz describes it as "Median settled throughput across 130 measured blocks." |
| How throughput is measured | Chain transactions per block-timestamp second, summed over the 1807-s window when every load generator was in its mixed phase and divided by its length. | The methodology page defines TPS as "Transactions per second (derived from gas / avg tx gas)": gas used divided by the average gas of one transaction, reported as the median over the measured blocks. |
| Run length | 49.9 min merged mixed window; each generator was set to a 40 min mixed phase | 2m 39s for the 5 Oct 2026 nightly run (130 measured blocks at a 687 ms average block time), per perf.tempo.xyz. |
| Validators and placement | 10 validators and 1 observer across 5 regions: australia-southeast1, europe-west2, europe-west4, northamerica-northeast1, us-west2; load generators: one blaster per validator, in that validator's zone, sending only to it | perf.tempo.xyz labels the scenario "Public mix · multi-region". The methodology page says "Benchmarks run two local validators in consensus on a dedicated bare-metal server with 100 GiB pre-bloated state." The public tempoxyz/tempo repo has nightly workflows for both kinds of setup: bench-e2e runs on one bare-metal runner, and bench-e2e-multi-region defaults to 10 validators on cloud VMs (its regions input reads "Empty uses the selected cloud's five defaults."). The run page does not say which setup produced this number. |
| Hardware | c3-highcpu-8, 8 vCPUs, 16 GB memory; disk: hyperdisk-balanced, 100 GB, 20,000 IOPS, 800 MiB/s | The methodology page lists a dedicated bare-metal server: "AMD EPYC 4585PX (16C/32T, 4.3/5.7 GHz)", "128 GB 3600 MHz" memory and "2×960 GB NVMe + 2×1.92 TB NVMe via schelk". The repo's multi-region workflow uses cloud VMs instead (c4d-standard-48-lssd on GCP, c6id.8xlarge on AWS). |
| Workload | batched_self_send, contract_deploy, faucet_drip, native_transfer, stablecoin_approve, stablecoin_transfer, uniswap_v2_add_liquidity, uniswap_v2_remove_liquidity, uniswap_v2_swap, uniswap_v3_swap, uniswap_v4_swap, wtel_unwrap, wtel_wrap; 10 generators × 600 senders each | perf.tempo.xyz labels the scenario "Public mix"; the methodology page says "100% TIP-20 token transfers (ERC-20 equivalent)". The run used 1,000 accounts and 4 TIP-20 tokens. |
| State | Fresh chain from genesis for every run | The methodology page says "100 GiB pre-bloated state". |
| Latency (as each source defines it) | Time to land, median: 3.17 s; Time to land, p95: 26.07 s; Commit latency, p95: 4.77–4.88 s | Block time of 687 ms, which the methodology page defines as "Average wall-clock time between blocks". The run page gives no per-transaction latency. Separately, tempo.xyz says "Blocks finalize in ~0.6 seconds with no re-orgs." (a network claim, not a figure from this benchmark). |
| Source and fetch date | This page; every figure names its source field. | perf.tempo.xyz nightly run of 5 Oct 2026 (commit de5f5d3), the perf.tempo.xyz methodology page, tempo.xyz, and the benchmark workflows in the tempoxyz/tempo repo. All fetched on 5 Oct 2026. |
Sources for Tempo (fetched 2026-10-05):
- perf.tempo.xyz (nightly benchmark results)
- perf.tempo.xyz methodology
- tempo.xyz
- tempoxyz/tempo repository
- bench-e2e workflow (bare-metal)
- bench-e2e-multi-region workflow
Tempo's setup may not match ours: its methodology page describes two validators on one bare-metal server, while our runs use ten validators across five regions, so read this table as context, not a ranking.
This table is context, not a ranking.
Caveats
- The all-generators window (02:00:09 to 02:30:16 UTC, 1,807 s) contains the 301-second stall. From 02:10:17 all nine running validators stayed at consensus round 1787 of epoch 2 and re-proposed their headers every 500 ms. Six of them had their round-1787 headers certified; validators 03, 04 and 10 logged repeated failures to collect votes for theirs. Headers in this run named at least seven parent certificates from the round before, and the logged quorum was seven of ten. The round moved on at 02:15:17 UTC, less than a second after validator-05's own round-1787 header was certified. Our working hypothesis is that the next round needed seven certificates and only six existed until validator-05 returned. The logs do not show why three of the nine running validators could not collect votes, so the cause of the stall is not established.
- Epoch 2 was due to close at 02:10:37 UTC. The nodes detected that boundary only at the first commit after the stall, at 02:15:17, and epoch 3 started at 02:15:22, so every later epoch boundary moved by about 4 minutes 40 seconds (the epoch-3 boundary fell at 02:35:16 instead of 02:30:37). Apart from the block that closed epoch 2, at 02:15:16, no block was produced from 02:10:15 until 02:15:36.
- Eight of the ten load generators, all but those for validators 05 and 06, stopped waiting for their setup transactions at the 600-second preamble limit with between 24 and 600 of their 600 senders still missing a setup step, and began their mixed phase about ten minutes late, at about 02:00 UTC. Generators 05 and 06 saw every setup transaction land within 8 s and ran from 01:50:16 to 02:30:16. The merged mixed window, 01:50:16 to 02:40:09 UTC (2,993 s), therefore has about ten minutes at each end with only two or eight generators running; it averages 3,329 tx/s. On 29 September three generators started late.
- The offered load was uneven. Generators 03 and 04 sent about 120,000 transactions each over the blast, against about 850,000 to 2.5 million for the other eight. Generator 04 had no sender whose setup completed and sent no stablecoin transfers, Uniswap transactions or WTEL unwraps. Their transactions also waited longest: median time to land 108.5 s for generator 03 and 76.9 s for generator 04 (95th percentile 577.7 s and 663.0 s), against 6 to 16 s at the 95th percentile for the other seven generators; these waits are part of the whole-blast p95 of 26.1 s.
- Validator-05 never reported catch-up mode. Its log shows it recovered its storage, booted directly as an active voting validator at 02:15:15 UTC and proposed its first header at 02:15:16, 16 s after the restart. The rejoin time in the recovery table, 02:15:28, is the first 15-second Prometheus sample after the restart, and the catch-up time is empty because no demotion was recorded.
- The schedule set the kill for 02:09:38 UTC and the restart for 02:14:38. The kill ran at 02:09:55, 42 s before the boundary rather than 59 s, the restart at 02:15:00 (305 s later) and the observer restart at 02:20:04. The observer was following the chain again by 02:20:09, and chain throughput in the two minutes after its restart, 3,778 tx/s, was in line with the two minutes before, 3,695 tx/s.
- The kill was timed differently in the two runs. On 29 September validator-05 was killed 15 minutes into the mixed phase, wherever that fell in the epoch, and the other nine validators kept the chain running; here the kill landed 42 s before an epoch was due to close and the chain stopped until the node returned. Recovery figures from the two runs are not like for like, and one run cannot say whether the stall comes from the kill's timing within the epoch, from the new build or delays, or from something else.
- The comparison run, r20260929-1647, used the devnet consensus timing (3 s maximum header delay, 1 s minimum header delay, 1 s maximum batch delay, read from its node logs; its page publishes no consensus block) and the image built from main at e85bd7a3f. Both the node build and the consensus timing changed between the runs, so no difference in throughput or latency can be assigned to either one alone.
- The node build is commit 964087e02, the head of the integration branch devnet-bench-10-5-2026, made for this benchmark, that merges seven fix and feature branches. It is not a tagged release, and the effect of any one of those branches cannot be separated from the rest.
- The published vote timeout, 5,000 ms, is the node's built-in default, since the parameters file does not set one; this build logs it at startup. The 29 September build did not log a vote timeout, so whether it differed between the two runs is not recorded.
- This is the first run of this image on this fleet, and each configuration has been run once, so run-to-run variation is not measured and the stall has been seen once.
- The tag adiri:mmap-subsecond-10-5-2026 is mutable. The harness resolved it to digest sha256:61fe433a… before any node was created, and every node ran that digest.
- Per-validator rows cover 9 validators; bench-validator-05 excluded as chaos target(s).
- Recovery: demoted / rejoined are the first 15-s Prometheus samples showing the change, so they trail the node's own log times by up to a scrape interval plus one step.
- Recovery: failed_reports integrates the 1-m rate at the 15-s step, which misses the increment that creates the counter series and so can read a few low.
Full report
Changes from the original report:
- The site adds a banner at the top of the report that links back to this page.
SHA-256 of the committed file, before the banner is added: 06f2cd79905e8b8addf24f8f60864d03f9118cbc3429746a0f3280b66f5af563