Hyperdisk Balanced on the c3 fleet
2026-09-29 · started 16:47 UTC · ended 18:56 UTC · merged mixed window 49 min 50 s
We re-ran the 23 September ten-validator benchmark on the same c3-highcpu-8 machines and changed one thing: the boot disks went from pd-ssd to Hyperdisk Balanced provisioned at 20,000 IOPS and 800 MiB/s. Against the 23 September c3 run on pd-ssd (fleet B, r20260923-0515), average chain throughput with all ten load generators running rose 63%, from 2,558 to 4,177 tx/s, and the median time for a transaction to land over the whole blast fell from 7.7 s to 4.0 s. A validator killed and restarted mid-run then spent about 25 minutes in catch-up mode, sealing the same batch about once a second without being able to report it, and rejoined only as the load eased.
Key figures
How it was run
Software
| Upstream commit | telcoin-network e85bd7a3f07f510c548cac9d783fb907fdf7fcca · main · commit verified from node startup logs |
|---|---|
| Node image | adiri:v0.1.0-devnet |
| Node image digest | sha256:f6fcef5db5f14fec4011a7d86ad8c85d01d8efc26635b597531fa16491e093d2 |
| Node version | 0.1.0 |
| Fork schedule | pre-subsecond |
| Load generator commit | 3644adc |
| Load generator image digest | sha256:811ab3a78637b436d69868ab10bdfeab5e50780bd5906d4643f1acc9de1425ca |
| Analysis commit | 1046acb |
| Harness commit | not recorded |
| Harness submodules | not recorded |
Fleet
| Fleet | c3-highcpu-8 + Hyperdisk Balanced (c3-hdb) |
|---|---|
| Validators | 10 |
| Observers | 1 |
| Machine type | c3-highcpu-8 |
| vCPUs per node | 8 |
| Memory per node | 16 GB |
| Disk | hyperdisk-balanced, 100 GB, 20,000 IOPS, 800 MiB/s |
| Regions | australia-southeast1, europe-west2, europe-west4, northamerica-northeast1, us-west2 (5) |
| Load generators | 10 generators × e2-standard-2 × 600 senders each |
| Load placement | one blaster per validator, in that validator's zone, sending only to it |
Chain
| Chain ID | 32286 |
|---|---|
| Network | bench |
| Epoch length | 1,200 s (20 min) |
Workload
| Warm-up | 120 s (2 min) |
|---|---|
| Mixed phase | 2,400 s (40 min) |
| Cool-down | 180 s (3 min) |
| Transaction types | batched_self_send, contract_deploy, faucet_drip, native_transfer, stablecoin_approve, stablecoin_transfer, uniswap_v2_add_liquidity, uniswap_v2_remove_liquidity, uniswap_v2_swap, uniswap_v3_swap, uniswap_v4_swap, wtel_unwrap, wtel_wrap |
Chaos events
| Marker | Time (UTC) | Node | Action | Detail |
|---|---|---|---|---|
| C1 | 2026-09-29 18:12:05 | bench-validator-05 | docker kill -s KILL telcoin | 5 min outage then restart (replay worst case); systemctl stop holds it down (unit has Restart=on-failure). bench-blaster-05 targets ONLY this node, so its sends fail for the outage by design; the generator resends on connection errors |
| C2 | 2026-09-29 18:17:01 | bench-validator-05 | systemctl start telcoin | docker start is impossible: the unit runs docker run --rm |
| C3 | 2026-09-29 18:22:10 | bench-observer-11 | systemctl restart telcoin | observer restart |
Steps
- Preflight checked access, tools, quotas and a clean genesis directory, confirmed the load-generator image built from tn-transaction-generator commit 3644adc, and pinned the node image by digest.
- The harness created ten validators and one observer on c3-highcpu-8 (8 vCPU, 16 GB), each with a 100 GB Hyperdisk Balanced boot disk provisioned at 20,000 IOPS and 800 MiB/s, two validators in each of five regions, and opened the consensus ports between the nodes only.
- It generated validator keys and built genesis for a separate benchmark chain (chain id 32286, 20-minute epochs) with the devnet default consensus parameters: maximum header delay 3 s, minimum header delay 1 s and maximum batch delay 1 s.
- It finalized each node's configuration and started the chain.
- It relabelled each node's metrics agent so that every sample from these nodes is recorded under the benchmark network, never the live devnet.
- It restarted the nodes one at a time with RPC open on the internal network, waiting after each until the node answered with the benchmark chain id and consensus rounds were advancing, then started an on-node sampler and checked that all eleven nodes reported and advanced.
- It deployed the devnet contract set that the mixed workload calls.
- A one-off funder VM funded 600 sender accounts for each of ten e2-standard-2 load generators, one per validator in that validator's zone and sending only to it, and the generators then started together.
- Each generator ran 120 s of warm-up, 2,400 s of mixed load across 13 transaction types and 180 s of cool-down.
- A fixed schedule, timed from the planned start of the mixed phase, killed validator-05's container with SIGKILL after 15 minutes and held it down, restarted it about five minutes later, and restarted the observer five minutes after that.
- Afterwards the harness collected each generator's results and the on-node sampler files, merged the generator results with aggregate.py, derived the sizing figures with derive-sizing.py, and pulled 60-second machine metrics from Cloud Monitoring and 15-second node metrics from Prometheus.
- A second blast with a separate container-pause test followed (not published here); after 35 idle minutes every node's logs were saved and the fleet was torn down at 20:50 UTC.
The harness scripts live in a private repository; the steps above describe what they do.
Results
| Metric | Value | Window | Source | Method |
|---|---|---|---|---|
| Average chain throughput, all load generators running | 4,177.2 tx/s | all generators active | combined.series.per_second[chain_tx] | Chain transactions per block-timestamp second, summed over the 1810-s window when every load generator was in its mixed phase and divided by its length. |
| Best 60 seconds, all load generators running | 5,242.8 tx/s | all generators active | combined.series.per_second[chain_tx] | Highest mean chain tx/s over any 60 consecutive seconds inside the all-generators window (aggregate.py best_window_mean). |
| Transactions landed over the whole blast | 13,772,022 tx | whole run | combined.totals.landed_chain | Sum of tx_count over every block the load generators saw during the blast, own and foreign transactions alike. |
| Blocks per second | 12.74 blocks/s | merged mixed window | combined.blocks.rows[timestamp] | Chain blocks whose timestamp falls in the merged mixed window, divided by its length (aggregate.py landed.capacity.blocks_per_sec over the same window). |
| Time to land, median | 4.04 s | whole run | combined.latency.ts_ms.p50 | Block timestamp minus submit time over every landed own transaction in the blast, from the load generators' merged latency histograms (aggregate.py merge_latency), in seconds. |
| Time to land, p95 | 14.36 s | whole run | combined.latency.ts_ms.p95 | Block timestamp minus submit time over every landed own transaction in the blast, from the load generators' merged latency histograms (aggregate.py merge_latency), in seconds. |
| Commit latency, p95 | 4.75–4.87 s | merged mixed window | metrics.prometheus.series.tn_primary_commit_latency_seconds.by_node[healthy validators].p95 | For each healthy validator, the nearest-rank p95 over the merged mixed window of its 15-s histogram_quantile(0.95) commit-latency series; reported as the range over validators. |
| Execution time per consensus output | 2.41–2.63 s | merged mixed window | metrics.prometheus.series.tn_engine_execution_duration_seconds.by_node[healthy validators].mean | For each healthy validator, the nearest-rank median over the merged mixed window of its 15-s mean execution-duration series; reported as the range over validators. |
| Write throughput, p95, busiest validator | 471.1 MiB/s | merged mixed window | metrics.gcp[bench-validator-09].disk_write_bytes_per_sec p95 | Nearest-rank p95 of the 60-s mean write-byte rate over the merged mixed window for each healthy validator; the highest (bench-validator-09) is reported. |
| Write operations, p95, busiest validator | 7,710 IOPS | merged mixed window | metrics.gcp[bench-validator-01].disk_write_ops_per_sec p95 | Nearest-rank p95 of the 60-s mean write-operation rate over the merged mixed window for each healthy validator; the highest (bench-validator-01) is reported. |
| CPU, p95, busiest validator | 32.8 % | merged mixed window | metrics.gcp[bench-validator-03].cpu_utilization p95 | Nearest-rank p95 of the 60-s CPU utilisation over the merged mixed window for each healthy validator; the highest (bench-validator-03) is reported. |
| Node resident memory, p95, busiest validator | 9.79 GiB | merged mixed window | metrics.gcp[bench-validator-01].telcoin_rss_bytes p95 | Nearest-rank p95 of the 60-s telcoin process resident memory over the merged mixed window for each healthy validator; the highest (bench-validator-01) is reported. |
| IO pressure, highest reading, range over validators | 26.1–30.3 % | merged mixed window | sampler[healthy validators].psi_io_some_avg10 max | For each healthy validator, the highest 10-s 'some' IO pressure reading the on-node sampler logged inside the merged mixed window; reported as the range over validators. |
| Failed batch reports, validator-05 after its restart (exact counter) (published figure) | 1,207 reports | — | published report (2026-09-29), exact collector counter | Increase of validator-05's tn_batch_builder_seal_failures_total{reason="failed_to_report"} counter as the collector recorded it, not integrated from 15-s rate samples (the recovery table's 1,203 is the sampled figure). |
| Anonymous memory per validator at the start of the idle period (19:53 UTC) (published figure) | 7.78–9.20 GB | idle after the second blast, 19:53–20:28 UTC | published report (2026-09-29), 'Memory is not returned after load stops': GB, read from each node's kernel | Range over the ten validators of the node's anonymous memory at the start of the 35-minute idle period. |
| Anonymous memory per validator after 35 idle minutes (20:28 UTC) (published figure) | 7.79–9.21 GB | idle after the second blast, 19:53–20:28 UTC | published report (2026-09-29), 'Memory is not returned after load stops': GB, read from each node's kernel | Range over the ten validators of the node's anonymous memory at the end of the idle period, with every pool empty, nothing executing and two epochs closed. |
| Validators whose anonymous memory did not move while idle (published figure) | 9 of 10 validators | idle after the second blast, 19:53–20:28 UTC | published report (2026-09-29), 'Memory is not returned after load stops': Anonymous memory on nine of ten validators did not move | Nine validators changed by at most 0.02 GB between 19:53 and 20:28 UTC; validator-04 fell 0.47 GB (9.16 to 8.69 GB). |
| File-backed memory per validator while idle (published figure) | 3.38–5.78 GB | idle after the second blast, 19:53–20:28 UTC | published report (2026-09-29), 'Memory is not returned after load stops': GB, read from each node's kernel | Range over the ten validators. File-backed memory is mapped database pages, which the kernel reclaims on demand. |
| Peak resident memory per validator (published figure) | 13.11–14.94 GB | idle after the second blast, 19:53–20:28 UTC | published report (2026-09-29), 'Memory is not returned after load stops': GB, read from each node's kernel | Range over the ten validators of the node's peak resident memory as the report's idle-memory table gives it; the machines have 16 GB. |
| Memory the kernel reported available while idle (published figure) | 6.4–7.8 GB | idle after the second blast, 19:53–20:28 UTC | published report (2026-09-29), 'Memory is not returned after load stops': Machines have 16 GB; the kernel reported 6.4 to 7.8 GB available and no memory pressure | — |
Recovery
| Node | bench-validator-05 |
|---|---|
| Killed | 2026-09-29 18:12:05 UTC |
| Restarted | 2026-09-29 18:17:01 UTC |
| Rejoined | 2026-09-29 18:42:23 UTC |
| Catch-up time | 1,500 s (25 min) |
| Failed batch reports | 1,203 |
Chain TPS per 5-minute interval
Each bar is the average chain TPS over one 5-minute interval of the merged mixed window (17:55–18:45 UTC); fainter bars had fewer generators active than the maximum, the dashed line is the average of all bars and amber lines mark the chaos events listed below.
- C1 2026-09-29 18:12:05 UTC: docker kill -s KILL telcoin on bench-validator-05 (5 min outage then restart (replay worst case); systemctl stop holds it down (unit has Restart=on-failure). bench-blaster-05 targets ONLY this node, so its sends fail for the outage by design; the generator resends on connection errors)
- C2 2026-09-29 18:17:01 UTC: systemctl start telcoin on bench-validator-05 (docker start is impossible: the unit runs docker run --rm)
- C3 2026-09-29 18:22:10 UTC: systemctl restart telcoin on bench-observer-11 (observer restart)
5-minute data (10 rows)
| start_utc | end_utc | generators_active | chain_tps |
|---|---|---|---|
| 2026-09-29T17:55:40Z | 2026-09-29T18:00:40Z | 6.8 | 6,008.4 |
| 2026-09-29T18:00:40Z | 2026-09-29T18:05:40Z | 7.1 | 5,530.7 |
| 2026-09-29T18:05:40Z | 2026-09-29T18:10:40Z | 10.0 | 4,549.3 |
| 2026-09-29T18:10:40Z | 2026-09-29T18:15:40Z | 10.0 | 4,368.1 |
| 2026-09-29T18:15:40Z | 2026-09-29T18:20:40Z | 10.0 | 4,329.3 |
| 2026-09-29T18:20:40Z | 2026-09-29T18:25:40Z | 10.0 | 4,029.3 |
| 2026-09-29T18:25:40Z | 2026-09-29T18:30:40Z | 10.0 | 3,821.5 |
| 2026-09-29T18:30:40Z | 2026-09-29T18:35:40Z | 10.0 | 3,935.7 |
| 2026-09-29T18:35:40Z | 2026-09-29T18:40:40Z | 3.2 | 2,579.9 |
| 2026-09-29T18:40:40Z | 2026-09-29T18:45:30Z | 3.0 | 2,887.5 |
The fine-grained chain throughput series (its step_s field gives the seconds per point) is published as series/tps-1s.json.
Other networks
| Measure | Telcoin Network (this run) | Tempo |
|---|---|---|
| Throughput | 4,177.2 tx/s (Avg chain TPS) | 16,455 TPS, shown as "Settled TPS" for the nightly run of 5 Oct 2026 (commit de5f5d3). perf.tempo.xyz describes it as "Median settled throughput across 130 measured blocks." |
| How throughput is measured | Chain transactions per block-timestamp second, summed over the 1810-s window when every load generator was in its mixed phase and divided by its length. | The methodology page defines TPS as "Transactions per second (derived from gas / avg tx gas)": gas used divided by the average gas of one transaction, reported as the median over the measured blocks. |
| Run length | 49.8 min merged mixed window; each generator was set to a 40 min mixed phase | 2m 39s for the 5 Oct 2026 nightly run (130 measured blocks at a 687 ms average block time), per perf.tempo.xyz. |
| Validators and placement | 10 validators and 1 observer across 5 regions: australia-southeast1, europe-west2, europe-west4, northamerica-northeast1, us-west2; load generators: one blaster per validator, in that validator's zone, sending only to it | perf.tempo.xyz labels the scenario "Public mix · multi-region". The methodology page says "Benchmarks run two local validators in consensus on a dedicated bare-metal server with 100 GiB pre-bloated state." The public tempoxyz/tempo repo has nightly workflows for both kinds of setup: bench-e2e runs on one bare-metal runner, and bench-e2e-multi-region defaults to 10 validators on cloud VMs (its regions input reads "Empty uses the selected cloud's five defaults."). The run page does not say which setup produced this number. |
| Hardware | c3-highcpu-8, 8 vCPUs, 16 GB memory; disk: hyperdisk-balanced, 100 GB, 20,000 IOPS, 800 MiB/s | The methodology page lists a dedicated bare-metal server: "AMD EPYC 4585PX (16C/32T, 4.3/5.7 GHz)", "128 GB 3600 MHz" memory and "2×960 GB NVMe + 2×1.92 TB NVMe via schelk". The repo's multi-region workflow uses cloud VMs instead (c4d-standard-48-lssd on GCP, c6id.8xlarge on AWS). |
| Workload | batched_self_send, contract_deploy, faucet_drip, native_transfer, stablecoin_approve, stablecoin_transfer, uniswap_v2_add_liquidity, uniswap_v2_remove_liquidity, uniswap_v2_swap, uniswap_v3_swap, uniswap_v4_swap, wtel_unwrap, wtel_wrap; 10 generators × 600 senders each | perf.tempo.xyz labels the scenario "Public mix"; the methodology page says "100% TIP-20 token transfers (ERC-20 equivalent)". The run used 1,000 accounts and 4 TIP-20 tokens. |
| State | Fresh chain from genesis for every run | The methodology page says "100 GiB pre-bloated state". |
| Latency (as each source defines it) | Time to land, median: 4.04 s; Time to land, p95: 14.36 s; Commit latency, p95: 4.75–4.87 s | Block time of 687 ms, which the methodology page defines as "Average wall-clock time between blocks". The run page gives no per-transaction latency. Separately, tempo.xyz says "Blocks finalize in ~0.6 seconds with no re-orgs." (a network claim, not a figure from this benchmark). |
| Source and fetch date | This page; every figure names its source field. | perf.tempo.xyz nightly run of 5 Oct 2026 (commit de5f5d3), the perf.tempo.xyz methodology page, tempo.xyz, and the benchmark workflows in the tempoxyz/tempo repo. All fetched on 5 Oct 2026. |
Sources for Tempo (fetched 2026-10-05):
- perf.tempo.xyz (nightly benchmark results)
- perf.tempo.xyz methodology
- tempo.xyz
- tempoxyz/tempo repository
- bench-e2e workflow (bare-metal)
- bench-e2e-multi-region workflow
Tempo's setup may not match ours: its methodology page describes two validators on one bare-metal server, while our runs use ten validators across five regions, so read this table as context, not a ranking.
This table is context, not a ranking.
Caveats
- Throughput figures marked 'all generators active' cover 18:05:30 to 18:35:40 UTC (1,810 s), when all ten load generators were in their mixed phase. Three of the ten generators (those for validators 01, 02 and 08) started their mixed phase about ten minutes late; in the fleet B baseline on 23 September only one generator did. The longer merged mixed window, 17:55:40 to 18:45:30 UTC, therefore includes stretches with fewer generators running; it averages about 4,200 tx/s. The original report compares the all-generators figures.
- The all-generators window (18:05:30 to 18:35:40 UTC) contains bench-validator-05's five-minute outage (killed 18:12:05, restarted 18:17:01) and its catch-up, which lasted past the end of the window, so from 18:12 onward the headline figures come from nine working validators.
- Recovery figures for validator-05 come from 15-second Prometheus samples: 1,500 s in catch-up mode (rejoined 18:42:23 UTC) and 1,203 failed batch reports. The original report read the validator's own logs and the collector's exact counter instead: demoted 18:17:04, rejoined 18:42:13, 25 min 09 s (1,509 s) in catch-up, and 1,207 failed reports, which is listed with the KPIs as a published figure.
- Write IOPS p95 is 7,710 here (7,710.48 before rounding). The original report printed 7,711 because it rounded an intermediate value of 7,710.5.
- The run held a second blast, a three-minute container pause of validator-03, as a separate test. It has not been through the report tooling and is not published here. The idle-memory figures were read from the nodes' kernels in the 35 idle minutes after it and are carried over from the original report as published figures.
- The nodes ran an image pinned by digest (sha256:f6fcef5d…). The tag v0.1.0-devnet shown with it is mutable; it pointed at this digest from 17 September through this run. The fleet B baseline on 23 September recorded only the tag, so its digest is inferred from the tag's history.
- The full report's baseline link opens the 23 September page for fleet A (r20260923-0708, e2-custom-4-8192); the run this report compares against, fleet B (r20260923-0515, c3-highcpu-8 on pd-ssd), appears on that page only as its comparison column.
- This is one run on each disk type. The differences are large against the spread between validators within a run, but that the disk caused them is inferred, not proven.
- Cloud Monitoring's disk-throttling series was unavailable in both runs, so the write ceiling is read from the peak one-second write rate.
- Per-validator rows cover 9 validators; bench-validator-05 excluded as chaos target(s).
- Recovery: demoted / rejoined are the first 15-s Prometheus samples showing the change, so they trail the node's own log times by up to a scrape interval plus one step.
- Recovery: failed_reports integrates the 1-m rate at the 15-s step, which misses the increment that creates the counter series and so can read a few low.
- Recovery: over the whole metrics window the batch-seal failure counter on bench-validator-05 rose by 1207 in total (metrics.prometheus.series.tn_batch_builder_seal_failures_total.count), outage or not.
- Consensus parameters are not published for this run: vote_timeout_ms not recorded.
Full report
Archived from: Internal report 'The disk was holding the c3 fleet back' (2026-09-29).
Changes from the original report:
- Wrapped the original fragment in a complete HTML document (doctype, head, charset, viewport).
- Baseline-report link repointed from an external artifact URL to ../r20260923-0708/report.html.
- Section 'Where the fixes stand' removed (internal status notes).
- The site adds a banner at the top of the report that links back to this page.
SHA-256 of the committed file, before the banner is added: 8ad610d7393625c6f848554b8478adef82c366e425fc116c7e2c4a6baa17c511