Hyperdisk Balanced on the c3 fleet

2026-09-29 · started 16:47 UTC · ended 18:56 UTC · merged mixed window 49 min 50 s

telcoin-network e85bd7a3f · main · commit verified from node startup logs

35 node starts across 11 instances logged telcoin-network 0.1.0 (e85bd7a3f07f510c548cac9d783fb907fdf7fcca) Node logs: fork schedule: seed_signature, multi_workers, prevrandao, and leader_seeded_ordering active from genesis Commit e85bd7a3f is "fix(network): refresh published node records (#1371)" on main.

We re-ran the 23 September ten-validator benchmark on the same c3-highcpu-8 machines and changed one thing: the boot disks went from pd-ssd to Hyperdisk Balanced provisioned at 20,000 IOPS and 800 MiB/s. Against the 23 September c3 run on pd-ssd (fleet B, r20260923-0515), average chain throughput with all ten load generators running rose 63%, from 2,558 to 4,177 tx/s, and the median time for a transaction to land over the whole blast fell from 7.7 s to 4.0 s. A validator killed and restarted mid-run then spent about 25 minutes in catch-up mode, sealing the same batch about once a second without being able to report it, and rejoined only as the load eased.

Key figures

4,177.2tx/s
Average chain throughput, all load generators running
all generators active
5,242.8tx/s
Best 60 seconds, all load generators running
all generators active
14.36s
Time to land, p95
whole run
4.75–4.87s
Commit latency, p95
merged mixed window

How it was run

Software

Upstream committelcoin-network e85bd7a3f07f510c548cac9d783fb907fdf7fcca · main · commit verified from node startup logs
Node imageadiri:v0.1.0-devnet
Node image digestsha256:f6fcef5db5f14fec4011a7d86ad8c85d01d8efc26635b597531fa16491e093d2
Node version0.1.0
Fork schedulepre-subsecond
Load generator commit3644adc
Load generator image digestsha256:811ab3a78637b436d69868ab10bdfeab5e50780bd5906d4643f1acc9de1425ca
Analysis commit1046acb
Harness commitnot recorded
Harness submodulesnot recorded

Fleet

Fleetc3-highcpu-8 + Hyperdisk Balanced (c3-hdb)
Validators10
Observers1
Machine typec3-highcpu-8
vCPUs per node8
Memory per node16 GB
Diskhyperdisk-balanced, 100 GB, 20,000 IOPS, 800 MiB/s
Regionsaustralia-southeast1, europe-west2, europe-west4, northamerica-northeast1, us-west2 (5)
Load generators10 generators × e2-standard-2 × 600 senders each
Load placementone blaster per validator, in that validator's zone, sending only to it

Chain

Chain ID32286
Networkbench
Epoch length1,200 s (20 min)

Workload

Warm-up120 s (2 min)
Mixed phase2,400 s (40 min)
Cool-down180 s (3 min)
Transaction typesbatched_self_send, contract_deploy, faucet_drip, native_transfer, stablecoin_approve, stablecoin_transfer, uniswap_v2_add_liquidity, uniswap_v2_remove_liquidity, uniswap_v2_swap, uniswap_v3_swap, uniswap_v4_swap, wtel_unwrap, wtel_wrap

Chaos events

MarkerTime (UTC)NodeActionDetail
C12026-09-29 18:12:05bench-validator-05docker kill -s KILL telcoin5 min outage then restart (replay worst case); systemctl stop holds it down (unit has Restart=on-failure). bench-blaster-05 targets ONLY this node, so its sends fail for the outage by design; the generator resends on connection errors
C22026-09-29 18:17:01bench-validator-05systemctl start telcoindocker start is impossible: the unit runs docker run --rm
C32026-09-29 18:22:10bench-observer-11systemctl restart telcoinobserver restart

Steps

  1. Preflight checked access, tools, quotas and a clean genesis directory, confirmed the load-generator image built from tn-transaction-generator commit 3644adc, and pinned the node image by digest.
  2. The harness created ten validators and one observer on c3-highcpu-8 (8 vCPU, 16 GB), each with a 100 GB Hyperdisk Balanced boot disk provisioned at 20,000 IOPS and 800 MiB/s, two validators in each of five regions, and opened the consensus ports between the nodes only.
  3. It generated validator keys and built genesis for a separate benchmark chain (chain id 32286, 20-minute epochs) with the devnet default consensus parameters: maximum header delay 3 s, minimum header delay 1 s and maximum batch delay 1 s.
  4. It finalized each node's configuration and started the chain.
  5. It relabelled each node's metrics agent so that every sample from these nodes is recorded under the benchmark network, never the live devnet.
  6. It restarted the nodes one at a time with RPC open on the internal network, waiting after each until the node answered with the benchmark chain id and consensus rounds were advancing, then started an on-node sampler and checked that all eleven nodes reported and advanced.
  7. It deployed the devnet contract set that the mixed workload calls.
  8. A one-off funder VM funded 600 sender accounts for each of ten e2-standard-2 load generators, one per validator in that validator's zone and sending only to it, and the generators then started together.
  9. Each generator ran 120 s of warm-up, 2,400 s of mixed load across 13 transaction types and 180 s of cool-down.
  10. A fixed schedule, timed from the planned start of the mixed phase, killed validator-05's container with SIGKILL after 15 minutes and held it down, restarted it about five minutes later, and restarted the observer five minutes after that.
  11. Afterwards the harness collected each generator's results and the on-node sampler files, merged the generator results with aggregate.py, derived the sizing figures with derive-sizing.py, and pulled 60-second machine metrics from Cloud Monitoring and 15-second node metrics from Prometheus.
  12. A second blast with a separate container-pause test followed (not published here); after 35 idle minutes every node's logs were saved and the fleet was torn down at 20:50 UTC.

The harness scripts live in a private repository; the steps above describe what they do.

Results

MetricValueWindowSourceMethod
Average chain throughput, all load generators running4,177.2 tx/sall generators activecombined.series.per_second[chain_tx]Chain transactions per block-timestamp second, summed over the 1810-s window when every load generator was in its mixed phase and divided by its length.
Best 60 seconds, all load generators running5,242.8 tx/sall generators activecombined.series.per_second[chain_tx]Highest mean chain tx/s over any 60 consecutive seconds inside the all-generators window (aggregate.py best_window_mean).
Transactions landed over the whole blast13,772,022 txwhole runcombined.totals.landed_chainSum of tx_count over every block the load generators saw during the blast, own and foreign transactions alike.
Blocks per second12.74 blocks/smerged mixed windowcombined.blocks.rows[timestamp]Chain blocks whose timestamp falls in the merged mixed window, divided by its length (aggregate.py landed.capacity.blocks_per_sec over the same window).
Time to land, median4.04 swhole runcombined.latency.ts_ms.p50Block timestamp minus submit time over every landed own transaction in the blast, from the load generators' merged latency histograms (aggregate.py merge_latency), in seconds.
Time to land, p9514.36 swhole runcombined.latency.ts_ms.p95Block timestamp minus submit time over every landed own transaction in the blast, from the load generators' merged latency histograms (aggregate.py merge_latency), in seconds.
Commit latency, p954.75–4.87 smerged mixed windowmetrics.prometheus.series.tn_primary_commit_latency_seconds.by_node[healthy validators].p95For each healthy validator, the nearest-rank p95 over the merged mixed window of its 15-s histogram_quantile(0.95) commit-latency series; reported as the range over validators.
Execution time per consensus output2.41–2.63 smerged mixed windowmetrics.prometheus.series.tn_engine_execution_duration_seconds.by_node[healthy validators].meanFor each healthy validator, the nearest-rank median over the merged mixed window of its 15-s mean execution-duration series; reported as the range over validators.
Write throughput, p95, busiest validator471.1 MiB/smerged mixed windowmetrics.gcp[bench-validator-09].disk_write_bytes_per_sec p95Nearest-rank p95 of the 60-s mean write-byte rate over the merged mixed window for each healthy validator; the highest (bench-validator-09) is reported.
Write operations, p95, busiest validator7,710 IOPSmerged mixed windowmetrics.gcp[bench-validator-01].disk_write_ops_per_sec p95Nearest-rank p95 of the 60-s mean write-operation rate over the merged mixed window for each healthy validator; the highest (bench-validator-01) is reported.
CPU, p95, busiest validator32.8 %merged mixed windowmetrics.gcp[bench-validator-03].cpu_utilization p95Nearest-rank p95 of the 60-s CPU utilisation over the merged mixed window for each healthy validator; the highest (bench-validator-03) is reported.
Node resident memory, p95, busiest validator9.79 GiBmerged mixed windowmetrics.gcp[bench-validator-01].telcoin_rss_bytes p95Nearest-rank p95 of the 60-s telcoin process resident memory over the merged mixed window for each healthy validator; the highest (bench-validator-01) is reported.
IO pressure, highest reading, range over validators26.1–30.3 %merged mixed windowsampler[healthy validators].psi_io_some_avg10 maxFor each healthy validator, the highest 10-s 'some' IO pressure reading the on-node sampler logged inside the merged mixed window; reported as the range over validators.
Failed batch reports, validator-05 after its restart (exact counter) (published figure)1,207 reports—published report (2026-09-29), exact collector counterIncrease of validator-05's tn_batch_builder_seal_failures_total{reason="failed_to_report"} counter as the collector recorded it, not integrated from 15-s rate samples (the recovery table's 1,203 is the sampled figure).
Anonymous memory per validator at the start of the idle period (19:53 UTC) (published figure)7.78–9.20 GBidle after the second blast, 19:53–20:28 UTCpublished report (2026-09-29), 'Memory is not returned after load stops': GB, read from each node's kernelRange over the ten validators of the node's anonymous memory at the start of the 35-minute idle period.
Anonymous memory per validator after 35 idle minutes (20:28 UTC) (published figure)7.79–9.21 GBidle after the second blast, 19:53–20:28 UTCpublished report (2026-09-29), 'Memory is not returned after load stops': GB, read from each node's kernelRange over the ten validators of the node's anonymous memory at the end of the idle period, with every pool empty, nothing executing and two epochs closed.
Validators whose anonymous memory did not move while idle (published figure)9 of 10 validatorsidle after the second blast, 19:53–20:28 UTCpublished report (2026-09-29), 'Memory is not returned after load stops': Anonymous memory on nine of ten validators did not moveNine validators changed by at most 0.02 GB between 19:53 and 20:28 UTC; validator-04 fell 0.47 GB (9.16 to 8.69 GB).
File-backed memory per validator while idle (published figure)3.38–5.78 GBidle after the second blast, 19:53–20:28 UTCpublished report (2026-09-29), 'Memory is not returned after load stops': GB, read from each node's kernelRange over the ten validators. File-backed memory is mapped database pages, which the kernel reclaims on demand.
Peak resident memory per validator (published figure)13.11–14.94 GBidle after the second blast, 19:53–20:28 UTCpublished report (2026-09-29), 'Memory is not returned after load stops': GB, read from each node's kernelRange over the ten validators of the node's peak resident memory as the report's idle-memory table gives it; the machines have 16 GB.
Memory the kernel reported available while idle (published figure)6.4–7.8 GBidle after the second blast, 19:53–20:28 UTCpublished report (2026-09-29), 'Memory is not returned after load stops': Machines have 16 GB; the kernel reported 6.4 to 7.8 GB available and no memory pressure—

Recovery

Nodebench-validator-05
Killed2026-09-29 18:12:05 UTC
Restarted2026-09-29 18:17:01 UTC
Rejoined2026-09-29 18:42:23 UTC
Catch-up time1,500 s (25 min)
Failed batch reports1,203

Chain TPS per 5-minute interval

Chain TPS per 5-minute interval 10 intervals from 17:55 to 18:45 UTC; average 4,204 TPS, highest 6,008 TPS. The merged mixed window ran from 17:55 to 18:45 UTC. TPS 0 2,000 4,000 6,000 8,000 17:55–18:00 UTC: 6,008 TPS, 6.8 generators active 18:00–18:05 UTC: 5,531 TPS, 7.1 generators active 18:05–18:10 UTC: 4,549 TPS, 10.0 generators active 18:10–18:15 UTC: 4,368 TPS, 10.0 generators active 18:15–18:20 UTC: 4,329 TPS, 10.0 generators active 18:20–18:25 UTC: 4,029 TPS, 10.0 generators active 18:25–18:30 UTC: 3,822 TPS, 10.0 generators active 18:30–18:35 UTC: 3,936 TPS, 10.0 generators active 18:35–18:40 UTC: 2,580 TPS, 3.2 generators active 18:40–18:45 UTC: 2,888 TPS, 3.0 generators active avg 4,204 17:55 UTC 18:05 UTC 18:15 UTC 18:25 UTC 18:35 UTC C1 2026-09-29 18:12:05 UTC: docker kill -s KILL telcoin on bench-validator-05 (5 min outage then restart (replay worst case); systemctl stop holds it down (unit has Restart=on-failure). bench-blaster-05 targets ONLY this node, so its sends fail for the outage by design; the generator resends on connection errors) C1 C2 2026-09-29 18:17:01 UTC: systemctl start telcoin on bench-validator-05 (docker start is impossible: the unit runs docker run --rm) C2 C3 2026-09-29 18:22:10 UTC: systemctl restart telcoin on bench-observer-11 (observer restart) C3

Each bar is the average chain TPS over one 5-minute interval of the merged mixed window (17:55–18:45 UTC); fainter bars had fewer generators active than the maximum, the dashed line is the average of all bars and amber lines mark the chaos events listed below.

  • C1 2026-09-29 18:12:05 UTC: docker kill -s KILL telcoin on bench-validator-05 (5 min outage then restart (replay worst case); systemctl stop holds it down (unit has Restart=on-failure). bench-blaster-05 targets ONLY this node, so its sends fail for the outage by design; the generator resends on connection errors)
  • C2 2026-09-29 18:17:01 UTC: systemctl start telcoin on bench-validator-05 (docker start is impossible: the unit runs docker run --rm)
  • C3 2026-09-29 18:22:10 UTC: systemctl restart telcoin on bench-observer-11 (observer restart)
5-minute data (10 rows)
start_utcend_utcgenerators_activechain_tps
2026-09-29T17:55:40Z2026-09-29T18:00:40Z6.86,008.4
2026-09-29T18:00:40Z2026-09-29T18:05:40Z7.15,530.7
2026-09-29T18:05:40Z2026-09-29T18:10:40Z10.04,549.3
2026-09-29T18:10:40Z2026-09-29T18:15:40Z10.04,368.1
2026-09-29T18:15:40Z2026-09-29T18:20:40Z10.04,329.3
2026-09-29T18:20:40Z2026-09-29T18:25:40Z10.04,029.3
2026-09-29T18:25:40Z2026-09-29T18:30:40Z10.03,821.5
2026-09-29T18:30:40Z2026-09-29T18:35:40Z10.03,935.7
2026-09-29T18:35:40Z2026-09-29T18:40:40Z3.22,579.9
2026-09-29T18:40:40Z2026-09-29T18:45:30Z3.02,887.5

The fine-grained chain throughput series (its step_s field gives the seconds per point) is published as series/tps-1s.json.

Other networks

MeasureTelcoin Network (this run)Tempo
Throughput4,177.2 tx/s (Avg chain TPS)16,455 TPS, shown as "Settled TPS" for the nightly run of 5 Oct 2026 (commit de5f5d3). perf.tempo.xyz describes it as "Median settled throughput across 130 measured blocks."
How throughput is measuredChain transactions per block-timestamp second, summed over the 1810-s window when every load generator was in its mixed phase and divided by its length.The methodology page defines TPS as "Transactions per second (derived from gas / avg tx gas)": gas used divided by the average gas of one transaction, reported as the median over the measured blocks.
Run length49.8 min merged mixed window; each generator was set to a 40 min mixed phase2m 39s for the 5 Oct 2026 nightly run (130 measured blocks at a 687 ms average block time), per perf.tempo.xyz.
Validators and placement10 validators and 1 observer across 5 regions: australia-southeast1, europe-west2, europe-west4, northamerica-northeast1, us-west2; load generators: one blaster per validator, in that validator's zone, sending only to itperf.tempo.xyz labels the scenario "Public mix · multi-region". The methodology page says "Benchmarks run two local validators in consensus on a dedicated bare-metal server with 100 GiB pre-bloated state." The public tempoxyz/tempo repo has nightly workflows for both kinds of setup: bench-e2e runs on one bare-metal runner, and bench-e2e-multi-region defaults to 10 validators on cloud VMs (its regions input reads "Empty uses the selected cloud's five defaults."). The run page does not say which setup produced this number.
Hardwarec3-highcpu-8, 8 vCPUs, 16 GB memory; disk: hyperdisk-balanced, 100 GB, 20,000 IOPS, 800 MiB/sThe methodology page lists a dedicated bare-metal server: "AMD EPYC 4585PX (16C/32T, 4.3/5.7 GHz)", "128 GB 3600 MHz" memory and "2×960 GB NVMe + 2×1.92 TB NVMe via schelk". The repo's multi-region workflow uses cloud VMs instead (c4d-standard-48-lssd on GCP, c6id.8xlarge on AWS).
Workloadbatched_self_send, contract_deploy, faucet_drip, native_transfer, stablecoin_approve, stablecoin_transfer, uniswap_v2_add_liquidity, uniswap_v2_remove_liquidity, uniswap_v2_swap, uniswap_v3_swap, uniswap_v4_swap, wtel_unwrap, wtel_wrap; 10 generators × 600 senders eachperf.tempo.xyz labels the scenario "Public mix"; the methodology page says "100% TIP-20 token transfers (ERC-20 equivalent)". The run used 1,000 accounts and 4 TIP-20 tokens.
StateFresh chain from genesis for every runThe methodology page says "100 GiB pre-bloated state".
Latency (as each source defines it)Time to land, median: 4.04 s; Time to land, p95: 14.36 s; Commit latency, p95: 4.75–4.87 sBlock time of 687 ms, which the methodology page defines as "Average wall-clock time between blocks". The run page gives no per-transaction latency. Separately, tempo.xyz says "Blocks finalize in ~0.6 seconds with no re-orgs." (a network claim, not a figure from this benchmark).
Source and fetch dateThis page; every figure names its source field.perf.tempo.xyz nightly run of 5 Oct 2026 (commit de5f5d3), the perf.tempo.xyz methodology page, tempo.xyz, and the benchmark workflows in the tempoxyz/tempo repo. All fetched on 5 Oct 2026.

Sources for Tempo (fetched 2026-10-05):

Tempo's setup may not match ours: its methodology page describes two validators on one bare-metal server, while our runs use ten validators across five regions, so read this table as context, not a ranking.

This table is context, not a ranking.

Caveats

  • Throughput figures marked 'all generators active' cover 18:05:30 to 18:35:40 UTC (1,810 s), when all ten load generators were in their mixed phase. Three of the ten generators (those for validators 01, 02 and 08) started their mixed phase about ten minutes late; in the fleet B baseline on 23 September only one generator did. The longer merged mixed window, 17:55:40 to 18:45:30 UTC, therefore includes stretches with fewer generators running; it averages about 4,200 tx/s. The original report compares the all-generators figures.
  • The all-generators window (18:05:30 to 18:35:40 UTC) contains bench-validator-05's five-minute outage (killed 18:12:05, restarted 18:17:01) and its catch-up, which lasted past the end of the window, so from 18:12 onward the headline figures come from nine working validators.
  • Recovery figures for validator-05 come from 15-second Prometheus samples: 1,500 s in catch-up mode (rejoined 18:42:23 UTC) and 1,203 failed batch reports. The original report read the validator's own logs and the collector's exact counter instead: demoted 18:17:04, rejoined 18:42:13, 25 min 09 s (1,509 s) in catch-up, and 1,207 failed reports, which is listed with the KPIs as a published figure.
  • Write IOPS p95 is 7,710 here (7,710.48 before rounding). The original report printed 7,711 because it rounded an intermediate value of 7,710.5.
  • The run held a second blast, a three-minute container pause of validator-03, as a separate test. It has not been through the report tooling and is not published here. The idle-memory figures were read from the nodes' kernels in the 35 idle minutes after it and are carried over from the original report as published figures.
  • The nodes ran an image pinned by digest (sha256:f6fcef5d…). The tag v0.1.0-devnet shown with it is mutable; it pointed at this digest from 17 September through this run. The fleet B baseline on 23 September recorded only the tag, so its digest is inferred from the tag's history.
  • The full report's baseline link opens the 23 September page for fleet A (r20260923-0708, e2-custom-4-8192); the run this report compares against, fleet B (r20260923-0515, c3-highcpu-8 on pd-ssd), appears on that page only as its comparison column.
  • This is one run on each disk type. The differences are large against the spread between validators within a run, but that the disk caused them is inferred, not proven.
  • Cloud Monitoring's disk-throttling series was unavailable in both runs, so the write ceiling is read from the peak one-second write rate.
  • Per-validator rows cover 9 validators; bench-validator-05 excluded as chaos target(s).
  • Recovery: demoted / rejoined are the first 15-s Prometheus samples showing the change, so they trail the node's own log times by up to a scrape interval plus one step.
  • Recovery: failed_reports integrates the 1-m rate at the 15-s step, which misses the increment that creates the counter series and so can read a few low.
  • Recovery: over the whole metrics window the batch-seal failure counter on bench-validator-05 rose by 1207 in total (metrics.prometheus.series.tn_batch_builder_seal_failures_total.count), outage or not.
  • Consensus parameters are not published for this run: vote_timeout_ms not recorded.

Full report

Full report HTML · 34 KB

Archived from: Internal report 'The disk was holding the c3 fleet back' (2026-09-29).

Changes from the original report:

  • Wrapped the original fragment in a complete HTML document (doctype, head, charset, viewport).
  • Baseline-report link repointed from an external artifact URL to ../r20260923-0708/report.html.
  • Section 'Where the fixes stand' removed (internal status notes).
  • The site adds a banner at the top of the report that links back to this page.

SHA-256 of the committed file, before the banner is added: 8ad610d7393625c6f848554b8478adef82c366e425fc116c7e2c4a6baa17c511