This is the archived full report of the run “Hyperdisk Balanced on the c3 fleet”. Back to the run page.
Telcoin Network · ten-validator benchmark

The disk was holding the c3 fleet back

We re-ran the 23 September benchmark on the same machines with faster disks. Average chain throughput rose 63%, and the write ceiling moved from about 288 to 800 MiB/s. The run also reproduced the stuck-validator bug on demand.

Run r20260929-1647 29 September 2026, 16:47 to 20:50 UTC 10 validators and 1 observer on c3-highcpu-8, five regions Same node build and load generator as 23 September
Average chain throughput, all ten load generators running
4,177 tx/s
▲ 63% from 2,558 tx/s on pd-ssd
Write throughput, p95 of the busiest validator
471 MiB/s
was 181 MiB/s
IO pressure, highest reading per validator
26–30%
was 50–60%
Median time for a transaction to land
4.0 s
was 7.7 s

Same fleet, one change: the disk

Both runs used 100 GB boot disks on the same machine type, zones, node build, load generator and fault schedule. On 23 September the disks were pd-ssd. This run used Hyperdisk Balanced provisioned at 20,000 IOPS and 800 MiB/s. The baseline is in the 23 September sizing report.

Chain throughput

Transactions per second landing on chain, in five-minute blocks from the start of load.

Hyperdisk Balanced, 29 Seppd-ssd, 23 Sepall ten load generators running
Table

Disk writes per validator

MiB/s, mean of the nine validators that stayed up. The dashed lines are the highest one-second rate in a typical minute.

Hyperdisk, minute meanHyperdisk, 1-second peakpd-ssd, minute meanpd-ssd, 1-second peak
Table

On pd-ssd the one-second write peaks sat at about 282 MiB/s from the first minute on every node, and the minute average flattened near 170. On Hyperdisk the peaks sit at about 800 to 820, and the average kept climbing through the run. Throughput falls over time in both runs, but the slowest five-minute block on Hyperdisk is 34% above the fastest on pd-ssd.

Measurepd-ssd, 23 SepHyperdisk, 29 SepChange
Average chain throughput, all ten load generators2,558 tx/s4,177 tx/s+63%
Best 60 seconds, all ten load generators3,518 tx/s5,243 tx/s+49%
Transactions landed over the whole blast8,149,27513,772,022+69%
Blocks per second7.4312.74+72%
Time to land, median and p957.7 s, 31.5 s4.0 s, 14.4 sabout half
Commit latency, p9522.0–23.0 s4.75–4.87 s−79%
Execution time per consensus output4.88–5.00 s2.41–2.63 sabout half
Write throughput, p95, busiest validator181 MiB/s471 MiB/s2.6×
Write operations, p95, busiest validator2,440 IOPS7,711 IOPS3.2×
IO pressure, highest reading, range over validators50.3–59.8%26.1–30.3%about half
CPU, p95, busiest validator24.6%32.8%+8 points
Node resident memory, p9510.0 GiB9.8 GiBunchanged

Validators only. The validator that the fault schedule kills and restarts is left out of the disk and CPU rows in both runs. Both result sets were built with the same scripts; re-running them on the 23 September data reproduced its files byte for byte.

What limits the fleet now

The disk is still the closest resource to its limit, but only in bursts. Averaged over a minute nothing is over its line. Second by second, every validator's writes reach the provisioned 800 MiB/s in most minutes.

Resource, busiest validatorReadingShare of limit
Disk throughput, one-second bursts800–810 MiB/s● at the limit
Disk throughput, p95 of minute averages471 MiB/s59% of 800
Memory in use, p9543.7%44% of 16 GB
Disk operations, p957,711 IOPS39% of 20,000
CPU, p9532.8%33% of 8 vCPU
Network out, p95112 Mbps11% of 1 Gbps

IO pressure peaked at 26 to 30%, above the 20% line the sizing report uses, though half of what it was on pd-ssd.

A validator that falls behind keeps sealing a batch it cannot report

When a validator falls behind it drops to catch-up mode. Its worker keeps sealing batches, peers keep voting on them, and the report to its own primary fails every time because nothing is listening. The transactions stay in its pool and the same batch is sealed again about once a second.

This run triggered it twice, once by killing and restarting a validator and once by pausing a container for three minutes. Both validators were out for about 25 minutes and rejoined only as load eased.

Paused for 3 minutes (validator-03, second blast)

Failed batch reports per minute.

failed reportsin catch-up mode
  • A 19:10 paused
  • B 19:13 resumed, demoted
  • C 19:28 epoch change
  • D 19:40 rejoined
Table

Killed and restarted (validator-05, first blast)

Failed batch reports per minute.

failed reportsin catch-up mode
  • A 18:12 killed
  • B 18:17 restarted, demoted
  • C 18:28 epoch change
  • D 18:42 rejoined
Table

The gap after each epoch change is not recovery. For those minutes the validator is still in the old epoch, so peers reject its batch at quorum before it gets as far as the report.

ValidatorTriggerDemotedRejoinedIn catch-upFailed reportsRejected at quorumTransactions stuck
validator-03Container paused 19:10:30 to 19:13:3019:13:3919:40:0826 min 29 s9151369,600
validator-05Killed 18:12:05, restarted 18:17:0118:17:0418:42:1325 min 09 s1,2072459,589

Times and counts are from the validators' own logs, which this run kept in full. All times UTC.

Memory is not returned after load stops

After the second blast the fleet sat idle for 35 minutes. Every pool was empty, nothing was executing, and two epochs closed. Anonymous memory on nine of ten validators did not move.

NodeAnonymous, 19:53Anonymous, 20:28ChangeFile-backedPeak resident

GB, read from each node's kernel. File-backed memory is mapped database pages, which the kernel reclaims on demand. Machines have 16 GB; the kernel reported 6.4 to 7.8 GB available and no memory pressure.

The batch cache is cleared at every epoch close, so it is not what holds the memory. What remains is either the allocator keeping freed memory or something long-lived in the node, and the node does not yet export a metric that tells them apart.

Also found

How far to trust this