We re-ran the 23 September benchmark on the same machines with faster disks. Average chain throughput rose 63%, and the write ceiling moved from about 288 to 800 MiB/s. The run also reproduced the stuck-validator bug on demand.
Both runs used 100 GB boot disks on the same machine type, zones, node build, load generator and fault schedule. On 23 September the disks were pd-ssd. This run used Hyperdisk Balanced provisioned at 20,000 IOPS and 800 MiB/s. The baseline is in the 23 September sizing report.
Transactions per second landing on chain, in five-minute blocks from the start of load.
MiB/s, mean of the nine validators that stayed up. The dashed lines are the highest one-second rate in a typical minute.
On pd-ssd the one-second write peaks sat at about 282 MiB/s from the first minute on every node, and the minute average flattened near 170. On Hyperdisk the peaks sit at about 800 to 820, and the average kept climbing through the run. Throughput falls over time in both runs, but the slowest five-minute block on Hyperdisk is 34% above the fastest on pd-ssd.
| Measure | pd-ssd, 23 Sep | Hyperdisk, 29 Sep | Change |
|---|---|---|---|
| Average chain throughput, all ten load generators | 2,558 tx/s | 4,177 tx/s | +63% |
| Best 60 seconds, all ten load generators | 3,518 tx/s | 5,243 tx/s | +49% |
| Transactions landed over the whole blast | 8,149,275 | 13,772,022 | +69% |
| Blocks per second | 7.43 | 12.74 | +72% |
| Time to land, median and p95 | 7.7 s, 31.5 s | 4.0 s, 14.4 s | about half |
| Commit latency, p95 | 22.0–23.0 s | 4.75–4.87 s | −79% |
| Execution time per consensus output | 4.88–5.00 s | 2.41–2.63 s | about half |
| Write throughput, p95, busiest validator | 181 MiB/s | 471 MiB/s | 2.6× |
| Write operations, p95, busiest validator | 2,440 IOPS | 7,711 IOPS | 3.2× |
| IO pressure, highest reading, range over validators | 50.3–59.8% | 26.1–30.3% | about half |
| CPU, p95, busiest validator | 24.6% | 32.8% | +8 points |
| Node resident memory, p95 | 10.0 GiB | 9.8 GiB | unchanged |
Validators only. The validator that the fault schedule kills and restarts is left out of the disk and CPU rows in both runs. Both result sets were built with the same scripts; re-running them on the 23 September data reproduced its files byte for byte.
The disk is still the closest resource to its limit, but only in bursts. Averaged over a minute nothing is over its line. Second by second, every validator's writes reach the provisioned 800 MiB/s in most minutes.
| Resource, busiest validator | Reading | Share of limit | |
|---|---|---|---|
| Disk throughput, one-second bursts | 800–810 MiB/s | ● at the limit | |
| Disk throughput, p95 of minute averages | 471 MiB/s | 59% of 800 | |
| Memory in use, p95 | 43.7% | 44% of 16 GB | |
| Disk operations, p95 | 7,711 IOPS | 39% of 20,000 | |
| CPU, p95 | 32.8% | 33% of 8 vCPU | |
| Network out, p95 | 112 Mbps | 11% of 1 Gbps |
IO pressure peaked at 26 to 30%, above the 20% line the sizing report uses, though half of what it was on pd-ssd.
When a validator falls behind it drops to catch-up mode. Its worker keeps sealing batches, peers keep voting on them, and the report to its own primary fails every time because nothing is listening. The transactions stay in its pool and the same batch is sealed again about once a second.
This run triggered it twice, once by killing and restarting a validator and once by pausing a container for three minutes. Both validators were out for about 25 minutes and rejoined only as load eased.
Failed batch reports per minute.
Failed batch reports per minute.
The gap after each epoch change is not recovery. For those minutes the validator is still in the old epoch, so peers reject its batch at quorum before it gets as far as the report.
| Validator | Trigger | Demoted | Rejoined | In catch-up | Failed reports | Rejected at quorum | Transactions stuck |
|---|---|---|---|---|---|---|---|
| validator-03 | Container paused 19:10:30 to 19:13:30 | 19:13:39 | 19:40:08 | 26 min 29 s | 915 | 136 | 9,600 |
| validator-05 | Killed 18:12:05, restarted 18:17:01 | 18:17:04 | 18:42:13 | 25 min 09 s | 1,207 | 245 | 9,589 |
Times and counts are from the validators' own logs, which this run kept in full. All times UTC.
After the second blast the fleet sat idle for 35 minutes. Every pool was empty, nothing was executing, and two epochs closed. Anonymous memory on nine of ten validators did not move.
| Node | Anonymous, 19:53 | Anonymous, 20:28 | Change | File-backed | Peak resident |
|---|
GB, read from each node's kernel. File-backed memory is mapped database pages, which the kernel reclaims on demand. Machines have 16 GB; the kernel reported 6.4 to 7.8 GB available and no memory pressure.
The batch cache is cleared at every epoch close, so it is not what holds the memory. What remains is either the allocator keeping freed memory or something long-lived in the node, and the node does not yet export a metric that tells them apart.