This is the full report of the run “The always-on vote fix and the order hold on tn-lmdb at 20,000 IOPS”. Back to the run page.
r20261115-1131 · Fleet c3-hdb · c3-highcpu-8

Telcoin Network benchmark: mixed workload

Throughput is measured from blocks the chain produced (own senders vs every sender), resource use from Cloud Monitoring, Prometheus and the on-node sampler. Summaries cover the mixed phase. Hover any number for the JSON field it came from.

10 blaster(s), 6K senders, chain 32286, schema tn-blast-combined/v2 · mixed phase 2026-11-15 13:27–14:08 UTC · 1 observers, 10 validators

Preflight: WTEL from genesis (0x0000…37e1), v3 fee tier 3000, v4 available, Multicall3 present · Cooldown ended: all_landed ×9, idle_30s ×1

Headline numbers

Sustained mixed TPS (own)
11,921.4 tx/s
best 60-s mean of our own landed tx in the mixed phase
Sustained mixed TPS (chain)
11,921.4 tx/s
all tx in blocks, same window
Landed / accepted
100.0 %
24.9M of 24.9M accepted
Reverted (own)
0
0.00 % of landed
Latency p50
1,479 ms
submit → block timestamp; seen p50 9,398 ms
Latency p95
7,763 ms
seen p95 17,327 ms
Gas / s (own)
466M gas/s
chain 466M gas/s
Capacity at mix
19,007.7 tx/s
gas-limit basis 12,311.5 tx/s; block gas limit 30M

Per transaction type

Landed counts only our own senders; latency is submit → block timestamp.

TypeSubmittedAcceptedLandedRevertedUnlandedSkipsGas limitGas used avgp50p95
native_transfer7.37M7.22M7.22M05024.1K21K1,471 ms7,743 ms
batched_self_send3.9M3.78M3.78M03024.1K21K1,525 ms7,836 ms
uniswap_v2_swap3.5M3.39M3.39M0100180K106K1,474 ms7,735 ms
stablecoin_transfer2.92M2.83M2.83M06075K43.9K1,472 ms7,733 ms
wtel_wrap2.34M2.27M2.27M03050K27.9K1,471 ms7,736 ms
wtel_unwrap1.55M1.51M1.51M01045K35.2K1,469 ms7,731 ms
stablecoin_approve1.17M1.13M1.13M02060K31.8K1,474 ms7,732 ms
uniswap_v2_add_liquidity779K755K755K010250K128K1,476 ms7,723 ms
contract_deploy586K568K568K010140K74.2K1,469 ms7,754 ms
uniswap_v4_swap583K565K565K000250K124K1,477 ms7,741 ms
uniswap_v3_swap393K380K380K000220K120K1,471 ms7,736 ms
uniswap_v2_remove_liquidity390K377K377K000220K129K1,474 ms7,759 ms
faucet_drip126K126K126K00845K150K107K1,081 ms12,142 ms

Submission by RPC node

NodeSubmittedAcceptedAccepted %Top errorsSend p50Send p95
bench-validator-01
bench-validator-01:8545
2.16M2.16M100.0 %none7 ms26 ms
bench-validator-02
bench-validator-02:8545
1.85M1.85M100.0 %none8 ms34 ms
bench-validator-03
bench-validator-03:8545
3.29M3.29M100.0 %none7 ms25 ms
bench-validator-04
bench-validator-04:8545
3.73M3.73M100.0 %none11 ms39 ms
bench-validator-05
bench-validator-05:8545
3.07M935K30.4 %other 2.14M1 ms17 ms
bench-validator-06
bench-validator-06:8545
1.73M1.73M100.0 %none11 ms44 ms
bench-validator-07
bench-validator-07:8545
2.24M2.24M100.0 %none6 ms23 ms
bench-validator-08
bench-validator-08:8545
3.68M3.68M100.0 %none11 ms36 ms
bench-validator-09
bench-validator-09:8545
2.8M2.8M100.0 %none5 ms19 ms
bench-validator-10
bench-validator-10:8545
2.54M2.54M100.0 %none7 ms28 ms

Throughput over time

Shaded: phases outside the mixed window. Hairlines: epoch closes. Dashed orange: chaos events.

Landed transactions per second

own senders vs chain total, per block-timestamp second

Gas per second

own vs chain gas used, per block-timestamp second

CPU and memory per node

CPU utilization

compute.googleapis.com/instance/cpu/utilization, 60-s mean, % of all vCPUs; summaries over the mixed phase

Table view
NodeClassMeanp95Peak 60-s meanMax
bench-observer-11observer27.5 %30.2 %34.8 %34.8 %
bench-validator-01validator47.4 %52.6 %68.2 %68.2 %
bench-validator-02validator47.4 %53.5 %64.3 %64.3 %
bench-validator-03validator49.0 %54.7 %70.4 %70.4 %
bench-validator-04validator49.4 %55.8 %70.3 %70.3 %
bench-validator-05validator36.9 %52.3 %68.8 %68.8 %
bench-validator-06validator47.0 %52.1 %64.0 %64.0 %
bench-validator-07validator47.4 %52.5 %67.6 %67.6 %
bench-validator-08validator48.9 %54.5 %69.6 %69.6 %
bench-validator-09validator47.9 %54.4 %68.5 %68.5 %
bench-validator-10validator47.9 %51.8 %64.0 %64.0 %

CPU: mean vs peak 60-s mean

One row per node, over the mixed phase.

telcoin resident memory

agent processes/rss_usage summed over telcoin pids; summaries over the mixed phase

Table view
NodeClassMeanp95Peak 60-s meanMax
bench-observer-11observer4.50 GiB5.84 GiB5.92 GiB5.92 GiB
bench-validator-01validator5.46 GiB6.60 GiB6.63 GiB6.63 GiB
bench-validator-02validator5.51 GiB6.67 GiB6.71 GiB6.71 GiB
bench-validator-03validator5.34 GiB6.39 GiB6.65 GiB6.65 GiB
bench-validator-04validator5.41 GiB6.53 GiB6.54 GiB6.54 GiB
bench-validator-05validator4.78 GiB7.20 GiB7.30 GiB7.30 GiB
bench-validator-06validator5.46 GiB6.65 GiB6.68 GiB6.68 GiB
bench-validator-07validator5.76 GiB7.07 GiB7.10 GiB7.10 GiB
bench-validator-08validator5.38 GiB6.49 GiB6.58 GiB6.58 GiB
bench-validator-09validator5.35 GiB6.41 GiB6.52 GiB6.52 GiB
bench-validator-10validator5.35 GiB6.46 GiB6.59 GiB6.59 GiB

telcoin RSS: mean vs peak 60-s mean

One row per node, over the mixed phase.

Memory used

agent memory/percent_used, state=used; summaries over the mixed phase

Table view
NodeClassMeanp95Peak 60-s meanMax
bench-observer-11observer10.6 %11.9 %12.0 %12.0 %
bench-validator-01validator16.1 %16.7 %16.9 %16.9 %
bench-validator-02validator16.3 %17.3 %17.3 %17.3 %
bench-validator-03validator15.1 %15.6 %15.6 %15.6 %
bench-validator-04validator15.5 %16.2 %16.3 %16.3 %
bench-validator-05validator15.1 %20.8 %21.1 %21.1 %
bench-validator-06validator16.2 %17.1 %17.2 %17.2 %
bench-validator-07validator17.7 %19.5 %19.7 %19.7 %
bench-validator-08validator15.4 %16.1 %16.4 %16.4 %
bench-validator-09validator15.3 %15.8 %15.9 %15.9 %
bench-validator-10validator15.6 %16.1 %16.1 %16.1 %

Memory used: mean vs peak 60-s mean

One row per node, over the mixed phase.

Disk

Disk throttling (validator)
not collected –
metric type unavailable in project
Disk performance degraded (validator)
none –
any sample

Disk read + write throughput

instance/disk/{read,write}_bytes_count rate, all devices; summaries over the mixed phase

Table view
NodeClassMeanp95Peak 60-s meanMaxLimit
bench-observer-11observer236.4 MiB/s274.6 MiB/s278.2 MiB/s278.2 MiB/s800.0 MiB/s
bench-validator-01validator229.4 MiB/s279.5 MiB/s280.5 MiB/s280.5 MiB/s800.0 MiB/s
bench-validator-02validator220.5 MiB/s276.9 MiB/s279.9 MiB/s279.9 MiB/s800.0 MiB/s
bench-validator-03validator240.0 MiB/s283.0 MiB/s287.5 MiB/s287.5 MiB/s800.0 MiB/s
bench-validator-04validator232.2 MiB/s286.7 MiB/s290.9 MiB/s290.9 MiB/s800.0 MiB/s
bench-validator-05validator175.1 MiB/s274.0 MiB/s281.0 MiB/s281.0 MiB/s800.0 MiB/s
bench-validator-06validator236.6 MiB/s281.5 MiB/s287.4 MiB/s287.4 MiB/s800.0 MiB/s
bench-validator-07validator227.4 MiB/s278.7 MiB/s282.6 MiB/s282.6 MiB/s800.0 MiB/s
bench-validator-08validator235.8 MiB/s284.3 MiB/s292.1 MiB/s292.1 MiB/s800.0 MiB/s
bench-validator-09validator226.9 MiB/s279.1 MiB/s283.7 MiB/s283.7 MiB/s800.0 MiB/s
bench-validator-10validator218.9 MiB/s281.6 MiB/s284.6 MiB/s284.6 MiB/s800.0 MiB/s

Disk read + write IOPS

instance/disk/{read,write}_ops_count rate, all devices; summaries over the mixed phase

Table view
NodeClassMeanp95Peak 60-s meanMaxLimit
bench-observer-11observer16.4K IOPS18.2K IOPS19.5K IOPS19.5K IOPS20K IOPS
bench-validator-01validator17.2K IOPS19.5K IOPS20.1K IOPS20.1K IOPS20K IOPS
bench-validator-02validator17.5K IOPS19.9K IOPS20K IOPS20K IOPS20K IOPS
bench-validator-03validator16.6K IOPS18.8K IOPS19.3K IOPS19.3K IOPS20K IOPS
bench-validator-04validator17.1K IOPS19.4K IOPS19.8K IOPS19.8K IOPS20K IOPS
bench-validator-05validator14K IOPS19.4K IOPS19.8K IOPS19.8K IOPS20K IOPS
bench-validator-06validator16.8K IOPS19.7K IOPS19.7K IOPS19.7K IOPS20K IOPS
bench-validator-07validator17.1K IOPS19.9K IOPS20.1K IOPS20.1K IOPS20K IOPS
bench-validator-08validator17K IOPS19.1K IOPS19.5K IOPS19.5K IOPS20K IOPS
bench-validator-09validator17.2K IOPS19.7K IOPS20K IOPS20K IOPS20K IOPS
bench-validator-10validator17.6K IOPS19.9K IOPS20K IOPS20K IOPS20K IOPS
Disk limits used
NodeMachinevCPUDiskIOPS limitMiB/s limitBasis
bench-observer-11c3-highcpu-88hyperdisk-balanced 100 GB20K IOPS800.0 MiB/shyperdisk-balanced 100 GB: provisioned 20,000 IOPS, provisioned 800 MiB/s; c3 8 vCPU cap 50,000 IOPS / 800 MiB/s; reads and writes share this budget
bench-validator-01c3-highcpu-88hyperdisk-balanced 100 GB20K IOPS800.0 MiB/shyperdisk-balanced 100 GB: provisioned 20,000 IOPS, provisioned 800 MiB/s; c3 8 vCPU cap 50,000 IOPS / 800 MiB/s; reads and writes share this budget
bench-validator-02c3-highcpu-88hyperdisk-balanced 100 GB20K IOPS800.0 MiB/shyperdisk-balanced 100 GB: provisioned 20,000 IOPS, provisioned 800 MiB/s; c3 8 vCPU cap 50,000 IOPS / 800 MiB/s; reads and writes share this budget
bench-validator-03c3-highcpu-88hyperdisk-balanced 100 GB20K IOPS800.0 MiB/shyperdisk-balanced 100 GB: provisioned 20,000 IOPS, provisioned 800 MiB/s; c3 8 vCPU cap 50,000 IOPS / 800 MiB/s; reads and writes share this budget
bench-validator-04c3-highcpu-88hyperdisk-balanced 100 GB20K IOPS800.0 MiB/shyperdisk-balanced 100 GB: provisioned 20,000 IOPS, provisioned 800 MiB/s; c3 8 vCPU cap 50,000 IOPS / 800 MiB/s; reads and writes share this budget
bench-validator-05c3-highcpu-88hyperdisk-balanced 100 GB20K IOPS800.0 MiB/shyperdisk-balanced 100 GB: provisioned 20,000 IOPS, provisioned 800 MiB/s; c3 8 vCPU cap 50,000 IOPS / 800 MiB/s; reads and writes share this budget
bench-validator-06c3-highcpu-88hyperdisk-balanced 100 GB20K IOPS800.0 MiB/shyperdisk-balanced 100 GB: provisioned 20,000 IOPS, provisioned 800 MiB/s; c3 8 vCPU cap 50,000 IOPS / 800 MiB/s; reads and writes share this budget
bench-validator-07c3-highcpu-88hyperdisk-balanced 100 GB20K IOPS800.0 MiB/shyperdisk-balanced 100 GB: provisioned 20,000 IOPS, provisioned 800 MiB/s; c3 8 vCPU cap 50,000 IOPS / 800 MiB/s; reads and writes share this budget
bench-validator-08c3-highcpu-88hyperdisk-balanced 100 GB20K IOPS800.0 MiB/shyperdisk-balanced 100 GB: provisioned 20,000 IOPS, provisioned 800 MiB/s; c3 8 vCPU cap 50,000 IOPS / 800 MiB/s; reads and writes share this budget
bench-validator-09c3-highcpu-88hyperdisk-balanced 100 GB20K IOPS800.0 MiB/shyperdisk-balanced 100 GB: provisioned 20,000 IOPS, provisioned 800 MiB/s; c3 8 vCPU cap 50,000 IOPS / 800 MiB/s; reads and writes share this budget
bench-validator-10c3-highcpu-88hyperdisk-balanced 100 GB20K IOPS800.0 MiB/shyperdisk-balanced 100 GB: provisioned 20,000 IOPS, provisioned 800 MiB/s; c3 8 vCPU cap 50,000 IOPS / 800 MiB/s; reads and writes share this budget

Network

Network sent

instance/network/sent_bytes_count rate; summaries over the mixed phase

Table view
NodeClassMeanp95Peak 60-s meanMax
bench-observer-11observer1.5 Mbps1.9 Mbps2.4 Mbps2.4 Mbps
bench-validator-01validator162.6 Mbps185.0 Mbps198.5 Mbps198.5 Mbps
bench-validator-02validator160.9 Mbps183.7 Mbps185.0 Mbps185.0 Mbps
bench-validator-03validator166.5 Mbps189.5 Mbps204.6 Mbps204.6 Mbps
bench-validator-04validator168.7 Mbps190.3 Mbps213.8 Mbps213.8 Mbps
bench-validator-05validator130.6 Mbps180.3 Mbps204.4 Mbps204.4 Mbps
bench-validator-06validator164.2 Mbps187.8 Mbps193.2 Mbps193.2 Mbps
bench-validator-07validator165.3 Mbps204.1 Mbps237.4 Mbps237.4 Mbps
bench-validator-08validator168.8 Mbps187.4 Mbps216.6 Mbps216.6 Mbps
bench-validator-09validator164.8 Mbps191.6 Mbps199.5 Mbps199.5 Mbps
bench-validator-10validator163.9 Mbps181.6 Mbps191.1 Mbps191.1 Mbps

Network received

instance/network/received_bytes_count rate; summaries over the mixed phase

Table view
NodeClassMeanp95Peak 60-s meanMax
bench-observer-11observer8.3 Mbps9.4 Mbps15.2 Mbps15.2 Mbps
bench-validator-01validator14.0 Mbps15.7 Mbps26.0 Mbps26.0 Mbps
bench-validator-02validator13.8 Mbps15.9 Mbps19.5 Mbps19.5 Mbps
bench-validator-03validator16.5 Mbps19.4 Mbps27.5 Mbps27.5 Mbps
bench-validator-04validator17.2 Mbps20.5 Mbps29.7 Mbps29.7 Mbps
bench-validator-05validator26.9 Mbps81.1 Mbps98.2 Mbps98.2 Mbps
bench-validator-06validator13.8 Mbps15.7 Mbps18.9 Mbps18.9 Mbps
bench-validator-07validator14.3 Mbps16.1 Mbps26.3 Mbps26.3 Mbps
bench-validator-08validator17.2 Mbps20.0 Mbps29.9 Mbps29.9 Mbps
bench-validator-09validator15.4 Mbps19.4 Mbps25.2 Mbps25.2 Mbps
bench-validator-10validator15.1 Mbps16.7 Mbps23.6 Mbps23.6 Mbps

Consensus health

MetricQueryClassNodesMedian of node meansWorst peak 60-s meanMax
tn_primary_commit_latency_secondsp95validator101,267 ms3,910 ms5,250 ms
tn_engine_execution_duration_secondsp95validator102,073 ms29.0 s29.0 s
tn_engine_execution_duration_secondsp95observer11,783 ms2,485 ms3,083 ms
tn_worker_batch_size_bytesp95validator9242.1 KiB246.4 KiB246.4 KiB
tn_worker_batch_transactionsp95validator105031.48K1.49K
tn_executor_output_transactionsp95validator920.6K31.3K35.6K
tn_primary_consensus_output_bytesmaxvalidator10709.1 KiB1.8 MiB2.5 MiB
tn_primary_consensus_output_bytesmaxobserver1778.7 KiB1.5 MiB2.5 MiB
tn_engine_queued_outputsmaxvalidator106.7788
tn_engine_queued_outputsmaxobserver11.958
tn_primary_roundmaxvalidator101.49K3.12K3.15K
tn_primary_roundmaxobserver11.49K3.1K3.15K
tn_primary_committed_roundmaxvalidator101.48K3.12K3.15K
tn_primary_committed_roundmaxobserver11.49K3.1K3.15K
tn_epoch_currentmaxvalidator105.7877
tn_epoch_currentmaxobserver15.7777
tn_batch_builder_pending_pool_transactionsmaxvalidator101.12K5.19K9.9K
tn_batch_builder_pending_pool_transactionsmaxobserver1000
reth_process_resident_memory_bytesmaxvalidator105.43 GiB7.43 GiB7.88 GiB
reth_process_resident_memory_bytesmaxobserver14.49 GiB5.90 GiB5.94 GiB
tn_primary_subdags_committed_totalratevalidator101.121.431.44
tn_primary_subdags_committed_totalrateobserver1000
tn_primary_round_laglagvalidator102.163.757
tn_primary_round_laglagobserver1000
tn_batch_builder_seal_failures_totalratevalidator900.01670.0222
tn_node_modemaxvalidator10011
tn_node_modemaxobserver1000
tn_primary_vote_requests_totalratevalidator1019.632.632.7
tn_primary_vote_requests_totalrateobserver1000
tn_primary_votes_created_totalratevalidator1018.725.725.8
tn_primary_votes_created_totalrateobserver1000
tn_primary_votes_recast_totalratevalidator10000
tn_primary_votes_recast_totalrateobserver1000
tn_primary_vote_requests_too_new_totalratevalidator10000
tn_primary_vote_requests_too_new_totalrateobserver1000
tn_primary_vote_timeouts_total{stage="exec_wait"}ratevalidator10000
tn_primary_vote_timeouts_total{stage="exec_wait"}rateobserver1000
tn_primary_vote_timeouts_total{stage="lead_sleep"}ratevalidator10000
tn_primary_vote_timeouts_total{stage="lead_sleep"}rateobserver1000
tn_primary_vote_timeouts_total{stage="parents"}ratevalidator1000.05980.0798
tn_primary_vote_timeouts_total{stage="parents"}rateobserver1000
tn_primary_vote_timeouts_total{stage="persist"}ratevalidator10000
tn_primary_vote_timeouts_total{stage="persist"}rateobserver1000
tn_primary_vote_timeouts_total{stage="prelock_wait"}ratevalidator10000
tn_primary_vote_timeouts_total{stage="prelock_wait"}rateobserver1000
tn_primary_vote_timeouts_total{stage="sync_batches"}ratevalidator10000
tn_primary_vote_timeouts_total{stage="sync_batches"}rateobserver1000
tn_primary_vote_timeouts_total{stage="validate"}ratevalidator10000
tn_primary_vote_timeouts_total{stage="validate"}rateobserver1000
tn_primary_vote_timeouts_total{stage="vote_store"}ratevalidator10000
tn_primary_vote_timeouts_total{stage="vote_store"}rateobserver1000
tn_primary_vote_stage_seconds{stage="exec_wait"}p95validator90 ms0 ms0 ms
tn_primary_vote_stage_seconds{stage="lead_sleep"}p95validator90 ms0 ms0 ms
tn_primary_vote_stage_seconds{stage="lock"}p95validator90 ms1,500 ms2,392 ms
tn_primary_vote_stage_seconds{stage="parents"}p95validator91 ms2 ms2 ms
tn_primary_vote_stage_seconds{stage="persist"}p95validator912 ms30 ms33 ms
tn_primary_vote_stage_seconds{stage="prelock_wait"}p95validator9523 ms2,397 ms2,399 ms
tn_primary_vote_stage_seconds{stage="sync_batches"}p95validator90 ms0 ms0 ms
tn_primary_vote_stage_seconds{stage="validate"}p95validator94 ms6 ms6 ms
tn_primary_vote_stage_seconds{stage="vote_store"}p95validator91 ms3 ms3 ms
tn_primary_leader_rounds_total{leader="4SxMDsdwEfaWymUitMjsGtCJfdeY8Ey8a6vorue2GNvm",swapped="false",found="false"}ratevalidator100.01090.05560.0667
tn_primary_leader_rounds_total{leader="4SxMDsdwEfaWymUitMjsGtCJfdeY8Ey8a6vorue2GNvm",swapped="false",found="true"}ratevalidator100.1060.1560.156
tn_primary_leader_rounds_total{leader="4SxMDsdwEfaWymUitMjsGtCJfdeY8Ey8a6vorue2GNvm",swapped="true",found="false"}ratevalidator100.002410.050.0667
tn_primary_leader_rounds_total{leader="4SxMDsdwEfaWymUitMjsGtCJfdeY8Ey8a6vorue2GNvm",swapped="true",found="true"}ratevalidator100.01930.1330.133
tn_primary_leader_rounds_total{leader="4jfiSawn73qjdDrEEyUtwyHnyTb7PuSGhZLWKeW46nXy",swapped="false",found="false"}ratevalidator10000
tn_primary_leader_rounds_total{leader="4jfiSawn73qjdDrEEyUtwyHnyTb7PuSGhZLWKeW46nXy",swapped="false",found="true"}ratevalidator100.1170.1560.156
tn_primary_leader_rounds_total{leader="4jfiSawn73qjdDrEEyUtwyHnyTb7PuSGhZLWKeW46nXy",swapped="true",found="false"}ratevalidator100.0003520.01630.0217
tn_primary_leader_rounds_total{leader="4jfiSawn73qjdDrEEyUtwyHnyTb7PuSGhZLWKeW46nXy",swapped="true",found="true"}ratevalidator100.02140.09440.111
tn_primary_leader_rounds_total{leader="64SZkNxHc7bADaTYSLs2SMw6rL2GdC38AYmjY27P4dPr",swapped="false",found="false"}ratevalidator100.01880.1330.133
tn_primary_leader_rounds_total{leader="64SZkNxHc7bADaTYSLs2SMw6rL2GdC38AYmjY27P4dPr",swapped="false",found="true"}ratevalidator100.04440.1560.156
tn_primary_leader_rounds_total{leader="64SZkNxHc7bADaTYSLs2SMw6rL2GdC38AYmjY27P4dPr",swapped="true",found="false"}ratevalidator100.0003520.01630.0217
tn_primary_leader_rounds_total{leader="64SZkNxHc7bADaTYSLs2SMw6rL2GdC38AYmjY27P4dPr",swapped="true",found="true"}ratevalidator100.002060.0370.0444
tn_primary_leader_rounds_total{leader="86iS1yBLyqxNBpBK2kUb7QSpw5gdRBqFzD9ZvbNZVHWp",swapped="false",found="false"}ratevalidator100.0003520.01630.0217
tn_primary_leader_rounds_total{leader="86iS1yBLyqxNBpBK2kUb7QSpw5gdRBqFzD9ZvbNZVHWp",swapped="false",found="true"}ratevalidator100.05120.1440.156
tn_primary_leader_rounds_total{leader="86iS1yBLyqxNBpBK2kUb7QSpw5gdRBqFzD9ZvbNZVHWp",swapped="true",found="false"}ratevalidator10000
tn_primary_leader_rounds_total{leader="86iS1yBLyqxNBpBK2kUb7QSpw5gdRBqFzD9ZvbNZVHWp",swapped="true",found="true"}ratevalidator10000
tn_primary_leader_rounds_total{leader="AWgztxuoJShXittjoeL8thGdDWzywN1dg4yrJcofdweq",swapped="false",found="false"}ratevalidator10000
tn_primary_leader_rounds_total{leader="AWgztxuoJShXittjoeL8thGdDWzywN1dg4yrJcofdweq",swapped="false",found="true"}ratevalidator100.03540.1330.156
tn_primary_leader_rounds_total{leader="AWgztxuoJShXittjoeL8thGdDWzywN1dg4yrJcofdweq",swapped="true",found="false"}ratevalidator10000
tn_primary_leader_rounds_total{leader="AWgztxuoJShXittjoeL8thGdDWzywN1dg4yrJcofdweq",swapped="true",found="true"}ratevalidator100.001880.07320.0997
tn_primary_leader_rounds_total{leader="AfAVgJWz8RGmhhqfEgQdsuMZ9sTQuR1nvecqaxgmwezx",swapped="false",found="false"}ratevalidator100.001550.03250.0434
tn_primary_leader_rounds_total{leader="AfAVgJWz8RGmhhqfEgQdsuMZ9sTQuR1nvecqaxgmwezx",swapped="false",found="true"}ratevalidator100.1140.1560.156
tn_primary_leader_rounds_total{leader="AfAVgJWz8RGmhhqfEgQdsuMZ9sTQuR1nvecqaxgmwezx",swapped="true",found="false"}ratevalidator10000
tn_primary_leader_rounds_total{leader="AfAVgJWz8RGmhhqfEgQdsuMZ9sTQuR1nvecqaxgmwezx",swapped="true",found="true"}ratevalidator100.02260.07220.0889
tn_primary_leader_rounds_total{leader="BUNbjcD7XDj59dPfHZzebLkPbrcnmVk2s9V9RiZiNEtA",swapped="false",found="false"}ratevalidator100.002370.03270.0434
tn_primary_leader_rounds_total{leader="BUNbjcD7XDj59dPfHZzebLkPbrcnmVk2s9V9RiZiNEtA",swapped="false",found="true"}ratevalidator100.1140.1560.156
tn_primary_leader_rounds_total{leader="BUNbjcD7XDj59dPfHZzebLkPbrcnmVk2s9V9RiZiNEtA",swapped="true",found="false"}ratevalidator100.0007650.01670.0222
tn_primary_leader_rounds_total{leader="BUNbjcD7XDj59dPfHZzebLkPbrcnmVk2s9V9RiZiNEtA",swapped="true",found="true"}ratevalidator100.030.1060.133
tn_primary_leader_rounds_total{leader="GrMgdCJ8xiqgeg14d81g28EbqAaPzukJZ1sE9NMBkcrL",swapped="false",found="false"}ratevalidator10000
tn_primary_leader_rounds_total{leader="GrMgdCJ8xiqgeg14d81g28EbqAaPzukJZ1sE9NMBkcrL",swapped="false",found="true"}ratevalidator100.1160.1560.156
tn_primary_leader_rounds_total{leader="GrMgdCJ8xiqgeg14d81g28EbqAaPzukJZ1sE9NMBkcrL",swapped="true",found="false"}ratevalidator10000
tn_primary_leader_rounds_total{leader="GrMgdCJ8xiqgeg14d81g28EbqAaPzukJZ1sE9NMBkcrL",swapped="true",found="true"}ratevalidator100.02510.08890.111
tn_primary_leader_rounds_total{leader="H3mEWkRevyzSnYMcL6Z8M5AwDWYhUo1HR2bgFZzmKWSu",swapped="false",found="false"}ratevalidator10000
tn_primary_leader_rounds_total{leader="H3mEWkRevyzSnYMcL6Z8M5AwDWYhUo1HR2bgFZzmKWSu",swapped="false",found="true"}ratevalidator100.1160.1560.156
tn_primary_leader_rounds_total{leader="H3mEWkRevyzSnYMcL6Z8M5AwDWYhUo1HR2bgFZzmKWSu",swapped="true",found="false"}ratevalidator10000
tn_primary_leader_rounds_total{leader="H3mEWkRevyzSnYMcL6Z8M5AwDWYhUo1HR2bgFZzmKWSu",swapped="true",found="true"}ratevalidator100.0380.1060.133
tn_primary_leader_rounds_total{leader="H4fbN7HsCDw74LeFeaSgzzfzwTTpKNcRcc4312uuD3D9",swapped="false",found="false"}ratevalidator10000
tn_primary_leader_rounds_total{leader="H4fbN7HsCDw74LeFeaSgzzfzwTTpKNcRcc4312uuD3D9",swapped="false",found="true"}ratevalidator100.1140.1560.156
tn_primary_leader_rounds_total{leader="H4fbN7HsCDw74LeFeaSgzzfzwTTpKNcRcc4312uuD3D9",swapped="true",found="false"}ratevalidator10000
tn_primary_leader_rounds_total{leader="H4fbN7HsCDw74LeFeaSgzzfzwTTpKNcRcc4312uuD3D9",swapped="true",found="true"}ratevalidator100.03690.10.133
tn_primary_leader_commits_total{leader="4SxMDsdwEfaWymUitMjsGtCJfdeY8Ey8a6vorue2GNvm"}ratevalidator100.1250.2890.289
tn_primary_leader_commits_total{leader="4jfiSawn73qjdDrEEyUtwyHnyTb7PuSGhZLWKeW46nXy"}ratevalidator100.1380.2110.244
tn_primary_leader_commits_total{leader="64SZkNxHc7bADaTYSLs2SMw6rL2GdC38AYmjY27P4dPr"}ratevalidator100.04650.1850.2
tn_primary_leader_commits_total{leader="86iS1yBLyqxNBpBK2kUb7QSpw5gdRBqFzD9ZvbNZVHWp"}ratevalidator100.05120.1440.156
tn_primary_leader_commits_total{leader="AWgztxuoJShXittjoeL8thGdDWzywN1dg4yrJcofdweq"}ratevalidator100.03740.1780.244
tn_primary_leader_commits_total{leader="AfAVgJWz8RGmhhqfEgQdsuMZ9sTQuR1nvecqaxgmwezx"}ratevalidator100.1370.2170.244
tn_primary_leader_commits_total{leader="BUNbjcD7XDj59dPfHZzebLkPbrcnmVk2s9V9RiZiNEtA"}ratevalidator100.1440.2390.267
tn_primary_leader_commits_total{leader="GrMgdCJ8xiqgeg14d81g28EbqAaPzukJZ1sE9NMBkcrL"}ratevalidator100.1410.2330.267
tn_primary_leader_commits_total{leader="H3mEWkRevyzSnYMcL6Z8M5AwDWYhUo1HR2bgFZzmKWSu"}ratevalidator100.1540.2440.244
tn_primary_leader_commits_total{leader="H4fbN7HsCDw74LeFeaSgzzfzwTTpKNcRcc4312uuD3D9"}ratevalidator100.1510.2280.267
tn_engine_queue_wait_secondsp95validator109,201 ms30.0 s30.0 s
tn_engine_queue_wait_secondsp95observer12,757 ms7,878 ms8,705 ms
tn_engine_persist_duration_secondsp95validator10894 ms29.0 s29.0 s
tn_engine_persist_duration_secondsp95observer1793 ms1,649 ms1,810 ms
tn_engine_busy_milliseconds_totalratevalidator109841.07K1.21K
tn_engine_busy_milliseconds_totalrateobserver18839781.03K
tn_reth_persist_save_blocks_secondsp95validator9297 ms29.0 s29.0 s
tn_reth_persist_save_blocks_secondsp95observer1263 ms363 ms392 ms
tn_reth_persist_commit_secondsp95validator10584 ms1,181 ms1,400 ms
tn_reth_persist_commit_secondsp95observer1641 ms968 ms995 ms
tn_reth_persisted_accounts_totalratevalidator104.13K10.3K10.3K
tn_reth_persisted_accounts_totalrateobserver14.12K10.1K10.1K
tn_reth_persisted_storage_slots_totalratevalidator105.39K6.7K6.92K
tn_reth_persisted_storage_slots_totalrateobserver15.36K6.76K7.07K
tn_executor_rejoin_blocked_total{reason="attempts_exhausted"}ratevalidator1000
tn_executor_rejoin_blocked_total{reason="epoch_closed"}ratevalidator10.3151.061.18
tn_executor_rejoin_blocked_total{reason="epoch_mismatch"}ratevalidator10.003810.07520.1
tn_executor_rejoin_blocked_total{reason="no_known_header"}ratevalidator1000
tn_executor_rejoin_blocked_total{reason="round_gap"}ratevalidator10.6191.241.27
tn_executor_rejoin_blocked_total{reason="stale_head"}ratevalidator10.03350.1950.261
tn_worker_sync_serves_denied_total{worker="0",tier="committee"}ratevalidator10000
tn_worker_sync_serves_denied_total{worker="0",tier="committee"}rateobserver1000
tn_worker_sync_serves_denied_total{worker="0",tier="public"}ratevalidator10000
tn_worker_sync_serves_denied_total{worker="0",tier="public"}rateobserver1000
tn_worker_batch_requests_denied_total{worker="0",peer="212J1mA7zZQ51ukqQKvWVRwywSEPLh1NsvT9wy4cKoUmNmbmYxLdSJaL3oXaLTxN6AjpxNkj6fTCbU9wrx7cwCACTHYVDi7j53Z3bRjF2vGPucmAGL8TTkSQigRqFXnp7rBt"}ratevalidator3000
tn_worker_batch_requests_denied_total{worker="0",peer="21YCnFzx4v4VtmLqYj7CNQqX4v7PUXFWsQhSmLE9f4Kfm4LSiZxpQZNN28cjNyTPR2XxfgD1fe4DR5cVWhftSgnMh64gmGCS8NxyKDPAytbTSdR5wmcofTsP9Xfr4wWk93Xk"}ratevalidator3000
tn_worker_batch_requests_denied_total{worker="0",peer="21iPCy34tzHPBsuC9yb1Q4pdsYYAxiQS9LYCVPBxoCzHiwCHruunBs9eP6YKM3nm2NVbuCtvYvJRTthqrqvMww2xvrbLashwQkzpXrVZMTyMQrsD35o7qkkUdX37fE2tY8rX"}ratevalidator3000
tn_worker_batch_requests_denied_total{worker="0",peer="22F8ZnogXF6477tTpccswX2XiEYZT254kzbKcXQDuPhVKkxefa2xnqcbtzDV7fxDFmFmnkccaCx45LNUuXaQnTG7eafuJmz9Jh9hA7RzXaKRT97JKtKfS7qFTeD6yBWezhjX"}ratevalidator2000
tn_worker_batch_requests_denied_total{worker="0",peer="nENWWkwd3NxS22EmtXU8hYQPgFjsDG1XxRwSxnLwRntnDXs4cdiMxAWo535Srphnyo49dcFdGi4pkFxCQpK8BcY4jYYWeG6GkX9fMAyfNggZqSBnD2MuMHS8rJ2k4tgWXAE"}ratevalidator1000
tn_worker_batch_requests_denied_total{worker="0",peer="nb8kLGNaqv8Gx3FtgKNLLCaM2eXpV58hwtdhLUWvwsQzsa88Um8u7ppNN3zRmFePE95VLhwFoHfnmwW4DZPHLzhKxJk5Ng1Sg1rXNKuGqsnXA6i34LUXnv7QyAkaZz2Zw2x"}ratevalidator3000
tn_worker_batch_requests_denied_total{worker="0",peer="smREePKNZPDrerYFLbctThvtgXu33jm9yDxD59eZkvb3L7Y6bhVEQbeTjYzGAqe14Kvw4f64g2mgwbHwCA9ZCKRp2iuSy7UCaSSdvQQCfJhiD7tExoULx8BZwQQx8Awpje3"}ratevalidator70.001960.050.0667
tn_worker_batch_requests_denied_total{worker="0",peer="tzxt1iN85tXzs8BABJ48vTMWTrDoQHpnHmke5rBXxes6Tf5bSta8yKRBqe64f94Ya4ZGGKh6YZ9csG9sZFro3szhkjYPD2Smg5rLyd5bgigTxho82234QaL1izWsXRMehWA"}ratevalidator3000
tn_worker_batch_requests_denied_total{worker="0",peer="z6P9RtGujezPz2Y2aSJcxJUdTx2CuUQ3fMHhs1xZfkX66BqjEPknJ9hwPhnoGstvNRjmkoQXMhnZu1vNKD8C3X2PEM1SVgSApd1J9VQD1sLgMjTxk1uUzHVUyVZWuA7TJyG"}ratevalidator1000
tn_worker_batch_requests_failed_total{worker="0",peer="212J1mA7zZQ51ukqQKvWVRwywSEPLh1NsvT9wy4cKoUmNmbmYxLdSJaL3oXaLTxN6AjpxNkj6fTCbU9wrx7cwCACTHYVDi7j53Z3bRjF2vGPucmAGL8TTkSQigRqFXnp7rBt"}ratevalidator3000
tn_worker_batch_requests_failed_total{worker="0",peer="21YCnFzx4v4VtmLqYj7CNQqX4v7PUXFWsQhSmLE9f4Kfm4LSiZxpQZNN28cjNyTPR2XxfgD1fe4DR5cVWhftSgnMh64gmGCS8NxyKDPAytbTSdR5wmcofTsP9Xfr4wWk93Xk"}ratevalidator300.03890.0444
tn_worker_batch_requests_failed_total{worker="0",peer="21iPCy34tzHPBsuC9yb1Q4pdsYYAxiQS9LYCVPBxoCzHiwCHruunBs9eP6YKM3nm2NVbuCtvYvJRTthqrqvMww2xvrbLashwQkzpXrVZMTyMQrsD35o7qkkUdX37fE2tY8rX"}ratevalidator30.002340.05450.0667
tn_worker_batch_requests_failed_total{worker="0",peer="22F8ZnogXF6477tTpccswX2XiEYZT254kzbKcXQDuPhVKkxefa2xnqcbtzDV7fxDFmFmnkccaCx45LNUuXaQnTG7eafuJmz9Jh9hA7RzXaKRT97JKtKfS7qFTeD6yBWezhjX"}ratevalidator2000
tn_worker_batch_requests_failed_total{worker="0",peer="nENWWkwd3NxS22EmtXU8hYQPgFjsDG1XxRwSxnLwRntnDXs4cdiMxAWo535Srphnyo49dcFdGi4pkFxCQpK8BcY4jYYWeG6GkX9fMAyfNggZqSBnD2MuMHS8rJ2k4tgWXAE"}ratevalidator1000
tn_worker_batch_requests_failed_total{worker="0",peer="nb8kLGNaqv8Gx3FtgKNLLCaM2eXpV58hwtdhLUWvwsQzsa88Um8u7ppNN3zRmFePE95VLhwFoHfnmwW4DZPHLzhKxJk5Ng1Sg1rXNKuGqsnXA6i34LUXnv7QyAkaZz2Zw2x"}ratevalidator3000
tn_worker_batch_requests_failed_total{worker="0",peer="smREePKNZPDrerYFLbctThvtgXu33jm9yDxD59eZkvb3L7Y6bhVEQbeTjYzGAqe14Kvw4f64g2mgwbHwCA9ZCKRp2iuSy7UCaSSdvQQCfJhiD7tExoULx8BZwQQx8Awpje3"}ratevalidator700.01670.0222
tn_worker_batch_requests_failed_total{worker="0",peer="tzxt1iN85tXzs8BABJ48vTMWTrDoQHpnHmke5rBXxes6Tf5bSta8yKRBqe64f94Ya4ZGGKh6YZ9csG9sZFro3szhkjYPD2Smg5rLyd5bgigTxho82234QaL1izWsXRMehWA"}ratevalidator300.01670.0222
tn_worker_batch_requests_failed_total{worker="0",peer="z6P9RtGujezPz2Y2aSJcxJUdTx2CuUQ3fMHhs1xZfkX66BqjEPknJ9hwPhnoGstvNRjmkoQXMhnZu1vNKD8C3X2PEM1SVgSApd1J9VQD1sLgMjTxk1uUzHVUyVZWuA7TJyG"}ratevalidator1000
tn_node_epoch_record_signersmaxvalidator108.381010
tn_node_epoch_record_signersmaxobserver18.651010
tn_node_epoch_record_missing_signers_total{signer="4SxMDsdwEfaWymUitMjsGtCJfdeY8Ey8a6vorue2GNvm"}ratevalidator10000
tn_node_epoch_record_missing_signers_total{signer="4SxMDsdwEfaWymUitMjsGtCJfdeY8Ey8a6vorue2GNvm"}rateobserver1000
tn_node_epoch_record_missing_signers_total{signer="4jfiSawn73qjdDrEEyUtwyHnyTb7PuSGhZLWKeW46nXy"}ratevalidator1000.01430.0191
tn_node_epoch_record_missing_signers_total{signer="4jfiSawn73qjdDrEEyUtwyHnyTb7PuSGhZLWKeW46nXy"}rateobserver1000
tn_node_epoch_record_missing_signers_total{signer="64SZkNxHc7bADaTYSLs2SMw6rL2GdC38AYmjY27P4dPr"}ratevalidator100.0007390.01670.0222
tn_node_epoch_record_missing_signers_total{signer="64SZkNxHc7bADaTYSLs2SMw6rL2GdC38AYmjY27P4dPr"}rateobserver10.000520.01530.0204
tn_node_epoch_record_missing_signers_total{signer="86iS1yBLyqxNBpBK2kUb7QSpw5gdRBqFzD9ZvbNZVHWp"}ratevalidator100.0003280.01630.0217
tn_node_epoch_record_missing_signers_total{signer="86iS1yBLyqxNBpBK2kUb7QSpw5gdRBqFzD9ZvbNZVHWp"}rateobserver10.000520.01530.0204
tn_node_epoch_record_missing_signers_total{signer="AWgztxuoJShXittjoeL8thGdDWzywN1dg4yrJcofdweq"}ratevalidator10000
tn_node_epoch_record_missing_signers_total{signer="AWgztxuoJShXittjoeL8thGdDWzywN1dg4yrJcofdweq"}rateobserver1000
tn_node_epoch_record_missing_signers_total{signer="AfAVgJWz8RGmhhqfEgQdsuMZ9sTQuR1nvecqaxgmwezx"}ratevalidator10000
tn_node_epoch_record_missing_signers_total{signer="AfAVgJWz8RGmhhqfEgQdsuMZ9sTQuR1nvecqaxgmwezx"}rateobserver1000
tn_node_epoch_record_missing_signers_total{signer="BUNbjcD7XDj59dPfHZzebLkPbrcnmVk2s9V9RiZiNEtA"}ratevalidator10000
tn_node_epoch_record_missing_signers_total{signer="BUNbjcD7XDj59dPfHZzebLkPbrcnmVk2s9V9RiZiNEtA"}rateobserver1000
tn_node_epoch_record_missing_signers_total{signer="GrMgdCJ8xiqgeg14d81g28EbqAaPzukJZ1sE9NMBkcrL"}ratevalidator1000.01430.0191
tn_node_epoch_record_missing_signers_total{signer="GrMgdCJ8xiqgeg14d81g28EbqAaPzukJZ1sE9NMBkcrL"}rateobserver1000
tn_node_epoch_record_missing_signers_total{signer="H3mEWkRevyzSnYMcL6Z8M5AwDWYhUo1HR2bgFZzmKWSu"}ratevalidator10000
tn_node_epoch_record_missing_signers_total{signer="H3mEWkRevyzSnYMcL6Z8M5AwDWYhUo1HR2bgFZzmKWSu"}rateobserver1000
tn_node_epoch_record_missing_signers_total{signer="H4fbN7HsCDw74LeFeaSgzzfzwTTpKNcRcc4312uuD3D9"}ratevalidator100.0003280.01630.0217
tn_node_epoch_record_missing_signers_total{signer="H4fbN7HsCDw74LeFeaSgzzfzwTTpKNcRcc4312uuD3D9"}rateobserver10.000520.01530.0204
tn_network_outbound_request_duration_seconds{outcome="failure",class="batch"}p95validator611.6 s29.0 s29.0 s
tn_network_outbound_request_duration_seconds{outcome="failure",class="epoch_record"}p95validator36,300 ms29.0 s29.0 s
tn_network_outbound_request_duration_seconds{outcome="failure",class="vote"}p95validator28,996 ms29.0 s29.0 s
tn_network_outbound_request_duration_seconds{outcome="response",class="batch"}p95validator6454 ms929 ms931 ms
tn_network_outbound_request_duration_seconds{outcome="response",class="epoch_record"}p95validator5437 ms481 ms482 ms
tn_network_outbound_request_duration_seconds{outcome="response",class="epoch_record"}p95observer1236 ms241 ms241 ms
tn_network_outbound_request_duration_seconds{outcome="response",class="vote"}p95validator61,179 ms2,353 ms2,356 ms
tn_network_closed_admission_denied_total{network="bench",direction="in"}ratevalidator10000
tn_network_closed_admission_denied_total{network="bench",direction="in"}rateobserver1000
tn_network_closed_admission_denied_total{network="bench",direction="out"}ratevalidator10000
tn_network_closed_admission_denied_total{network="bench",direction="out"}rateobserver1000
tn_network_source_admission_denied_total{network="bench",direction="in",reason="address_full"}ratevalidator10000
tn_network_source_admission_denied_total{network="bench",direction="in",reason="address_full"}rateobserver1000
tn_network_source_admission_denied_total{network="bench",direction="in",reason="duplicate_connection"}ratevalidator10000
tn_network_source_admission_denied_total{network="bench",direction="in",reason="duplicate_connection"}rateobserver1000
tn_network_source_admission_denied_total{network="bench",direction="in",reason="invalid_limits"}ratevalidator10000
tn_network_source_admission_denied_total{network="bench",direction="in",reason="invalid_limits"}rateobserver1000
tn_network_source_admission_denied_total{network="bench",direction="in",reason="peer_full"}ratevalidator10000
tn_network_source_admission_denied_total{network="bench",direction="in",reason="peer_full"}rateobserver1000
tn_network_source_admission_denied_total{network="bench",direction="in",reason="poisoned"}ratevalidator10000
tn_network_source_admission_denied_total{network="bench",direction="in",reason="poisoned"}rateobserver1000
tn_network_source_admission_denied_total{network="bench",direction="in",reason="prefix_full"}ratevalidator10000
tn_network_source_admission_denied_total{network="bench",direction="in",reason="prefix_full"}rateobserver1000
tn_network_source_admission_denied_total{network="bench",direction="in",reason="process_full"}ratevalidator10000
tn_network_source_admission_denied_total{network="bench",direction="in",reason="process_full"}rateobserver1000
tn_network_source_admission_denied_total{network="bench",direction="in",reason="sources_full"}ratevalidator10000
tn_network_source_admission_denied_total{network="bench",direction="in",reason="sources_full"}rateobserver1000
tn_network_source_admission_denied_total{network="bench",direction="in",reason="unsupported_address"}ratevalidator10000
tn_network_source_admission_denied_total{network="bench",direction="in",reason="unsupported_address"}rateobserver1000
tn_network_source_admission_denied_total{network="bench",direction="out",reason="address_full"}ratevalidator10000
tn_network_source_admission_denied_total{network="bench",direction="out",reason="address_full"}rateobserver1000
tn_network_source_admission_denied_total{network="bench",direction="out",reason="duplicate_connection"}ratevalidator10000
tn_network_source_admission_denied_total{network="bench",direction="out",reason="duplicate_connection"}rateobserver1000
tn_network_source_admission_denied_total{network="bench",direction="out",reason="invalid_limits"}ratevalidator10000
tn_network_source_admission_denied_total{network="bench",direction="out",reason="invalid_limits"}rateobserver1000
tn_network_source_admission_denied_total{network="bench",direction="out",reason="peer_full"}ratevalidator10000
tn_network_source_admission_denied_total{network="bench",direction="out",reason="peer_full"}rateobserver1000
tn_network_source_admission_denied_total{network="bench",direction="out",reason="poisoned"}ratevalidator10000
tn_network_source_admission_denied_total{network="bench",direction="out",reason="poisoned"}rateobserver1000
tn_network_source_admission_denied_total{network="bench",direction="out",reason="prefix_full"}ratevalidator10000
tn_network_source_admission_denied_total{network="bench",direction="out",reason="prefix_full"}rateobserver1000
tn_network_source_admission_denied_total{network="bench",direction="out",reason="process_full"}ratevalidator10000
tn_network_source_admission_denied_total{network="bench",direction="out",reason="process_full"}rateobserver1000
tn_network_source_admission_denied_total{network="bench",direction="out",reason="sources_full"}ratevalidator10000
tn_network_source_admission_denied_total{network="bench",direction="out",reason="sources_full"}rateobserver1000
tn_network_source_admission_denied_total{network="bench",direction="out",reason="unsupported_address"}ratevalidator10000
tn_network_source_admission_denied_total{network="bench",direction="out",reason="unsupported_address"}rateobserver1000
tn_network_admission_modemaxvalidator10222
tn_network_admission_modemaxobserver1222
tn_network_admission_fallbackmaxvalidator10011
tn_network_admission_fallbackmaxobserver1000
tn_network_outbound_request_failures_total{kind="connection"}ratevalidator10000
tn_network_outbound_request_failures_total{kind="connection"}rateobserver1000
tn_network_outbound_request_failures_total{kind="dial"}ratevalidator1025.8107129
tn_network_outbound_request_failures_total{kind="dial"}rateobserver1000
tn_network_outbound_request_failures_total{kind="io"}ratevalidator1010.758.464.4
tn_network_outbound_request_failures_total{kind="io"}rateobserver10.3120.9781.27
tn_network_outbound_request_failures_total{kind="timeout"}ratevalidator101.164.465.49
tn_network_outbound_request_failures_total{kind="timeout"}rateobserver10.002690.02130.0222
tn_network_outbound_request_failures_total{kind="unsupported"}ratevalidator10000
tn_network_outbound_request_failures_total{kind="unsupported"}rateobserver1000
tn_network_inbound_requests_shed_total{class="batch",reason="admission"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="batch",reason="admission"}rateobserver1000
tn_network_inbound_requests_shed_total{class="batch",reason="queue_budget"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="batch",reason="queue_budget"}rateobserver1000
tn_network_inbound_requests_shed_total{class="batch",reason="queue_full"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="batch",reason="queue_full"}rateobserver1000
tn_network_inbound_requests_shed_total{class="batch",reason="serve_budget"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="batch",reason="serve_budget"}rateobserver1000
tn_network_inbound_requests_shed_total{class="batch",reason="unsubscribed"}ratevalidator100.013334430
tn_network_inbound_requests_shed_total{class="batch",reason="unsubscribed"}rateobserver1000
tn_network_inbound_requests_shed_total{class="certificate_sync",reason="admission"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="certificate_sync",reason="admission"}rateobserver1000
tn_network_inbound_requests_shed_total{class="certificate_sync",reason="queue_budget"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="certificate_sync",reason="queue_budget"}rateobserver1000
tn_network_inbound_requests_shed_total{class="certificate_sync",reason="queue_full"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="certificate_sync",reason="queue_full"}rateobserver1000
tn_network_inbound_requests_shed_total{class="certificate_sync",reason="serve_budget"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="certificate_sync",reason="serve_budget"}rateobserver1000
tn_network_inbound_requests_shed_total{class="certificate_sync",reason="unsubscribed"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="certificate_sync",reason="unsubscribed"}rateobserver1000
tn_network_inbound_requests_shed_total{class="epoch_record",reason="admission"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="epoch_record",reason="admission"}rateobserver1000
tn_network_inbound_requests_shed_total{class="epoch_record",reason="queue_budget"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="epoch_record",reason="queue_budget"}rateobserver1000
tn_network_inbound_requests_shed_total{class="epoch_record",reason="queue_full"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="epoch_record",reason="queue_full"}rateobserver1000
tn_network_inbound_requests_shed_total{class="epoch_record",reason="serve_budget"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="epoch_record",reason="serve_budget"}rateobserver1000
tn_network_inbound_requests_shed_total{class="epoch_record",reason="unsubscribed"}ratevalidator100.3971.312
tn_network_inbound_requests_shed_total{class="epoch_record",reason="unsubscribed"}rateobserver10.1720.5360.695
tn_network_inbound_requests_shed_total{class="gossip",reason="admission"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="gossip",reason="admission"}rateobserver1000
tn_network_inbound_requests_shed_total{class="gossip",reason="queue_budget"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="gossip",reason="queue_budget"}rateobserver1000
tn_network_inbound_requests_shed_total{class="gossip",reason="queue_full"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="gossip",reason="queue_full"}rateobserver1000
tn_network_inbound_requests_shed_total{class="gossip",reason="serve_budget"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="gossip",reason="serve_budget"}rateobserver1000
tn_network_inbound_requests_shed_total{class="gossip",reason="unsubscribed"}ratevalidator1000.08330.111
tn_network_inbound_requests_shed_total{class="gossip",reason="unsubscribed"}rateobserver10.04940.2040.204
tn_network_inbound_requests_shed_total{class="other",reason="admission"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="other",reason="admission"}rateobserver1000
tn_network_inbound_requests_shed_total{class="other",reason="queue_budget"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="other",reason="queue_budget"}rateobserver1000
tn_network_inbound_requests_shed_total{class="other",reason="queue_full"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="other",reason="queue_full"}rateobserver1000
tn_network_inbound_requests_shed_total{class="other",reason="serve_budget"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="other",reason="serve_budget"}rateobserver1000
tn_network_inbound_requests_shed_total{class="other",reason="unsubscribed"}ratevalidator100.07730.4860.595
tn_network_inbound_requests_shed_total{class="other",reason="unsubscribed"}rateobserver1000
tn_network_inbound_requests_shed_total{class="vote",reason="admission"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="vote",reason="admission"}rateobserver1000
tn_network_inbound_requests_shed_total{class="vote",reason="queue_budget"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="vote",reason="queue_budget"}rateobserver1000
tn_network_inbound_requests_shed_total{class="vote",reason="queue_full"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="vote",reason="queue_full"}rateobserver1000
tn_network_inbound_requests_shed_total{class="vote",reason="serve_budget"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="vote",reason="serve_budget"}rateobserver1000
tn_network_inbound_requests_shed_total{class="vote",reason="unsubscribed"}ratevalidator100.255138212
tn_network_inbound_requests_shed_total{class="vote",reason="unsubscribed"}rateobserver1000
tn_network_inbound_request_service_seconds{class="batch"}p95validator699 ms350 ms350 ms
tn_network_inbound_request_service_seconds{class="epoch_record"}p95validator75 ms134 ms262 ms
tn_network_inbound_request_service_seconds{class="epoch_record"}p95observer15 ms5 ms5 ms
tn_network_inbound_request_service_seconds{class="vote"}p95validator7299 ms2,344 ms2,469 ms
tn_network_established_connectionsmaxvalidator1019.82121
tn_network_established_connectionsmaxobserver119.122.523
tn_worker_sync_streams_shed_total{reason="denied"}ratevalidator1000
tn_worker_batch_fetch_duration_secondsp95validator106 ms12 ms13 ms
tn_executor_batch_fetch_failures_totalratevalidator10000
tn_executor_batch_fetch_failures_totalrateobserver1000
tn_primary_votes_received_totalratevalidator1013.917.217.2
tn_primary_votes_received_totalrateobserver1000
tn_primary_vote_request_failures_totalratevalidator100.0070.6060.756
tn_primary_vote_request_failures_totalrateobserver1000
tn_primary_certificates_formed_totalratevalidator102.282.862.87
tn_primary_certificates_formed_totalrateobserver1000
tn_primary_certificate_form_duration_secondsp95validator10483 ms1,599 ms2,050 ms
tn_engine_outputs_executed_totalratevalidator101.121.471.49
tn_engine_outputs_executed_totalrateobserver11.121.461.53
tn_engine_persist_failures_totalratevalidator10000
tn_engine_persist_failures_totalrateobserver1000
tn_node_consensus_sync_distancemaxvalidator108.99576578
tn_node_consensus_sync_distancemaxobserver12.495.759
reth_db_table_size{table="AccountsHistory"}maxvalidator1067 MiB105 MiB106 MiB
reth_db_table_size{table="AccountsHistory"}maxobserver167 MiB105 MiB105 MiB
reth_db_table_size{table="AccountsTrie"}maxvalidator106.9 MiB11 MiB11 MiB
reth_db_table_size{table="AccountsTrie"}maxobserver16.9 MiB11 MiB11 MiB
reth_db_table_size{table="BlockBodyIndices"}maxvalidator101.4 MiB2.3 MiB2.3 MiB
reth_db_table_size{table="BlockBodyIndices"}maxobserver11.4 MiB2.3 MiB2.3 MiB
reth_db_table_size{table="HashedAccounts"}maxvalidator1043 MiB72 MiB73 MiB
reth_db_table_size{table="HashedAccounts"}maxobserver143 MiB72 MiB73 MiB
reth_db_table_size{table="HashedStorages"}maxvalidator1044 MiB46 MiB46 MiB
reth_db_table_size{table="HashedStorages"}maxobserver144 MiB46 MiB46 MiB
reth_db_table_size{table="HeaderNumbers"}maxvalidator104.3 MiB6.9 MiB7.0 MiB
reth_db_table_size{table="HeaderNumbers"}maxobserver14.3 MiB6.9 MiB7.0 MiB
reth_db_table_size{table="StoragesHistory"}maxvalidator10118 MiB147 MiB148 MiB
reth_db_table_size{table="StoragesHistory"}maxobserver1118 MiB147 MiB148 MiB
reth_db_table_size{table="StoragesTrie"}maxvalidator1017 MiB17 MiB17 MiB
reth_db_table_size{table="StoragesTrie"}maxobserver117 MiB17 MiB17 MiB
reth_db_table_size{table="TransactionBlocks"}maxvalidator101.5 MiB2.5 MiB2.5 MiB
reth_db_table_size{table="TransactionBlocks"}maxobserver11.5 MiB2.5 MiB2.5 MiB
reth_db_table_size{table="TransactionHashNumbers"}maxvalidator101.28 GiB2.00 GiB2.02 GiB
reth_db_table_size{table="TransactionHashNumbers"}maxobserver11.28 GiB2.00 GiB2.01 GiB
reth_db_freelistmaxvalidator1080.9K109K113K
reth_db_freelistmaxobserver180.6K108K112K
tn_engine_recover_secondsp95validator10292 ms955 ms969 ms
tn_engine_recover_secondsp95observer1257 ms550 ms871 ms
tn_engine_evm_secondsp95validator10446 ms2,184 ms2,184 ms
tn_engine_evm_secondsp95observer1393 ms694 ms765 ms
tn_engine_root_secondsp95validator9224 ms895 ms895 ms
tn_engine_root_secondsp95observer1188 ms291 ms353 ms
tn_engine_finish_secondsp95validator10369 ms708 ms825 ms
tn_engine_finish_secondsp95observer1235 ms392 ms432 ms
tn_engine_forward_wait_secondsp95validator102,939 ms30.0 s30.0 s
tn_engine_forward_wait_secondsp95observer11 ms3 ms5 ms
tn_engine_channel_depthmaxvalidator101.551011
tn_engine_channel_depthmaxobserver1000
tn_engine_executed_transactions_totalratevalidator109.46K21.1K21.1K
tn_engine_executed_transactions_totalrateobserver19.44K20.6K20.6K
tn_executor_execution_wait_secondsp95validator10237 ms9,256 ms9,368 ms
tn_primary_commit_run_ahead_blocked_totalratevalidator10000
tn_primary_commit_run_ahead_blocked_totalrateobserver1000
tn_primary_consensus_exec_lag_roundsmaxvalidator1018.740.544
tn_primary_consensus_exec_lag_roundsmaxobserver14.781118
tn_primary_commit_lag_roundsmaxvalidator102.163.757
tn_primary_commit_lag_roundsmaxobserver1000
tn_primary_rejoin_grace_holds_totalratevalidator10000
tn_primary_rejoin_grace_holds_totalrateobserver1000
tn_primary_votes_refused_inactive_totalratevalidator10032.632.7
tn_primary_votes_refused_inactive_totalrateobserver1000
tn_primary_votes_refused_execution_behind_totalratevalidator100.04311.091.29
tn_primary_votes_refused_execution_behind_totalrateobserver1000
tn_batch_builder_pool_age_secondsp95validator105,817 ms30.0 s30.0 s
tn_batch_builder_builds_total{reason="bytes_full"}ratevalidator10000
tn_batch_builder_builds_total{reason="bytes_full"}rateobserver1000
tn_batch_builder_builds_total{reason="drained"}ratevalidator101.392.532.53
tn_batch_builder_builds_total{reason="drained"}rateobserver1000
tn_batch_builder_builds_total{reason="gas_full"}ratevalidator102.285.295.58
tn_batch_builder_builds_total{reason="gas_full"}rateobserver1000
tn_worker_batch_report_secondsp95validator90 ms3 ms3 ms
tn_primary_digest_wait_secondsp95validator101,449 ms30.0 s30.0 s
tn_primary_headers_proposed_total{reason="max_delay"}ratevalidator100.0420.1780.2
tn_primary_headers_proposed_total{reason="min_delay"}ratevalidator102.222.862.87
tn_primary_headers_proposed_total{reason="parents_and_threshold"}ratevalidator100.09652.192.36
tn_primary_digests_requeued_total{cause="canceled"}ratevalidator100.1535.087
tn_primary_digests_requeued_total{cause="committed"}ratevalidator100.01570.8611.29
tn_executor_consensus_pack_persist_secondsp95validator1025 ms54 ms69 ms
tn_batch_builder_seal_duration_secondsp95validator10325 ms950 ms950 ms
tn_worker_quorum_wait_secondsp95validator10318 ms947 ms947 ms
tn_primary_proposer_pending_digestsmaxvalidator101.4419.229
tn_primary_proposer_pending_digestsmaxobserver1000
reth_transaction_pool_pending_pool_transactionsmaxvalidator101.12K5.19K9.9K
reth_transaction_pool_pending_pool_transactionsmaxobserver1000
reth_transaction_pool_queued_pool_transactionsmaxvalidator102742.51K2.68K
reth_transaction_pool_queued_pool_transactionsmaxobserver1000
reth_transaction_pool_all_transactions_by_hashmaxvalidator101.95K9.6K9.9K
reth_transaction_pool_all_transactions_by_hashmaxobserver1000
reth_storage_providers_database_save_blocks_totalp95validator10233 ms13.5 s17.5 s
reth_storage_providers_database_save_blocks_totalp95observer1203 ms277 ms310 ms
reth_storage_providers_database_save_blocks_mdbxp95validator10233 ms13.5 s17.5 s
reth_storage_providers_database_save_blocks_mdbxp95observer1203 ms277 ms310 ms
reth_storage_providers_database_save_blocks_sfp95validator957 ms87 ms110 ms
reth_storage_providers_database_save_blocks_sfp95observer146 ms66 ms73 ms
reth_storage_providers_database_save_blocks_commit_mdbxp95validator10465 ms944 ms962 ms
reth_storage_providers_database_save_blocks_commit_mdbxp95observer1499 ms792 ms882 ms
reth_storage_providers_database_save_blocks_commit_sfp95validator101 ms1 ms1 ms
reth_storage_providers_database_save_blocks_commit_sfp95observer10 ms1 ms1 ms
reth_storage_providers_database_save_blocks_write_statep95validator100 ms1 ms1 ms
reth_storage_providers_database_save_blocks_write_statep95observer10 ms0 ms0 ms
reth_storage_providers_database_save_blocks_write_hashed_statep95validator1027 ms1,470 ms1,662 ms
reth_storage_providers_database_save_blocks_write_hashed_statep95observer123 ms30 ms35 ms
reth_storage_providers_database_save_blocks_write_trie_updatesp95validator1038 ms963 ms1,002 ms
reth_storage_providers_database_save_blocks_write_trie_updatesp95observer126 ms40 ms45 ms
tn_node_switch{name="digest_order_hold"}maxvalidator10111
tn_node_switch{name="digest_order_hold"}maxobserver1111
tn_node_switch{name="epoch_record_collect_past_uncertified"}maxvalidator10111
tn_node_switch{name="epoch_record_collect_past_uncertified"}maxobserver1111
tn_node_switch{name="epoch_record_conflict_halt"}maxvalidator10111
tn_node_switch{name="epoch_record_conflict_halt"}maxobserver1111
tn_node_switch{name="epoch_record_recertify"}maxvalidator10111
tn_node_switch{name="epoch_record_recertify"}maxobserver1111
tn_node_switch{name="quorum_exec_block"}maxvalidator10111
tn_node_switch{name="quorum_exec_block"}maxobserver1111
tn_node_switch{name="quorum_exec_block_starved_rounds"}maxvalidator10000
tn_node_switch{name="quorum_exec_block_starved_rounds"}maxobserver1000
tn_node_switch{name="vote_queue_fix"}maxvalidator10111
tn_node_switch{name="vote_queue_fix"}maxobserver1111
tn_primary_epoch_entry_rounds_behindmaxvalidator102.981919
tn_primary_epoch_entry_rounds_behindmaxobserver1000
reth_transaction_pool_pending_pool_size_bytesmaxvalidator10399.1 KiB1.8 MiB3.4 MiB
reth_transaction_pool_pending_pool_size_bytesmaxobserver10 B0 B0 B
reth_transaction_pool_queued_pool_size_bytesmaxvalidator1097.8 KiB899.5 KiB953.0 KiB
reth_transaction_pool_queued_pool_size_bytesmaxobserver10 B0 B0 B
reth_transaction_pool_pending_transactions_evictedmaxvalidator10000
reth_transaction_pool_pending_transactions_evictedmaxobserver1000
reth_transaction_pool_queued_transactions_evictedmaxvalidator10000
reth_transaction_pool_queued_transactions_evictedmaxobserver1000
reth_transaction_pool_basefee_transactions_evictedmaxvalidator10000
reth_transaction_pool_basefee_transactions_evictedmaxobserver1000
reth_transaction_pool_removed_transactionsmaxvalidator101.85M4.28M4.29M
reth_transaction_pool_removed_transactionsmaxobserver1000
reth_transaction_pool_invalid_transactionsmaxvalidator10000
reth_transaction_pool_invalid_transactionsmaxobserver1000
tn_engine_recovery_fallback_totalmaxvalidator10000
tn_engine_recovery_fallback_totalmaxobserver1000

Commit latency p95

tn_primary_commit_latency_seconds

Round lag

tn_primary_round − tn_primary_committed_round

Engine queued outputs

tn_engine_queued_outputs, max by node

Certification by node

The share of each validator's own headers that formed a certificate, per window (tools/cert-rate.py on the node logs). bench-validator-06 and bench-validator-08 certified fewer than 90 % of their own headers in a window where they proposed at least 50. Own leader % is the validator's share of the committed leaders and not committed % the share of its certificates no committed leader reached, both over the run window; the fair share is 100 / 10 validators = 10.0 %. Greyed: fewer than 50 headers proposed, not judged.

ValidatorRegionblast 0 preambleblast 0 warmupblast 0 mixedblast 0 cooldownblast 1 preambleblast 1 warmupblast 1 mixedblast 1 cooldownblast 2 preambleblast 2 warmupblast 2 mixedblast 2 cooldownrunOwn leader %Not committed %
bench-validator-01us-west2 (Los Angeles)99.3100.0100.099.5100.1100.0100.099.998.9100.0100.0100.0100.012.80.1
bench-validator-02europe-west2 (London)99.299.099.7100.599.699.699.899.7100.099.098.7100.099.12.246.9
bench-validator-03australia-southeast1 (Sydney)99.3100.3100.199.5100.1100.1100.0100.0100.099.898.2100.099.112.60.1
bench-validator-04asia-northeast1 (Tokyo)99.3100.3100.0100.0100.1100.0100.099.998.999.895.696.398.012.80.1
bench-validator-05us-east4 (Ashburn)100.899.7100.099.5100.1100.0100.099.9100.0100.0100.0–100.010.50.1
bench-validator-06 below 90 %europe-west4 (Netherlands)99.299.099.7100.099.299.199.499.4100.0100.083.6100.092.11.651.7
bench-validator-07asia-southeast1 (Singapore)100.0100.3100.099.5100.099.9100.0100.098.999.899.9100.0100.013.30.1
bench-validator-08 below 90 %asia-east2 (Hong Kong)99.399.7100.099.5100.0100.0100.099.998.999.585.596.393.411.90.3
bench-validator-09asia-northeast3 (Seoul)99.3100.0100.099.5100.199.9100.0100.198.9100.099.5100.099.713.70.1
bench-validator-10asia-south1 (Mumbai)100.099.7100.099.5100.1100.0100.0100.098.999.897.798.898.911.62.6

On-node sampler

NodeClassPeak RssAnonPeak RssFilePeak cachePSI memPSI ioPSI cpudb growthstatic_files growthRoot usedOOM Δ
bench-observer-11observer1.01 GiB4.93 GiB– 0.0 %7.7 %2.1 %3.04 GiB9.43 GiB22.0 %0
bench-validator-01validator1.77 GiB4.89 GiB– 0.0 %8.1 %11.5 %3.28 GiB9.43 GiB24.0 %0
bench-validator-02validator1.89 GiB4.90 GiB– 0.0 %10.2 %10.2 %3.28 GiB9.43 GiB24.0 %0
bench-validator-03validator1.61 GiB5.06 GiB– 0.0 %6.7 %15.3 %3.28 GiB9.43 GiB25.0 %0
bench-validator-04validator1.71 GiB4.99 GiB– 0.0 %10.8 %18.4 %3.28 GiB9.43 GiB25.0 %0
bench-validator-05validator2.51 GiB4.83 GiB– 0.0 %14.6 %16.1 %3.56 GiB9.17 GiB24.0 %0
bench-validator-06validator1.84 GiB4.89 GiB– 0.0 %8.2 %10.4 %3.28 GiB9.43 GiB24.0 %0
bench-validator-07validator2.30 GiB4.90 GiB– 0.0 %9.9 %14.5 %3.28 GiB9.43 GiB25.0 %0
bench-validator-08validator1.69 GiB4.93 GiB– 0.0 %7.6 %16.2 %3.28 GiB9.43 GiB25.0 %0
bench-validator-09validator1.65 GiB4.90 GiB– 0.0 %8.2 %12.6 %3.28 GiB9.43 GiB25.0 %0
bench-validator-10validator1.69 GiB4.96 GiB– 0.0 %12.7 %9.4 %3.28 GiB9.43 GiB24.0 %0

RssAnon vs RssFile

/proc/<pid>/status of the telcoin process on the node with the largest RSS

consensus-db/cache size

Not collected: consensus-db/cache size.

Pressure stall (some, avg10)

/proc/pressure on the node with the highest memory PSI

Headroom

Per role and per node, from sizing.json (p95 over the landed window; PSI and RssAnon max over the whole sampler run). The lines follow the hardware doc's tier rules: CPU p95 over 70 % of vCPUs; memory used p95 over 75 % of RAM, or telcoin RSS p95 over 85 % of RAM; disk write p95 over 60 % of the PD MiB/s limit, or write IOPS p95 over 60 % of the IOPS limit; IO pressure over 20 % or memory pressure over 5 % (PSI some avg10, max); network p95 over 50 % of 1 Gbps; root disk over 80 %. A line crossed on any node puts the role at the limit; a nonzero reading within 15 points of its line is tight; otherwise the role has headroom. OOM kills, disk throttling and degraded disk performance are flagged too.

SignalClassWorst nodeValueShare of capacityLineNodes over / tightStatus
CPU p95validatorbench-validator-0455.8 %55.8 %> 70 %0 / 1 tight
Memory used p95validatorbench-validator-0520.8 %20.8 %> 75 %0 / 0 headroom
telcoin RSS p95validatorbench-validator-077.59 GB44.2 %> 85 %0 / 0 headroom
PSI memory maxvalidatorbench-validator-010.0 %0.0 %> 5 %0 / 0 headroom
RssAnon peakvalidatorbench-validator-052.72 GB15.8 %info– info
Disk write p95validatorbench-validator-04301 MB/s35.8 %> 60 %0 / 0 headroom
Write IOPS p95validatorbench-validator-0219,92499.6 %> 60 %10 / 0 at the limit
PSI io maxvalidatorbench-validator-0514.6 %14.6 %> 20 %0 / 10 tight
Root disk maxvalidatorbench-validator-0324.1 %24.1 %> 80 %0 / 0 headroom
Network p95validatorbench-validator-07204 Mbps20.4 %> 50 %0 / 0 headroom
Consensus cache peakvalidator–––info– not collected
OOM killsvalidator––anyany– none
Disk throttlingvalidator––metric type unavailable in projectany– not collected
Disk performance degradedvalidator––any sampleany– none
CPU p95observerbench-observer-1130.2 %30.2 %> 70 %0 / 0 headroom
Memory used p95observerbench-observer-1111.9 %11.9 %> 75 %0 / 0 headroom
telcoin RSS p95observerbench-observer-116.27 GB36.5 %> 85 %0 / 0 headroom
PSI memory maxobserverbench-observer-110.0 %0.0 %> 5 %0 / 0 headroom
RssAnon peakobserverbench-observer-111.08 GB6.3 %info– info
Disk write p95observerbench-observer-11288 MB/s34.3 %> 60 %0 / 0 headroom
Write IOPS p95observerbench-observer-1118,19491.0 %> 60 %1 / 0 at the limit
PSI io maxobserverbench-observer-117.7 %7.6 %> 20 %0 / 1 tight
Root disk maxobserverbench-observer-1121.7 %21.7 %> 80 %0 / 0 headroom
Network p95observerbench-observer-119 Mbps0.9 %> 50 %0 / 0 headroom
Consensus cache peakobserver–––info– not collected
OOM killsobserver––anyany– none
Disk throttlingobserver––metric type unavailable in projectany– not collected
Disk performance degradedobserver––any sampleany– none

Per node

NodeClassCPU p95Memory used p95telcoin RSS p95PSI memory maxRssAnon peakDisk write p95Write IOPS p95PSI io maxRoot disk maxNetwork p95Consensus cache peak
Line> 70 %> 75 %> 85 %> 5 %info> 60 %> 60 %> 20 %> 80 %> 50 %info
bench-validator-01validator52.6 %16.7 %7.09 GB 41 %0.0 %1.90 GB 11 %293 MB/s 35 %19,540 98 %8.1 %23.8 %185 Mbps 18 %–
bench-validator-02validator53.5 %17.3 %7.16 GB 42 %0.0 %2.03 GB 12 %290 MB/s 35 %19,924 100 %10.2 %23.8 %184 Mbps 18 %–
bench-validator-03validator54.7 %15.6 %6.86 GB 40 %0.0 %1.73 GB 10 %297 MB/s 35 %18,789 94 %6.7 %24.1 %189 Mbps 19 %–
bench-validator-04validator55.8 %16.2 %7.01 GB 41 %0.0 %1.84 GB 11 %301 MB/s 36 %19,420 97 %10.8 %23.9 %190 Mbps 19 %–
bench-validator-05validator52.3 %20.8 %7.38 GB 43 %0.0 %2.72 GB 16 %287 MB/s 34 %19,352 97 %14.6 %23.0 %180 Mbps 18 %–
bench-validator-06validator52.1 %17.1 %7.14 GB 42 %0.0 %1.98 GB 12 %295 MB/s 35 %19,651 98 %8.2 %23.8 %188 Mbps 19 %–
bench-validator-07validator52.5 %19.5 %7.59 GB 44 %0.0 %2.47 GB 14 %292 MB/s 35 %19,860 99 %9.9 %23.9 %204 Mbps 20 %–
bench-validator-08validator54.5 %16.1 %6.97 GB 41 %0.0 %1.82 GB 11 %298 MB/s 36 %19,071 95 %7.6 %24.0 %187 Mbps 19 %–
bench-validator-09validator54.4 %15.8 %6.88 GB 40 %0.0 %1.77 GB 10 %293 MB/s 35 %19,659 98 %8.2 %23.9 %192 Mbps 19 %–
bench-validator-10validator51.8 %16.1 %6.94 GB 40 %0.0 %1.82 GB 11 %295 MB/s 35 %19,877 99 %12.7 %23.7 %182 Mbps 18 %–
bench-observer-11observer30.2 %11.9 %6.27 GB 36 %0.0 %1.08 GB 6 %288 MB/s 34 %18,194 91 %7.7 %21.7 %9 Mbps 1 %–

Small figures are the share of capacity: RAM for RSS, the PD limit for disk write (VM cap applied), 1 Gbps for network. Hover a cell for its JSON path.

Verdict

Fleet A vs Fleet B

A = Fleet c3-hdb · c3-highcpu-8 · r20261115-1131
B = Fleet c3-hdb · c3-highcpu-8 · r20261113-1351

Verdict

at the limit c3-highcpu-8 validators at 11,921 sustained TPS: at the limit on disk. Disk at the limit: write IOPS p95 18,789–19,924, up to 99.6 % of the 20,000 IOPS limit, over the 60 % line on all 10 validators (bench-validator-01, 02, 03, 04, 05, 06, 07, 08, 09, 10); IO pressure (PSI some avg10) max 14.6 % (bench-validator-05), tight against the 20 % line. CPU tight: CPU p95 55.8 % of 8 vCPUs (bench-validator-04), tight against the 70 % line. Memory has headroom: memory used p95 20.8 % of RAM (bench-validator-05); telcoin RSS p95 7.59 GB, 44.2 % of 16 GiB RAM (bench-validator-07); memory pressure (PSI some avg10) max 0.0 % (bench-validator-01). Network has headroom: network p95 204 Mbps, 20.4 % of 1 Gbps (bench-validator-07). Next step: a faster volume (pd-ssd adds 30 IOPS and 0.48 MiB/s per GB up to the per-VM cap; past the cap, a larger VM).

at the limit c3-highcpu-8 observers at 11,921 sustained TPS: at the limit on disk. Disk at the limit: write IOPS p95 18,194, 91.0 % of the 20,000 IOPS limit on bench-observer-11, over the 60 % line; IO pressure (PSI some avg10) max 7.7 % (bench-observer-11), tight against the 20 % line. CPU has headroom: CPU p95 30.2 % of 8 vCPUs (bench-observer-11). Memory has headroom: memory used p95 11.9 % of RAM (bench-observer-11); telcoin RSS p95 6.27 GB, 36.5 % of 16 GiB RAM (bench-observer-11); memory pressure (PSI some avg10) max 0.0 % (bench-observer-11). Network has headroom: network p95 9 Mbps, 0.9 % of 1 Gbps (bench-observer-11). Next step: a faster volume (pd-ssd adds 30 IOPS and 0.48 MiB/s per GB up to the per-VM cap; past the cap, a larger VM).

at the limit c3-highcpu-8 validators at 8,065 sustained TPS: at the limit on disk. Disk at the limit: write IOPS p95 20,050–20,146, up to 100.7 % of the 20,000 IOPS limit, over the 60 % line on all 10 validators (bench-validator-01, 02, 03, 04, 05, 06, 07, 08, 09, 10); IO pressure (PSI some avg10) max 14.9 % (bench-validator-08), tight against the 20 % line. CPU has headroom: CPU p95 36.0 % of 8 vCPUs (bench-validator-03). Memory has headroom: memory used p95 22.7 % of RAM (bench-validator-06); telcoin RSS p95 6.87 GB, 40.0 % of 16 GiB RAM (bench-validator-06); memory pressure (PSI some avg10) max 0.0 % (bench-validator-01). Network has headroom: network p95 130 Mbps, 13.0 % of 1 Gbps (bench-validator-07). Next step: a faster volume (pd-ssd adds 30 IOPS and 0.48 MiB/s per GB up to the per-VM cap; past the cap, a larger VM).

at the limit c3-highcpu-8 observers at 8,065 sustained TPS: at the limit on disk. Disk at the limit: write IOPS p95 20,071, 100.4 % of the 20,000 IOPS limit on bench-observer-11, over the 60 % line; IO pressure (PSI some avg10) max 13.7 % (bench-observer-11), tight against the 20 % line. CPU has headroom: CPU p95 20.4 % of 8 vCPUs (bench-observer-11). Memory has headroom: memory used p95 12.4 % of RAM (bench-observer-11); telcoin RSS p95 4.54 GB, 26.4 % of 16 GiB RAM (bench-observer-11); memory pressure (PSI some avg10) max 0.0 % (bench-observer-11). Network has headroom: network p95 12 Mbps, 1.2 % of 1 Gbps (bench-observer-11). Next step: a faster volume (pd-ssd adds 30 IOPS and 0.48 MiB/s per GB up to the per-VM cap; past the cap, a larger VM).

Validators: Fleet A (c3-highcpu-8) is at the limit on disk, tight on CPU, with headroom on memory and network; Fleet B (c3-highcpu-8) is at the limit on disk, with headroom on CPU, memory and network. The same disk limit binds both fleets: every validator runs a 100 GB hyperdisk-balanced, which allows 20,000 IOPS on both; its throughput limit is 800 MiB/s on both. Disk is at the limit on both (Fleet A (c3-highcpu-8): write IOPS p95 19,924, 99.6 % of the 20,000 IOPS limit; Fleet B (c3-highcpu-8): write IOPS p95 20,146, 100.7 % of the 20,000 IOPS limit). Changing the machine shape did not clear it; the volume is the binding resource.

Observers: Fleet A (c3-highcpu-8) is at the limit on disk, with headroom on CPU, memory and network; Fleet B (c3-highcpu-8) is at the limit on disk, with headroom on CPU, memory and network. The same disk limit binds both fleets: every observer runs a 100 GB hyperdisk-balanced, which allows 20,000 IOPS on both; its throughput limit is 800 MiB/s on both. Disk is at the limit on both (Fleet A (c3-highcpu-8): write IOPS p95 18,194, 91.0 % of the 20,000 IOPS limit; Fleet B (c3-highcpu-8): write IOPS p95 20,071, 100.4 % of the 20,000 IOPS limit). Changing the machine shape did not clear it; the volume is the binding resource.

KPIFleet AFleet BB ÷ A
Sustained mixed TPS (own)11,921.4 tx/s8,065.4 tx/s0.68×
Sustained mixed TPS (chain)11,921.4 tx/s8,065.4 tx/s0.68×
Landed / accepted100.0 %100.0 %1.00×
Reverted (own)00–
Latency p501,479 ms679 ms0.46×
Latency p957,763 ms14,662 ms1.89×
Gas / s (own)466M gas/s269M gas/s0.58×
Capacity at mix19,007.7 tx/s14,103.0 tx/s0.74×

Headroom, worst node per class

SignalClassFleet AFleet BLine
CPU p95validator55.8 % tight36.0 % headroom> 70 %
Memory used p95validator20.8 % headroom22.7 % headroom> 75 %
telcoin RSS p95validator7.59 GB 44.2 % headroom6.87 GB 40.0 % headroom> 85 %
PSI memory maxvalidator0.0 % headroom0.0 % headroom> 5 %
Disk write p95validator301 MB/s 35.8 % headroom110 MB/s 13.1 % headroom> 60 %
Write IOPS p95validator19,924 99.6 % at the limit20,146 100.7 % at the limit> 60 %
PSI io maxvalidator14.6 % tight14.9 % tight> 20 %
Root disk maxvalidator24.1 % headroom17.5 % headroom> 80 %
Network p95validator204 Mbps 20.4 % headroom130 Mbps 13.0 % headroom> 50 %
CPU p95observer30.2 % headroom20.4 % headroom> 70 %
Memory used p95observer11.9 % headroom12.4 % headroom> 75 %
telcoin RSS p95observer6.27 GB 36.5 % headroom4.54 GB 26.4 % headroom> 85 %
PSI memory maxobserver0.0 % headroom0.0 % headroom> 5 %
Disk write p95observer288 MB/s 34.3 % headroom97 MB/s 11.5 % headroom> 60 %
Write IOPS p95observer18,194 91.0 % at the limit20,071 100.4 % at the limit> 60 %
PSI io maxobserver7.7 % tight13.7 % tight> 20 %
Root disk maxobserver21.7 % headroom15.9 % headroom> 80 %
Network p95observer9 Mbps 0.9 % headroom12 Mbps 1.2 % headroom> 50 %

Own TPS, trailing 60-s mean

aligned on each fleet's mixed-phase start

CPU, hottest validator

aligned on each fleet's mixed-phase start

Memory used, hottest validator

aligned on each fleet's mixed-phase start

telcoin RSS, hottest validator

aligned on each fleet's mixed-phase start

Disk MiB/s, hottest validator

aligned on each fleet's mixed-phase start

Network sent, hottest validator

aligned on each fleet's mixed-phase start

Reference chains

Not collected: reference chains.