This is the full report of the run “The adaptive vote fix and the order hold on tn-lmdb at 20,000 IOPS”. Back to the run page.
r20261115-1653 · Fleet c3-hdb · c3-highcpu-8

Telcoin Network benchmark: mixed workload

Throughput is measured from blocks the chain produced (own senders vs every sender), resource use from Cloud Monitoring, Prometheus and the on-node sampler. Summaries cover the mixed phase. Hover any number for the JSON field it came from.

10 blaster(s), 6K senders, chain 32286, schema tn-blast-combined/v2 · mixed phase 2026-11-15 18:25–19:05 UTC · 1 observers, 10 validators

Preflight: WTEL from genesis (0x0000…37e1), v3 fee tier 3000, v4 available, Multicall3 present · Cooldown ended: all_landed ×9, idle_30s ×1

Headline numbers

Sustained mixed TPS (own)
12,593.5 tx/s
best 60-s mean of our own landed tx in the mixed phase
Sustained mixed TPS (chain)
12,593.5 tx/s
all tx in blocks, same window
Landed / accepted
100.0 %
25.1M of 25.1M accepted
Reverted (own)
0
0.00 % of landed
Latency p50
1,662 ms
submit → block timestamp; seen p50 8,617 ms
Latency p95
8,771 ms
seen p95 24,177 ms
Gas / s (own)
471M gas/s
chain 471M gas/s
Capacity at mix
18,696.2 tx/s
gas-limit basis 12,134.9 tx/s; block gas limit 30M

Per transaction type

Landed counts only our own senders; latency is submit → block timestamp.

TypeSubmittedAcceptedLandedRevertedUnlandedSkipsGas limitGas used avgp50p95
native_transfer7.44M7.34M7.34M0480024.1K21K1,654 ms8,698 ms
batched_self_send3.87M3.79M3.79M0311024.1K21K1,706 ms9,148 ms
uniswap_v2_swap3.48M3.41M3.41M03430180K106K1,656 ms8,713 ms
stablecoin_transfer2.9M2.84M2.84M0264075K43.9K1,654 ms8,648 ms
wtel_wrap2.31M2.27M2.27M0220050K27.9K1,655 ms8,635 ms
wtel_unwrap1.54M1.51M1.51M0140045K35.2K1,655 ms8,704 ms
stablecoin_approve1.16M1.14M1.14M0113060K31.8K1,654 ms8,650 ms
uniswap_v2_add_liquidity773K758K758K0820250K128K1,658 ms8,693 ms
uniswap_v4_swap580K568K568K0660250K124K1,655 ms8,806 ms
contract_deploy574K562K562K0430140K74.2K1,654 ms8,583 ms
uniswap_v2_remove_liquidity389K381K381K0370220K129K1,654 ms8,618 ms
uniswap_v3_swap386K379K379K0380220K120K1,657 ms8,683 ms
faucet_drip126K126K126K00839K150K107K1,299 ms9,779 ms

Submission by RPC node

NodeSubmittedAcceptedAccepted %Top errorsSend p50Send p95
bench-validator-01
bench-validator-01:8545
2.75M2.75M100.0 %none7 ms28 ms
bench-validator-02
bench-validator-02:8545
2.02M2.02M100.0 %none11 ms43 ms
bench-validator-03
bench-validator-03:8545
2.9M2.9M100.0 %none8 ms27 ms
bench-validator-04
bench-validator-04:8545
1.88M1.88M100.0 %none8 ms35 ms
bench-validator-05
bench-validator-05:8545
3.1M1.74M56.0 %other 1.37M3 ms30 ms
bench-validator-06
bench-validator-06:8545
1.89M1.89M100.0 %none16 ms49 ms
bench-validator-07
bench-validator-07:8545
2.95M2.95M100.0 %none9 ms33 ms
bench-validator-08
bench-validator-08:8545
3.13M3.13M100.0 %none8 ms29 ms
bench-validator-09
bench-validator-09:8545
2.87M2.87M100.0 %none6 ms23 ms
bench-validator-10
bench-validator-10:8545
3.01M3.01M100.0 %none9 ms38 ms

Throughput over time

Shaded: phases outside the mixed window. Hairlines: epoch closes. Dashed orange: chaos events.

Landed transactions per second

own senders vs chain total, per block-timestamp second

Gas per second

own vs chain gas used, per block-timestamp second

CPU and memory per node

CPU utilization

compute.googleapis.com/instance/cpu/utilization, 60-s mean, % of all vCPUs; summaries over the mixed phase

Table view
NodeClassMeanp95Peak 60-s meanMax
bench-observer-11observer27.8 %32.1 %35.2 %35.2 %
bench-validator-01validator47.9 %56.2 %70.2 %70.2 %
bench-validator-02validator48.1 %53.6 %69.1 %69.1 %
bench-validator-03validator48.8 %56.5 %69.8 %69.8 %
bench-validator-04validator46.7 %53.8 %70.2 %70.2 %
bench-validator-05validator39.6 %56.3 %70.1 %70.1 %
bench-validator-06validator47.7 %55.7 %66.7 %66.7 %
bench-validator-07validator49.5 %55.1 %72.7 %72.7 %
bench-validator-08validator49.4 %54.7 %71.0 %71.0 %
bench-validator-09validator48.6 %53.3 %71.6 %71.6 %
bench-validator-10validator49.4 %55.4 %70.2 %70.2 %

CPU: mean vs peak 60-s mean

One row per node, over the mixed phase.

telcoin resident memory

agent processes/rss_usage summed over telcoin pids; summaries over the mixed phase

Table view
NodeClassMeanp95Peak 60-s meanMax
bench-observer-11observer4.49 GiB5.94 GiB6.01 GiB6.01 GiB
bench-validator-01validator5.61 GiB6.83 GiB6.93 GiB6.93 GiB
bench-validator-02validator5.54 GiB6.78 GiB6.92 GiB6.92 GiB
bench-validator-03validator5.56 GiB6.76 GiB6.83 GiB6.83 GiB
bench-validator-04validator5.71 GiB7.09 GiB7.25 GiB7.25 GiB
bench-validator-05validator4.90 GiB7.29 GiB7.33 GiB7.33 GiB
bench-validator-06validator5.53 GiB6.71 GiB6.83 GiB6.83 GiB
bench-validator-07validator5.46 GiB6.65 GiB6.71 GiB6.71 GiB
bench-validator-08validator5.47 GiB6.68 GiB6.80 GiB6.80 GiB
bench-validator-09validator5.49 GiB6.72 GiB6.74 GiB6.74 GiB
bench-validator-10validator5.54 GiB6.64 GiB6.72 GiB6.72 GiB

telcoin RSS: mean vs peak 60-s mean

One row per node, over the mixed phase.

Memory used

agent memory/percent_used, state=used; summaries over the mixed phase

Table view
NodeClassMeanp95Peak 60-s meanMax
bench-observer-11observer10.5 %11.4 %11.6 %11.6 %
bench-validator-01validator17.1 %18.1 %18.2 %18.2 %
bench-validator-02validator16.7 %17.8 %17.9 %17.9 %
bench-validator-03validator16.8 %17.5 %17.6 %17.6 %
bench-validator-04validator17.8 %19.9 %20.3 %20.3 %
bench-validator-05validator15.4 %21.0 %21.4 %21.4 %
bench-validator-06validator16.6 %17.4 %17.5 %17.5 %
bench-validator-07validator16.2 %16.9 %17.0 %17.0 %
bench-validator-08validator16.1 %17.0 %17.2 %17.2 %
bench-validator-09validator16.3 %17.2 %17.3 %17.3 %
bench-validator-10validator16.6 %17.0 %17.2 %17.2 %

Memory used: mean vs peak 60-s mean

One row per node, over the mixed phase.

Disk

Disk throttling (validator)
not collected –
metric type unavailable in project
Disk performance degraded (validator)
none –
any sample

Disk read + write throughput

instance/disk/{read,write}_bytes_count rate, all devices; summaries over the mixed phase

Table view
NodeClassMeanp95Peak 60-s meanMaxLimit
bench-observer-11observer241.2 MiB/s289.4 MiB/s304.5 MiB/s304.5 MiB/s800.0 MiB/s
bench-validator-01validator237.9 MiB/s282.7 MiB/s291.4 MiB/s291.4 MiB/s800.0 MiB/s
bench-validator-02validator216.4 MiB/s283.5 MiB/s284.1 MiB/s284.1 MiB/s800.0 MiB/s
bench-validator-03validator214.9 MiB/s277.1 MiB/s279.9 MiB/s279.9 MiB/s800.0 MiB/s
bench-validator-04validator244.8 MiB/s282.7 MiB/s283.4 MiB/s283.4 MiB/s800.0 MiB/s
bench-validator-05validator166.8 MiB/s278.1 MiB/s283.1 MiB/s283.1 MiB/s800.0 MiB/s
bench-validator-06validator231.9 MiB/s286.0 MiB/s294.1 MiB/s294.1 MiB/s800.0 MiB/s
bench-validator-07validator211.0 MiB/s279.3 MiB/s285.2 MiB/s285.2 MiB/s800.0 MiB/s
bench-validator-08validator225.2 MiB/s284.2 MiB/s290.2 MiB/s290.2 MiB/s800.0 MiB/s
bench-validator-09validator223.6 MiB/s278.4 MiB/s281.1 MiB/s281.1 MiB/s800.0 MiB/s
bench-validator-10validator209.7 MiB/s275.5 MiB/s280.3 MiB/s280.3 MiB/s800.0 MiB/s

Disk read + write IOPS

instance/disk/{read,write}_ops_count rate, all devices; summaries over the mixed phase

Table view
NodeClassMeanp95Peak 60-s meanMaxLimit
bench-observer-11observer15.9K IOPS17.7K IOPS18K IOPS18K IOPS20K IOPS
bench-validator-01validator16.5K IOPS19.4K IOPS19.8K IOPS19.8K IOPS20K IOPS
bench-validator-02validator17.5K IOPS19.8K IOPS20.3K IOPS20.3K IOPS20K IOPS
bench-validator-03validator17.4K IOPS20K IOPS20.2K IOPS20.2K IOPS20K IOPS
bench-validator-04validator16K IOPS18.6K IOPS20.1K IOPS20.1K IOPS20K IOPS
bench-validator-05validator14.6K IOPS20K IOPS20.1K IOPS20.1K IOPS20K IOPS
bench-validator-06validator16.8K IOPS19.7K IOPS19.9K IOPS19.9K IOPS20K IOPS
bench-validator-07validator17.7K IOPS20K IOPS20.1K IOPS20.1K IOPS20K IOPS
bench-validator-08validator17.1K IOPS19.6K IOPS20K IOPS20K IOPS20K IOPS
bench-validator-09validator17.2K IOPS19.1K IOPS19.4K IOPS19.4K IOPS20K IOPS
bench-validator-10validator17.7K IOPS19.7K IOPS19.9K IOPS19.9K IOPS20K IOPS
Disk limits used
NodeMachinevCPUDiskIOPS limitMiB/s limitBasis
bench-observer-11c3-highcpu-88hyperdisk-balanced 100 GB20K IOPS800.0 MiB/shyperdisk-balanced 100 GB: provisioned 20,000 IOPS, provisioned 800 MiB/s; c3 8 vCPU cap 50,000 IOPS / 800 MiB/s; reads and writes share this budget
bench-validator-01c3-highcpu-88hyperdisk-balanced 100 GB20K IOPS800.0 MiB/shyperdisk-balanced 100 GB: provisioned 20,000 IOPS, provisioned 800 MiB/s; c3 8 vCPU cap 50,000 IOPS / 800 MiB/s; reads and writes share this budget
bench-validator-02c3-highcpu-88hyperdisk-balanced 100 GB20K IOPS800.0 MiB/shyperdisk-balanced 100 GB: provisioned 20,000 IOPS, provisioned 800 MiB/s; c3 8 vCPU cap 50,000 IOPS / 800 MiB/s; reads and writes share this budget
bench-validator-03c3-highcpu-88hyperdisk-balanced 100 GB20K IOPS800.0 MiB/shyperdisk-balanced 100 GB: provisioned 20,000 IOPS, provisioned 800 MiB/s; c3 8 vCPU cap 50,000 IOPS / 800 MiB/s; reads and writes share this budget
bench-validator-04c3-highcpu-88hyperdisk-balanced 100 GB20K IOPS800.0 MiB/shyperdisk-balanced 100 GB: provisioned 20,000 IOPS, provisioned 800 MiB/s; c3 8 vCPU cap 50,000 IOPS / 800 MiB/s; reads and writes share this budget
bench-validator-05c3-highcpu-88hyperdisk-balanced 100 GB20K IOPS800.0 MiB/shyperdisk-balanced 100 GB: provisioned 20,000 IOPS, provisioned 800 MiB/s; c3 8 vCPU cap 50,000 IOPS / 800 MiB/s; reads and writes share this budget
bench-validator-06c3-highcpu-88hyperdisk-balanced 100 GB20K IOPS800.0 MiB/shyperdisk-balanced 100 GB: provisioned 20,000 IOPS, provisioned 800 MiB/s; c3 8 vCPU cap 50,000 IOPS / 800 MiB/s; reads and writes share this budget
bench-validator-07c3-highcpu-88hyperdisk-balanced 100 GB20K IOPS800.0 MiB/shyperdisk-balanced 100 GB: provisioned 20,000 IOPS, provisioned 800 MiB/s; c3 8 vCPU cap 50,000 IOPS / 800 MiB/s; reads and writes share this budget
bench-validator-08c3-highcpu-88hyperdisk-balanced 100 GB20K IOPS800.0 MiB/shyperdisk-balanced 100 GB: provisioned 20,000 IOPS, provisioned 800 MiB/s; c3 8 vCPU cap 50,000 IOPS / 800 MiB/s; reads and writes share this budget
bench-validator-09c3-highcpu-88hyperdisk-balanced 100 GB20K IOPS800.0 MiB/shyperdisk-balanced 100 GB: provisioned 20,000 IOPS, provisioned 800 MiB/s; c3 8 vCPU cap 50,000 IOPS / 800 MiB/s; reads and writes share this budget
bench-validator-10c3-highcpu-88hyperdisk-balanced 100 GB20K IOPS800.0 MiB/shyperdisk-balanced 100 GB: provisioned 20,000 IOPS, provisioned 800 MiB/s; c3 8 vCPU cap 50,000 IOPS / 800 MiB/s; reads and writes share this budget

Network

Network sent

instance/network/sent_bytes_count rate; summaries over the mixed phase

Table view
NodeClassMeanp95Peak 60-s meanMax
bench-observer-11observer1.4 Mbps1.7 Mbps1.8 Mbps1.8 Mbps
bench-validator-01validator166.8 Mbps195.1 Mbps221.2 Mbps221.2 Mbps
bench-validator-02validator167.9 Mbps196.4 Mbps271.2 Mbps271.2 Mbps
bench-validator-03validator167.4 Mbps196.1 Mbps207.3 Mbps207.3 Mbps
bench-validator-04validator162.1 Mbps189.8 Mbps210.1 Mbps210.1 Mbps
bench-validator-05validator137.6 Mbps195.0 Mbps210.6 Mbps210.6 Mbps
bench-validator-06validator164.1 Mbps189.2 Mbps199.3 Mbps199.3 Mbps
bench-validator-07validator168.4 Mbps187.5 Mbps205.9 Mbps205.9 Mbps
bench-validator-08validator167.9 Mbps185.0 Mbps206.6 Mbps206.6 Mbps
bench-validator-09validator168.7 Mbps193.8 Mbps210.2 Mbps210.2 Mbps
bench-validator-10validator169.2 Mbps190.6 Mbps205.3 Mbps205.3 Mbps

Network received

instance/network/received_bytes_count rate; summaries over the mixed phase

Table view
NodeClassMeanp95Peak 60-s meanMax
bench-observer-11observer8.5 Mbps9.9 Mbps17.2 Mbps17.2 Mbps
bench-validator-01validator15.5 Mbps20.1 Mbps29.1 Mbps29.1 Mbps
bench-validator-02validator14.3 Mbps15.5 Mbps26.2 Mbps26.2 Mbps
bench-validator-03validator16.1 Mbps19.8 Mbps27.2 Mbps27.2 Mbps
bench-validator-04validator14.0 Mbps18.4 Mbps29.1 Mbps29.1 Mbps
bench-validator-05validator22.9 Mbps48.7 Mbps149.6 Mbps149.6 Mbps
bench-validator-06validator14.4 Mbps16.8 Mbps25.6 Mbps25.6 Mbps
bench-validator-07validator16.0 Mbps17.7 Mbps28.8 Mbps28.8 Mbps
bench-validator-08validator16.4 Mbps18.0 Mbps30.3 Mbps30.3 Mbps
bench-validator-09validator15.9 Mbps18.3 Mbps30.9 Mbps30.9 Mbps
bench-validator-10validator16.3 Mbps18.3 Mbps28.5 Mbps28.5 Mbps

Consensus health

MetricQueryClassNodesMedian of node meansWorst peak 60-s meanMax
tn_primary_commit_latency_secondsp95validator91,402 ms18.0 s25.3 s
tn_engine_execution_duration_secondsp95validator92,184 ms29.0 s29.0 s
tn_engine_execution_duration_secondsp95observer11,965 ms2,425 ms2,562 ms
tn_worker_batch_size_bytesp95validator9240.8 KiB246.4 KiB246.4 KiB
tn_worker_batch_transactionsp95validator95001.48K1.48K
tn_executor_output_transactionsp95validator921.8K37.5K37.5K
tn_primary_consensus_output_bytesmaxvalidator10784.8 KiB2.3 MiB3.0 MiB
tn_primary_consensus_output_bytesmaxobserver1773.2 KiB1.6 MiB2.5 MiB
tn_engine_queued_outputsmaxvalidator105.9588
tn_engine_queued_outputsmaxobserver11.864.257
tn_primary_roundmaxvalidator101.36K2.92K2.93K
tn_primary_roundmaxobserver11.34K2.86K2.91K
tn_primary_committed_roundmaxvalidator101.36K2.92K2.93K
tn_primary_committed_roundmaxobserver11.34K2.86K2.91K
tn_epoch_currentmaxvalidator104.515.56
tn_epoch_currentmaxobserver14.525.56
tn_batch_builder_pending_pool_transactionsmaxvalidator109536.17K9.89K
tn_batch_builder_pending_pool_transactionsmaxobserver1000
reth_process_resident_memory_bytesmaxvalidator105.57 GiB7.51 GiB7.59 GiB
reth_process_resident_memory_bytesmaxobserver14.51 GiB6.02 GiB6.07 GiB
tn_primary_subdags_committed_totalratevalidator101.091.41.4
tn_primary_subdags_committed_totalrateobserver1000
tn_primary_round_laglagvalidator102.243.55
tn_primary_round_laglagobserver1000
tn_batch_builder_seal_failures_totalratevalidator100.0005520.01670.0222
tn_node_modemaxvalidator10011
tn_node_modemaxobserver1000
tn_primary_vote_requests_totalratevalidator1019.131.231.6
tn_primary_vote_requests_totalrateobserver1000
tn_primary_votes_created_totalratevalidator1017.825.125.2
tn_primary_votes_created_totalrateobserver1000
tn_primary_votes_recast_totalratevalidator10000
tn_primary_votes_recast_totalrateobserver1000
tn_primary_vote_requests_too_new_totalratevalidator10000
tn_primary_vote_requests_too_new_totalrateobserver1000
tn_primary_vote_timeouts_total{stage="exec_wait"}ratevalidator10000
tn_primary_vote_timeouts_total{stage="exec_wait"}rateobserver1000
tn_primary_vote_timeouts_total{stage="lead_sleep"}ratevalidator10000
tn_primary_vote_timeouts_total{stage="lead_sleep"}rateobserver1000
tn_primary_vote_timeouts_total{stage="parents"}ratevalidator10000
tn_primary_vote_timeouts_total{stage="parents"}rateobserver1000
tn_primary_vote_timeouts_total{stage="persist"}ratevalidator10000
tn_primary_vote_timeouts_total{stage="persist"}rateobserver1000
tn_primary_vote_timeouts_total{stage="prelock_wait"}ratevalidator10000
tn_primary_vote_timeouts_total{stage="prelock_wait"}rateobserver1000
tn_primary_vote_timeouts_total{stage="sync_batches"}ratevalidator10000
tn_primary_vote_timeouts_total{stage="sync_batches"}rateobserver1000
tn_primary_vote_timeouts_total{stage="validate"}ratevalidator10000
tn_primary_vote_timeouts_total{stage="validate"}rateobserver1000
tn_primary_vote_timeouts_total{stage="vote_store"}ratevalidator10000
tn_primary_vote_timeouts_total{stage="vote_store"}rateobserver1000
tn_primary_vote_stage_seconds{stage="exec_wait"}p95validator100 ms0 ms0 ms
tn_primary_vote_stage_seconds{stage="lead_sleep"}p95validator100 ms0 ms0 ms
tn_primary_vote_stage_seconds{stage="lock"}p95validator100 ms0 ms0 ms
tn_primary_vote_stage_seconds{stage="parents"}p95validator101 ms2 ms2 ms
tn_primary_vote_stage_seconds{stage="persist"}p95validator1011 ms28 ms33 ms
tn_primary_vote_stage_seconds{stage="prelock_wait"}p95validator10484 ms2,425 ms2,425 ms
tn_primary_vote_stage_seconds{stage="sync_batches"}p95validator100 ms0 ms0 ms
tn_primary_vote_stage_seconds{stage="validate"}p95validator104 ms6 ms6 ms
tn_primary_vote_stage_seconds{stage="vote_store"}p95validator101 ms2 ms2 ms
tn_primary_leader_rounds_total{leader="2DCK5j4xmRh2S7BwQ5R6BST3UpcUJo2ubKzbTrwKX8Jh",swapped="false",found="false"}ratevalidator100.0003810.01660.0221
tn_primary_leader_rounds_total{leader="2DCK5j4xmRh2S7BwQ5R6BST3UpcUJo2ubKzbTrwKX8Jh",swapped="false",found="true"}ratevalidator100.09620.150.156
tn_primary_leader_rounds_total{leader="2DCK5j4xmRh2S7BwQ5R6BST3UpcUJo2ubKzbTrwKX8Jh",swapped="true",found="false"}ratevalidator10000
tn_primary_leader_rounds_total{leader="2DCK5j4xmRh2S7BwQ5R6BST3UpcUJo2ubKzbTrwKX8Jh",swapped="true",found="true"}ratevalidator100.02020.07220.0889
tn_primary_leader_rounds_total{leader="2i2dxF6uefXyS87MLN8Hic8wMJ64po5hGwGiQjsEJWnW",swapped="false",found="false"}ratevalidator100.005440.1060.111
tn_primary_leader_rounds_total{leader="2i2dxF6uefXyS87MLN8Hic8wMJ64po5hGwGiQjsEJWnW",swapped="false",found="true"}ratevalidator100.04680.150.156
tn_primary_leader_rounds_total{leader="2i2dxF6uefXyS87MLN8Hic8wMJ64po5hGwGiQjsEJWnW",swapped="true",found="false"}ratevalidator100.001180.04430.0664
tn_primary_leader_rounds_total{leader="2i2dxF6uefXyS87MLN8Hic8wMJ64po5hGwGiQjsEJWnW",swapped="true",found="true"}ratevalidator100.004550.06110.0667
tn_primary_leader_rounds_total{leader="6azyk4NmEfHamXEY9kQK3cnNH3W3rL8R14sN1ixzSgsg",swapped="false",found="false"}ratevalidator100.0007950.01670.0222
tn_primary_leader_rounds_total{leader="6azyk4NmEfHamXEY9kQK3cnNH3W3rL8R14sN1ixzSgsg",swapped="false",found="true"}ratevalidator100.1110.150.156
tn_primary_leader_rounds_total{leader="6azyk4NmEfHamXEY9kQK3cnNH3W3rL8R14sN1ixzSgsg",swapped="true",found="false"}ratevalidator100.0003830.01660.0221
tn_primary_leader_rounds_total{leader="6azyk4NmEfHamXEY9kQK3cnNH3W3rL8R14sN1ixzSgsg",swapped="true",found="true"}ratevalidator100.03980.1330.178
tn_primary_leader_rounds_total{leader="8h6kX97Cs66wqtsswGym86GkuRbR8fWnKkwtZn5VeJzs",swapped="false",found="false"}ratevalidator10000
tn_primary_leader_rounds_total{leader="8h6kX97Cs66wqtsswGym86GkuRbR8fWnKkwtZn5VeJzs",swapped="false",found="true"}ratevalidator100.004550.10.133
tn_primary_leader_rounds_total{leader="8h6kX97Cs66wqtsswGym86GkuRbR8fWnKkwtZn5VeJzs",swapped="true",found="false"}ratevalidator10000
tn_primary_leader_rounds_total{leader="8h6kX97Cs66wqtsswGym86GkuRbR8fWnKkwtZn5VeJzs",swapped="true",found="true"}ratevalidator100.0007730.03320.0443
tn_primary_leader_rounds_total{leader="9REQ4Cn46SwZffUpbaUZo7xcEszesF325wDVY5QhYQQP",swapped="false",found="false"}ratevalidator100.00190.01670.0222
tn_primary_leader_rounds_total{leader="9REQ4Cn46SwZffUpbaUZo7xcEszesF325wDVY5QhYQQP",swapped="false",found="true"}ratevalidator100.1090.150.156
tn_primary_leader_rounds_total{leader="9REQ4Cn46SwZffUpbaUZo7xcEszesF325wDVY5QhYQQP",swapped="true",found="false"}ratevalidator100.0003830.01660.0221
tn_primary_leader_rounds_total{leader="9REQ4Cn46SwZffUpbaUZo7xcEszesF325wDVY5QhYQQP",swapped="true",found="true"}ratevalidator100.03930.1040.111
tn_primary_leader_rounds_total{leader="BCUVdtwG3Xjwi99f6gXHSbC95G461jJsiSLVb1C7M3hy",swapped="false",found="false"}ratevalidator100.001210.02220.0222
tn_primary_leader_rounds_total{leader="BCUVdtwG3Xjwi99f6gXHSbC95G461jJsiSLVb1C7M3hy",swapped="false",found="true"}ratevalidator100.110.150.156
tn_primary_leader_rounds_total{leader="BCUVdtwG3Xjwi99f6gXHSbC95G461jJsiSLVb1C7M3hy",swapped="true",found="false"}ratevalidator100.0003810.01660.0221
tn_primary_leader_rounds_total{leader="BCUVdtwG3Xjwi99f6gXHSbC95G461jJsiSLVb1C7M3hy",swapped="true",found="true"}ratevalidator100.04470.1170.156
tn_primary_leader_rounds_total{leader="C2S8yukvLNoFHrZV89mQZJzsPZ2zQuiZg4dkwxwh88un",swapped="false",found="false"}ratevalidator100.0004120.01660.0222
tn_primary_leader_rounds_total{leader="C2S8yukvLNoFHrZV89mQZJzsPZ2zQuiZg4dkwxwh88un",swapped="false",found="true"}ratevalidator100.1080.150.156
tn_primary_leader_rounds_total{leader="C2S8yukvLNoFHrZV89mQZJzsPZ2zQuiZg4dkwxwh88un",swapped="true",found="false"}ratevalidator100.0003830.01660.0221
tn_primary_leader_rounds_total{leader="C2S8yukvLNoFHrZV89mQZJzsPZ2zQuiZg4dkwxwh88un",swapped="true",found="true"}ratevalidator100.03740.1280.156
tn_primary_leader_rounds_total{leader="EdzGQYtW4x9VTADCFHz6JURdVfwtwzAb4eF9HAs28Jsj",swapped="false",found="false"}ratevalidator100.0003810.01660.0221
tn_primary_leader_rounds_total{leader="EdzGQYtW4x9VTADCFHz6JURdVfwtwzAb4eF9HAs28Jsj",swapped="false",found="true"}ratevalidator100.03050.1170.133
tn_primary_leader_rounds_total{leader="EdzGQYtW4x9VTADCFHz6JURdVfwtwzAb4eF9HAs28Jsj",swapped="true",found="false"}ratevalidator100.0003810.01660.0221
tn_primary_leader_rounds_total{leader="EdzGQYtW4x9VTADCFHz6JURdVfwtwzAb4eF9HAs28Jsj",swapped="true",found="true"}ratevalidator100.00160.05450.0664
tn_primary_leader_rounds_total{leader="FK99Q69fo6Ge5PqybG2dqAVn7x4A2SZUiNaya8rgVSay",swapped="false",found="false"}ratevalidator100.001210.01670.0222
tn_primary_leader_rounds_total{leader="FK99Q69fo6Ge5PqybG2dqAVn7x4A2SZUiNaya8rgVSay",swapped="false",found="true"}ratevalidator100.110.150.156
tn_primary_leader_rounds_total{leader="FK99Q69fo6Ge5PqybG2dqAVn7x4A2SZUiNaya8rgVSay",swapped="true",found="false"}ratevalidator100.0003830.01660.0221
tn_primary_leader_rounds_total{leader="FK99Q69fo6Ge5PqybG2dqAVn7x4A2SZUiNaya8rgVSay",swapped="true",found="true"}ratevalidator100.04180.10.111
tn_primary_leader_rounds_total{leader="L5cHimQmhttFbobChPjTVUVUyBXQZP3XcohGburWJRu",swapped="false",found="false"}ratevalidator100.00410.04440.0667
tn_primary_leader_rounds_total{leader="L5cHimQmhttFbobChPjTVUVUyBXQZP3XcohGburWJRu",swapped="false",found="true"}ratevalidator100.1080.150.156
tn_primary_leader_rounds_total{leader="L5cHimQmhttFbobChPjTVUVUyBXQZP3XcohGburWJRu",swapped="true",found="false"}ratevalidator100.0003830.01660.025
tn_primary_leader_rounds_total{leader="L5cHimQmhttFbobChPjTVUVUyBXQZP3XcohGburWJRu",swapped="true",found="true"}ratevalidator100.0290.08890.111
tn_primary_leader_commits_total{leader="2DCK5j4xmRh2S7BwQ5R6BST3UpcUJo2ubKzbTrwKX8Jh"}ratevalidator100.1160.2060.222
tn_primary_leader_commits_total{leader="2i2dxF6uefXyS87MLN8Hic8wMJ64po5hGwGiQjsEJWnW"}ratevalidator100.05130.1890.222
tn_primary_leader_commits_total{leader="6azyk4NmEfHamXEY9kQK3cnNH3W3rL8R14sN1ixzSgsg"}ratevalidator100.150.2610.311
tn_primary_leader_commits_total{leader="8h6kX97Cs66wqtsswGym86GkuRbR8fWnKkwtZn5VeJzs"}ratevalidator100.005380.1330.178
tn_primary_leader_commits_total{leader="9REQ4Cn46SwZffUpbaUZo7xcEszesF325wDVY5QhYQQP"}ratevalidator100.1490.2370.244
tn_primary_leader_commits_total{leader="BCUVdtwG3Xjwi99f6gXHSbC95G461jJsiSLVb1C7M3hy"}ratevalidator100.1540.2440.267
tn_primary_leader_commits_total{leader="C2S8yukvLNoFHrZV89mQZJzsPZ2zQuiZg4dkwxwh88un"}ratevalidator100.1460.2330.267
tn_primary_leader_commits_total{leader="EdzGQYtW4x9VTADCFHz6JURdVfwtwzAb4eF9HAs28Jsj"}ratevalidator100.03190.150.178
tn_primary_leader_commits_total{leader="FK99Q69fo6Ge5PqybG2dqAVn7x4A2SZUiNaya8rgVSay"}ratevalidator100.1510.2280.244
tn_primary_leader_commits_total{leader="L5cHimQmhttFbobChPjTVUVUyBXQZP3XcohGburWJRu"}ratevalidator100.1370.2170.244
tn_engine_queue_wait_secondsp95validator108,159 ms30.0 s30.0 s
tn_engine_queue_wait_secondsp95observer12,816 ms6,074 ms6,875 ms
tn_engine_persist_duration_secondsp95validator9878 ms29.0 s29.0 s
tn_engine_persist_duration_secondsp95observer1832 ms1,457 ms1,731 ms
tn_engine_busy_milliseconds_totalratevalidator109821.09K1.25K
tn_engine_busy_milliseconds_totalrateobserver18801K1.01K
tn_reth_persist_save_blocks_secondsp95validator10299 ms29.0 s29.0 s
tn_reth_persist_save_blocks_secondsp95observer1259 ms402 ms425 ms
tn_reth_persist_commit_secondsp95validator10574 ms1,442 ms1,825 ms
tn_reth_persist_commit_secondsp95observer1592 ms905 ms927 ms
tn_reth_persisted_accounts_totalratevalidator104.23K10.2K10.2K
tn_reth_persisted_accounts_totalrateobserver14.2K9.53K9.53K
tn_reth_persisted_storage_slots_totalratevalidator105.51K7.13K7.31K
tn_reth_persisted_storage_slots_totalrateobserver15.48K7.24K7.45K
tn_executor_rejoin_blocked_total{reason="attempts_exhausted"}ratevalidator1000
tn_executor_rejoin_blocked_total{reason="epoch_closed"}ratevalidator10.04450.3110.378
tn_executor_rejoin_blocked_total{reason="epoch_mismatch"}ratevalidator1000
tn_executor_rejoin_blocked_total{reason="no_known_header"}ratevalidator1000
tn_executor_rejoin_blocked_total{reason="round_gap"}ratevalidator10.81.171.27
tn_executor_rejoin_blocked_total{reason="stale_head"}ratevalidator10.02880.10.2
tn_worker_sync_serves_denied_total{worker="0",tier="committee"}ratevalidator10000
tn_worker_sync_serves_denied_total{worker="0",tier="committee"}rateobserver1000
tn_worker_sync_serves_denied_total{worker="0",tier="public"}ratevalidator10000
tn_worker_sync_serves_denied_total{worker="0",tier="public"}rateobserver1000
tn_worker_batch_requests_denied_total{worker="0",peer="22nvxLQcXRosZbgj1jUJRLgnXYQ4QjBYM6n7XHLwMyG7dnEVTaygo62RNhY5qJaEwKhN596vWYq8BB53FUqGWN1TD2x48gWwupC4aLuGN53RuEjt8CJU8coxHAsjRHYKJwmG"}ratevalidator20.00840.3160.421
tn_worker_batch_requests_denied_total{worker="0",peer="23y7SCzzCBZqCPUUcY833pU1gF74pXrGturtYCDTaasP2KXfW699mJP2h6ULkwSQ2HTW7tKaLid3maDpEmevRAfrWrU1RA7naUwZQ5ky7eTxj33nDMfD2QNfMqh8fJR12uat"}ratevalidator20.009580.250.333
tn_worker_batch_requests_denied_total{worker="0",peer="26eqXGzLWGaemHqS4Hy3ageCGgJBqCut756XfFtJ8yyAJhu2J2FaMxPohzKuEmXsM9Rh4XUZvT7hvtjSEKeSuhtLya3itYYQqejCoKSona2V1PNRVm29ERyNazV2FDciPBCf"}ratevalidator600.1710.228
tn_worker_batch_requests_denied_total{worker="0",peer="nsxwdTeAJZEYe72jViRAc1Gv3S76NRD2g2t4RvntAHYbRxaHVB51zehNzMkhKpnujAYjGjwddz8scF3FaSqqSrYSan9NafuRtY139xVjDLiDAbH1njdUMTBYR4pNY4hbm5A"}ratevalidator20.001130.02220.0889
tn_worker_batch_requests_denied_total{worker="0",peer="oYq6NTTeRyZsGtQ9zi58WDY7Fu1t6o5shWTLu5CFmuJHH7vDt9p66pXAWUqxCtq93AH6mpAW7dJKE77JKQCg8DHGnUdQJnwNW3G6wvWD13jMxvZQCxLsqrEk6xbpuszHUuz"}ratevalidator2000
tn_worker_batch_requests_denied_total{worker="0",peer="otLeJWfYvzUbns4hYikGKYoGz226KKiyAibo7YgrzZ75zRPHyUeR5gNML1JwfMmdKvp7JY7AFs5AVQhjLoUeyPB7b84WdSz49rADMoCxLEQ9JdyxGvWwhZtAidJHa4rKVZa"}ratevalidator20.001410.02780.111
tn_worker_batch_requests_denied_total{worker="0",peer="sVf6rdVnb2JprGawcrHv651LauKHsNrU9YFyj8CfQQWuSBaCcxCZZZyNxH4Cd7jwYwiH4XgBZvGmzKdzrg3qchfPrUSEKFUronouRWo3w4NmZaV7PhsidfjX4vpbUJhT6hD"}ratevalidator20.005920.2110.281
tn_worker_batch_requests_denied_total{worker="0",peer="tAd3NTiH6dVTnnPBVQGACmdWMVFCWvU5RdpMWH9BuAinEi3pkhzLXdnqp2rcJiGjypmEdK3juQLF6FmdZmPYg31jEBvceUKKUNBTp5ozy8swv9VmZiZh8trCm7V29mSx1XY"}ratevalidator70.001240.05560.133
tn_worker_batch_requests_denied_total{worker="0",peer="uWh9aawJ2gNHzV7pSryUJX72r3fHe6F8iDB1kC5q7YXxQkjjkWb7KUWLWqBc8JgdXYAfEkrZ8sjDVMWkXDyWmKuCT61bkm8Lmrqvp2NrQ769ebTiaRGW7xHjARnvFWH8crA"}ratevalidator300.03330.133
tn_worker_batch_requests_denied_total{worker="0",peer="yPh8F4LCWcQjfyKZatWqrq5H7qwS5zyTDw7TAi2BdyZvMQegULXoQbwpkpRDSG37SxWBu2SFAbfLbvvdyouXsdoAHW5L9KMir3bNzPWi6tAWwudwbEsZmPCPgAadfPdHK84"}ratevalidator300.1970.263
tn_worker_batch_requests_failed_total{worker="0",peer="22nvxLQcXRosZbgj1jUJRLgnXYQ4QjBYM6n7XHLwMyG7dnEVTaygo62RNhY5qJaEwKhN596vWYq8BB53FUqGWN1TD2x48gWwupC4aLuGN53RuEjt8CJU8coxHAsjRHYKJwmG"}ratevalidator2000
tn_worker_batch_requests_failed_total{worker="0",peer="23y7SCzzCBZqCPUUcY833pU1gF74pXrGturtYCDTaasP2KXfW699mJP2h6ULkwSQ2HTW7tKaLid3maDpEmevRAfrWrU1RA7naUwZQ5ky7eTxj33nDMfD2QNfMqh8fJR12uat"}ratevalidator20.00210.01670.0222
tn_worker_batch_requests_failed_total{worker="0",peer="26eqXGzLWGaemHqS4Hy3ageCGgJBqCut756XfFtJ8yyAJhu2J2FaMxPohzKuEmXsM9Rh4XUZvT7hvtjSEKeSuhtLya3itYYQqejCoKSona2V1PNRVm29ERyNazV2FDciPBCf"}ratevalidator60.0004140.03890.0444
tn_worker_batch_requests_failed_total{worker="0",peer="nsxwdTeAJZEYe72jViRAc1Gv3S76NRD2g2t4RvntAHYbRxaHVB51zehNzMkhKpnujAYjGjwddz8scF3FaSqqSrYSan9NafuRtY139xVjDLiDAbH1njdUMTBYR4pNY4hbm5A"}ratevalidator2000
tn_worker_batch_requests_failed_total{worker="0",peer="oYq6NTTeRyZsGtQ9zi58WDY7Fu1t6o5shWTLu5CFmuJHH7vDt9p66pXAWUqxCtq93AH6mpAW7dJKE77JKQCg8DHGnUdQJnwNW3G6wvWD13jMxvZQCxLsqrEk6xbpuszHUuz"}ratevalidator2000
tn_worker_batch_requests_failed_total{worker="0",peer="otLeJWfYvzUbns4hYikGKYoGz226KKiyAibo7YgrzZ75zRPHyUeR5gNML1JwfMmdKvp7JY7AFs5AVQhjLoUeyPB7b84WdSz49rADMoCxLEQ9JdyxGvWwhZtAidJHa4rKVZa"}ratevalidator2000
tn_worker_batch_requests_failed_total{worker="0",peer="sVf6rdVnb2JprGawcrHv651LauKHsNrU9YFyj8CfQQWuSBaCcxCZZZyNxH4Cd7jwYwiH4XgBZvGmzKdzrg3qchfPrUSEKFUronouRWo3w4NmZaV7PhsidfjX4vpbUJhT6hD"}ratevalidator2000
tn_worker_batch_requests_failed_total{worker="0",peer="tAd3NTiH6dVTnnPBVQGACmdWMVFCWvU5RdpMWH9BuAinEi3pkhzLXdnqp2rcJiGjypmEdK3juQLF6FmdZmPYg31jEBvceUKKUNBTp5ozy8swv9VmZiZh8trCm7V29mSx1XY"}ratevalidator700.01670.0222
tn_worker_batch_requests_failed_total{worker="0",peer="uWh9aawJ2gNHzV7pSryUJX72r3fHe6F8iDB1kC5q7YXxQkjjkWb7KUWLWqBc8JgdXYAfEkrZ8sjDVMWkXDyWmKuCT61bkm8Lmrqvp2NrQ769ebTiaRGW7xHjARnvFWH8crA"}ratevalidator300.03330.0444
tn_worker_batch_requests_failed_total{worker="0",peer="yPh8F4LCWcQjfyKZatWqrq5H7qwS5zyTDw7TAi2BdyZvMQegULXoQbwpkpRDSG37SxWBu2SFAbfLbvvdyouXsdoAHW5L9KMir3bNzPWi6tAWwudwbEsZmPCPgAadfPdHK84"}ratevalidator3000
tn_node_epoch_record_signersmaxvalidator108.491010
tn_node_epoch_record_signersmaxobserver19.071010
tn_node_epoch_record_missing_signers_total{signer="2DCK5j4xmRh2S7BwQ5R6BST3UpcUJo2ubKzbTrwKX8Jh"}ratevalidator10000
tn_node_epoch_record_missing_signers_total{signer="2DCK5j4xmRh2S7BwQ5R6BST3UpcUJo2ubKzbTrwKX8Jh"}rateobserver1000
tn_node_epoch_record_missing_signers_total{signer="2i2dxF6uefXyS87MLN8Hic8wMJ64po5hGwGiQjsEJWnW"}ratevalidator100.0004730.01660.0222
tn_node_epoch_record_missing_signers_total{signer="2i2dxF6uefXyS87MLN8Hic8wMJ64po5hGwGiQjsEJWnW"}rateobserver10.0004960.01450.0193
tn_node_epoch_record_missing_signers_total{signer="6azyk4NmEfHamXEY9kQK3cnNH3W3rL8R14sN1ixzSgsg"}ratevalidator10000
tn_node_epoch_record_missing_signers_total{signer="6azyk4NmEfHamXEY9kQK3cnNH3W3rL8R14sN1ixzSgsg"}rateobserver1000
tn_node_epoch_record_missing_signers_total{signer="8h6kX97Cs66wqtsswGym86GkuRbR8fWnKkwtZn5VeJzs"}ratevalidator10000
tn_node_epoch_record_missing_signers_total{signer="8h6kX97Cs66wqtsswGym86GkuRbR8fWnKkwtZn5VeJzs"}rateobserver1000
tn_node_epoch_record_missing_signers_total{signer="9REQ4Cn46SwZffUpbaUZo7xcEszesF325wDVY5QhYQQP"}ratevalidator10000
tn_node_epoch_record_missing_signers_total{signer="9REQ4Cn46SwZffUpbaUZo7xcEszesF325wDVY5QhYQQP"}rateobserver1000
tn_node_epoch_record_missing_signers_total{signer="BCUVdtwG3Xjwi99f6gXHSbC95G461jJsiSLVb1C7M3hy"}ratevalidator100.0003350.01660.0221
tn_node_epoch_record_missing_signers_total{signer="BCUVdtwG3Xjwi99f6gXHSbC95G461jJsiSLVb1C7M3hy"}rateobserver10.0004960.01450.0193
tn_node_epoch_record_missing_signers_total{signer="C2S8yukvLNoFHrZV89mQZJzsPZ2zQuiZg4dkwxwh88un"}ratevalidator100.0004730.01660.0222
tn_node_epoch_record_missing_signers_total{signer="C2S8yukvLNoFHrZV89mQZJzsPZ2zQuiZg4dkwxwh88un"}rateobserver10.0004960.01450.0193
tn_node_epoch_record_missing_signers_total{signer="EdzGQYtW4x9VTADCFHz6JURdVfwtwzAb4eF9HAs28Jsj"}ratevalidator10000
tn_node_epoch_record_missing_signers_total{signer="EdzGQYtW4x9VTADCFHz6JURdVfwtwzAb4eF9HAs28Jsj"}rateobserver1000
tn_node_epoch_record_missing_signers_total{signer="FK99Q69fo6Ge5PqybG2dqAVn7x4A2SZUiNaya8rgVSay"}ratevalidator100.0001170.005420.0217
tn_node_epoch_record_missing_signers_total{signer="FK99Q69fo6Ge5PqybG2dqAVn7x4A2SZUiNaya8rgVSay"}rateobserver1000
tn_node_epoch_record_missing_signers_total{signer="L5cHimQmhttFbobChPjTVUVUyBXQZP3XcohGburWJRu"}ratevalidator10000
tn_node_epoch_record_missing_signers_total{signer="L5cHimQmhttFbobChPjTVUVUyBXQZP3XcohGburWJRu"}rateobserver1000
tn_network_outbound_request_duration_seconds{outcome="failure",class="batch"}p95validator413.3 s29.0 s29.0 s
tn_network_outbound_request_duration_seconds{outcome="failure",class="epoch_record"}p95validator48,218 ms29.0 s29.0 s
tn_network_outbound_request_duration_seconds{outcome="failure",class="epoch_record"}p95observer1242 ms242 ms242 ms
tn_network_outbound_request_duration_seconds{outcome="failure",class="vote"}p95validator610.1 s29.0 s29.0 s
tn_network_outbound_request_duration_seconds{outcome="response",class="batch"}p95validator5455 ms898 ms898 ms
tn_network_outbound_request_duration_seconds{outcome="response",class="epoch_record"}p95validator3372 ms479 ms479 ms
tn_network_outbound_request_duration_seconds{outcome="response",class="vote"}p95validator31,460 ms2,246 ms2,276 ms
tn_network_closed_admission_denied_total{network="bench",direction="in"}ratevalidator10000
tn_network_closed_admission_denied_total{network="bench",direction="in"}rateobserver1000
tn_network_closed_admission_denied_total{network="bench",direction="out"}ratevalidator10000
tn_network_closed_admission_denied_total{network="bench",direction="out"}rateobserver1000
tn_network_source_admission_denied_total{network="bench",direction="in",reason="address_full"}ratevalidator10000
tn_network_source_admission_denied_total{network="bench",direction="in",reason="address_full"}rateobserver1000
tn_network_source_admission_denied_total{network="bench",direction="in",reason="duplicate_connection"}ratevalidator10000
tn_network_source_admission_denied_total{network="bench",direction="in",reason="duplicate_connection"}rateobserver1000
tn_network_source_admission_denied_total{network="bench",direction="in",reason="invalid_limits"}ratevalidator10000
tn_network_source_admission_denied_total{network="bench",direction="in",reason="invalid_limits"}rateobserver1000
tn_network_source_admission_denied_total{network="bench",direction="in",reason="peer_full"}ratevalidator10000
tn_network_source_admission_denied_total{network="bench",direction="in",reason="peer_full"}rateobserver1000
tn_network_source_admission_denied_total{network="bench",direction="in",reason="poisoned"}ratevalidator10000
tn_network_source_admission_denied_total{network="bench",direction="in",reason="poisoned"}rateobserver1000
tn_network_source_admission_denied_total{network="bench",direction="in",reason="prefix_full"}ratevalidator10000
tn_network_source_admission_denied_total{network="bench",direction="in",reason="prefix_full"}rateobserver1000
tn_network_source_admission_denied_total{network="bench",direction="in",reason="process_full"}ratevalidator10000
tn_network_source_admission_denied_total{network="bench",direction="in",reason="process_full"}rateobserver1000
tn_network_source_admission_denied_total{network="bench",direction="in",reason="sources_full"}ratevalidator10000
tn_network_source_admission_denied_total{network="bench",direction="in",reason="sources_full"}rateobserver1000
tn_network_source_admission_denied_total{network="bench",direction="in",reason="unsupported_address"}ratevalidator10000
tn_network_source_admission_denied_total{network="bench",direction="in",reason="unsupported_address"}rateobserver1000
tn_network_source_admission_denied_total{network="bench",direction="out",reason="address_full"}ratevalidator10000
tn_network_source_admission_denied_total{network="bench",direction="out",reason="address_full"}rateobserver1000
tn_network_source_admission_denied_total{network="bench",direction="out",reason="duplicate_connection"}ratevalidator10000
tn_network_source_admission_denied_total{network="bench",direction="out",reason="duplicate_connection"}rateobserver1000
tn_network_source_admission_denied_total{network="bench",direction="out",reason="invalid_limits"}ratevalidator10000
tn_network_source_admission_denied_total{network="bench",direction="out",reason="invalid_limits"}rateobserver1000
tn_network_source_admission_denied_total{network="bench",direction="out",reason="peer_full"}ratevalidator10000
tn_network_source_admission_denied_total{network="bench",direction="out",reason="peer_full"}rateobserver1000
tn_network_source_admission_denied_total{network="bench",direction="out",reason="poisoned"}ratevalidator10000
tn_network_source_admission_denied_total{network="bench",direction="out",reason="poisoned"}rateobserver1000
tn_network_source_admission_denied_total{network="bench",direction="out",reason="prefix_full"}ratevalidator10000
tn_network_source_admission_denied_total{network="bench",direction="out",reason="prefix_full"}rateobserver1000
tn_network_source_admission_denied_total{network="bench",direction="out",reason="process_full"}ratevalidator10000
tn_network_source_admission_denied_total{network="bench",direction="out",reason="process_full"}rateobserver1000
tn_network_source_admission_denied_total{network="bench",direction="out",reason="sources_full"}ratevalidator10000
tn_network_source_admission_denied_total{network="bench",direction="out",reason="sources_full"}rateobserver1000
tn_network_source_admission_denied_total{network="bench",direction="out",reason="unsupported_address"}ratevalidator10000
tn_network_source_admission_denied_total{network="bench",direction="out",reason="unsupported_address"}rateobserver1000
tn_network_admission_modemaxvalidator10222
tn_network_admission_modemaxobserver1222
tn_network_admission_fallbackmaxvalidator10011
tn_network_admission_fallbackmaxobserver1000
tn_network_outbound_request_failures_total{kind="connection"}ratevalidator1001.692.7
tn_network_outbound_request_failures_total{kind="connection"}rateobserver1000
tn_network_outbound_request_failures_total{kind="dial"}ratevalidator1021.1107130
tn_network_outbound_request_failures_total{kind="dial"}rateobserver1000
tn_network_outbound_request_failures_total{kind="io"}ratevalidator105.7746.650.8
tn_network_outbound_request_failures_total{kind="io"}rateobserver10.3090.8521.08
tn_network_outbound_request_failures_total{kind="timeout"}ratevalidator100.74.025.24
tn_network_outbound_request_failures_total{kind="timeout"}rateobserver10.004930.03040.0387
tn_network_outbound_request_failures_total{kind="unsupported"}ratevalidator10000
tn_network_outbound_request_failures_total{kind="unsupported"}rateobserver1000
tn_network_inbound_requests_shed_total{class="batch",reason="admission"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="batch",reason="admission"}rateobserver1000
tn_network_inbound_requests_shed_total{class="batch",reason="queue_budget"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="batch",reason="queue_budget"}rateobserver1000
tn_network_inbound_requests_shed_total{class="batch",reason="queue_full"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="batch",reason="queue_full"}rateobserver1000
tn_network_inbound_requests_shed_total{class="batch",reason="serve_budget"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="batch",reason="serve_budget"}rateobserver1000
tn_network_inbound_requests_shed_total{class="batch",reason="unsubscribed"}ratevalidator100.00692215271
tn_network_inbound_requests_shed_total{class="batch",reason="unsubscribed"}rateobserver1000
tn_network_inbound_requests_shed_total{class="certificate_sync",reason="admission"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="certificate_sync",reason="admission"}rateobserver1000
tn_network_inbound_requests_shed_total{class="certificate_sync",reason="queue_budget"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="certificate_sync",reason="queue_budget"}rateobserver1000
tn_network_inbound_requests_shed_total{class="certificate_sync",reason="queue_full"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="certificate_sync",reason="queue_full"}rateobserver1000
tn_network_inbound_requests_shed_total{class="certificate_sync",reason="serve_budget"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="certificate_sync",reason="serve_budget"}rateobserver1000
tn_network_inbound_requests_shed_total{class="certificate_sync",reason="unsubscribed"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="certificate_sync",reason="unsubscribed"}rateobserver1000
tn_network_inbound_requests_shed_total{class="epoch_record",reason="admission"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="epoch_record",reason="admission"}rateobserver1000
tn_network_inbound_requests_shed_total{class="epoch_record",reason="queue_budget"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="epoch_record",reason="queue_budget"}rateobserver1000
tn_network_inbound_requests_shed_total{class="epoch_record",reason="queue_full"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="epoch_record",reason="queue_full"}rateobserver1000
tn_network_inbound_requests_shed_total{class="epoch_record",reason="serve_budget"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="epoch_record",reason="serve_budget"}rateobserver1000
tn_network_inbound_requests_shed_total{class="epoch_record",reason="unsubscribed"}ratevalidator100.4051.181.56
tn_network_inbound_requests_shed_total{class="epoch_record",reason="unsubscribed"}rateobserver10.1810.5180.657
tn_network_inbound_requests_shed_total{class="gossip",reason="admission"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="gossip",reason="admission"}rateobserver1000
tn_network_inbound_requests_shed_total{class="gossip",reason="queue_budget"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="gossip",reason="queue_budget"}rateobserver1000
tn_network_inbound_requests_shed_total{class="gossip",reason="queue_full"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="gossip",reason="queue_full"}rateobserver1000
tn_network_inbound_requests_shed_total{class="gossip",reason="serve_budget"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="gossip",reason="serve_budget"}rateobserver1000
tn_network_inbound_requests_shed_total{class="gossip",reason="unsubscribed"}ratevalidator100.008940.1730.559
tn_network_inbound_requests_shed_total{class="gossip",reason="unsubscribed"}rateobserver10.05240.1520.193
tn_network_inbound_requests_shed_total{class="other",reason="admission"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="other",reason="admission"}rateobserver1000
tn_network_inbound_requests_shed_total{class="other",reason="queue_budget"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="other",reason="queue_budget"}rateobserver1000
tn_network_inbound_requests_shed_total{class="other",reason="queue_full"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="other",reason="queue_full"}rateobserver1000
tn_network_inbound_requests_shed_total{class="other",reason="serve_budget"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="other",reason="serve_budget"}rateobserver1000
tn_network_inbound_requests_shed_total{class="other",reason="unsubscribed"}ratevalidator100.04540.3190.452
tn_network_inbound_requests_shed_total{class="other",reason="unsubscribed"}rateobserver1000
tn_network_inbound_requests_shed_total{class="vote",reason="admission"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="vote",reason="admission"}rateobserver1000
tn_network_inbound_requests_shed_total{class="vote",reason="queue_budget"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="vote",reason="queue_budget"}rateobserver1000
tn_network_inbound_requests_shed_total{class="vote",reason="queue_full"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="vote",reason="queue_full"}rateobserver1000
tn_network_inbound_requests_shed_total{class="vote",reason="serve_budget"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="vote",reason="serve_budget"}rateobserver1000
tn_network_inbound_requests_shed_total{class="vote",reason="unsubscribed"}ratevalidator100.307115145
tn_network_inbound_requests_shed_total{class="vote",reason="unsubscribed"}rateobserver1000
tn_network_inbound_request_service_seconds{class="batch"}p95validator5104 ms416 ms416 ms
tn_network_inbound_request_service_seconds{class="epoch_record"}p95validator65 ms10 ms24 ms
tn_network_inbound_request_service_seconds{class="vote"}p95validator4770 ms2,425 ms2,425 ms
tn_network_established_connectionsmaxvalidator1019.52020
tn_network_established_connectionsmaxobserver118.42020
tn_worker_sync_streams_shed_total{reason="denied"}ratevalidator80.001410.820.82
tn_worker_batch_fetch_duration_secondsp95validator95 ms259 ms412 ms
tn_executor_batch_fetch_failures_totalratevalidator10000
tn_executor_batch_fetch_failures_totalrateobserver1000
tn_primary_votes_received_totalratevalidator1013.316.716.8
tn_primary_votes_received_totalrateobserver1000
tn_primary_vote_request_failures_totalratevalidator100.01151.542.07
tn_primary_vote_request_failures_totalrateobserver1000
tn_primary_certificates_formed_totalratevalidator102.192.792.8
tn_primary_certificates_formed_totalrateobserver1000
tn_primary_certificate_form_duration_secondsp95validator10521 ms3,950 ms7,000 ms
tn_engine_outputs_executed_totalratevalidator101.091.421.44
tn_engine_outputs_executed_totalrateobserver11.091.411.42
tn_engine_persist_failures_totalratevalidator10000
tn_engine_persist_failures_totalrateobserver1000
tn_node_consensus_sync_distancemaxvalidator108.98492494
tn_node_consensus_sync_distancemaxobserver12.424.757
reth_db_table_size{table="AccountsHistory"}maxvalidator1068 MiB105 MiB106 MiB
reth_db_table_size{table="AccountsHistory"}maxobserver168 MiB105 MiB106 MiB
reth_db_table_size{table="AccountsTrie"}maxvalidator106.8 MiB11 MiB11 MiB
reth_db_table_size{table="AccountsTrie"}maxobserver16.8 MiB11 MiB11 MiB
reth_db_table_size{table="BlockBodyIndices"}maxvalidator101.4 MiB2.2 MiB2.3 MiB
reth_db_table_size{table="BlockBodyIndices"}maxobserver11.4 MiB2.2 MiB2.3 MiB
reth_db_table_size{table="HashedAccounts"}maxvalidator1042 MiB71 MiB72 MiB
reth_db_table_size{table="HashedAccounts"}maxobserver142 MiB71 MiB72 MiB
reth_db_table_size{table="HashedStorages"}maxvalidator1044 MiB46 MiB46 MiB
reth_db_table_size{table="HashedStorages"}maxobserver144 MiB46 MiB46 MiB
reth_db_table_size{table="HeaderNumbers"}maxvalidator104.2 MiB6.8 MiB6.8 MiB
reth_db_table_size{table="HeaderNumbers"}maxobserver14.3 MiB6.8 MiB6.8 MiB
reth_db_table_size{table="StoragesHistory"}maxvalidator10117 MiB139 MiB139 MiB
reth_db_table_size{table="StoragesHistory"}maxobserver1117 MiB139 MiB139 MiB
reth_db_table_size{table="StoragesTrie"}maxvalidator1017 MiB17 MiB17 MiB
reth_db_table_size{table="StoragesTrie"}maxobserver117 MiB17 MiB17 MiB
reth_db_table_size{table="TransactionBlocks"}maxvalidator101.5 MiB2.4 MiB2.4 MiB
reth_db_table_size{table="TransactionBlocks"}maxobserver11.5 MiB2.4 MiB2.4 MiB
reth_db_table_size{table="TransactionHashNumbers"}maxvalidator101.28 GiB2.01 GiB2.02 GiB
reth_db_table_size{table="TransactionHashNumbers"}maxobserver11.29 GiB2.01 GiB2.02 GiB
reth_db_freelistmaxvalidator1081K113K117K
reth_db_freelistmaxobserver180.8K112K116K
tn_engine_recover_secondsp95validator10294 ms2,425 ms2,425 ms
tn_engine_recover_secondsp95observer1278 ms847 ms871 ms
tn_engine_evm_secondsp95validator10472 ms2,214 ms2,214 ms
tn_engine_evm_secondsp95observer1411 ms743 ms781 ms
tn_engine_root_secondsp95validator9237 ms905 ms905 ms
tn_engine_root_secondsp95observer1205 ms244 ms256 ms
tn_engine_finish_secondsp95validator10372 ms892 ms900 ms
tn_engine_finish_secondsp95observer1232 ms345 ms384 ms
tn_engine_forward_wait_secondsp95validator103,339 ms30.0 s30.0 s
tn_engine_forward_wait_secondsp95observer11 ms4 ms6 ms
tn_engine_channel_depthmaxvalidator101.559.2511
tn_engine_channel_depthmaxobserver1000
tn_engine_executed_transactions_totalratevalidator109.54K21.5K21.5K
tn_engine_executed_transactions_totalrateobserver19.47K20.3K20.3K
tn_executor_execution_wait_secondsp95validator10173 ms29.0 s29.0 s
tn_primary_commit_run_ahead_blocked_totalratevalidator10000
tn_primary_commit_run_ahead_blocked_totalrateobserver1000
tn_primary_consensus_exec_lag_roundsmaxvalidator1016.969.574
tn_primary_consensus_exec_lag_roundsmaxobserver14.529.514
tn_primary_commit_lag_roundsmaxvalidator102.243.55
tn_primary_commit_lag_roundsmaxobserver1000
tn_primary_rejoin_grace_holds_totalratevalidator10000
tn_primary_rejoin_grace_holds_totalrateobserver1000
tn_primary_votes_refused_inactive_totalratevalidator10031.231.6
tn_primary_votes_refused_inactive_totalrateobserver1000
tn_primary_votes_refused_execution_behind_totalratevalidator100.08231.271.49
tn_primary_votes_refused_execution_behind_totalrateobserver1000
tn_batch_builder_pool_age_secondsp95validator105,256 ms30.0 s30.0 s
tn_batch_builder_builds_total{reason="bytes_full"}ratevalidator10000
tn_batch_builder_builds_total{reason="bytes_full"}rateobserver1000
tn_batch_builder_builds_total{reason="drained"}ratevalidator101.192.372.51
tn_batch_builder_builds_total{reason="drained"}rateobserver1000
tn_batch_builder_builds_total{reason="gas_full"}ratevalidator102.554.875.18
tn_batch_builder_builds_total{reason="gas_full"}rateobserver1000
tn_worker_batch_report_secondsp95validator100 ms1 ms1 ms
tn_primary_digest_wait_secondsp95validator9936 ms30.0 s30.0 s
tn_primary_headers_proposed_total{reason="max_delay"}ratevalidator100.0240.1940.222
tn_primary_headers_proposed_total{reason="min_delay"}ratevalidator102.112.792.8
tn_primary_headers_proposed_total{reason="parents_and_threshold"}ratevalidator100.1662.072.24
tn_primary_digests_requeued_total{cause="canceled"}ratevalidator100.1098.8411.3
tn_primary_digests_requeued_total{cause="committed"}ratevalidator100.009111.221.42
tn_executor_consensus_pack_persist_secondsp95validator1022 ms57 ms68 ms
tn_batch_builder_seal_duration_secondsp95validator10318 ms953 ms953 ms
tn_worker_quorum_wait_secondsp95validator9340 ms947 ms947 ms
tn_primary_proposer_pending_digestsmaxvalidator101.3718.531
tn_primary_proposer_pending_digestsmaxobserver1000
reth_transaction_pool_pending_pool_transactionsmaxvalidator109536.17K9.89K
reth_transaction_pool_pending_pool_transactionsmaxobserver1000
reth_transaction_pool_queued_pool_transactionsmaxvalidator102873.01K4.99K
reth_transaction_pool_queued_pool_transactionsmaxobserver1000
reth_transaction_pool_all_transactions_by_hashmaxvalidator101.71K7.51K9.89K
reth_transaction_pool_all_transactions_by_hashmaxobserver1000
reth_storage_providers_database_save_blocks_totalp95validator10237 ms16.3 s19.9 s
reth_storage_providers_database_save_blocks_totalp95observer1205 ms296 ms327 ms
reth_storage_providers_database_save_blocks_mdbxp95validator10236 ms16.3 s19.9 s
reth_storage_providers_database_save_blocks_mdbxp95observer1193 ms279 ms307 ms
reth_storage_providers_database_save_blocks_sfp95validator1058 ms104 ms110 ms
reth_storage_providers_database_save_blocks_sfp95observer145 ms66 ms67 ms
reth_storage_providers_database_save_blocks_commit_mdbxp95validator10452 ms851 ms940 ms
reth_storage_providers_database_save_blocks_commit_mdbxp95observer1480 ms793 ms820 ms
reth_storage_providers_database_save_blocks_commit_sfp95validator101 ms1 ms1 ms
reth_storage_providers_database_save_blocks_commit_sfp95observer10 ms0 ms0 ms
reth_storage_providers_database_save_blocks_write_statep95validator100 ms1 ms1 ms
reth_storage_providers_database_save_blocks_write_statep95observer10 ms0 ms0 ms
reth_storage_providers_database_save_blocks_write_hashed_statep95validator1026 ms1,770 ms1,877 ms
reth_storage_providers_database_save_blocks_write_hashed_statep95observer123 ms29 ms30 ms
reth_storage_providers_database_save_blocks_write_trie_updatesp95validator938 ms1,043 ms1,067 ms
reth_storage_providers_database_save_blocks_write_trie_updatesp95observer132 ms40 ms45 ms
tn_node_switch{name="digest_order_hold"}maxvalidator10111
tn_node_switch{name="digest_order_hold"}maxobserver1111
tn_node_switch{name="epoch_record_collect_past_uncertified"}maxvalidator10111
tn_node_switch{name="epoch_record_collect_past_uncertified"}maxobserver1111
tn_node_switch{name="epoch_record_conflict_halt"}maxvalidator10111
tn_node_switch{name="epoch_record_conflict_halt"}maxobserver1111
tn_node_switch{name="epoch_record_recertify"}maxvalidator10111
tn_node_switch{name="epoch_record_recertify"}maxobserver1111
tn_node_switch{name="quorum_exec_block"}maxvalidator10111
tn_node_switch{name="quorum_exec_block"}maxobserver1111
tn_node_switch{name="quorum_exec_block_starved_rounds"}maxvalidator10444
tn_node_switch{name="quorum_exec_block_starved_rounds"}maxobserver1444
tn_node_switch{name="vote_queue_fix"}maxvalidator10111
tn_node_switch{name="vote_queue_fix"}maxobserver1111
tn_primary_epoch_entry_rounds_behindmaxvalidator1033636
tn_primary_epoch_entry_rounds_behindmaxobserver1000
reth_transaction_pool_pending_pool_size_bytesmaxvalidator10339.3 KiB2.1 MiB3.5 MiB
reth_transaction_pool_pending_pool_size_bytesmaxobserver10 B0 B0 B
reth_transaction_pool_queued_pool_size_bytesmaxvalidator10102.6 KiB867.8 KiB1.3 MiB
reth_transaction_pool_queued_pool_size_bytesmaxobserver10 B0 B0 B
reth_transaction_pool_pending_transactions_evictedmaxvalidator10000
reth_transaction_pool_pending_transactions_evictedmaxobserver1000
reth_transaction_pool_queued_transactions_evictedmaxvalidator10000
reth_transaction_pool_queued_transactions_evictedmaxobserver1000
reth_transaction_pool_basefee_transactions_evictedmaxvalidator10000
reth_transaction_pool_basefee_transactions_evictedmaxobserver1000
reth_transaction_pool_removed_transactionsmaxvalidator102.03M3.66M3.68M
reth_transaction_pool_removed_transactionsmaxobserver1000
reth_transaction_pool_invalid_transactionsmaxvalidator10000
reth_transaction_pool_invalid_transactionsmaxobserver1000
tn_engine_recovery_fallback_totalmaxvalidator10000
tn_engine_recovery_fallback_totalmaxobserver1000

Commit latency p95

tn_primary_commit_latency_seconds

Round lag

tn_primary_round − tn_primary_committed_round

Engine queued outputs

tn_engine_queued_outputs, max by node

Certification by node

The share of each validator's own headers that formed a certificate, per window (tools/cert-rate.py on the node logs). bench-validator-06 certified fewer than 90 % of its own headers in a window where it proposed at least 50. Own leader % is the validator's share of the committed leaders and not committed % the share of its certificates no committed leader reached, both over the run window; the fair share is 100 / 10 validators = 10.0 %. Greyed: fewer than 50 headers proposed, not judged.

ValidatorRegionblast 0 preambleblast 0 warmupblast 0 mixedblast 0 cooldownblast 1 preambleblast 1 warmupblast 1 mixedblast 1 cooldownblast 2 preambleblast 2 warmupblast 2 mixedblast 2 cooldownrunOwn leader %Not committed %
bench-validator-01us-west2 (Los Angeles)100.0100.0100.0101.198.3100.0100.1102.0102.097.194.398.897.012.50.1
bench-validator-02europe-west2 (London)100.0100.0100.0101.198.2100.099.698.098.097.497.098.898.31.739.7
bench-validator-03australia-southeast1 (Sydney)112.5100.099.9101.1101.8100.0100.0100.0100.099.198.498.899.213.50.2
bench-validator-04asia-northeast1 (Tokyo)100.0100.699.9101.1101.8100.0100.098.0100.098.899.697.599.812.90.1
bench-validator-05us-east4 (Ashburn)88.999.499.0100.0100.099.599.9100.0102.098.890.8–96.310.11.1
bench-validator-06 below 90 %europe-west4 (Netherlands)112.599.499.498.998.299.599.598.0100.096.576.4100.087.50.446.8
bench-validator-07asia-southeast1 (Singapore)88.9100.699.9100.098.3100.0100.1100.0100.098.398.698.899.213.40.1
bench-validator-08asia-east2 (Hong Kong)100.0100.699.9100.0100.0100.0100.1100.0100.098.398.598.899.213.30.1
bench-validator-09asia-northeast3 (Seoul)100.0100.699.9100.0101.8100.0100.0100.0100.097.799.098.899.414.10.1
bench-validator-10asia-south1 (Mumbai)100.0100.099.9101.1100.0100.0100.0100.0102.096.895.596.397.610.76.3

On-node sampler

NodeClassPeak RssAnonPeak RssFilePeak cachePSI memPSI ioPSI cpudb growthstatic_files growthRoot usedOOM Δ
bench-observer-11observer955 MiB5.14 GiB– 0.0 %7.3 %2.8 %3.29 GiB9.48 GiB22.0 %0
bench-validator-01validator1.99 GiB4.96 GiB– 0.0 %9.1 %14.7 %3.28 GiB9.48 GiB24.0 %0
bench-validator-02validator1.96 GiB4.98 GiB– 0.0 %11.3 %13.1 %3.28 GiB9.48 GiB24.0 %0
bench-validator-03validator1.89 GiB5.04 GiB– 0.0 %11.2 %14.3 %3.28 GiB9.48 GiB24.0 %0
bench-validator-04validator2.31 GiB4.98 GiB– 0.0 %10.3 %11.7 %3.28 GiB9.48 GiB24.0 %0
bench-validator-05validator2.51 GiB5.08 GiB– 0.0 %14.1 %15.6 %2.83 GiB9.29 GiB24.0 %0
bench-validator-06validator1.88 GiB5.00 GiB– 0.0 %10.0 %14.6 %3.28 GiB9.48 GiB24.0 %0
bench-validator-07validator1.81 GiB4.95 GiB– 0.0 %9.1 %16.2 %3.28 GiB9.48 GiB24.0 %0
bench-validator-08validator1.85 GiB5.06 GiB– 0.0 %11.4 %15.6 %3.28 GiB9.48 GiB24.0 %0
bench-validator-09validator1.87 GiB4.99 GiB– 0.0 %7.7 %14.7 %3.28 GiB9.48 GiB24.0 %0
bench-validator-10validator1.82 GiB4.95 GiB– 0.0 %11.6 %12.7 %3.28 GiB9.48 GiB24.0 %0

RssAnon vs RssFile

/proc/<pid>/status of the telcoin process on the node with the largest RSS

consensus-db/cache size

Not collected: consensus-db/cache size.

Pressure stall (some, avg10)

/proc/pressure on the node with the highest memory PSI

Headroom

Per role and per node, from sizing.json (p95 over the landed window; PSI and RssAnon max over the whole sampler run). The lines follow the hardware doc's tier rules: CPU p95 over 70 % of vCPUs; memory used p95 over 75 % of RAM, or telcoin RSS p95 over 85 % of RAM; disk write p95 over 60 % of the PD MiB/s limit, or write IOPS p95 over 60 % of the IOPS limit; IO pressure over 20 % or memory pressure over 5 % (PSI some avg10, max); network p95 over 50 % of 1 Gbps; root disk over 80 %. A line crossed on any node puts the role at the limit; a nonzero reading within 15 points of its line is tight; otherwise the role has headroom. OOM kills, disk throttling and degraded disk performance are flagged too.

SignalClassWorst nodeValueShare of capacityLineNodes over / tightStatus
CPU p95validatorbench-validator-0356.5 %56.5 %> 70 %0 / 6 tight
Memory used p95validatorbench-validator-0521.0 %21.0 %> 75 %0 / 0 headroom
telcoin RSS p95validatorbench-validator-057.70 GB44.8 %> 85 %0 / 0 headroom
PSI memory maxvalidatorbench-validator-010.0 %0.0 %> 5 %0 / 0 headroom
RssAnon peakvalidatorbench-validator-052.84 GB16.5 %info– info
Disk write p95validatorbench-validator-06300 MB/s35.7 %> 60 %0 / 0 headroom
Write IOPS p95validatorbench-validator-0320,039100.2 %> 60 %10 / 0 at the limit
PSI io maxvalidatorbench-validator-0514.1 %14.1 %> 20 %0 / 10 tight
Root disk maxvalidatorbench-validator-0923.7 %23.7 %> 80 %0 / 0 headroom
Network p95validatorbench-validator-02196 Mbps19.6 %> 50 %0 / 0 headroom
Consensus cache peakvalidator–––info– not collected
OOM killsvalidator––anyany– none
Disk throttlingvalidator––metric type unavailable in projectany– not collected
Disk performance degradedvalidator––any sampleany– none
CPU p95observerbench-observer-1132.1 %32.1 %> 70 %0 / 0 headroom
Memory used p95observerbench-observer-1111.4 %11.4 %> 75 %0 / 0 headroom
telcoin RSS p95observerbench-observer-116.37 GB37.1 %> 85 %0 / 0 headroom
PSI memory maxobserverbench-observer-110.0 %0.0 %> 5 %0 / 0 headroom
RssAnon peakobserverbench-observer-111.01 GB5.9 %info– info
Disk write p95observerbench-observer-11303 MB/s36.2 %> 60 %0 / 0 headroom
Write IOPS p95observerbench-observer-1117,72188.6 %> 60 %1 / 0 at the limit
PSI io maxobserverbench-observer-117.3 %7.3 %> 20 %0 / 1 tight
Root disk maxobserverbench-observer-1121.8 %21.8 %> 80 %0 / 0 headroom
Network p95observerbench-observer-1110 Mbps1.0 %> 50 %0 / 0 headroom
Consensus cache peakobserver–––info– not collected
OOM killsobserver––anyany– none
Disk throttlingobserver––metric type unavailable in projectany– not collected
Disk performance degradedobserver––any sampleany– none

Per node

NodeClassCPU p95Memory used p95telcoin RSS p95PSI memory maxRssAnon peakDisk write p95Write IOPS p95PSI io maxRoot disk maxNetwork p95Consensus cache peak
Line> 70 %> 75 %> 85 %> 5 %info> 60 %> 60 %> 20 %> 80 %> 50 %info
bench-validator-01validator56.2 %18.1 %7.34 GB 43 %0.0 %2.14 GB 12 %296 MB/s 35 %19,390 97 %9.1 %23.4 %195 Mbps 20 %–
bench-validator-02validator53.6 %17.8 %7.29 GB 42 %0.0 %2.10 GB 12 %297 MB/s 35 %19,838 99 %11.3 %23.5 %196 Mbps 20 %–
bench-validator-03validator56.5 %17.5 %7.26 GB 42 %0.0 %2.03 GB 12 %291 MB/s 35 %20,039 100 %11.2 %23.6 %196 Mbps 20 %–
bench-validator-04validator53.8 %19.9 %7.61 GB 44 %0.0 %2.53 GB 15 %296 MB/s 35 %18,596 93 %10.3 %23.4 %190 Mbps 19 %–
bench-validator-05validator56.3 %21.0 %7.70 GB 45 %0.0 %2.84 GB 17 %292 MB/s 35 %19,992 100 %14.1 %22.7 %195 Mbps 20 %–
bench-validator-06validator55.7 %17.4 %7.20 GB 42 %0.0 %2.02 GB 12 %300 MB/s 36 %19,728 99 %10.0 %23.6 %189 Mbps 19 %–
bench-validator-07validator55.1 %16.9 %7.14 GB 42 %0.0 %1.94 GB 11 %293 MB/s 35 %20,026 100 %9.1 %23.5 %188 Mbps 19 %–
bench-validator-08validator54.7 %17.0 %7.17 GB 42 %0.0 %1.99 GB 12 %298 MB/s 36 %19,584 98 %11.4 %23.7 %185 Mbps 18 %–
bench-validator-09validator53.3 %17.2 %7.22 GB 42 %0.0 %2.02 GB 12 %292 MB/s 35 %19,092 95 %7.7 %23.7 %194 Mbps 19 %–
bench-validator-10validator55.4 %17.0 %7.13 GB 42 %0.0 %1.95 GB 11 %289 MB/s 34 %19,704 99 %11.6 %23.7 %191 Mbps 19 %–
bench-observer-11observer32.1 %11.4 %6.37 GB 37 %0.0 %1.01 GB 6 %303 MB/s 36 %17,721 89 %7.3 %21.8 %10 Mbps 1 %–

Small figures are the share of capacity: RAM for RSS, the PD limit for disk write (VM cap applied), 1 Gbps for network. Hover a cell for its JSON path.

Verdict

Fleet A vs Fleet B

A = Fleet c3-hdb · c3-highcpu-8 · r20261115-1653
B = Fleet c3-hdb · c3-highcpu-8 · r20261115-1131

Verdict

at the limit c3-highcpu-8 validators at 12,594 sustained TPS: at the limit on disk. Disk at the limit: write IOPS p95 18,596–20,039, up to 100.2 % of the 20,000 IOPS limit, over the 60 % line on all 10 validators (bench-validator-01, 02, 03, 04, 05, 06, 07, 08, 09, 10); IO pressure (PSI some avg10) max 14.1 % (bench-validator-05), tight against the 20 % line. CPU tight: CPU p95 56.5 % of 8 vCPUs (bench-validator-03), tight against the 70 % line. Memory has headroom: memory used p95 21.0 % of RAM (bench-validator-05); telcoin RSS p95 7.70 GB, 44.8 % of 16 GiB RAM (bench-validator-05); memory pressure (PSI some avg10) max 0.0 % (bench-validator-01). Network has headroom: network p95 196 Mbps, 19.6 % of 1 Gbps (bench-validator-02). Next step: a faster volume (pd-ssd adds 30 IOPS and 0.48 MiB/s per GB up to the per-VM cap; past the cap, a larger VM).

at the limit c3-highcpu-8 observers at 12,594 sustained TPS: at the limit on disk. Disk at the limit: write IOPS p95 17,721, 88.6 % of the 20,000 IOPS limit on bench-observer-11, over the 60 % line; IO pressure (PSI some avg10) max 7.3 % (bench-observer-11), tight against the 20 % line. CPU has headroom: CPU p95 32.1 % of 8 vCPUs (bench-observer-11). Memory has headroom: memory used p95 11.4 % of RAM (bench-observer-11); telcoin RSS p95 6.37 GB, 37.1 % of 16 GiB RAM (bench-observer-11); memory pressure (PSI some avg10) max 0.0 % (bench-observer-11). Network has headroom: network p95 10 Mbps, 1.0 % of 1 Gbps (bench-observer-11). Next step: a faster volume (pd-ssd adds 30 IOPS and 0.48 MiB/s per GB up to the per-VM cap; past the cap, a larger VM).

at the limit c3-highcpu-8 validators at 11,921 sustained TPS: at the limit on disk. Disk at the limit: write IOPS p95 18,789–19,924, up to 99.6 % of the 20,000 IOPS limit, over the 60 % line on all 10 validators (bench-validator-01, 02, 03, 04, 05, 06, 07, 08, 09, 10); IO pressure (PSI some avg10) max 14.6 % (bench-validator-05), tight against the 20 % line. CPU tight: CPU p95 55.8 % of 8 vCPUs (bench-validator-04), tight against the 70 % line. Memory has headroom: memory used p95 20.8 % of RAM (bench-validator-05); telcoin RSS p95 7.59 GB, 44.2 % of 16 GiB RAM (bench-validator-07); memory pressure (PSI some avg10) max 0.0 % (bench-validator-01). Network has headroom: network p95 204 Mbps, 20.4 % of 1 Gbps (bench-validator-07). Next step: a faster volume (pd-ssd adds 30 IOPS and 0.48 MiB/s per GB up to the per-VM cap; past the cap, a larger VM).

at the limit c3-highcpu-8 observers at 11,921 sustained TPS: at the limit on disk. Disk at the limit: write IOPS p95 18,194, 91.0 % of the 20,000 IOPS limit on bench-observer-11, over the 60 % line; IO pressure (PSI some avg10) max 7.7 % (bench-observer-11), tight against the 20 % line. CPU has headroom: CPU p95 30.2 % of 8 vCPUs (bench-observer-11). Memory has headroom: memory used p95 11.9 % of RAM (bench-observer-11); telcoin RSS p95 6.27 GB, 36.5 % of 16 GiB RAM (bench-observer-11); memory pressure (PSI some avg10) max 0.0 % (bench-observer-11). Network has headroom: network p95 9 Mbps, 0.9 % of 1 Gbps (bench-observer-11). Next step: a faster volume (pd-ssd adds 30 IOPS and 0.48 MiB/s per GB up to the per-VM cap; past the cap, a larger VM).

Validators: Fleet A (c3-highcpu-8) is at the limit on disk, tight on CPU, with headroom on memory and network; Fleet B (c3-highcpu-8) is at the limit on disk, tight on CPU, with headroom on memory and network. The same disk limit binds both fleets: every validator runs a 100 GB hyperdisk-balanced, which allows 20,000 IOPS on both; its throughput limit is 800 MiB/s on both. Disk is at the limit on both (Fleet A (c3-highcpu-8): write IOPS p95 20,039, 100.2 % of the 20,000 IOPS limit; Fleet B (c3-highcpu-8): write IOPS p95 19,924, 99.6 % of the 20,000 IOPS limit). Changing the machine shape did not clear it; the volume is the binding resource.

Observers: Fleet A (c3-highcpu-8) is at the limit on disk, with headroom on CPU, memory and network; Fleet B (c3-highcpu-8) is at the limit on disk, with headroom on CPU, memory and network. The same disk limit binds both fleets: every observer runs a 100 GB hyperdisk-balanced, which allows 20,000 IOPS on both; its throughput limit is 800 MiB/s on both. Disk is at the limit on both (Fleet A (c3-highcpu-8): write IOPS p95 17,721, 88.6 % of the 20,000 IOPS limit; Fleet B (c3-highcpu-8): write IOPS p95 18,194, 91.0 % of the 20,000 IOPS limit). Changing the machine shape did not clear it; the volume is the binding resource.

KPIFleet AFleet BB ÷ A
Sustained mixed TPS (own)12,593.5 tx/s11,921.4 tx/s0.95×
Sustained mixed TPS (chain)12,593.5 tx/s11,921.4 tx/s0.95×
Landed / accepted100.0 %100.0 %1.00×
Reverted (own)00–
Latency p501,662 ms1,479 ms0.89×
Latency p958,771 ms7,763 ms0.89×
Gas / s (own)471M gas/s466M gas/s0.99×
Capacity at mix18,696.2 tx/s19,007.7 tx/s1.02×

Headroom, worst node per class

SignalClassFleet AFleet BLine
CPU p95validator56.5 % tight55.8 % tight> 70 %
Memory used p95validator21.0 % headroom20.8 % headroom> 75 %
telcoin RSS p95validator7.70 GB 44.8 % headroom7.59 GB 44.2 % headroom> 85 %
PSI memory maxvalidator0.0 % headroom0.0 % headroom> 5 %
Disk write p95validator300 MB/s 35.7 % headroom301 MB/s 35.8 % headroom> 60 %
Write IOPS p95validator20,039 100.2 % at the limit19,924 99.6 % at the limit> 60 %
PSI io maxvalidator14.1 % tight14.6 % tight> 20 %
Root disk maxvalidator23.7 % headroom24.1 % headroom> 80 %
Network p95validator196 Mbps 19.6 % headroom204 Mbps 20.4 % headroom> 50 %
CPU p95observer32.1 % headroom30.2 % headroom> 70 %
Memory used p95observer11.4 % headroom11.9 % headroom> 75 %
telcoin RSS p95observer6.37 GB 37.1 % headroom6.27 GB 36.5 % headroom> 85 %
PSI memory maxobserver0.0 % headroom0.0 % headroom> 5 %
Disk write p95observer303 MB/s 36.2 % headroom288 MB/s 34.3 % headroom> 60 %
Write IOPS p95observer17,721 88.6 % at the limit18,194 91.0 % at the limit> 60 %
PSI io maxobserver7.3 % tight7.7 % tight> 20 %
Root disk maxobserver21.8 % headroom21.7 % headroom> 80 %
Network p95observer10 Mbps 1.0 % headroom9 Mbps 0.9 % headroom> 50 %

Own TPS, trailing 60-s mean

aligned on each fleet's mixed-phase start

CPU, hottest validator

aligned on each fleet's mixed-phase start

Memory used, hottest validator

aligned on each fleet's mixed-phase start

telcoin RSS, hottest validator

aligned on each fleet's mixed-phase start

Disk MiB/s, hottest validator

aligned on each fleet's mixed-phase start

Network sent, hottest validator

aligned on each fleet's mixed-phase start

Reference chains

Not collected: reference chains.