This is the full report of the run “The main branch with the commit-lag, rejoin and execution fixes”. Back to the run page.
r20261107-1504 · Fleet c3-hdb · c3-highcpu-8

Telcoin Network benchmark: mixed workload

Throughput is measured from blocks the chain produced (own senders vs every sender), resource use from Cloud Monitoring, Prometheus and the on-node sampler. Summaries cover the mixed phase. Hover any number for the JSON field it came from.

10 blaster(s), 6K senders, chain 32286, schema tn-blast-combined/v2 · mixed phase 2026-11-07 17:41–18:21 UTC · 1 observers, 10 validators

Preflight: WTEL from genesis (0x0000…37e1), v3 fee tier 3000, v4 available, Multicall3 present · Cooldown ended: idle_30s ×10

Headline numbers

Sustained mixed TPS (own)
4,939.2 tx/s
best 60-s mean of our own landed tx in the mixed phase
Sustained mixed TPS (chain)
4,939.2 tx/s
all tx in blocks, same window
Landed / accepted
99.6 %
9.22M of 9.26M accepted
Reverted (own)
0
0.00 % of landed
Latency p50
2,809 ms
submit → block timestamp; seen p50 14,179 ms
Latency p95
14,404 ms
seen p95 53,387 ms
Gas / s (own)
174M gas/s
chain 174M gas/s
Capacity at mix
7,705.4 tx/s
gas-limit basis 4,966.6 tx/s; block gas limit 30M

Per transaction type

Landed counts only our own senders; latency is submit → block timestamp.

TypeSubmittedAcceptedLandedRevertedUnlandedSkipsGas limitGas used avgp50p95
native_transfer2.78M2.66M2.65M07.18K024.1K21K2,801 ms14,359 ms
batched_self_send1.49M1.4M1.39M06.17K024.1K21K2,895 ms14,770 ms
uniswap_v2_swap1.34M1.25M1.25M05.28K0180K106K2,807 ms14,349 ms
stablecoin_transfer1.12M1.05M1.04M04.22K075K43.9K2,804 ms14,377 ms
wtel_wrap888K831K827K03.41K050K27.9K2,808 ms14,409 ms
wtel_unwrap594K555K553K02.25K045K35.2K2,805 ms14,386 ms
stablecoin_approve448K419K418K01.72K060K31.8K2,798 ms14,321 ms
uniswap_v2_add_liquidity297K278K277K01.14K0250K128K2,805 ms14,271 ms
uniswap_v4_swap225K210K209K08730250K124K2,793 ms14,322 ms
contract_deploy224K209K208K08570140K74.2K2,804 ms14,324 ms
uniswap_v3_swap149K140K139K05850220K120K2,819 ms14,339 ms
uniswap_v2_remove_liquidity149K139K139K05870220K129K2,796 ms14,283 ms
faucet_drip125K122K122K097244K150K107K2,153 ms7,925 ms

Submission by RPC node

NodeSubmittedAcceptedAccepted %Top errorsSend p50Send p95
bench-validator-01
bench-validator-01:8545
695K690K99.3 %known 1.58K6 ms24 ms
bench-validator-02
bench-validator-02:8545
1.17M1.14M97.4 %known 15.9K7 ms29 ms
bench-validator-03
bench-validator-03:8545
992K989K99.7 %known 1.33K9 ms32 ms
bench-validator-04
bench-validator-04:8545
940K903K96.1 %known 10.6K12 ms37 ms
bench-validator-05
bench-validator-05:8545
1.94M226K11.7 %other 1.7M, known 5.55K1 ms10 ms
bench-validator-06
bench-validator-06:8545
1.03M994K96.8 %known 15.6K8 ms29 ms
bench-validator-07
bench-validator-07:8545
1.5M1.48M98.9 %known 8.02K7 ms23 ms
bench-validator-08
bench-validator-08:8545
575K530K92.1 %known 10.9K9 ms36 ms
bench-validator-09
bench-validator-09:8545
1.52M1.5M98.8 %known 8.55K7 ms24 ms
bench-validator-10
bench-validator-10:8545
899K865K96.2 %known 14.8K8 ms29 ms

Throughput over time

Shaded: phases outside the mixed window. Hairlines: epoch closes. Dashed orange: chaos events.

Landed transactions per second

own senders vs chain total, per block-timestamp second

Gas per second

own vs chain gas used, per block-timestamp second

CPU and memory per node

CPU utilization

compute.googleapis.com/instance/cpu/utilization, 60-s mean, % of all vCPUs; summaries over the mixed phase

Table view
NodeClassMeanp95Peak 60-s meanMax
bench-observer-11observer10.7 %12.0 %13.4 %13.4 %
bench-validator-01validator17.5 %20.5 %20.7 %20.7 %
bench-validator-02validator18.2 %19.8 %20.5 %20.5 %
bench-validator-03validator17.7 %19.8 %22.2 %22.2 %
bench-validator-04validator18.0 %20.6 %20.9 %20.9 %
bench-validator-05validator16.0 %20.3 %21.8 %21.8 %
bench-validator-06validator18.1 %20.6 %20.9 %20.9 %
bench-validator-07validator18.5 %20.2 %20.5 %20.5 %
bench-validator-08validator17.4 %20.4 %22.2 %22.2 %
bench-validator-09validator18.4 %20.5 %20.8 %20.8 %
bench-validator-10validator18.1 %19.5 %19.9 %19.9 %

CPU: mean vs peak 60-s mean

One row per node, over the mixed phase.

telcoin resident memory

agent processes/rss_usage summed over telcoin pids; summaries over the mixed phase

Table view
NodeClassMeanp95Peak 60-s meanMax
bench-observer-11observer2.73 GiB3.70 GiB3.90 GiB3.90 GiB
bench-validator-01validator5.59 GiB7.32 GiB7.52 GiB7.52 GiB
bench-validator-02validator6.22 GiB7.65 GiB8.11 GiB8.11 GiB
bench-validator-03validator5.66 GiB7.10 GiB7.25 GiB7.25 GiB
bench-validator-04validator5.57 GiB6.85 GiB7.10 GiB7.10 GiB
bench-validator-05validator4.56 GiB5.60 GiB5.98 GiB5.98 GiB
bench-validator-06validator5.84 GiB7.36 GiB7.51 GiB7.51 GiB
bench-validator-07validator5.66 GiB6.97 GiB7.16 GiB7.16 GiB
bench-validator-08validator6.06 GiB7.30 GiB7.41 GiB7.41 GiB
bench-validator-09validator5.72 GiB7.16 GiB7.33 GiB7.33 GiB
bench-validator-10validator5.97 GiB7.53 GiB7.88 GiB7.88 GiB

telcoin RSS: mean vs peak 60-s mean

One row per node, over the mixed phase.

Memory used

agent memory/percent_used, state=used; summaries over the mixed phase

Table view
NodeClassMeanp95Peak 60-s meanMax
bench-observer-11observer10.4 %11.7 %12.5 %12.5 %
bench-validator-01validator27.1 %32.3 %33.1 %33.1 %
bench-validator-02validator30.7 %34.6 %36.4 %36.4 %
bench-validator-03validator27.2 %30.7 %32.0 %32.0 %
bench-validator-04validator26.7 %29.6 %30.6 %30.6 %
bench-validator-05validator18.8 %27.0 %27.1 %27.1 %
bench-validator-06validator28.3 %32.8 %33.4 %33.4 %
bench-validator-07validator27.2 %30.4 %30.6 %30.6 %
bench-validator-08validator29.5 %31.9 %32.0 %32.0 %
bench-validator-09validator27.6 %31.1 %31.9 %31.9 %
bench-validator-10validator29.2 %34.1 %35.4 %35.4 %

Memory used: mean vs peak 60-s mean

One row per node, over the mixed phase.

Disk

Disk throttling (validator)
not collected –
metric type unavailable in project
Disk performance degraded (validator)
none –
any sample

Disk read + write throughput

instance/disk/{read,write}_bytes_count rate, all devices; summaries over the mixed phase

Table view
NodeClassMeanp95Peak 60-s meanMaxLimit
bench-observer-11observer480.9 MiB/s559.1 MiB/s567.1 MiB/s567.1 MiB/s800.0 MiB/s
bench-validator-01validator499.5 MiB/s548.8 MiB/s552.1 MiB/s552.1 MiB/s800.0 MiB/s
bench-validator-02validator509.5 MiB/s565.1 MiB/s571.9 MiB/s571.9 MiB/s800.0 MiB/s
bench-validator-03validator495.0 MiB/s551.6 MiB/s554.4 MiB/s554.4 MiB/s800.0 MiB/s
bench-validator-04validator523.0 MiB/s563.7 MiB/s567.7 MiB/s567.7 MiB/s800.0 MiB/s
bench-validator-05validator401.7 MiB/s516.5 MiB/s544.4 MiB/s544.4 MiB/s800.0 MiB/s
bench-validator-06validator521.9 MiB/s562.8 MiB/s567.1 MiB/s567.1 MiB/s800.0 MiB/s
bench-validator-07validator496.0 MiB/s560.5 MiB/s563.2 MiB/s563.2 MiB/s800.0 MiB/s
bench-validator-08validator469.9 MiB/s558.1 MiB/s569.2 MiB/s569.2 MiB/s800.0 MiB/s
bench-validator-09validator497.1 MiB/s565.4 MiB/s567.5 MiB/s567.5 MiB/s800.0 MiB/s
bench-validator-10validator520.2 MiB/s567.6 MiB/s572.8 MiB/s572.8 MiB/s800.0 MiB/s

Disk read + write IOPS

instance/disk/{read,write}_ops_count rate, all devices; summaries over the mixed phase

Table view
NodeClassMeanp95Peak 60-s meanMaxLimit
bench-observer-11observer3.14K IOPS5.21K IOPS5.73K IOPS5.73K IOPS20K IOPS
bench-validator-01validator2.98K IOPS5.06K IOPS5.51K IOPS5.51K IOPS20K IOPS
bench-validator-02validator2.83K IOPS4.18K IOPS4.57K IOPS4.57K IOPS20K IOPS
bench-validator-03validator3.08K IOPS5.32K IOPS5.38K IOPS5.38K IOPS20K IOPS
bench-validator-04validator2.48K IOPS2.68K IOPS2.7K IOPS2.7K IOPS20K IOPS
bench-validator-05validator4.04K IOPS8.91K IOPS9.46K IOPS9.46K IOPS20K IOPS
bench-validator-06validator2.55K IOPS2.87K IOPS2.89K IOPS2.89K IOPS20K IOPS
bench-validator-07validator3.08K IOPS4.89K IOPS5.11K IOPS5.11K IOPS20K IOPS
bench-validator-08validator3.61K IOPS5.9K IOPS6.5K IOPS6.5K IOPS20K IOPS
bench-validator-09validator3.05K IOPS4.84K IOPS5.5K IOPS5.5K IOPS20K IOPS
bench-validator-10validator2.56K IOPS2.83K IOPS2.94K IOPS2.94K IOPS20K IOPS
Disk limits used
NodeMachinevCPUDiskIOPS limitMiB/s limitBasis
bench-observer-11c3-highcpu-88hyperdisk-balanced 100 GB20K IOPS800.0 MiB/shyperdisk-balanced 100 GB: provisioned 20,000 IOPS, provisioned 800 MiB/s; c3 8 vCPU cap 50,000 IOPS / 800 MiB/s; reads and writes share this budget
bench-validator-01c3-highcpu-88hyperdisk-balanced 100 GB20K IOPS800.0 MiB/shyperdisk-balanced 100 GB: provisioned 20,000 IOPS, provisioned 800 MiB/s; c3 8 vCPU cap 50,000 IOPS / 800 MiB/s; reads and writes share this budget
bench-validator-02c3-highcpu-88hyperdisk-balanced 100 GB20K IOPS800.0 MiB/shyperdisk-balanced 100 GB: provisioned 20,000 IOPS, provisioned 800 MiB/s; c3 8 vCPU cap 50,000 IOPS / 800 MiB/s; reads and writes share this budget
bench-validator-03c3-highcpu-88hyperdisk-balanced 100 GB20K IOPS800.0 MiB/shyperdisk-balanced 100 GB: provisioned 20,000 IOPS, provisioned 800 MiB/s; c3 8 vCPU cap 50,000 IOPS / 800 MiB/s; reads and writes share this budget
bench-validator-04c3-highcpu-88hyperdisk-balanced 100 GB20K IOPS800.0 MiB/shyperdisk-balanced 100 GB: provisioned 20,000 IOPS, provisioned 800 MiB/s; c3 8 vCPU cap 50,000 IOPS / 800 MiB/s; reads and writes share this budget
bench-validator-05c3-highcpu-88hyperdisk-balanced 100 GB20K IOPS800.0 MiB/shyperdisk-balanced 100 GB: provisioned 20,000 IOPS, provisioned 800 MiB/s; c3 8 vCPU cap 50,000 IOPS / 800 MiB/s; reads and writes share this budget
bench-validator-06c3-highcpu-88hyperdisk-balanced 100 GB20K IOPS800.0 MiB/shyperdisk-balanced 100 GB: provisioned 20,000 IOPS, provisioned 800 MiB/s; c3 8 vCPU cap 50,000 IOPS / 800 MiB/s; reads and writes share this budget
bench-validator-07c3-highcpu-88hyperdisk-balanced 100 GB20K IOPS800.0 MiB/shyperdisk-balanced 100 GB: provisioned 20,000 IOPS, provisioned 800 MiB/s; c3 8 vCPU cap 50,000 IOPS / 800 MiB/s; reads and writes share this budget
bench-validator-08c3-highcpu-88hyperdisk-balanced 100 GB20K IOPS800.0 MiB/shyperdisk-balanced 100 GB: provisioned 20,000 IOPS, provisioned 800 MiB/s; c3 8 vCPU cap 50,000 IOPS / 800 MiB/s; reads and writes share this budget
bench-validator-09c3-highcpu-88hyperdisk-balanced 100 GB20K IOPS800.0 MiB/shyperdisk-balanced 100 GB: provisioned 20,000 IOPS, provisioned 800 MiB/s; c3 8 vCPU cap 50,000 IOPS / 800 MiB/s; reads and writes share this budget
bench-validator-10c3-highcpu-88hyperdisk-balanced 100 GB20K IOPS800.0 MiB/shyperdisk-balanced 100 GB: provisioned 20,000 IOPS, provisioned 800 MiB/s; c3 8 vCPU cap 50,000 IOPS / 800 MiB/s; reads and writes share this budget

Network

Network sent

instance/network/sent_bytes_count rate; summaries over the mixed phase

Table view
NodeClassMeanp95Peak 60-s meanMax
bench-observer-11observer1.0 Mbps1.2 Mbps1.4 Mbps1.4 Mbps
bench-validator-01validator59.0 Mbps70.9 Mbps82.5 Mbps82.5 Mbps
bench-validator-02validator61.1 Mbps70.9 Mbps71.9 Mbps71.9 Mbps
bench-validator-03validator60.0 Mbps70.7 Mbps79.1 Mbps79.1 Mbps
bench-validator-04validator61.2 Mbps73.6 Mbps76.4 Mbps76.4 Mbps
bench-validator-05validator55.0 Mbps77.8 Mbps84.6 Mbps84.6 Mbps
bench-validator-06validator60.9 Mbps69.8 Mbps70.2 Mbps70.2 Mbps
bench-validator-07validator62.9 Mbps71.6 Mbps73.6 Mbps73.6 Mbps
bench-validator-08validator62.9 Mbps81.6 Mbps99.6 Mbps99.6 Mbps
bench-validator-09validator62.3 Mbps71.3 Mbps73.6 Mbps73.6 Mbps
bench-validator-10validator60.2 Mbps67.5 Mbps72.2 Mbps72.2 Mbps

Network received

instance/network/received_bytes_count rate; summaries over the mixed phase

Table view
NodeClassMeanp95Peak 60-s meanMax
bench-observer-11observer3.3 Mbps3.8 Mbps4.2 Mbps4.2 Mbps
bench-validator-01validator5.4 Mbps6.5 Mbps7.3 Mbps7.3 Mbps
bench-validator-02validator6.1 Mbps7.4 Mbps8.1 Mbps8.1 Mbps
bench-validator-03validator5.8 Mbps7.9 Mbps8.0 Mbps8.0 Mbps
bench-validator-04validator5.8 Mbps8.2 Mbps8.4 Mbps8.4 Mbps
bench-validator-05validator12.2 Mbps31.7 Mbps43.9 Mbps43.9 Mbps
bench-validator-06validator6.0 Mbps7.1 Mbps7.7 Mbps7.7 Mbps
bench-validator-07validator6.7 Mbps7.8 Mbps7.9 Mbps7.9 Mbps
bench-validator-08validator5.3 Mbps7.2 Mbps7.5 Mbps7.5 Mbps
bench-validator-09validator6.7 Mbps8.5 Mbps8.9 Mbps8.9 Mbps
bench-validator-10validator5.8 Mbps7.4 Mbps8.2 Mbps8.2 Mbps

Consensus health

MetricQueryClassNodesMedian of node meansWorst peak 60-s meanMax
tn_primary_commit_latency_secondsp95validator103,378 ms26.9 s30.0 s
tn_engine_execution_duration_secondsp95validator104,255 ms12.2 s17.0 s
tn_engine_execution_duration_secondsp95observer13,984 ms6,219 ms7,000 ms
tn_worker_batch_size_bytesp95validator10237.8 KiB245.8 KiB246.2 KiB
tn_worker_batch_transactionsp95validator104871.48K1.48K
tn_executor_output_transactionsp95validator1020.4K51.2K75K
tn_primary_consensus_output_bytesmaxvalidator10677.2 KiB2.4 MiB4.4 MiB
tn_primary_consensus_output_bytesmaxobserver1691.5 KiB2.0 MiB2.3 MiB
tn_engine_queued_outputsmaxvalidator104.3788
tn_engine_queued_outputsmaxobserver12.137.758
tn_primary_roundmaxvalidator108031.78K1.78K
tn_primary_roundmaxobserver17861.75K1.78K
tn_primary_committed_roundmaxvalidator108011.78K1.78K
tn_primary_committed_roundmaxobserver17861.75K1.78K
tn_epoch_currentmaxvalidator107.7799
tn_epoch_currentmaxobserver17.7799
tn_batch_builder_pending_pool_transactionsmaxvalidator103879.74K9.74K
tn_batch_builder_pending_pool_transactionsmaxobserver1000
reth_process_resident_memory_bytesmaxvalidator105.71 GiB8.08 GiB8.12 GiB
reth_process_resident_memory_bytesmaxobserver12.74 GiB3.89 GiB3.91 GiB
tn_primary_subdags_committed_totalratevalidator100.4720.7670.844
tn_primary_subdags_committed_totalrateobserver1000
tn_primary_round_laglagvalidator102.134.757
tn_primary_round_laglagobserver1000
tn_batch_builder_seal_failures_totalratevalidator5000
tn_node_modemaxvalidator10011
tn_node_modemaxobserver1000
tn_primary_vote_requests_totalratevalidator108.72727.8
tn_primary_vote_requests_totalrateobserver1000
tn_primary_votes_created_totalratevalidator107.241414
tn_primary_votes_created_totalrateobserver1000
tn_primary_votes_recast_totalratevalidator10000
tn_primary_votes_recast_totalrateobserver1000
tn_primary_vote_requests_too_new_totalratevalidator10000
tn_primary_vote_requests_too_new_totalrateobserver1000
tn_primary_vote_timeouts_total{stage="exec_wait"}ratevalidator10000
tn_primary_vote_timeouts_total{stage="exec_wait"}rateobserver1000
tn_primary_vote_timeouts_total{stage="lead_sleep"}ratevalidator10000
tn_primary_vote_timeouts_total{stage="lead_sleep"}rateobserver1000
tn_primary_vote_timeouts_total{stage="parents"}ratevalidator10000
tn_primary_vote_timeouts_total{stage="parents"}rateobserver1000
tn_primary_vote_timeouts_total{stage="persist"}ratevalidator10000
tn_primary_vote_timeouts_total{stage="persist"}rateobserver1000
tn_primary_vote_timeouts_total{stage="sync_batches"}ratevalidator10000
tn_primary_vote_timeouts_total{stage="sync_batches"}rateobserver1000
tn_primary_vote_timeouts_total{stage="validate"}ratevalidator10000
tn_primary_vote_timeouts_total{stage="validate"}rateobserver1000
tn_primary_vote_timeouts_total{stage="vote_store"}ratevalidator10000
tn_primary_vote_timeouts_total{stage="vote_store"}rateobserver1000
tn_primary_vote_stage_seconds{stage="exec_wait"}p95validator101,503 ms2,425 ms2,425 ms
tn_primary_vote_stage_seconds{stage="lead_sleep"}p95validator100 ms0 ms0 ms
tn_primary_vote_stage_seconds{stage="lock"}p95validator101,562 ms9,709 ms9,739 ms
tn_primary_vote_stage_seconds{stage="parents"}p95validator101 ms19 ms27 ms
tn_primary_vote_stage_seconds{stage="persist"}p95validator10190 ms232 ms246 ms
tn_primary_vote_stage_seconds{stage="sync_batches"}p95validator100 ms0 ms0 ms
tn_primary_vote_stage_seconds{stage="validate"}p95validator102 ms4 ms4 ms
tn_primary_vote_stage_seconds{stage="vote_store"}p95validator1012 ms55 ms80 ms
tn_primary_leader_rounds_total{leader="2xQKratGEXurhrZgwZV1kaxJTtUDbskRJvyJTYRg4gAw",swapped="false",found="false"}ratevalidator100.002640.03330.0444
tn_primary_leader_rounds_total{leader="2xQKratGEXurhrZgwZV1kaxJTtUDbskRJvyJTYRg4gAw",swapped="false",found="true"}ratevalidator100.01820.08330.111
tn_primary_leader_rounds_total{leader="2xQKratGEXurhrZgwZV1kaxJTtUDbskRJvyJTYRg4gAw",swapped="true",found="false"}ratevalidator10000
tn_primary_leader_rounds_total{leader="2xQKratGEXurhrZgwZV1kaxJTtUDbskRJvyJTYRg4gAw",swapped="true",found="true"}ratevalidator100.003470.03330.0444
tn_primary_leader_rounds_total{leader="35UGYEik14WmgYJYWzf8e97o1jZtv6FDaLVz8SoCJiTU",swapped="false",found="false"}ratevalidator100.004170.050.0667
tn_primary_leader_rounds_total{leader="35UGYEik14WmgYJYWzf8e97o1jZtv6FDaLVz8SoCJiTU",swapped="false",found="true"}ratevalidator100.04570.08890.111
tn_primary_leader_rounds_total{leader="35UGYEik14WmgYJYWzf8e97o1jZtv6FDaLVz8SoCJiTU",swapped="true",found="false"}ratevalidator100.0008330.01670.0222
tn_primary_leader_rounds_total{leader="35UGYEik14WmgYJYWzf8e97o1jZtv6FDaLVz8SoCJiTU",swapped="true",found="true"}ratevalidator100.01120.08150.0889
tn_primary_leader_rounds_total{leader="3cTfbmwcFSdAFQE5G4Chtdjc5H2zBtutnJDes8KuYXov",swapped="false",found="false"}ratevalidator10000
tn_primary_leader_rounds_total{leader="3cTfbmwcFSdAFQE5G4Chtdjc5H2zBtutnJDes8KuYXov",swapped="false",found="true"}ratevalidator100.04990.08890.111
tn_primary_leader_rounds_total{leader="3cTfbmwcFSdAFQE5G4Chtdjc5H2zBtutnJDes8KuYXov",swapped="true",found="false"}ratevalidator10000
tn_primary_leader_rounds_total{leader="3cTfbmwcFSdAFQE5G4Chtdjc5H2zBtutnJDes8KuYXov",swapped="true",found="true"}ratevalidator100.01070.06670.0667
tn_primary_leader_rounds_total{leader="4zoSra4cxG6XAr6MS7duBukTBdvV7SswEswuFhKdTRX3",swapped="false",found="false"}ratevalidator100.002030.04440.0444
tn_primary_leader_rounds_total{leader="4zoSra4cxG6XAr6MS7duBukTBdvV7SswEswuFhKdTRX3",swapped="false",found="true"}ratevalidator100.03320.08890.111
tn_primary_leader_rounds_total{leader="4zoSra4cxG6XAr6MS7duBukTBdvV7SswEswuFhKdTRX3",swapped="true",found="false"}ratevalidator10000
tn_primary_leader_rounds_total{leader="4zoSra4cxG6XAr6MS7duBukTBdvV7SswEswuFhKdTRX3",swapped="true",found="true"}ratevalidator100.005420.03890.0444
tn_primary_leader_rounds_total{leader="9qbgQeg8qruCGKrjgbE3RZCeCzppUYPwzC5ZGs5zbrWN",swapped="false",found="false"}ratevalidator100.001160.03310.0441
tn_primary_leader_rounds_total{leader="9qbgQeg8qruCGKrjgbE3RZCeCzppUYPwzC5ZGs5zbrWN",swapped="false",found="true"}ratevalidator100.04860.08890.0889
tn_primary_leader_rounds_total{leader="9qbgQeg8qruCGKrjgbE3RZCeCzppUYPwzC5ZGs5zbrWN",swapped="true",found="false"}ratevalidator10000
tn_primary_leader_rounds_total{leader="9qbgQeg8qruCGKrjgbE3RZCeCzppUYPwzC5ZGs5zbrWN",swapped="true",found="true"}ratevalidator100.01080.050.0667
tn_primary_leader_rounds_total{leader="Br3WMnwqL5zXNtK4VwriZZyoRze3GRR9jzoYt4kRKaMu",swapped="false",found="false"}ratevalidator10000
tn_primary_leader_rounds_total{leader="Br3WMnwqL5zXNtK4VwriZZyoRze3GRR9jzoYt4kRKaMu",swapped="false",found="true"}ratevalidator100.04970.08330.111
tn_primary_leader_rounds_total{leader="Br3WMnwqL5zXNtK4VwriZZyoRze3GRR9jzoYt4kRKaMu",swapped="true",found="false"}ratevalidator10000
tn_primary_leader_rounds_total{leader="Br3WMnwqL5zXNtK4VwriZZyoRze3GRR9jzoYt4kRKaMu",swapped="true",found="true"}ratevalidator100.01120.06110.0889
tn_primary_leader_rounds_total{leader="CraLT3xtoP81xHLUVnUPzhnTcrbssD1DXga1dFjSTbxy",swapped="false",found="false"}ratevalidator100.008630.07220.0889
tn_primary_leader_rounds_total{leader="CraLT3xtoP81xHLUVnUPzhnTcrbssD1DXga1dFjSTbxy",swapped="false",found="true"}ratevalidator100.02030.08890.0889
tn_primary_leader_rounds_total{leader="CraLT3xtoP81xHLUVnUPzhnTcrbssD1DXga1dFjSTbxy",swapped="true",found="false"}ratevalidator100.0007440.03310.0441
tn_primary_leader_rounds_total{leader="CraLT3xtoP81xHLUVnUPzhnTcrbssD1DXga1dFjSTbxy",swapped="true",found="true"}ratevalidator100.009580.06670.0667
tn_primary_leader_rounds_total{leader="Cy8GPpAWuRTXmb3MRTxMLnehaF3Q9VL6WedTrNQkLT4J",swapped="false",found="false"}ratevalidator100.004170.050.0667
tn_primary_leader_rounds_total{leader="Cy8GPpAWuRTXmb3MRTxMLnehaF3Q9VL6WedTrNQkLT4J",swapped="false",found="true"}ratevalidator100.01640.07220.0889
tn_primary_leader_rounds_total{leader="Cy8GPpAWuRTXmb3MRTxMLnehaF3Q9VL6WedTrNQkLT4J",swapped="true",found="false"}ratevalidator10000
tn_primary_leader_rounds_total{leader="Cy8GPpAWuRTXmb3MRTxMLnehaF3Q9VL6WedTrNQkLT4J",swapped="true",found="true"}ratevalidator100.00250.03330.0444
tn_primary_leader_rounds_total{leader="FKvYJLZuwBRW37NmRrSh9VH7BBv95yLfGSb5Ckn1xhZp",swapped="false",found="false"}ratevalidator10000
tn_primary_leader_rounds_total{leader="FKvYJLZuwBRW37NmRrSh9VH7BBv95yLfGSb5Ckn1xhZp",swapped="false",found="true"}ratevalidator100.04970.08330.0889
tn_primary_leader_rounds_total{leader="FKvYJLZuwBRW37NmRrSh9VH7BBv95yLfGSb5Ckn1xhZp",swapped="true",found="false"}ratevalidator10000
tn_primary_leader_rounds_total{leader="FKvYJLZuwBRW37NmRrSh9VH7BBv95yLfGSb5Ckn1xhZp",swapped="true",found="true"}ratevalidator100.01410.05560.0667
tn_primary_leader_rounds_total{leader="GBSG38zYEybFE4ELTMRQFQWkSLZLn5goracd4TXx2tX2",swapped="false",found="false"}ratevalidator100.0003630.01650.0221
tn_primary_leader_rounds_total{leader="GBSG38zYEybFE4ELTMRQFQWkSLZLn5goracd4TXx2tX2",swapped="false",found="true"}ratevalidator100.04880.08890.0889
tn_primary_leader_rounds_total{leader="GBSG38zYEybFE4ELTMRQFQWkSLZLn5goracd4TXx2tX2",swapped="true",found="false"}ratevalidator10000
tn_primary_leader_rounds_total{leader="GBSG38zYEybFE4ELTMRQFQWkSLZLn5goracd4TXx2tX2",swapped="true",found="true"}ratevalidator100.01330.06110.0889
tn_primary_leader_commits_total{leader="2xQKratGEXurhrZgwZV1kaxJTtUDbskRJvyJTYRg4gAw"}ratevalidator100.02170.08330.111
tn_primary_leader_commits_total{leader="35UGYEik14WmgYJYWzf8e97o1jZtv6FDaLVz8SoCJiTU"}ratevalidator100.05690.1670.178
tn_primary_leader_commits_total{leader="3cTfbmwcFSdAFQE5G4Chtdjc5H2zBtutnJDes8KuYXov"}ratevalidator100.06010.1560.156
tn_primary_leader_commits_total{leader="4zoSra4cxG6XAr6MS7duBukTBdvV7SswEswuFhKdTRX3"}ratevalidator100.03860.10.111
tn_primary_leader_commits_total{leader="9qbgQeg8qruCGKrjgbE3RZCeCzppUYPwzC5ZGs5zbrWN"}ratevalidator100.05940.1280.156
tn_primary_leader_commits_total{leader="Br3WMnwqL5zXNtK4VwriZZyoRze3GRR9jzoYt4kRKaMu"}ratevalidator100.0610.1220.156
tn_primary_leader_commits_total{leader="CraLT3xtoP81xHLUVnUPzhnTcrbssD1DXga1dFjSTbxy"}ratevalidator100.02990.150.156
tn_primary_leader_commits_total{leader="Cy8GPpAWuRTXmb3MRTxMLnehaF3Q9VL6WedTrNQkLT4J"}ratevalidator100.01890.08890.111
tn_primary_leader_commits_total{leader="FKvYJLZuwBRW37NmRrSh9VH7BBv95yLfGSb5Ckn1xhZp"}ratevalidator100.06390.1280.156
tn_primary_leader_commits_total{leader="GBSG38zYEybFE4ELTMRQFQWkSLZLn5goracd4TXx2tX2"}ratevalidator100.06210.1390.178
tn_engine_queue_wait_secondsp95validator1018.9 s29.0 s29.0 s
tn_engine_queue_wait_secondsp95observer17,649 ms28.9 s29.0 s
tn_engine_persist_duration_secondsp95validator102,473 ms5,652 ms6,750 ms
tn_engine_persist_duration_secondsp95observer12,417 ms2,919 ms3,500 ms
tn_engine_busy_milliseconds_totalratevalidator109791.04K1.2K
tn_engine_busy_milliseconds_totalrateobserver19201.02K1.05K
tn_reth_persist_save_blocks_secondsp95validator10398 ms2,731 ms3,375 ms
tn_reth_persist_save_blocks_secondsp95observer1307 ms622 ms700 ms
tn_reth_persist_commit_secondsp95validator102,340 ms2,425 ms2,425 ms
tn_reth_persist_commit_secondsp95observer12,334 ms2,425 ms2,425 ms
tn_reth_persisted_accounts_totalratevalidator101.57K2.58K2.86K
tn_reth_persisted_accounts_totalrateobserver11.56K2.15K2.35K
tn_reth_persisted_storage_slots_totalratevalidator102.17K3.41K3.78K
tn_reth_persisted_storage_slots_totalrateobserver12.15K2.85K3.1K
tn_node_mode_transitions_total{to="cvv_inactive",reason="epoch_behind"}ratevalidator1000
tn_executor_rejoin_blocked_total{reason="attempts_exhausted"}ratevalidator1000
tn_executor_rejoin_blocked_total{reason="epoch_closed"}ratevalidator10.110.5280.578
tn_executor_rejoin_blocked_total{reason="epoch_mismatch"}ratevalidator1000
tn_executor_rejoin_blocked_total{reason="no_known_header"}ratevalidator1000
tn_executor_rejoin_blocked_total{reason="round_gap"}ratevalidator10.2810.5330.578
tn_executor_rejoin_blocked_total{reason="stale_head"}ratevalidator10.09410.3440.422
tn_worker_sync_serves_denied_total{worker="0",tier="committee"}ratevalidator10000
tn_worker_sync_serves_denied_total{worker="0",tier="committee"}rateobserver1000
tn_worker_sync_serves_denied_total{worker="0",tier="public"}ratevalidator10000
tn_worker_sync_serves_denied_total{worker="0",tier="public"}rateobserver1000
tn_worker_batch_requests_denied_total{worker="0",peer="21N2vp3GaZwbCt1bqcBkrg3e5ShjP3Ss4pDF92MtrtYNAVrxq1LNQyFeAty226eBSt7ywnRkAg8XcxztrJcLkSw82uJREWMvzAhnkZzMYLhSAxEggfxpnk41L398JkQheqr4"}ratevalidator20.1463.524.69
tn_worker_batch_requests_denied_total{worker="0",peer="22Pq8MwWk8jvrnMCaqVCSkTW3JKgC8sJwJ4V9sDAL7nypan2zh2YntVGVYnjM1J67BW6iw87wHL3GGxttJd8kv1evHyzv8X7wyCH6wsGP83Ua7f83EdWFELtZpCLzBpi7NRs"}ratevalidator20.0590.9331.2
tn_worker_batch_requests_denied_total{worker="0",peer="237jbxzcELWRFg5wb9XtWG7Vw9kacjAceJsiJ8P65XGgPgaA8wMRZKp1j7noKyimEdMEgbf5Xt1pZiGi7MSWDmyriPS3ENeZ7bqw7qKkYskJ4qNPz5F7aaCCGdfc9LZd1D9t"}ratevalidator30.09592.673.56
tn_worker_batch_requests_denied_total{worker="0",peer="241nv7f9rufDfN5aw3izxGs4awWbc2EeBnX8JeczXZb18KCxvgb7NPjGsdcMjQE8LA7TeFbeK7qW9W4LggWM15UphDvCVENdU7SwKD7qro7bzcQmVCssHULsDKcSMcxHYfqT"}ratevalidator301.371.37
tn_worker_batch_requests_denied_total{worker="0",peer="242iCdm1PnngYQ4Y3WTxt9ScYT9BmEJumr77vEaVnDMCj8b5bagbaRBafTLgt1Y4HuBtbC25CURfxhdm5jv3Yoen85Zy5BRHrXcjnJyGxGNmpiqzDPxsAojWpz7EH5g4STRw"}ratevalidator20.1042.553.4
tn_worker_batch_requests_denied_total{worker="0",peer="mY4mb1gk2adED3HTP3r8aLZhGqcCZmNDfzirXWyomvXB72kkd7SxQKNz857yzy98TS12X7LckY23G8coxSowXyMm93MxfbGviStaxD2g1cRp6Jmh68DbLoN9VVCT8ZX9Zxs"}ratevalidator30.003232.283
tn_worker_batch_requests_denied_total{worker="0",peer="n6cEqCe39mRWFoZ3mGZE4nbyGbhun8VSFMCWDvAJugFVR1WPLrgNiTyrBHoqFixkvQzr3ADuRD6X1XVfwjR5S4iQvq89DndaFwk8UGLze2JwuKD9zhazyQTwqrh7j5YKcPc"}ratevalidator20.1233.024.02
tn_worker_batch_requests_denied_total{worker="0",peer="ojV9SY5hHE4EEsZJzZLwqVpDfVfDBqUPnEsmNX9NBLUarrhPiggeSZonTDsctzH5bYoU9Wiu9RKc9sjja7hGHek8JHyZHwmYgiLD4awMtwJVa5Maig318iRyqXdw8cGsavt"}ratevalidator20.3154.255.67
tn_worker_batch_requests_denied_total{worker="0",peer="orWZphKCbZg4MyMgHMp5mzQ8gtRwYsSnzNGbgZDU73oRMbtSjag6qYmEdjqGJquoooJRRKPHX2H7WdioR6SUmvmgPomMje5g3VspvcseJF1VoFpMA8VMdWB9ug2WS31HdLM"}ratevalidator20.09392.33.07
tn_worker_batch_requests_denied_total{worker="0",peer="s434VVj1mKtfRuhV9ATgLq5tp1xEixtd42JZh7uYbjMUZdVPmq4YY6yLCoU497LfdukihSg3ytdqwgjGk7vxKqviUsWXQyGhyckLXLSvmBF87cz9e3ZcANSL55TK54eHp84"}ratevalidator20.08642.122.82
tn_worker_batch_requests_denied_total{worker="0",peer="z38HGKMnkMqbCiuDq3WbgzUxjSB3LvTm25K1d7Qg3DHLdmDuZwwaGGEaxxVc3FZQ5M5xVs8kzCys6TJE3FdUfnqQftxXMB4rivn4mZXfaMe4vgD5kQib9pp8zj6hZQMHbiQ"}ratevalidator20.07621.872.49
tn_worker_batch_requests_failed_total{worker="0",peer="21N2vp3GaZwbCt1bqcBkrg3e5ShjP3Ss4pDF92MtrtYNAVrxq1LNQyFeAty226eBSt7ywnRkAg8XcxztrJcLkSw82uJREWMvzAhnkZzMYLhSAxEggfxpnk41L398JkQheqr4"}ratevalidator20.02640.6460.8
tn_worker_batch_requests_failed_total{worker="0",peer="22Pq8MwWk8jvrnMCaqVCSkTW3JKgC8sJwJ4V9sDAL7nypan2zh2YntVGVYnjM1J67BW6iw87wHL3GGxttJd8kv1evHyzv8X7wyCH6wsGP83Ua7f83EdWFELtZpCLzBpi7NRs"}ratevalidator20.03950.9671.29
tn_worker_batch_requests_failed_total{worker="0",peer="237jbxzcELWRFg5wb9XtWG7Vw9kacjAceJsiJ8P65XGgPgaA8wMRZKp1j7noKyimEdMEgbf5Xt1pZiGi7MSWDmyriPS3ENeZ7bqw7qKkYskJ4qNPz5F7aaCCGdfc9LZd1D9t"}ratevalidator30.03480.9331.24
tn_worker_batch_requests_failed_total{worker="0",peer="241nv7f9rufDfN5aw3izxGs4awWbc2EeBnX8JeczXZb18KCxvgb7NPjGsdcMjQE8LA7TeFbeK7qW9W4LggWM15UphDvCVENdU7SwKD7qro7bzcQmVCssHULsDKcSMcxHYfqT"}ratevalidator300.3580.422
tn_worker_batch_requests_failed_total{worker="0",peer="242iCdm1PnngYQ4Y3WTxt9ScYT9BmEJumr77vEaVnDMCj8b5bagbaRBafTLgt1Y4HuBtbC25CURfxhdm5jv3Yoen85Zy5BRHrXcjnJyGxGNmpiqzDPxsAojWpz7EH5g4STRw"}ratevalidator20.041511.33
tn_worker_batch_requests_failed_total{worker="0",peer="mY4mb1gk2adED3HTP3r8aLZhGqcCZmNDfzirXWyomvXB72kkd7SxQKNz857yzy98TS12X7LckY23G8coxSowXyMm93MxfbGviStaxD2g1cRp6Jmh68DbLoN9VVCT8ZX9Zxs"}ratevalidator300.5990.797
tn_worker_batch_requests_failed_total{worker="0",peer="n6cEqCe39mRWFoZ3mGZE4nbyGbhun8VSFMCWDvAJugFVR1WPLrgNiTyrBHoqFixkvQzr3ADuRD6X1XVfwjR5S4iQvq89DndaFwk8UGLze2JwuKD9zhazyQTwqrh7j5YKcPc"}ratevalidator20.02290.4670.622
tn_worker_batch_requests_failed_total{worker="0",peer="ojV9SY5hHE4EEsZJzZLwqVpDfVfDBqUPnEsmNX9NBLUarrhPiggeSZonTDsctzH5bYoU9Wiu9RKc9sjja7hGHek8JHyZHwmYgiLD4awMtwJVa5Maig318iRyqXdw8cGsavt"}ratevalidator20.05750.8311.11
tn_worker_batch_requests_failed_total{worker="0",peer="orWZphKCbZg4MyMgHMp5mzQ8gtRwYsSnzNGbgZDU73oRMbtSjag6qYmEdjqGJquoooJRRKPHX2H7WdioR6SUmvmgPomMje5g3VspvcseJF1VoFpMA8VMdWB9ug2WS31HdLM"}ratevalidator20.0470.7290.933
tn_worker_batch_requests_failed_total{worker="0",peer="s434VVj1mKtfRuhV9ATgLq5tp1xEixtd42JZh7uYbjMUZdVPmq4YY6yLCoU497LfdukihSg3ytdqwgjGk7vxKqviUsWXQyGhyckLXLSvmBF87cz9e3ZcANSL55TK54eHp84"}ratevalidator20.03470.851.13
tn_worker_batch_requests_failed_total{worker="0",peer="z38HGKMnkMqbCiuDq3WbgzUxjSB3LvTm25K1d7Qg3DHLdmDuZwwaGGEaxxVc3FZQ5M5xVs8kzCys6TJE3FdUfnqQftxXMB4rivn4mZXfaMe4vgD5kQib9pp8zj6hZQMHbiQ"}ratevalidator20.09421.341.77
tn_node_epoch_record_signersmaxvalidator108.121010
tn_node_epoch_record_signersmaxobserver18.0399
tn_node_epoch_record_missing_signers_total{signer="2xQKratGEXurhrZgwZV1kaxJTtUDbskRJvyJTYRg4gAw"}ratevalidator10000
tn_node_epoch_record_missing_signers_total{signer="2xQKratGEXurhrZgwZV1kaxJTtUDbskRJvyJTYRg4gAw"}rateobserver1000
tn_node_epoch_record_missing_signers_total{signer="35UGYEik14WmgYJYWzf8e97o1jZtv6FDaLVz8SoCJiTU"}ratevalidator10000
tn_node_epoch_record_missing_signers_total{signer="35UGYEik14WmgYJYWzf8e97o1jZtv6FDaLVz8SoCJiTU"}rateobserver1000
tn_node_epoch_record_missing_signers_total{signer="3cTfbmwcFSdAFQE5G4Chtdjc5H2zBtutnJDes8KuYXov"}ratevalidator10000
tn_node_epoch_record_missing_signers_total{signer="3cTfbmwcFSdAFQE5G4Chtdjc5H2zBtutnJDes8KuYXov"}rateobserver1000
tn_node_epoch_record_missing_signers_total{signer="4zoSra4cxG6XAr6MS7duBukTBdvV7SswEswuFhKdTRX3"}ratevalidator10000
tn_node_epoch_record_missing_signers_total{signer="4zoSra4cxG6XAr6MS7duBukTBdvV7SswEswuFhKdTRX3"}rateobserver1000
tn_node_epoch_record_missing_signers_total{signer="9qbgQeg8qruCGKrjgbE3RZCeCzppUYPwzC5ZGs5zbrWN"}ratevalidator100.000780.01670.0222
tn_node_epoch_record_missing_signers_total{signer="9qbgQeg8qruCGKrjgbE3RZCeCzppUYPwzC5ZGs5zbrWN"}rateobserver10.0005750.01670.0222
tn_node_epoch_record_missing_signers_total{signer="Br3WMnwqL5zXNtK4VwriZZyoRze3GRR9jzoYt4kRKaMu"}ratevalidator10000
tn_node_epoch_record_missing_signers_total{signer="Br3WMnwqL5zXNtK4VwriZZyoRze3GRR9jzoYt4kRKaMu"}rateobserver1000
tn_node_epoch_record_missing_signers_total{signer="CraLT3xtoP81xHLUVnUPzhnTcrbssD1DXga1dFjSTbxy"}ratevalidator100.000780.01670.0222
tn_node_epoch_record_missing_signers_total{signer="CraLT3xtoP81xHLUVnUPzhnTcrbssD1DXga1dFjSTbxy"}rateobserver10.0005750.01670.0222
tn_node_epoch_record_missing_signers_total{signer="Cy8GPpAWuRTXmb3MRTxMLnehaF3Q9VL6WedTrNQkLT4J"}ratevalidator10000
tn_node_epoch_record_missing_signers_total{signer="Cy8GPpAWuRTXmb3MRTxMLnehaF3Q9VL6WedTrNQkLT4J"}rateobserver1000
tn_node_epoch_record_missing_signers_total{signer="FKvYJLZuwBRW37NmRrSh9VH7BBv95yLfGSb5Ckn1xhZp"}ratevalidator10000
tn_node_epoch_record_missing_signers_total{signer="FKvYJLZuwBRW37NmRrSh9VH7BBv95yLfGSb5Ckn1xhZp"}rateobserver1000
tn_node_epoch_record_missing_signers_total{signer="GBSG38zYEybFE4ELTMRQFQWkSLZLn5goracd4TXx2tX2"}ratevalidator100.000780.01670.0222
tn_node_epoch_record_missing_signers_total{signer="GBSG38zYEybFE4ELTMRQFQWkSLZLn5goracd4TXx2tX2"}rateobserver10.0005380.01560.0208
tn_network_outbound_request_duration_seconds{outcome="failure",class="batch"}p95validator421.1 s29.0 s29.0 s
tn_network_outbound_request_duration_seconds{outcome="failure",class="epoch_record"}p95validator329.0 s29.0 s29.0 s
tn_network_outbound_request_duration_seconds{outcome="failure",class="vote"}p95validator525.6 s29.0 s29.0 s
tn_network_outbound_request_duration_seconds{outcome="response",class="batch"}p95validator4467 ms622 ms622 ms
tn_network_outbound_request_duration_seconds{outcome="response",class="epoch_record"}p95validator2445 ms542 ms700 ms
tn_network_outbound_request_duration_seconds{outcome="response",class="epoch_record"}p95observer1243 ms248 ms250 ms
tn_network_outbound_request_duration_seconds{outcome="response",class="vote"}p95validator45,654 ms9,464 ms9,490 ms
tn_network_closed_admission_denied_total{network="bench",direction="in"}ratevalidator10000
tn_network_closed_admission_denied_total{network="bench",direction="in"}rateobserver1000
tn_network_closed_admission_denied_total{network="bench",direction="out"}ratevalidator10000
tn_network_closed_admission_denied_total{network="bench",direction="out"}rateobserver1000
tn_network_source_admission_denied_total{network="bench",direction="in",reason="address_full"}ratevalidator10000
tn_network_source_admission_denied_total{network="bench",direction="in",reason="address_full"}rateobserver1000
tn_network_source_admission_denied_total{network="bench",direction="in",reason="duplicate_connection"}ratevalidator10000
tn_network_source_admission_denied_total{network="bench",direction="in",reason="duplicate_connection"}rateobserver1000
tn_network_source_admission_denied_total{network="bench",direction="in",reason="invalid_limits"}ratevalidator10000
tn_network_source_admission_denied_total{network="bench",direction="in",reason="invalid_limits"}rateobserver1000
tn_network_source_admission_denied_total{network="bench",direction="in",reason="peer_full"}ratevalidator10000
tn_network_source_admission_denied_total{network="bench",direction="in",reason="peer_full"}rateobserver1000
tn_network_source_admission_denied_total{network="bench",direction="in",reason="poisoned"}ratevalidator10000
tn_network_source_admission_denied_total{network="bench",direction="in",reason="poisoned"}rateobserver1000
tn_network_source_admission_denied_total{network="bench",direction="in",reason="prefix_full"}ratevalidator10000
tn_network_source_admission_denied_total{network="bench",direction="in",reason="prefix_full"}rateobserver1000
tn_network_source_admission_denied_total{network="bench",direction="in",reason="process_full"}ratevalidator10000
tn_network_source_admission_denied_total{network="bench",direction="in",reason="process_full"}rateobserver1000
tn_network_source_admission_denied_total{network="bench",direction="in",reason="sources_full"}ratevalidator10000
tn_network_source_admission_denied_total{network="bench",direction="in",reason="sources_full"}rateobserver1000
tn_network_source_admission_denied_total{network="bench",direction="in",reason="unsupported_address"}ratevalidator10000
tn_network_source_admission_denied_total{network="bench",direction="in",reason="unsupported_address"}rateobserver1000
tn_network_source_admission_denied_total{network="bench",direction="out",reason="address_full"}ratevalidator10000
tn_network_source_admission_denied_total{network="bench",direction="out",reason="address_full"}rateobserver1000
tn_network_source_admission_denied_total{network="bench",direction="out",reason="duplicate_connection"}ratevalidator10000
tn_network_source_admission_denied_total{network="bench",direction="out",reason="duplicate_connection"}rateobserver1000
tn_network_source_admission_denied_total{network="bench",direction="out",reason="invalid_limits"}ratevalidator10000
tn_network_source_admission_denied_total{network="bench",direction="out",reason="invalid_limits"}rateobserver1000
tn_network_source_admission_denied_total{network="bench",direction="out",reason="peer_full"}ratevalidator10000
tn_network_source_admission_denied_total{network="bench",direction="out",reason="peer_full"}rateobserver1000
tn_network_source_admission_denied_total{network="bench",direction="out",reason="poisoned"}ratevalidator10000
tn_network_source_admission_denied_total{network="bench",direction="out",reason="poisoned"}rateobserver1000
tn_network_source_admission_denied_total{network="bench",direction="out",reason="prefix_full"}ratevalidator10000
tn_network_source_admission_denied_total{network="bench",direction="out",reason="prefix_full"}rateobserver1000
tn_network_source_admission_denied_total{network="bench",direction="out",reason="process_full"}ratevalidator10000
tn_network_source_admission_denied_total{network="bench",direction="out",reason="process_full"}rateobserver1000
tn_network_source_admission_denied_total{network="bench",direction="out",reason="sources_full"}ratevalidator10000
tn_network_source_admission_denied_total{network="bench",direction="out",reason="sources_full"}rateobserver1000
tn_network_source_admission_denied_total{network="bench",direction="out",reason="unsupported_address"}ratevalidator10000
tn_network_source_admission_denied_total{network="bench",direction="out",reason="unsupported_address"}rateobserver1000
tn_network_admission_modemaxvalidator10222
tn_network_admission_modemaxobserver1222
tn_network_admission_fallbackmaxvalidator10011
tn_network_admission_fallbackmaxobserver1000
tn_network_outbound_request_failures_total{kind="connection"}ratevalidator10000
tn_network_outbound_request_failures_total{kind="connection"}rateobserver1000
tn_network_outbound_request_failures_total{kind="dial"}ratevalidator1011.454.765.9
tn_network_outbound_request_failures_total{kind="dial"}rateobserver1000
tn_network_outbound_request_failures_total{kind="io"}ratevalidator101.577.088.53
tn_network_outbound_request_failures_total{kind="io"}rateobserver10.2220.7771.02
tn_network_outbound_request_failures_total{kind="timeout"}ratevalidator1017.953.469.8
tn_network_outbound_request_failures_total{kind="timeout"}rateobserver10.003960.02670.0416
tn_network_outbound_request_failures_total{kind="unsupported"}ratevalidator10000
tn_network_outbound_request_failures_total{kind="unsupported"}rateobserver1000
tn_network_inbound_requests_shed_total{class="batch",reason="admission"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="batch",reason="admission"}rateobserver1000
tn_network_inbound_requests_shed_total{class="batch",reason="queue_budget"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="batch",reason="queue_budget"}rateobserver1000
tn_network_inbound_requests_shed_total{class="batch",reason="queue_full"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="batch",reason="queue_full"}rateobserver1000
tn_network_inbound_requests_shed_total{class="batch",reason="serve_budget"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="batch",reason="serve_budget"}rateobserver1000
tn_network_inbound_requests_shed_total{class="batch",reason="unsubscribed"}ratevalidator100.015330.139
tn_network_inbound_requests_shed_total{class="batch",reason="unsubscribed"}rateobserver1000
tn_network_inbound_requests_shed_total{class="certificate_sync",reason="admission"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="certificate_sync",reason="admission"}rateobserver1000
tn_network_inbound_requests_shed_total{class="certificate_sync",reason="queue_budget"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="certificate_sync",reason="queue_budget"}rateobserver1000
tn_network_inbound_requests_shed_total{class="certificate_sync",reason="queue_full"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="certificate_sync",reason="queue_full"}rateobserver1000
tn_network_inbound_requests_shed_total{class="certificate_sync",reason="serve_budget"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="certificate_sync",reason="serve_budget"}rateobserver1000
tn_network_inbound_requests_shed_total{class="certificate_sync",reason="unsubscribed"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="certificate_sync",reason="unsubscribed"}rateobserver1000
tn_network_inbound_requests_shed_total{class="epoch_record",reason="admission"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="epoch_record",reason="admission"}rateobserver1000
tn_network_inbound_requests_shed_total{class="epoch_record",reason="queue_budget"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="epoch_record",reason="queue_budget"}rateobserver1000
tn_network_inbound_requests_shed_total{class="epoch_record",reason="queue_full"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="epoch_record",reason="queue_full"}rateobserver1000
tn_network_inbound_requests_shed_total{class="epoch_record",reason="serve_budget"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="epoch_record",reason="serve_budget"}rateobserver1000
tn_network_inbound_requests_shed_total{class="epoch_record",reason="unsubscribed"}ratevalidator100.2390.9961.21
tn_network_inbound_requests_shed_total{class="epoch_record",reason="unsubscribed"}rateobserver10.01830.06390.0833
tn_network_inbound_requests_shed_total{class="gossip",reason="admission"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="gossip",reason="admission"}rateobserver1000
tn_network_inbound_requests_shed_total{class="gossip",reason="queue_budget"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="gossip",reason="queue_budget"}rateobserver1000
tn_network_inbound_requests_shed_total{class="gossip",reason="queue_full"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="gossip",reason="queue_full"}rateobserver1000
tn_network_inbound_requests_shed_total{class="gossip",reason="serve_budget"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="gossip",reason="serve_budget"}rateobserver1000
tn_network_inbound_requests_shed_total{class="gossip",reason="unsubscribed"}ratevalidator1000.1890.245
tn_network_inbound_requests_shed_total{class="gossip",reason="unsubscribed"}rateobserver1000
tn_network_inbound_requests_shed_total{class="other",reason="admission"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="other",reason="admission"}rateobserver1000
tn_network_inbound_requests_shed_total{class="other",reason="queue_budget"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="other",reason="queue_budget"}rateobserver1000
tn_network_inbound_requests_shed_total{class="other",reason="queue_full"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="other",reason="queue_full"}rateobserver1000
tn_network_inbound_requests_shed_total{class="other",reason="serve_budget"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="other",reason="serve_budget"}rateobserver1000
tn_network_inbound_requests_shed_total{class="other",reason="unsubscribed"}ratevalidator100.02160.1170.141
tn_network_inbound_requests_shed_total{class="other",reason="unsubscribed"}rateobserver1000
tn_network_inbound_requests_shed_total{class="vote",reason="admission"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="vote",reason="admission"}rateobserver1000
tn_network_inbound_requests_shed_total{class="vote",reason="queue_budget"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="vote",reason="queue_budget"}rateobserver1000
tn_network_inbound_requests_shed_total{class="vote",reason="queue_full"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="vote",reason="queue_full"}rateobserver1000
tn_network_inbound_requests_shed_total{class="vote",reason="serve_budget"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="vote",reason="serve_budget"}rateobserver1000
tn_network_inbound_requests_shed_total{class="vote",reason="unsubscribed"}ratevalidator100.23719.426.3
tn_network_inbound_requests_shed_total{class="vote",reason="unsubscribed"}rateobserver1000
tn_network_inbound_request_service_seconds{class="batch"}p95validator451 ms80 ms102 ms
tn_network_inbound_request_service_seconds{class="epoch_record"}p95validator432 ms1,295 ms1,750 ms
tn_network_inbound_request_service_seconds{class="epoch_record"}p95observer134 ms180 ms195 ms
tn_network_inbound_request_service_seconds{class="vote"}p95validator63,401 ms9,710 ms9,718 ms
tn_network_established_connectionsmaxvalidator1019.42121
tn_network_established_connectionsmaxobserver118.82020
tn_worker_sync_streams_shed_total{reason="budget_exhausted"}ratevalidator90.000680.09460.0946
tn_worker_sync_streams_shed_total{reason="budget_exhausted"}rateobserver1000
tn_worker_sync_streams_shed_total{reason="denied"}ratevalidator100.2848.5911.4
tn_worker_sync_streams_shed_total{reason="denied"}rateobserver10.1774.044.04
tn_worker_batch_fetch_duration_secondsp95validator106 ms3,262 ms4,583 ms
tn_executor_batch_fetch_failures_totalratevalidator10000
tn_executor_batch_fetch_failures_totalrateobserver1000
tn_primary_votes_received_totalratevalidator105.829.4710.4
tn_primary_votes_received_totalrateobserver1000
tn_primary_vote_request_failures_totalratevalidator100.04272.383.03
tn_primary_vote_request_failures_totalrateobserver1000
tn_primary_certificates_formed_totalratevalidator100.9691.581.73
tn_primary_certificates_formed_totalrateobserver1000
tn_primary_certificate_form_duration_secondsp95validator101,203 ms9,750 ms9,750 ms
tn_engine_outputs_executed_totalratevalidator100.4740.7670.822
tn_engine_outputs_executed_totalrateobserver10.4720.7720.778
tn_engine_persist_failures_totalratevalidator10000
tn_engine_persist_failures_totalrateobserver1000
tn_node_consensus_sync_distancemaxvalidator104.86181183
tn_node_consensus_sync_distancemaxobserver12.678.511
reth_db_table_size{table="AccountChangeSets"}maxvalidator10280 MiB418 MiB419 MiB
reth_db_table_size{table="AccountChangeSets"}maxobserver1281 MiB417 MiB419 MiB
reth_db_table_size{table="AccountsHistory"}maxvalidator1030 MiB48 MiB48 MiB
reth_db_table_size{table="AccountsHistory"}maxobserver130 MiB48 MiB48 MiB
reth_db_table_size{table="HashedAccounts"}maxvalidator1019 MiB31 MiB31 MiB
reth_db_table_size{table="HashedAccounts"}maxobserver119 MiB31 MiB31 MiB
reth_db_table_size{table="HashedStorages"}maxvalidator1042 MiB46 MiB46 MiB
reth_db_table_size{table="HashedStorages"}maxobserver142 MiB46 MiB46 MiB
reth_db_table_size{table="PlainAccountState"}maxvalidator1016 MiB26 MiB26 MiB
reth_db_table_size{table="PlainAccountState"}maxobserver116 MiB26 MiB26 MiB
reth_db_table_size{table="PlainStorageState"}maxvalidator1041 MiB46 MiB46 MiB
reth_db_table_size{table="PlainStorageState"}maxobserver142 MiB46 MiB46 MiB
reth_db_table_size{table="StorageChangeSets"}maxvalidator10341 MiB539 MiB541 MiB
reth_db_table_size{table="StorageChangeSets"}maxobserver1341 MiB538 MiB541 MiB
reth_db_table_size{table="StoragesHistory"}maxvalidator1092 MiB109 MiB109 MiB
reth_db_table_size{table="StoragesHistory"}maxobserver192 MiB109 MiB109 MiB
reth_db_table_size{table="TransactionHashNumbers"}maxvalidator10646 MiB937 MiB939 MiB
reth_db_table_size{table="TransactionHashNumbers"}maxobserver1646 MiB936 MiB939 MiB
reth_db_table_size{table="TransactionSenders"}maxvalidator10332 MiB482 MiB483 MiB
reth_db_table_size{table="TransactionSenders"}maxobserver1333 MiB481 MiB483 MiB
reth_db_freelistmaxvalidator1038.8K77.2K80.7K
reth_db_freelistmaxobserver138.9K76.3K79.9K
reth_database_transaction_commit_write_duration_secondsp95validator105 ms5 ms5 ms
reth_database_transaction_commit_write_duration_secondsp95observer15 ms5 ms5 ms
reth_database_transaction_commit_sync_duration_secondsp95validator102,333 ms2,425 ms2,425 ms
reth_database_transaction_commit_sync_duration_secondsp95observer12,333 ms2,425 ms2,425 ms
reth_database_transaction_commit_whole_duration_secondsp95validator102,333 ms2,425 ms2,425 ms
reth_database_transaction_commit_whole_duration_secondsp95observer12,333 ms2,425 ms2,425 ms
tn_engine_recover_secondsp95validator10258 ms2,324 ms2,369 ms
tn_engine_recover_secondsp95observer1245 ms634 ms700 ms
tn_engine_evm_secondsp95validator10510 ms1,432 ms1,825 ms
tn_engine_evm_secondsp95observer1479 ms1,347 ms1,788 ms
tn_engine_root_secondsp95validator10945 ms2,948 ms3,750 ms
tn_engine_root_secondsp95observer1841 ms2,188 ms2,350 ms
tn_engine_finish_secondsp95validator10288 ms1,025 ms1,300 ms
tn_engine_finish_secondsp95observer1207 ms553 ms762 ms
tn_engine_forward_wait_secondsp95validator10278 ms9,077 ms9,571 ms
tn_engine_forward_wait_secondsp95observer1113 ms3,703 ms4,708 ms
tn_engine_channel_depthmaxvalidator100.012514
tn_engine_channel_depthmaxobserver10.006250.251
tn_engine_executed_transactions_totalratevalidator103.49K5.45K5.96K
tn_engine_executed_transactions_totalrateobserver13.47K4.41K4.91K
tn_executor_execution_wait_secondsp95validator105 ms30.0 s30.0 s
tn_primary_commit_run_ahead_blocked_totalratevalidator10000
tn_primary_commit_run_ahead_blocked_totalrateobserver1000
tn_primary_consensus_exec_lag_roundsmaxvalidator1010.458.562
tn_primary_consensus_exec_lag_roundsmaxobserver15.341924
tn_primary_commit_lag_roundsmaxvalidator102.134.757
tn_primary_commit_lag_roundsmaxobserver1000
tn_primary_rejoin_grace_holds_totalratevalidator10000
tn_primary_rejoin_grace_holds_totalrateobserver1000
tn_primary_votes_refused_inactive_totalratevalidator1002727.8
tn_primary_votes_refused_inactive_totalrateobserver1000
tn_primary_votes_refused_execution_behind_totalratevalidator101.057.998
tn_primary_votes_refused_execution_behind_totalrateobserver1000
tn_batch_builder_pool_age_secondsp95validator1013.3 s30.0 s30.0 s
tn_batch_builder_builds_total{reason="bytes_full"}ratevalidator10000
tn_batch_builder_builds_total{reason="bytes_full"}rateobserver1000
tn_batch_builder_builds_total{reason="drained"}ratevalidator100.6151.521.56
tn_batch_builder_builds_total{reason="drained"}rateobserver1000
tn_batch_builder_builds_total{reason="gas_full"}ratevalidator100.8062.873.31
tn_batch_builder_builds_total{reason="gas_full"}rateobserver1000
tn_worker_batch_report_secondsp95validator100 ms2 ms2 ms
tn_primary_digest_wait_secondsp95validator102,563 ms24.5 s30.0 s
tn_primary_headers_proposed_total{reason="max_delay"}ratevalidator100.03240.1440.178
tn_primary_headers_proposed_total{reason="min_delay"}ratevalidator100.8491.541.62
tn_primary_headers_proposed_total{reason="parents_and_threshold"}ratevalidator100.1131.241.29
tn_primary_digests_requeued_total{cause="canceled"}ratevalidator100.052912.913.3
tn_primary_digests_requeued_total{cause="committed"}ratevalidator100.02960.3560.422
tn_executor_consensus_pack_persist_secondsp95validator10229 ms445 ms462 ms
tn_batch_builder_seal_duration_secondsp95validator10274 ms788 ms788 ms
tn_worker_quorum_wait_secondsp95validator10270 ms788 ms788 ms
tn_primary_proposer_pending_digestsmaxvalidator101.3977.883
tn_primary_proposer_pending_digestsmaxobserver1000
reth_transaction_pool_pending_pool_transactionsmaxvalidator103868.94K9.74K
reth_transaction_pool_pending_pool_transactionsmaxobserver1000
reth_transaction_pool_queued_pool_transactionsmaxvalidator101.95K9.32K9.32K
reth_transaction_pool_queued_pool_transactionsmaxobserver1000
reth_transaction_pool_all_transactions_by_hashmaxvalidator104.06K10K13.9K
reth_transaction_pool_all_transactions_by_hashmaxobserver1000
reth_storage_providers_database_save_blocks_totalp95validator10277 ms1,901 ms2,473 ms
reth_storage_providers_database_save_blocks_totalp95observer1217 ms373 ms373 ms
reth_storage_providers_database_save_blocks_mdbxp95validator10277 ms1,901 ms2,473 ms
reth_storage_providers_database_save_blocks_mdbxp95observer1217 ms373 ms373 ms
reth_storage_providers_database_save_blocks_sfp95validator1046 ms369 ms553 ms
reth_storage_providers_database_save_blocks_sfp95observer158 ms142 ms186 ms
reth_storage_providers_database_save_blocks_commit_mdbxp95validator101,419 ms1,889 ms1,907 ms
reth_storage_providers_database_save_blocks_commit_mdbxp95observer11,513 ms1,771 ms1,803 ms
reth_storage_providers_database_save_blocks_commit_sfp95validator10218 ms1,102 ms1,121 ms
reth_storage_providers_database_save_blocks_commit_sfp95observer1109 ms271 ms277 ms
reth_storage_providers_database_save_blocks_write_statep95validator1054 ms373 ms501 ms
reth_storage_providers_database_save_blocks_write_statep95observer143 ms74 ms85 ms
reth_storage_providers_database_save_blocks_write_hashed_statep95validator1033 ms391 ms483 ms
reth_storage_providers_database_save_blocks_write_hashed_statep95observer126 ms37 ms45 ms
reth_storage_providers_database_save_blocks_write_trie_updatesp95validator1040 ms198 ms259 ms
reth_storage_providers_database_save_blocks_write_trie_updatesp95observer131 ms45 ms48 ms
reth_database_transaction_commit_preparation_duration_secondsp95validator105 ms22 ms29 ms
reth_database_transaction_commit_preparation_duration_secondsp95observer15 ms5 ms5 ms
reth_database_transaction_commit_gc_wallclock_duration_secondsp95validator105 ms49 ms70 ms
reth_database_transaction_commit_gc_wallclock_duration_secondsp95observer15 ms5 ms5 ms
reth_database_transaction_commit_gc_cputime_duration_secondsp95validator105 ms5 ms5 ms
reth_database_transaction_commit_gc_cputime_duration_secondsp95observer15 ms5 ms5 ms
reth_database_transaction_commit_audit_duration_secondsp95validator105 ms5 ms5 ms
reth_database_transaction_commit_audit_duration_secondsp95observer15 ms5 ms5 ms
reth_database_transaction_commit_ending_duration_secondsp95validator105 ms5 ms5 ms
reth_database_transaction_commit_ending_duration_secondsp95observer15 ms5 ms5 ms

Commit latency p95

tn_primary_commit_latency_seconds

Round lag

tn_primary_round − tn_primary_committed_round

Engine queued outputs

tn_engine_queued_outputs, max by node

Certification by node

The share of each validator's own headers that formed a certificate, per window (tools/cert-rate.py on the node logs). bench-validator-04, bench-validator-08 and bench-validator-10 certified fewer than 90 % of their own headers in a window where they proposed at least 50. Own leader % is the validator's share of the committed leaders and not committed % the share of its certificates no committed leader reached, both over the run window; the fair share is 100 / 10 validators = 10.0 %. Greyed: fewer than 50 headers proposed, not judged.

ValidatorRegionphase preamblephase warmupphase mixedphase cooldownrunOwn leader %Not committed %
bench-validator-01us-west2 (Los Angeles)100.599.599.8100.099.912.60.2
bench-validator-02europe-west2 (London)98.499.598.399.898.610.88.5
bench-validator-03australia-southeast1 (Sydney)100.0100.099.799.899.813.20.7
bench-validator-04 below 90 %asia-northeast1 (Tokyo)99.553.049.699.860.26.72.7
bench-validator-05us-east4 (Ashburn)100.599.5100.1100.0100.112.28.6
bench-validator-06europe-west4 (Netherlands)99.5100.098.999.899.110.49.4
bench-validator-07asia-southeast1 (Singapore)100.599.597.799.898.313.02.0
bench-validator-08 below 90 %asia-east2 (Hong Kong)100.054.044.499.856.46.12.6
bench-validator-09asia-northeast3 (Seoul)100.0100.090.299.892.812.12.0
bench-validator-10 below 90 %asia-south1 (Mumbai)97.997.567.999.676.28.97.0

On-node sampler

NodeClassPeak RssAnonPeak RssFilePeak cachePSI memPSI ioPSI cpudb growthstatic_files growthRoot usedOOM Δ
bench-observer-11observer1011 MiB2.93 GiB– 0.0 %32.8 %0.4 %0 B2.90 GiB14.0 %0
bench-validator-01validator4.37 GiB3.18 GiB– 0.0 %30.6 %1.7 %0 B2.90 GiB16.0 %0
bench-validator-02validator4.93 GiB3.19 GiB– 0.0 %31.3 %2.2 %0 B2.90 GiB16.0 %0
bench-validator-03validator4.22 GiB3.21 GiB– 0.0 %31.4 %2.0 %0 B2.90 GiB16.0 %0
bench-validator-04validator3.97 GiB3.15 GiB– 0.0 %32.1 %2.7 %0 B2.90 GiB16.0 %0
bench-validator-05validator3.37 GiB3.28 GiB– 0.0 %29.4 %1.1 %0 B2.90 GiB16.0 %0
bench-validator-06validator4.44 GiB3.15 GiB– 0.0 %31.6 %2.0 %0 B2.90 GiB16.0 %0
bench-validator-07validator4.05 GiB3.18 GiB– 0.0 %30.9 %2.0 %0 B2.90 GiB16.0 %0
bench-validator-08validator4.25 GiB3.21 GiB– 0.0 %34.2 %1.5 %0 B2.90 GiB16.0 %0
bench-validator-09validator4.19 GiB3.20 GiB– 0.0 %31.6 %2.0 %0 B2.90 GiB16.0 %0
bench-validator-10validator4.76 GiB3.10 GiB– 0.0 %32.4 %2.0 %0 B2.90 GiB16.0 %0

RssAnon vs RssFile

/proc/<pid>/status of the telcoin process on the node with the largest RSS

consensus-db/cache size

Not collected: consensus-db/cache size.

Pressure stall (some, avg10)

/proc/pressure on the node with the highest memory PSI

Headroom

Per role and per node, from sizing.json (p95 over the landed window; PSI and RssAnon max over the whole sampler run). The lines follow the hardware doc's tier rules: CPU p95 over 70 % of vCPUs; memory used p95 over 75 % of RAM, or telcoin RSS p95 over 85 % of RAM; disk write p95 over 60 % of the PD MiB/s limit, or write IOPS p95 over 60 % of the IOPS limit; IO pressure over 20 % or memory pressure over 5 % (PSI some avg10, max); network p95 over 50 % of 1 Gbps; root disk over 80 %. A line crossed on any node puts the role at the limit; a nonzero reading within 15 points of its line is tight; otherwise the role has headroom. OOM kills, disk throttling and degraded disk performance are flagged too.

SignalClassWorst nodeValueShare of capacityLineNodes over / tightStatus
CPU p95validatorbench-validator-0822.2 %22.2 %> 70 %0 / 0 headroom
Memory used p95validatorbench-validator-0234.2 %34.2 %> 75 %0 / 0 headroom
telcoin RSS p95validatorbench-validator-028.11 GB47.2 %> 85 %0 / 0 headroom
PSI memory maxvalidatorbench-validator-010.0 %0.0 %> 5 %0 / 0 headroom
RssAnon peakvalidatorbench-validator-025.29 GB30.8 %info– info
Disk write p95validatorbench-validator-10595 MB/s70.9 %> 60 %10 / 0 at the limit
Write IOPS p95validatorbench-validator-058,90944.5 %> 60 %0 / 0 headroom
PSI io maxvalidatorbench-validator-0834.2 %34.2 %> 20 %10 / 0 at the limit
Root disk maxvalidatorbench-validator-0715.6 %15.6 %> 80 %0 / 0 headroom
Network p95validatorbench-validator-0882 Mbps8.2 %> 50 %0 / 0 headroom
Consensus cache peakvalidator–––info– not collected
OOM killsvalidator––anyany– none
Disk throttlingvalidator––metric type unavailable in projectany– not collected
Disk performance degradedvalidator––any sampleany– none
CPU p95observerbench-observer-1113.2 %13.2 %> 70 %0 / 0 headroom
Memory used p95observerbench-observer-1111.7 %11.7 %> 75 %0 / 0 headroom
telcoin RSS p95observerbench-observer-113.79 GB22.1 %> 85 %0 / 0 headroom
PSI memory maxobserverbench-observer-110.0 %0.0 %> 5 %0 / 0 headroom
RssAnon peakobserverbench-observer-111.07 GB6.2 %info– info
Disk write p95observerbench-observer-11588 MB/s70.1 %> 60 %1 / 0 at the limit
Write IOPS p95observerbench-observer-115,20626.0 %> 60 %0 / 0 headroom
PSI io maxobserverbench-observer-1132.8 %32.8 %> 20 %1 / 0 at the limit
Root disk maxobserverbench-observer-1113.2 %13.2 %> 80 %0 / 0 headroom
Network p95observerbench-observer-114 Mbps0.4 %> 50 %0 / 0 headroom
Consensus cache peakobserver–––info– not collected
OOM killsobserver––anyany– none
Disk throttlingobserver––metric type unavailable in projectany– not collected
Disk performance degradedobserver––any sampleany– none

Per node

NodeClassCPU p95Memory used p95telcoin RSS p95PSI memory maxRssAnon peakDisk write p95Write IOPS p95PSI io maxRoot disk maxNetwork p95Consensus cache peak
Line> 70 %> 75 %> 85 %> 5 %info> 60 %> 60 %> 20 %> 80 %> 50 %info
bench-validator-01validator20.7 %31.4 %7.49 GB 44 %0.0 %4.69 GB 27 %576 MB/s 69 %5,057 25 %30.6 %15.6 %71 Mbps 7 %–
bench-validator-02validator20.5 %34.2 %8.11 GB 47 %0.0 %5.29 GB 31 %593 MB/s 71 %4,137 21 %31.3 %15.6 %71 Mbps 7 %–
bench-validator-03validator22.2 %30.1 %7.47 GB 43 %0.0 %4.53 GB 26 %580 MB/s 69 %5,323 27 %31.4 %15.5 %73 Mbps 7 %–
bench-validator-04validator20.9 %29.0 %7.30 GB 42 %0.0 %4.26 GB 25 %591 MB/s 70 %2,683 13 %32.1 %15.5 %74 Mbps 7 %–
bench-validator-05validator21.2 %27.0 %5.88 GB 34 %0.0 %3.62 GB 21 %547 MB/s 65 %8,909 45 %29.4 %15.1 %78 Mbps 8 %–
bench-validator-06validator20.9 %32.0 %7.78 GB 45 %0.0 %4.77 GB 28 %590 MB/s 70 %2,866 14 %31.6 %15.5 %70 Mbps 7 %–
bench-validator-07validator20.5 %30.0 %7.40 GB 43 %0.0 %4.35 GB 25 %588 MB/s 70 %4,885 24 %30.9 %15.6 %72 Mbps 7 %–
bench-validator-08validator22.2 %31.9 %7.75 GB 45 %0.0 %4.56 GB 27 %591 MB/s 70 %5,897 29 %34.2 %15.6 %82 Mbps 8 %–
bench-validator-09validator20.8 %30.7 %7.53 GB 44 %0.0 %4.50 GB 26 %593 MB/s 71 %4,838 24 %33.4 %15.6 %73 Mbps 7 %–
bench-validator-10validator19.9 %33.3 %7.87 GB 46 %0.0 %5.11 GB 30 %595 MB/s 71 %2,829 14 %32.4 %15.5 %68 Mbps 7 %–
bench-observer-11observer13.2 %11.7 %3.79 GB 22 %0.0 %1.07 GB 6 %588 MB/s 70 %5,206 26 %32.8 %13.2 %4 Mbps 0 %–

Small figures are the share of capacity: RAM for RSS, the PD limit for disk write (VM cap applied), 1 Gbps for network. Hover a cell for its JSON path.

Verdict

Fleet A vs Fleet B

A = Fleet c3-hdb · c3-highcpu-8 · r20261107-1504
B = Fleet c3-hdb · c3-highcpu-8 · r20261105-0803

Verdict

at the limit c3-highcpu-8 validators at 4,939 sustained TPS: at the limit on disk. Disk at the limit: write p95 547–595 MB/s, up to 70.9 % of the 800 MiB/s limit, over the 60 % line on all 10 validators (bench-validator-01, 02, 03, 04, 05, 06, 07, 08, 09, 10); IO pressure (PSI some avg10) max 29.4–34.2 %, over the 20 % line on all 10 validators (bench-validator-01, 02, 03, 04, 05, 06, 07, 08, 09, 10). CPU has headroom: CPU p95 22.2 % of 8 vCPUs (bench-validator-08). Memory has headroom: memory used p95 34.2 % of RAM (bench-validator-02); telcoin RSS p95 8.11 GB, 47.2 % of 16 GiB RAM (bench-validator-02); memory pressure (PSI some avg10) max 0.0 % (bench-validator-01). Network has headroom: network p95 82 Mbps, 8.2 % of 1 Gbps (bench-validator-08). Next step: a faster volume (pd-ssd adds 30 IOPS and 0.48 MiB/s per GB up to the per-VM cap; past the cap, a larger VM).

at the limit c3-highcpu-8 observers at 4,939 sustained TPS: at the limit on disk. Disk at the limit: write p95 588 MB/s, 70.1 % of the 800 MiB/s limit on bench-observer-11, over the 60 % line; IO pressure (PSI some avg10) max 32.8 % on bench-observer-11, over the 20 % line. CPU has headroom: CPU p95 13.2 % of 8 vCPUs (bench-observer-11). Memory has headroom: memory used p95 11.7 % of RAM (bench-observer-11); telcoin RSS p95 3.79 GB, 22.1 % of 16 GiB RAM (bench-observer-11); memory pressure (PSI some avg10) max 0.0 % (bench-observer-11). Network has headroom: network p95 4 Mbps, 0.4 % of 1 Gbps (bench-observer-11). Next step: a faster volume (pd-ssd adds 30 IOPS and 0.48 MiB/s per GB up to the per-VM cap; past the cap, a larger VM).

tight c3-highcpu-8 validators at 5,525 sustained TPS: tight on memory and disk. Memory tight: memory pressure (PSI some avg10) max 0.1 % (bench-validator-08), tight against the 5 % line. Disk tight: write IOPS p95 9,882, 49.4 % of the 20,000 IOPS limit (bench-validator-10), tight against the 60 % line; IO pressure (PSI some avg10) max 17.4 % (bench-validator-08), tight against the 20 % line. CPU has headroom: CPU p95 36.7 % of 8 vCPUs (bench-validator-09). Network has headroom: network p95 103 Mbps, 10.3 % of 1 Gbps (bench-validator-09).

tight c3-highcpu-8 observers at 5,525 sustained TPS: tight on disk. Disk tight: IO pressure (PSI some avg10) max 12.6 % (bench-observer-11), tight against the 20 % line. CPU has headroom: CPU p95 27.4 % of 8 vCPUs (bench-observer-11). Memory has headroom: memory used p95 13.5 % of RAM (bench-observer-11); telcoin RSS p95 3.45 GB, 20.1 % of 16 GiB RAM (bench-observer-11); memory pressure (PSI some avg10) max 0.0 % (bench-observer-11). Network has headroom: network p95 13 Mbps, 1.3 % of 1 Gbps (bench-observer-11).

Validators: Fleet A (c3-highcpu-8) is at the limit on disk, with headroom on CPU, memory and network; Fleet B (c3-highcpu-8) is tight on memory and disk, with headroom on CPU and network. Disk is at the limit on Fleet A (c3-highcpu-8) only. Fleet B (c3-highcpu-8) stays under the disk lines that Fleet A (c3-highcpu-8) crosses.

Observers: Fleet A (c3-highcpu-8) is at the limit on disk, with headroom on CPU, memory and network; Fleet B (c3-highcpu-8) is tight on disk, with headroom on CPU, memory and network. Disk is at the limit on Fleet A (c3-highcpu-8) only. Fleet B (c3-highcpu-8) stays under the disk lines that Fleet A (c3-highcpu-8) crosses.

KPIFleet AFleet BB ÷ A
Sustained mixed TPS (own)4,939.2 tx/s5,524.8 tx/s1.12×
Sustained mixed TPS (chain)4,939.2 tx/s5,524.8 tx/s1.12×
Landed / accepted99.6 %99.9 %1.00×
Reverted (own)00–
Latency p502,809 ms3,303 ms1.18×
Latency p9514,404 ms12,837 ms0.89×
Gas / s (own)174M gas/s231M gas/s1.33×
Capacity at mix7,705.4 tx/s9,431.4 tx/s1.22×

Headroom, worst node per class

SignalClassFleet AFleet BLine
CPU p95validator22.2 % headroom36.7 % headroom> 70 %
Memory used p95validator34.2 % headroom43.5 % headroom> 75 %
telcoin RSS p95validator8.11 GB 47.2 % headroom9.28 GB 54.0 % headroom> 85 %
PSI memory maxvalidator0.0 % headroom0.1 % tight> 5 %
Disk write p95validator595 MB/s 70.9 % at the limit325 MB/s 38.7 % headroom> 60 %
Write IOPS p95validator8,909 44.5 % headroom9,882 49.4 % tight> 60 %
PSI io maxvalidator34.2 % at the limit17.4 % tight> 20 %
Root disk maxvalidator15.6 % headroom63.9 % headroom> 80 %
Network p95validator82 Mbps 8.2 % headroom103 Mbps 10.3 % headroom> 50 %
CPU p95observer13.2 % headroom27.4 % headroom> 70 %
Memory used p95observer11.7 % headroom13.5 % headroom> 75 %
telcoin RSS p95observer3.79 GB 22.1 % headroom3.45 GB 20.1 % headroom> 85 %
PSI memory maxobserver0.0 % headroom0.0 % headroom> 5 %
Disk write p95observer588 MB/s 70.1 % at the limit325 MB/s 38.7 % headroom> 60 %
Write IOPS p95observer5,206 26.0 % headroom8,830 44.1 % headroom> 60 %
PSI io maxobserver32.8 % at the limit12.6 % tight> 20 %
Root disk maxobserver13.2 % headroom62.1 % headroom> 80 %
Network p95observer4 Mbps 0.4 % headroom13 Mbps 1.3 % headroom> 50 %

Own TPS, trailing 60-s mean

aligned on each fleet's mixed-phase start

CPU, hottest validator

aligned on each fleet's mixed-phase start

Memory used, hottest validator

aligned on each fleet's mixed-phase start

telcoin RSS, hottest validator

aligned on each fleet's mixed-phase start

Disk MiB/s, hottest validator

aligned on each fleet's mixed-phase start

Network sent, hottest validator

aligned on each fleet's mixed-phase start

Reference chains

Not collected: reference chains.