This is the full report of the run “The always-on vote fix on Hyperdisk Balanced at 20,000 IOPS”. Back to the run page.
r20261113-1351 · Fleet c3-hdb · c3-highcpu-8

Telcoin Network benchmark: mixed workload

Throughput is measured from blocks the chain produced (own senders vs every sender), resource use from Cloud Monitoring, Prometheus and the on-node sampler. Summaries cover the mixed phase. Hover any number for the JSON field it came from.

10 blaster(s), 6K senders, chain 32286, schema tn-blast-combined/v2 · mixed phase 2026-11-13 15:16–15:56 UTC · 1 observers, 10 validators

Preflight: WTEL from genesis (0x0000…37e1), v3 fee tier 3000, v4 available, Multicall3 present · Cooldown ended: all_landed ×9, idle_30s ×1

Headline numbers

Sustained mixed TPS (own)
8,065.4 tx/s
best 60-s mean of our own landed tx in the mixed phase
Sustained mixed TPS (chain)
8,065.4 tx/s
all tx in blocks, same window
Landed / accepted
100.0 %
14.6M of 14.6M accepted
Reverted (own)
0
0.00 % of landed
Latency p50
679 ms
submit → block timestamp; seen p50 12,990 ms
Latency p95
14,662 ms
seen p95 30,892 ms
Gas / s (own)
269M gas/s
chain 269M gas/s
Capacity at mix
14,103.0 tx/s
gas-limit basis 9,249.7 tx/s; block gas limit 30M

Per transaction type

Landed counts only our own senders; latency is submit → block timestamp.

TypeSubmittedAcceptedLandedRevertedUnlandedSkipsGas limitGas used avgp50p95
native_transfer4.48M4.4M4.4M037024.1K21K674 ms14,431 ms
batched_self_send2.22M2.16M2.16M054024.1K21K698 ms14,586 ms
uniswap_v2_swap2M1.94M1.94M0240180K106K675 ms14,418 ms
stablecoin_transfer1.66M1.61M1.61M020075K43.9K673 ms14,360 ms
wtel_wrap1.33M1.29M1.29M015050K27.9K673 ms14,404 ms
wtel_unwrap888K863K863K010045K35.2K673 ms14,394 ms
stablecoin_approve668K649K649K05060K31.8K673 ms14,415 ms
uniswap_v2_add_liquidity443K431K431K060250K128K674 ms14,360 ms
contract_deploy333K324K324K020140K74.2K673 ms14,528 ms
uniswap_v4_swap332K323K323K030250K124K676 ms14,555 ms
uniswap_v2_remove_liquidity223K216K216K060220K129K673 ms14,421 ms
uniswap_v3_swap222K215K215K010220K120K674 ms14,328 ms
faucet_drip126K126K126K00430K150K107K876 ms40,498 ms

Submission by RPC node

NodeSubmittedAcceptedAccepted %Top errorsSend p50Send p95
bench-validator-01
bench-validator-01:8545
1.8M1.8M100.0 %none5 ms19 ms
bench-validator-02
bench-validator-02:8545
549K534K97.2 %known 8.29K14 ms47 ms
bench-validator-03
bench-validator-03:8545
1.88M1.88M100.0 %none4 ms16 ms
bench-validator-04
bench-validator-04:8545
1.82M1.82M100.0 %none5 ms19 ms
bench-validator-05
bench-validator-05:8545
1.89M779K41.2 %other 1.11M, known 811 ms23 ms
bench-validator-06
bench-validator-06:8545
474K460K97.1 %known 8K15 ms48 ms
bench-validator-07
bench-validator-07:8545
1.88M1.88M100.0 %none5 ms21 ms
bench-validator-08
bench-validator-08:8545
1.88M1.88M100.0 %none5 ms18 ms
bench-validator-09
bench-validator-09:8545
1.91M1.91M100.0 %none3 ms14 ms
bench-validator-10
bench-validator-10:8545
1.68M1.68M100.0 %known 1384 ms17 ms

Throughput over time

Shaded: phases outside the mixed window. Hairlines: epoch closes. Dashed orange: chaos events.

Landed transactions per second

own senders vs chain total, per block-timestamp second

Gas per second

own vs chain gas used, per block-timestamp second

CPU and memory per node

CPU utilization

compute.googleapis.com/instance/cpu/utilization, 60-s mean, % of all vCPUs; summaries over the mixed phase

Table view
NodeClassMeanp95Peak 60-s meanMax
bench-observer-11observer17.9 %20.4 %26.6 %26.6 %
bench-validator-01validator32.2 %35.3 %51.0 %51.0 %
bench-validator-02validator29.9 %33.6 %46.6 %46.6 %
bench-validator-03validator32.2 %36.0 %52.4 %52.4 %
bench-validator-04validator32.0 %35.3 %51.9 %51.9 %
bench-validator-05validator27.9 %34.9 %49.0 %49.0 %
bench-validator-06validator30.4 %34.8 %48.0 %48.0 %
bench-validator-07validator31.8 %35.1 %51.2 %51.2 %
bench-validator-08validator31.8 %35.0 %51.3 %51.3 %
bench-validator-09validator31.5 %34.1 %50.9 %50.9 %
bench-validator-10validator32.1 %35.4 %51.1 %51.1 %

CPU: mean vs peak 60-s mean

One row per node, over the mixed phase.

telcoin resident memory

agent processes/rss_usage summed over telcoin pids; summaries over the mixed phase

Table view
NodeClassMeanp95Peak 60-s meanMax
bench-observer-11observer3.20 GiB4.23 GiB4.38 GiB4.38 GiB
bench-validator-01validator4.79 GiB5.48 GiB5.61 GiB5.61 GiB
bench-validator-02validator4.98 GiB5.91 GiB6.09 GiB6.09 GiB
bench-validator-03validator5.06 GiB6.07 GiB6.29 GiB6.29 GiB
bench-validator-04validator5.01 GiB5.83 GiB6.06 GiB6.06 GiB
bench-validator-05validator4.45 GiB5.29 GiB5.29 GiB5.29 GiB
bench-validator-06validator5.21 GiB6.40 GiB6.43 GiB6.43 GiB
bench-validator-07validator4.56 GiB5.52 GiB5.80 GiB5.80 GiB
bench-validator-08validator5.03 GiB5.92 GiB6.18 GiB6.18 GiB
bench-validator-09validator4.61 GiB5.55 GiB5.79 GiB5.79 GiB
bench-validator-10validator5.11 GiB6.26 GiB6.43 GiB6.43 GiB

telcoin RSS: mean vs peak 60-s mean

One row per node, over the mixed phase.

Memory used

agent memory/percent_used, state=used; summaries over the mixed phase

Table view
NodeClassMeanp95Peak 60-s meanMax
bench-observer-11observer10.5 %12.4 %12.5 %12.5 %
bench-validator-01validator20.1 %20.7 %20.8 %20.8 %
bench-validator-02validator20.6 %21.3 %21.4 %21.4 %
bench-validator-03validator20.8 %21.3 %21.4 %21.4 %
bench-validator-04validator21.0 %21.7 %21.9 %21.9 %
bench-validator-05validator16.5 %20.2 %20.3 %20.3 %
bench-validator-06validator21.8 %22.7 %22.9 %22.9 %
bench-validator-07validator17.9 %18.3 %18.4 %18.4 %
bench-validator-08validator20.8 %21.6 %21.7 %21.7 %
bench-validator-09validator18.1 %18.5 %18.7 %18.7 %
bench-validator-10validator20.9 %21.6 %21.6 %21.6 %

Memory used: mean vs peak 60-s mean

One row per node, over the mixed phase.

Disk

Disk throttling (validator)
not collected –
metric type unavailable in project
Disk performance degraded (validator)
none –
any sample

Disk read + write throughput

instance/disk/{read,write}_bytes_count rate, all devices; summaries over the mixed phase

Table view
NodeClassMeanp95Peak 60-s meanMaxLimit
bench-observer-11observer87.5 MiB/s92.3 MiB/s96.3 MiB/s96.3 MiB/s800.0 MiB/s
bench-validator-01validator98.7 MiB/s104.3 MiB/s110.5 MiB/s110.5 MiB/s800.0 MiB/s
bench-validator-02validator98.4 MiB/s103.7 MiB/s109.9 MiB/s109.9 MiB/s800.0 MiB/s
bench-validator-03validator98.5 MiB/s104.5 MiB/s108.9 MiB/s108.9 MiB/s800.0 MiB/s
bench-validator-04validator98.6 MiB/s104.2 MiB/s109.0 MiB/s109.0 MiB/s800.0 MiB/s
bench-validator-05validator84.5 MiB/s103.0 MiB/s109.1 MiB/s109.1 MiB/s800.0 MiB/s
bench-validator-06validator98.2 MiB/s103.6 MiB/s110.3 MiB/s110.3 MiB/s800.0 MiB/s
bench-validator-07validator98.3 MiB/s102.7 MiB/s109.6 MiB/s109.6 MiB/s800.0 MiB/s
bench-validator-08validator98.4 MiB/s102.7 MiB/s109.4 MiB/s109.4 MiB/s800.0 MiB/s
bench-validator-09validator98.5 MiB/s103.2 MiB/s108.8 MiB/s108.8 MiB/s800.0 MiB/s
bench-validator-10validator98.6 MiB/s103.2 MiB/s108.7 MiB/s108.7 MiB/s800.0 MiB/s

Disk read + write IOPS

instance/disk/{read,write}_ops_count rate, all devices; summaries over the mixed phase

Table view
NodeClassMeanp95Peak 60-s meanMaxLimit
bench-observer-11observer19.7K IOPS20.1K IOPS20.1K IOPS20.1K IOPS20K IOPS
bench-validator-01validator19.9K IOPS20.1K IOPS20.2K IOPS20.2K IOPS20K IOPS
bench-validator-02validator19.9K IOPS20.1K IOPS20.3K IOPS20.3K IOPS20K IOPS
bench-validator-03validator19.9K IOPS20.1K IOPS20.4K IOPS20.4K IOPS20K IOPS
bench-validator-04validator19.9K IOPS20.1K IOPS20.1K IOPS20.1K IOPS20K IOPS
bench-validator-05validator17K IOPS20.1K IOPS20.2K IOPS20.2K IOPS20K IOPS
bench-validator-06validator19.9K IOPS20.1K IOPS20.4K IOPS20.4K IOPS20K IOPS
bench-validator-07validator19.8K IOPS20.1K IOPS20.1K IOPS20.1K IOPS20K IOPS
bench-validator-08validator19.8K IOPS20.1K IOPS20.2K IOPS20.2K IOPS20K IOPS
bench-validator-09validator19.9K IOPS20.1K IOPS20.1K IOPS20.1K IOPS20K IOPS
bench-validator-10validator19.8K IOPS20.1K IOPS20.1K IOPS20.1K IOPS20K IOPS
Disk limits used
NodeMachinevCPUDiskIOPS limitMiB/s limitBasis
bench-observer-11c3-highcpu-88hyperdisk-balanced 100 GB20K IOPS800.0 MiB/shyperdisk-balanced 100 GB: provisioned 20,000 IOPS, provisioned 800 MiB/s; c3 8 vCPU cap 50,000 IOPS / 800 MiB/s; reads and writes share this budget
bench-validator-01c3-highcpu-88hyperdisk-balanced 100 GB20K IOPS800.0 MiB/shyperdisk-balanced 100 GB: provisioned 20,000 IOPS, provisioned 800 MiB/s; c3 8 vCPU cap 50,000 IOPS / 800 MiB/s; reads and writes share this budget
bench-validator-02c3-highcpu-88hyperdisk-balanced 100 GB20K IOPS800.0 MiB/shyperdisk-balanced 100 GB: provisioned 20,000 IOPS, provisioned 800 MiB/s; c3 8 vCPU cap 50,000 IOPS / 800 MiB/s; reads and writes share this budget
bench-validator-03c3-highcpu-88hyperdisk-balanced 100 GB20K IOPS800.0 MiB/shyperdisk-balanced 100 GB: provisioned 20,000 IOPS, provisioned 800 MiB/s; c3 8 vCPU cap 50,000 IOPS / 800 MiB/s; reads and writes share this budget
bench-validator-04c3-highcpu-88hyperdisk-balanced 100 GB20K IOPS800.0 MiB/shyperdisk-balanced 100 GB: provisioned 20,000 IOPS, provisioned 800 MiB/s; c3 8 vCPU cap 50,000 IOPS / 800 MiB/s; reads and writes share this budget
bench-validator-05c3-highcpu-88hyperdisk-balanced 100 GB20K IOPS800.0 MiB/shyperdisk-balanced 100 GB: provisioned 20,000 IOPS, provisioned 800 MiB/s; c3 8 vCPU cap 50,000 IOPS / 800 MiB/s; reads and writes share this budget
bench-validator-06c3-highcpu-88hyperdisk-balanced 100 GB20K IOPS800.0 MiB/shyperdisk-balanced 100 GB: provisioned 20,000 IOPS, provisioned 800 MiB/s; c3 8 vCPU cap 50,000 IOPS / 800 MiB/s; reads and writes share this budget
bench-validator-07c3-highcpu-88hyperdisk-balanced 100 GB20K IOPS800.0 MiB/shyperdisk-balanced 100 GB: provisioned 20,000 IOPS, provisioned 800 MiB/s; c3 8 vCPU cap 50,000 IOPS / 800 MiB/s; reads and writes share this budget
bench-validator-08c3-highcpu-88hyperdisk-balanced 100 GB20K IOPS800.0 MiB/shyperdisk-balanced 100 GB: provisioned 20,000 IOPS, provisioned 800 MiB/s; c3 8 vCPU cap 50,000 IOPS / 800 MiB/s; reads and writes share this budget
bench-validator-09c3-highcpu-88hyperdisk-balanced 100 GB20K IOPS800.0 MiB/shyperdisk-balanced 100 GB: provisioned 20,000 IOPS, provisioned 800 MiB/s; c3 8 vCPU cap 50,000 IOPS / 800 MiB/s; reads and writes share this budget
bench-validator-10c3-highcpu-88hyperdisk-balanced 100 GB20K IOPS800.0 MiB/shyperdisk-balanced 100 GB: provisioned 20,000 IOPS, provisioned 800 MiB/s; c3 8 vCPU cap 50,000 IOPS / 800 MiB/s; reads and writes share this budget

Network

Network sent

instance/network/sent_bytes_count rate; summaries over the mixed phase

Table view
NodeClassMeanp95Peak 60-s meanMax
bench-observer-11observer2.1 Mbps2.2 Mbps2.4 Mbps2.4 Mbps
bench-validator-01validator97.8 Mbps110.9 Mbps135.8 Mbps135.8 Mbps
bench-validator-02validator92.8 Mbps101.7 Mbps126.8 Mbps126.8 Mbps
bench-validator-03validator98.1 Mbps109.2 Mbps140.3 Mbps140.3 Mbps
bench-validator-04validator97.8 Mbps107.5 Mbps133.0 Mbps133.0 Mbps
bench-validator-05validator83.4 Mbps107.1 Mbps124.1 Mbps124.1 Mbps
bench-validator-06validator92.7 Mbps103.6 Mbps124.4 Mbps124.4 Mbps
bench-validator-07validator100.1 Mbps130.4 Mbps152.4 Mbps152.4 Mbps
bench-validator-08validator99.4 Mbps111.7 Mbps137.3 Mbps137.3 Mbps
bench-validator-09validator98.1 Mbps111.3 Mbps137.1 Mbps137.1 Mbps
bench-validator-10validator97.1 Mbps110.7 Mbps133.4 Mbps133.4 Mbps

Network received

instance/network/received_bytes_count rate; summaries over the mixed phase

Table view
NodeClassMeanp95Peak 60-s meanMax
bench-observer-11observer6.9 Mbps11.5 Mbps27.1 Mbps27.1 Mbps
bench-validator-01validator10.5 Mbps11.6 Mbps19.0 Mbps19.0 Mbps
bench-validator-02validator8.2 Mbps9.5 Mbps14.3 Mbps14.3 Mbps
bench-validator-03validator10.7 Mbps12.0 Mbps20.7 Mbps20.7 Mbps
bench-validator-04validator10.7 Mbps11.5 Mbps19.3 Mbps19.3 Mbps
bench-validator-05validator13.9 Mbps22.1 Mbps96.6 Mbps96.6 Mbps
bench-validator-06validator8.3 Mbps10.0 Mbps15.1 Mbps15.1 Mbps
bench-validator-07validator10.7 Mbps12.1 Mbps20.3 Mbps20.3 Mbps
bench-validator-08validator10.7 Mbps12.2 Mbps20.5 Mbps20.5 Mbps
bench-validator-09validator10.7 Mbps12.5 Mbps21.5 Mbps21.5 Mbps
bench-validator-10validator10.4 Mbps11.9 Mbps19.9 Mbps19.9 Mbps

Consensus health

MetricQueryClassNodesMedian of node meansWorst peak 60-s meanMax
tn_primary_commit_latency_secondsp95validator10975 ms1,312 ms1,600 ms
tn_engine_execution_duration_secondsp95validator101,250 ms2,778 ms3,250 ms
tn_engine_execution_duration_secondsp95observer11,219 ms2,358 ms2,442 ms
tn_worker_batch_size_bytesp95validator10232.7 KiB246.4 KiB246.4 KiB
tn_worker_batch_transactionsp95validator104931.48K1.48K
tn_executor_output_transactionsp95validator108.44K23K26.9K
tn_primary_consensus_output_bytesmaxvalidator10335.3 KiB1.8 MiB2.3 MiB
tn_primary_consensus_output_bytesmaxobserver1318.4 KiB670.9 KiB995.2 KiB
tn_engine_queued_outputsmaxvalidator107.8888
tn_engine_queued_outputsmaxobserver17.7588
tn_primary_roundmaxvalidator101.69K3.39K3.42K
tn_primary_roundmaxobserver11.65K3.34K3.41K
tn_primary_committed_roundmaxvalidator101.69K3.38K3.42K
tn_primary_committed_roundmaxobserver11.65K3.34K3.41K
tn_epoch_currentmaxvalidator104.2155
tn_epoch_currentmaxobserver14.2255
tn_batch_builder_pending_pool_transactionsmaxvalidator103084.98K9.84K
tn_batch_builder_pending_pool_transactionsmaxobserver1000
reth_process_resident_memory_bytesmaxvalidator105.03 GiB6.48 GiB6.56 GiB
reth_process_resident_memory_bytesmaxobserver13.21 GiB4.42 GiB4.47 GiB
tn_primary_subdags_committed_totalratevalidator101.381.441.44
tn_primary_subdags_committed_totalrateobserver1000
tn_primary_round_laglagvalidator102.0933
tn_primary_round_laglagobserver1000
tn_batch_builder_seal_failures_totalratevalidator90.0004120.01670.0222
tn_node_modemaxvalidator10011
tn_node_modemaxobserver1000
tn_primary_vote_requests_totalratevalidator1023.833.333.4
tn_primary_vote_requests_totalrateobserver1000
tn_primary_votes_created_totalratevalidator1023.725.825.8
tn_primary_votes_created_totalrateobserver1000
tn_primary_votes_recast_totalratevalidator10000
tn_primary_votes_recast_totalrateobserver1000
tn_primary_vote_requests_too_new_totalratevalidator10000
tn_primary_vote_requests_too_new_totalrateobserver1000
tn_primary_vote_timeouts_total{stage="exec_wait"}ratevalidator10000
tn_primary_vote_timeouts_total{stage="exec_wait"}rateobserver1000
tn_primary_vote_timeouts_total{stage="lead_sleep"}ratevalidator10000
tn_primary_vote_timeouts_total{stage="lead_sleep"}rateobserver1000
tn_primary_vote_timeouts_total{stage="parents"}ratevalidator10000
tn_primary_vote_timeouts_total{stage="parents"}rateobserver1000
tn_primary_vote_timeouts_total{stage="persist"}ratevalidator10000
tn_primary_vote_timeouts_total{stage="persist"}rateobserver1000
tn_primary_vote_timeouts_total{stage="prelock_wait"}ratevalidator10000
tn_primary_vote_timeouts_total{stage="prelock_wait"}rateobserver1000
tn_primary_vote_timeouts_total{stage="sync_batches"}ratevalidator10000
tn_primary_vote_timeouts_total{stage="sync_batches"}rateobserver1000
tn_primary_vote_timeouts_total{stage="validate"}ratevalidator10000
tn_primary_vote_timeouts_total{stage="validate"}rateobserver1000
tn_primary_vote_timeouts_total{stage="vote_store"}ratevalidator10000
tn_primary_vote_timeouts_total{stage="vote_store"}rateobserver1000
tn_primary_vote_stage_seconds{stage="exec_wait"}p95validator100 ms0 ms0 ms
tn_primary_vote_stage_seconds{stage="lead_sleep"}p95validator100 ms0 ms0 ms
tn_primary_vote_stage_seconds{stage="lock"}p95validator100 ms0 ms0 ms
tn_primary_vote_stage_seconds{stage="parents"}p95validator101 ms2 ms2 ms
tn_primary_vote_stage_seconds{stage="persist"}p95validator1036 ms45 ms46 ms
tn_primary_vote_stage_seconds{stage="prelock_wait"}p95validator100 ms97 ms211 ms
tn_primary_vote_stage_seconds{stage="sync_batches"}p95validator100 ms0 ms0 ms
tn_primary_vote_stage_seconds{stage="validate"}p95validator103 ms4 ms4 ms
tn_primary_vote_stage_seconds{stage="vote_store"}p95validator101 ms3 ms4 ms
tn_primary_leader_rounds_total{leader="3j3KF4X3yr9q8fFenDb8U8pBoYzCRdcZnwSEo6EqCXbf",swapped="false",found="false"}ratevalidator10000
tn_primary_leader_rounds_total{leader="3j3KF4X3yr9q8fFenDb8U8pBoYzCRdcZnwSEo6EqCXbf",swapped="false",found="true"}ratevalidator100.1240.1560.156
tn_primary_leader_rounds_total{leader="3j3KF4X3yr9q8fFenDb8U8pBoYzCRdcZnwSEo6EqCXbf",swapped="true",found="false"}ratevalidator10000
tn_primary_leader_rounds_total{leader="3j3KF4X3yr9q8fFenDb8U8pBoYzCRdcZnwSEo6EqCXbf",swapped="true",found="true"}ratevalidator100.03570.1220.156
tn_primary_leader_rounds_total{leader="3rgB5qYVyKYbKTe9H9yChbYHCFzrNRAP8ne9VHYgoRau",swapped="false",found="false"}ratevalidator10000
tn_primary_leader_rounds_total{leader="3rgB5qYVyKYbKTe9H9yChbYHCFzrNRAP8ne9VHYgoRau",swapped="false",found="true"}ratevalidator100.009470.150.156
tn_primary_leader_rounds_total{leader="3rgB5qYVyKYbKTe9H9yChbYHCFzrNRAP8ne9VHYgoRau",swapped="true",found="false"}ratevalidator10000
tn_primary_leader_rounds_total{leader="3rgB5qYVyKYbKTe9H9yChbYHCFzrNRAP8ne9VHYgoRau",swapped="true",found="true"}ratevalidator10000
tn_primary_leader_rounds_total{leader="4BzeDizv6RBRPHFiABmgio6TzJrgo61kQimHbVULGFrk",swapped="false",found="false"}ratevalidator10000
tn_primary_leader_rounds_total{leader="4BzeDizv6RBRPHFiABmgio6TzJrgo61kQimHbVULGFrk",swapped="false",found="true"}ratevalidator100.1380.1560.156
tn_primary_leader_rounds_total{leader="4BzeDizv6RBRPHFiABmgio6TzJrgo61kQimHbVULGFrk",swapped="true",found="false"}ratevalidator10000
tn_primary_leader_rounds_total{leader="4BzeDizv6RBRPHFiABmgio6TzJrgo61kQimHbVULGFrk",swapped="true",found="true"}ratevalidator100.04790.1330.178
tn_primary_leader_rounds_total{leader="Db9g4C5zk7Ce5huvb1JzXZ7AVZphwyuaEXakuXRmHosX",swapped="false",found="false"}ratevalidator10000
tn_primary_leader_rounds_total{leader="Db9g4C5zk7Ce5huvb1JzXZ7AVZphwyuaEXakuXRmHosX",swapped="false",found="true"}ratevalidator100.1380.1560.156
tn_primary_leader_rounds_total{leader="Db9g4C5zk7Ce5huvb1JzXZ7AVZphwyuaEXakuXRmHosX",swapped="true",found="false"}ratevalidator10000
tn_primary_leader_rounds_total{leader="Db9g4C5zk7Ce5huvb1JzXZ7AVZphwyuaEXakuXRmHosX",swapped="true",found="true"}ratevalidator100.05130.1280.156
tn_primary_leader_rounds_total{leader="ETvZruGgDgN4adQCn6WzZEf3KbWDa18x1ZxecCnwGBjg",swapped="false",found="false"}ratevalidator10000
tn_primary_leader_rounds_total{leader="ETvZruGgDgN4adQCn6WzZEf3KbWDa18x1ZxecCnwGBjg",swapped="false",found="true"}ratevalidator100.1380.1560.156
tn_primary_leader_rounds_total{leader="ETvZruGgDgN4adQCn6WzZEf3KbWDa18x1ZxecCnwGBjg",swapped="true",found="false"}ratevalidator10000
tn_primary_leader_rounds_total{leader="ETvZruGgDgN4adQCn6WzZEf3KbWDa18x1ZxecCnwGBjg",swapped="true",found="true"}ratevalidator100.04830.1220.156
tn_primary_leader_rounds_total{leader="GQE9Et1BLw4UTXmadgkuV5WAR5fzYctMx5muaPvn7tEU",swapped="false",found="false"}ratevalidator100.00370.1060.133
tn_primary_leader_rounds_total{leader="GQE9Et1BLw4UTXmadgkuV5WAR5fzYctMx5muaPvn7tEU",swapped="false",found="true"}ratevalidator100.0650.150.156
tn_primary_leader_rounds_total{leader="GQE9Et1BLw4UTXmadgkuV5WAR5fzYctMx5muaPvn7tEU",swapped="true",found="false"}ratevalidator10000
tn_primary_leader_rounds_total{leader="GQE9Et1BLw4UTXmadgkuV5WAR5fzYctMx5muaPvn7tEU",swapped="true",found="true"}ratevalidator100.008640.1110.133
tn_primary_leader_rounds_total{leader="GhBcDwgtgFxi4EGQggUgQgT9T6Evq6AxXmXM7dUB7w62",swapped="false",found="false"}ratevalidator10000
tn_primary_leader_rounds_total{leader="GhBcDwgtgFxi4EGQggUgQgT9T6Evq6AxXmXM7dUB7w62",swapped="false",found="true"}ratevalidator100.1380.1560.156
tn_primary_leader_rounds_total{leader="GhBcDwgtgFxi4EGQggUgQgT9T6Evq6AxXmXM7dUB7w62",swapped="true",found="false"}ratevalidator10000
tn_primary_leader_rounds_total{leader="GhBcDwgtgFxi4EGQggUgQgT9T6Evq6AxXmXM7dUB7w62",swapped="true",found="true"}ratevalidator100.04920.1220.156
tn_primary_leader_rounds_total{leader="Gq3UnvwZr5F4uXqKr2jwMFJF2b1Qp8BeN5UWeujxnnQH",swapped="false",found="false"}ratevalidator10000
tn_primary_leader_rounds_total{leader="Gq3UnvwZr5F4uXqKr2jwMFJF2b1Qp8BeN5UWeujxnnQH",swapped="false",found="true"}ratevalidator100.004120.1170.156
tn_primary_leader_rounds_total{leader="Gq3UnvwZr5F4uXqKr2jwMFJF2b1Qp8BeN5UWeujxnnQH",swapped="true",found="false"}ratevalidator10000
tn_primary_leader_rounds_total{leader="Gq3UnvwZr5F4uXqKr2jwMFJF2b1Qp8BeN5UWeujxnnQH",swapped="true",found="true"}ratevalidator10000
tn_primary_leader_rounds_total{leader="GyiPPV2w5XNa3c7a9JHCNqCybtGAiWh7uKJ9BNVEnFtG",swapped="false",found="false"}ratevalidator10000
tn_primary_leader_rounds_total{leader="GyiPPV2w5XNa3c7a9JHCNqCybtGAiWh7uKJ9BNVEnFtG",swapped="false",found="true"}ratevalidator100.1380.1560.156
tn_primary_leader_rounds_total{leader="GyiPPV2w5XNa3c7a9JHCNqCybtGAiWh7uKJ9BNVEnFtG",swapped="true",found="false"}ratevalidator10000
tn_primary_leader_rounds_total{leader="GyiPPV2w5XNa3c7a9JHCNqCybtGAiWh7uKJ9BNVEnFtG",swapped="true",found="true"}ratevalidator100.04710.1110.133
tn_primary_leader_rounds_total{leader="HLXJoW6Dh4yCaobZkphz23F98wB43CHrVq9AUFDu8G3T",swapped="false",found="false"}ratevalidator10000
tn_primary_leader_rounds_total{leader="HLXJoW6Dh4yCaobZkphz23F98wB43CHrVq9AUFDu8G3T",swapped="false",found="true"}ratevalidator100.1380.1560.156
tn_primary_leader_rounds_total{leader="HLXJoW6Dh4yCaobZkphz23F98wB43CHrVq9AUFDu8G3T",swapped="true",found="false"}ratevalidator10000
tn_primary_leader_rounds_total{leader="HLXJoW6Dh4yCaobZkphz23F98wB43CHrVq9AUFDu8G3T",swapped="true",found="true"}ratevalidator100.05790.1560.2
tn_primary_leader_commits_total{leader="3j3KF4X3yr9q8fFenDb8U8pBoYzCRdcZnwSEo6EqCXbf"}ratevalidator100.1590.2610.311
tn_primary_leader_commits_total{leader="3rgB5qYVyKYbKTe9H9yChbYHCFzrNRAP8ne9VHYgoRau"}ratevalidator100.009470.150.156
tn_primary_leader_commits_total{leader="4BzeDizv6RBRPHFiABmgio6TzJrgo61kQimHbVULGFrk"}ratevalidator100.1860.2830.333
tn_primary_leader_commits_total{leader="Db9g4C5zk7Ce5huvb1JzXZ7AVZphwyuaEXakuXRmHosX"}ratevalidator100.1890.2720.289
tn_primary_leader_commits_total{leader="ETvZruGgDgN4adQCn6WzZEf3KbWDa18x1ZxecCnwGBjg"}ratevalidator100.1860.2670.311
tn_primary_leader_commits_total{leader="GQE9Et1BLw4UTXmadgkuV5WAR5fzYctMx5muaPvn7tEU"}ratevalidator100.07370.2610.289
tn_primary_leader_commits_total{leader="GhBcDwgtgFxi4EGQggUgQgT9T6Evq6AxXmXM7dUB7w62"}ratevalidator100.1870.2720.311
tn_primary_leader_commits_total{leader="Gq3UnvwZr5F4uXqKr2jwMFJF2b1Qp8BeN5UWeujxnnQH"}ratevalidator100.004120.1170.156
tn_primary_leader_commits_total{leader="GyiPPV2w5XNa3c7a9JHCNqCybtGAiWh7uKJ9BNVEnFtG"}ratevalidator100.1840.2560.289
tn_primary_leader_commits_total{leader="HLXJoW6Dh4yCaobZkphz23F98wB43CHrVq9AUFDu8G3T"}ratevalidator100.1950.3060.333
tn_engine_queue_wait_secondsp95validator109,681 ms27.2 s27.2 s
tn_engine_queue_wait_secondsp95observer19,590 ms9,819 ms9,848 ms
tn_engine_persist_duration_secondsp95validator10572 ms973 ms1,000 ms
tn_engine_persist_duration_secondsp95observer1680 ms956 ms962 ms
tn_engine_busy_milliseconds_totalratevalidator109961.02K1.06K
tn_engine_busy_milliseconds_totalrateobserver19801.01K1.03K
tn_reth_persist_save_blocks_secondsp95validator10194 ms467 ms488 ms
tn_reth_persist_save_blocks_secondsp95observer1141 ms245 ms267 ms
tn_reth_persist_commit_secondsp95validator10478 ms547 ms683 ms
tn_reth_persist_commit_secondsp95observer1492 ms860 ms918 ms
tn_reth_persisted_accounts_totalratevalidator102.27K7.78K7.78K
tn_reth_persisted_accounts_totalrateobserver12.24K7.46K7.46K
tn_reth_persisted_storage_slots_totalratevalidator102.95K3.88K4.16K
tn_reth_persisted_storage_slots_totalrateobserver12.94K3.82K3.96K
tn_executor_rejoin_blocked_total{reason="attempts_exhausted"}ratevalidator1000
tn_executor_rejoin_blocked_total{reason="epoch_closed"}ratevalidator10.1030.8881.2
tn_executor_rejoin_blocked_total{reason="epoch_mismatch"}ratevalidator1000
tn_executor_rejoin_blocked_total{reason="no_known_header"}ratevalidator1000
tn_executor_rejoin_blocked_total{reason="round_gap"}ratevalidator11.11.391.42
tn_executor_rejoin_blocked_total{reason="stale_head"}ratevalidator10.01860.1910.255
tn_worker_sync_serves_denied_total{worker="0",tier="committee"}ratevalidator10000
tn_worker_sync_serves_denied_total{worker="0",tier="committee"}rateobserver1000
tn_worker_sync_serves_denied_total{worker="0",tier="public"}ratevalidator10000
tn_worker_sync_serves_denied_total{worker="0",tier="public"}rateobserver1000
tn_worker_batch_requests_denied_total{worker="0",peer="21Xm7N2ETof1EiczCkTxnvAT7RqncJynW7oDG9LdfXbftTGyimPKJScjFEtgu8jbz3zQvZqeVDSfuBUzZJsqVhbf7Gp48EjVd82BfYWUqaTq6Q4pxHFrH4cvX8rt6bPCJXj8"}ratevalidator20.00990.250.333
tn_worker_batch_requests_denied_total{worker="0",peer="other"}ratevalidator1000
tn_worker_batch_requests_denied_total{worker="0",peer="p9mw9r81Ez8WtYtDpGY2hisE8d2bSdbsBKxsLfXP14FJvQFx9FNSQTZ3QwVj5WFqhoCW8hWM1wi3CWnzBMToFSCmvj9AxjJGK9C9NVhSz1Mrqk4cPuTsZo2S2Z4mPSonatK"}ratevalidator300.4330.578
tn_worker_batch_requests_denied_total{worker="0",peer="pTvhwCdG6MJvMRXWZCCkXpnBRNnbTEMapA5Yb3r2uMaAbeyBbCMho1oGshFMR6HSzDSrvFFVJ9aDLXDXmXsbDVRJXmhVe1an8tatLUx22JEZ48oxWQbzi3kM7VSetHhmXiv"}ratevalidator300.150.2
tn_worker_batch_requests_denied_total{worker="0",peer="pyGb28dcEZzWZ1SvkNNBYWAHoKMahzr9HPfWirMkfTKiZ9EqMrUCj24FQnuzzXy3pbUEvr7CVCwdy56ctVseVfB4Spspgc8YTAGzihVVWxsQXan3z7tEx4t2mXpcGR68RsS"}ratevalidator300.350.467
tn_worker_batch_requests_denied_total{worker="0",peer="rJNJunJgoDxtiNRocRMQRemyibjCubR4DyXDyFrMi1ua3cjripqHz64w7ovUwmNnnqA16FWkYCbZNHfkxdRgdeC3H2mFaZeffCqJuZTJncWfC7mr8ne6AFRhPn4va2wj2uv"}ratevalidator300.2560.356
tn_worker_batch_requests_denied_total{worker="0",peer="u6Ar9c81k4sdSWY82JbijYWSVKkR6xLfywxgd4yxZEkG2EBZPdeM6HKUX61c7iFz2fosBvn9RgsCFFEUqVMnfpd1dwkAr6P8xK3hXG33vv2xNrQn8cmePd51rGeZBqAzkge"}ratevalidator20.01190.30.4
tn_worker_batch_requests_denied_total{worker="0",peer="u7ZWUHcAbH1aoouEbxvd9ofiRExo5kdcpZXN5kCyehc7Px9GJWaiefgLuNnHuBvEqWBCDbzhNXRjwVPv7Po19hZ8aGrrU4dhLJYE4PDXWV64nFtDtbSa44fWuX8qzdAyMin"}ratevalidator20.01230.4170.556
tn_worker_batch_requests_denied_total{worker="0",peer="xBZnXmyYqFi1mCwSH6KCF4f1W5xtdsXALBknGRb669vULDvaXqq89RXcW9hTfCi1SVABCSjavn3BWgLoeERA7SzM22Y7Lf74EvcRGa9X69mxReSJR9Lw4pX5VcM8Stdp1nz"}ratevalidator20.01450.3670.489
tn_worker_batch_requests_denied_total{worker="0",peer="ziGAo3Fw5vYzBuWHw8u8FUD72W3hE8p5kdW2Zd3NHmBWugD5UARi6jCW6aBGBo3UbMPgQwgCpBQ2nZWt52yk4PdQJtx7fBVkNTVqvjz8Lv2Y2iR8CcEaA59Au8atMzNJmks"}ratevalidator500.01670.0222
tn_worker_batch_requests_denied_total{worker="0",peer="zixBdNoyd77oeFw5LFYX7YADGxbYGj1shubF8Xbt2F6pdVrdNmV2G4BLVaiaZ7Y4NGqjFSLFQyTV6U5YG9JivwgVutT71ouveXcP9oPe3srBzb4zL25KLhCGj4HybzSMGFt"}ratevalidator20.01040.350.467
tn_worker_batch_requests_failed_total{worker="0",peer="21Xm7N2ETof1EiczCkTxnvAT7RqncJynW7oDG9LdfXbftTGyimPKJScjFEtgu8jbz3zQvZqeVDSfuBUzZJsqVhbf7Gp48EjVd82BfYWUqaTq6Q4pxHFrH4cvX8rt6bPCJXj8"}ratevalidator20.00660.1670.222
tn_worker_batch_requests_failed_total{worker="0",peer="other"}ratevalidator1000
tn_worker_batch_requests_failed_total{worker="0",peer="p9mw9r81Ez8WtYtDpGY2hisE8d2bSdbsBKxsLfXP14FJvQFx9FNSQTZ3QwVj5WFqhoCW8hWM1wi3CWnzBMToFSCmvj9AxjJGK9C9NVhSz1Mrqk4cPuTsZo2S2Z4mPSonatK"}ratevalidator300.08330.111
tn_worker_batch_requests_failed_total{worker="0",peer="pTvhwCdG6MJvMRXWZCCkXpnBRNnbTEMapA5Yb3r2uMaAbeyBbCMho1oGshFMR6HSzDSrvFFVJ9aDLXDXmXsbDVRJXmhVe1an8tatLUx22JEZ48oxWQbzi3kM7VSetHhmXiv"}ratevalidator300.1330.178
tn_worker_batch_requests_failed_total{worker="0",peer="pyGb28dcEZzWZ1SvkNNBYWAHoKMahzr9HPfWirMkfTKiZ9EqMrUCj24FQnuzzXy3pbUEvr7CVCwdy56ctVseVfB4Spspgc8YTAGzihVVWxsQXan3z7tEx4t2mXpcGR68RsS"}ratevalidator300.10.133
tn_worker_batch_requests_failed_total{worker="0",peer="rJNJunJgoDxtiNRocRMQRemyibjCubR4DyXDyFrMi1ua3cjripqHz64w7ovUwmNnnqA16FWkYCbZNHfkxdRgdeC3H2mFaZeffCqJuZTJncWfC7mr8ne6AFRhPn4va2wj2uv"}ratevalidator300.150.2
tn_worker_batch_requests_failed_total{worker="0",peer="u6Ar9c81k4sdSWY82JbijYWSVKkR6xLfywxgd4yxZEkG2EBZPdeM6HKUX61c7iFz2fosBvn9RgsCFFEUqVMnfpd1dwkAr6P8xK3hXG33vv2xNrQn8cmePd51rGeZBqAzkge"}ratevalidator20.00660.1670.222
tn_worker_batch_requests_failed_total{worker="0",peer="u7ZWUHcAbH1aoouEbxvd9ofiRExo5kdcpZXN5kCyehc7Px9GJWaiefgLuNnHuBvEqWBCDbzhNXRjwVPv7Po19hZ8aGrrU4dhLJYE4PDXWV64nFtDtbSa44fWuX8qzdAyMin"}ratevalidator20.002960.10.133
tn_worker_batch_requests_failed_total{worker="0",peer="xBZnXmyYqFi1mCwSH6KCF4f1W5xtdsXALBknGRb669vULDvaXqq89RXcW9hTfCi1SVABCSjavn3BWgLoeERA7SzM22Y7Lf74EvcRGa9X69mxReSJR9Lw4pX5VcM8Stdp1nz"}ratevalidator20.003960.10.133
tn_worker_batch_requests_failed_total{worker="0",peer="ziGAo3Fw5vYzBuWHw8u8FUD72W3hE8p5kdW2Zd3NHmBWugD5UARi6jCW6aBGBo3UbMPgQwgCpBQ2nZWt52yk4PdQJtx7fBVkNTVqvjz8Lv2Y2iR8CcEaA59Au8atMzNJmks"}ratevalidator5000
tn_worker_batch_requests_failed_total{worker="0",peer="zixBdNoyd77oeFw5LFYX7YADGxbYGj1shubF8Xbt2F6pdVrdNmV2G4BLVaiaZ7Y4NGqjFSLFQyTV6U5YG9JivwgVutT71ouveXcP9oPe3srBzb4zL25KLhCGj4HybzSMGFt"}ratevalidator20.0009880.03330.0444
tn_node_epoch_record_signersmaxvalidator108.211010
tn_node_epoch_record_signersmaxobserver18.6599
tn_node_epoch_record_missing_signers_total{signer="3j3KF4X3yr9q8fFenDb8U8pBoYzCRdcZnwSEo6EqCXbf"}ratevalidator10000
tn_node_epoch_record_missing_signers_total{signer="3j3KF4X3yr9q8fFenDb8U8pBoYzCRdcZnwSEo6EqCXbf"}rateobserver1000
tn_node_epoch_record_missing_signers_total{signer="3rgB5qYVyKYbKTe9H9yChbYHCFzrNRAP8ne9VHYgoRau"}ratevalidator1000.0160.0214
tn_node_epoch_record_missing_signers_total{signer="3rgB5qYVyKYbKTe9H9yChbYHCFzrNRAP8ne9VHYgoRau"}rateobserver1000
tn_node_epoch_record_missing_signers_total{signer="4BzeDizv6RBRPHFiABmgio6TzJrgo61kQimHbVULGFrk"}ratevalidator1000.0160.0214
tn_node_epoch_record_missing_signers_total{signer="4BzeDizv6RBRPHFiABmgio6TzJrgo61kQimHbVULGFrk"}rateobserver1000
tn_node_epoch_record_missing_signers_total{signer="Db9g4C5zk7Ce5huvb1JzXZ7AVZphwyuaEXakuXRmHosX"}ratevalidator10000
tn_node_epoch_record_missing_signers_total{signer="Db9g4C5zk7Ce5huvb1JzXZ7AVZphwyuaEXakuXRmHosX"}rateobserver1000
tn_node_epoch_record_missing_signers_total{signer="ETvZruGgDgN4adQCn6WzZEf3KbWDa18x1ZxecCnwGBjg"}ratevalidator10000
tn_node_epoch_record_missing_signers_total{signer="ETvZruGgDgN4adQCn6WzZEf3KbWDa18x1ZxecCnwGBjg"}rateobserver1000
tn_node_epoch_record_missing_signers_total{signer="GQE9Et1BLw4UTXmadgkuV5WAR5fzYctMx5muaPvn7tEU"}ratevalidator100.0008230.01670.0222
tn_node_epoch_record_missing_signers_total{signer="GQE9Et1BLw4UTXmadgkuV5WAR5fzYctMx5muaPvn7tEU"}rateobserver10.0009460.01670.0222
tn_node_epoch_record_missing_signers_total{signer="GhBcDwgtgFxi4EGQggUgQgT9T6Evq6AxXmXM7dUB7w62"}ratevalidator10000
tn_node_epoch_record_missing_signers_total{signer="GhBcDwgtgFxi4EGQggUgQgT9T6Evq6AxXmXM7dUB7w62"}rateobserver1000
tn_node_epoch_record_missing_signers_total{signer="Gq3UnvwZr5F4uXqKr2jwMFJF2b1Qp8BeN5UWeujxnnQH"}ratevalidator10000
tn_node_epoch_record_missing_signers_total{signer="Gq3UnvwZr5F4uXqKr2jwMFJF2b1Qp8BeN5UWeujxnnQH"}rateobserver1000
tn_node_epoch_record_missing_signers_total{signer="GyiPPV2w5XNa3c7a9JHCNqCybtGAiWh7uKJ9BNVEnFtG"}ratevalidator1000.01260.0168
tn_node_epoch_record_missing_signers_total{signer="GyiPPV2w5XNa3c7a9JHCNqCybtGAiWh7uKJ9BNVEnFtG"}rateobserver1000
tn_node_epoch_record_missing_signers_total{signer="HLXJoW6Dh4yCaobZkphz23F98wB43CHrVq9AUFDu8G3T"}ratevalidator10000
tn_node_epoch_record_missing_signers_total{signer="HLXJoW6Dh4yCaobZkphz23F98wB43CHrVq9AUFDu8G3T"}rateobserver1000
tn_network_outbound_request_duration_seconds{outcome="failure",class="batch"}p95validator418.4 s29.0 s29.0 s
tn_network_outbound_request_duration_seconds{outcome="failure",class="epoch_record"}p95validator714.5 s29.0 s29.0 s
tn_network_outbound_request_duration_seconds{outcome="failure",class="vote"}p95validator39,693 ms29.0 s29.0 s
tn_network_outbound_request_duration_seconds{outcome="response",class="batch"}p95validator6446 ms871 ms871 ms
tn_network_outbound_request_duration_seconds{outcome="response",class="epoch_record"}p95validator5437 ms479 ms481 ms
tn_network_outbound_request_duration_seconds{outcome="response",class="epoch_record"}p95observer1240 ms247 ms248 ms
tn_network_outbound_request_duration_seconds{outcome="response",class="vote"}p95validator5407 ms454 ms459 ms
tn_network_closed_admission_denied_total{network="bench",direction="in"}ratevalidator10000
tn_network_closed_admission_denied_total{network="bench",direction="in"}rateobserver1000
tn_network_closed_admission_denied_total{network="bench",direction="out"}ratevalidator10000
tn_network_closed_admission_denied_total{network="bench",direction="out"}rateobserver1000
tn_network_source_admission_denied_total{network="bench",direction="in",reason="address_full"}ratevalidator10000
tn_network_source_admission_denied_total{network="bench",direction="in",reason="address_full"}rateobserver1000
tn_network_source_admission_denied_total{network="bench",direction="in",reason="duplicate_connection"}ratevalidator10000
tn_network_source_admission_denied_total{network="bench",direction="in",reason="duplicate_connection"}rateobserver1000
tn_network_source_admission_denied_total{network="bench",direction="in",reason="invalid_limits"}ratevalidator10000
tn_network_source_admission_denied_total{network="bench",direction="in",reason="invalid_limits"}rateobserver1000
tn_network_source_admission_denied_total{network="bench",direction="in",reason="peer_full"}ratevalidator10000
tn_network_source_admission_denied_total{network="bench",direction="in",reason="peer_full"}rateobserver1000
tn_network_source_admission_denied_total{network="bench",direction="in",reason="poisoned"}ratevalidator10000
tn_network_source_admission_denied_total{network="bench",direction="in",reason="poisoned"}rateobserver1000
tn_network_source_admission_denied_total{network="bench",direction="in",reason="prefix_full"}ratevalidator10000
tn_network_source_admission_denied_total{network="bench",direction="in",reason="prefix_full"}rateobserver1000
tn_network_source_admission_denied_total{network="bench",direction="in",reason="process_full"}ratevalidator10000
tn_network_source_admission_denied_total{network="bench",direction="in",reason="process_full"}rateobserver1000
tn_network_source_admission_denied_total{network="bench",direction="in",reason="sources_full"}ratevalidator10000
tn_network_source_admission_denied_total{network="bench",direction="in",reason="sources_full"}rateobserver1000
tn_network_source_admission_denied_total{network="bench",direction="in",reason="unsupported_address"}ratevalidator10000
tn_network_source_admission_denied_total{network="bench",direction="in",reason="unsupported_address"}rateobserver1000
tn_network_source_admission_denied_total{network="bench",direction="out",reason="address_full"}ratevalidator10000
tn_network_source_admission_denied_total{network="bench",direction="out",reason="address_full"}rateobserver1000
tn_network_source_admission_denied_total{network="bench",direction="out",reason="duplicate_connection"}ratevalidator10000
tn_network_source_admission_denied_total{network="bench",direction="out",reason="duplicate_connection"}rateobserver1000
tn_network_source_admission_denied_total{network="bench",direction="out",reason="invalid_limits"}ratevalidator10000
tn_network_source_admission_denied_total{network="bench",direction="out",reason="invalid_limits"}rateobserver1000
tn_network_source_admission_denied_total{network="bench",direction="out",reason="peer_full"}ratevalidator10000
tn_network_source_admission_denied_total{network="bench",direction="out",reason="peer_full"}rateobserver1000
tn_network_source_admission_denied_total{network="bench",direction="out",reason="poisoned"}ratevalidator10000
tn_network_source_admission_denied_total{network="bench",direction="out",reason="poisoned"}rateobserver1000
tn_network_source_admission_denied_total{network="bench",direction="out",reason="prefix_full"}ratevalidator10000
tn_network_source_admission_denied_total{network="bench",direction="out",reason="prefix_full"}rateobserver1000
tn_network_source_admission_denied_total{network="bench",direction="out",reason="process_full"}ratevalidator10000
tn_network_source_admission_denied_total{network="bench",direction="out",reason="process_full"}rateobserver1000
tn_network_source_admission_denied_total{network="bench",direction="out",reason="sources_full"}ratevalidator10000
tn_network_source_admission_denied_total{network="bench",direction="out",reason="sources_full"}rateobserver1000
tn_network_source_admission_denied_total{network="bench",direction="out",reason="unsupported_address"}ratevalidator10000
tn_network_source_admission_denied_total{network="bench",direction="out",reason="unsupported_address"}rateobserver1000
tn_network_admission_modemaxvalidator10222
tn_network_admission_modemaxobserver11.9922
tn_network_admission_fallbackmaxvalidator10011
tn_network_admission_fallbackmaxobserver10.006170.251
tn_network_outbound_request_failures_total{kind="connection"}ratevalidator10000
tn_network_outbound_request_failures_total{kind="connection"}rateobserver1000
tn_network_outbound_request_failures_total{kind="dial"}ratevalidator1011.88298.6
tn_network_outbound_request_failures_total{kind="dial"}rateobserver1000
tn_network_outbound_request_failures_total{kind="io"}ratevalidator102.2914.115.6
tn_network_outbound_request_failures_total{kind="io"}rateobserver10.481.011.45
tn_network_outbound_request_failures_total{kind="timeout"}ratevalidator100.5173.073.4
tn_network_outbound_request_failures_total{kind="timeout"}rateobserver10.003320.0260.0372
tn_network_outbound_request_failures_total{kind="unsupported"}ratevalidator10000
tn_network_outbound_request_failures_total{kind="unsupported"}rateobserver1000
tn_network_inbound_requests_shed_total{class="batch",reason="admission"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="batch",reason="admission"}rateobserver1000
tn_network_inbound_requests_shed_total{class="batch",reason="queue_budget"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="batch",reason="queue_budget"}rateobserver1000
tn_network_inbound_requests_shed_total{class="batch",reason="queue_full"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="batch",reason="queue_full"}rateobserver1000
tn_network_inbound_requests_shed_total{class="batch",reason="serve_budget"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="batch",reason="serve_budget"}rateobserver1000
tn_network_inbound_requests_shed_total{class="batch",reason="unsubscribed"}ratevalidator100.025968.885.5
tn_network_inbound_requests_shed_total{class="batch",reason="unsubscribed"}rateobserver1000
tn_network_inbound_requests_shed_total{class="certificate_sync",reason="admission"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="certificate_sync",reason="admission"}rateobserver1000
tn_network_inbound_requests_shed_total{class="certificate_sync",reason="queue_budget"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="certificate_sync",reason="queue_budget"}rateobserver1000
tn_network_inbound_requests_shed_total{class="certificate_sync",reason="queue_full"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="certificate_sync",reason="queue_full"}rateobserver1000
tn_network_inbound_requests_shed_total{class="certificate_sync",reason="serve_budget"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="certificate_sync",reason="serve_budget"}rateobserver1000
tn_network_inbound_requests_shed_total{class="certificate_sync",reason="unsubscribed"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="certificate_sync",reason="unsubscribed"}rateobserver1000
tn_network_inbound_requests_shed_total{class="epoch_record",reason="admission"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="epoch_record",reason="admission"}rateobserver1000
tn_network_inbound_requests_shed_total{class="epoch_record",reason="queue_budget"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="epoch_record",reason="queue_budget"}rateobserver1000
tn_network_inbound_requests_shed_total{class="epoch_record",reason="queue_full"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="epoch_record",reason="queue_full"}rateobserver1000
tn_network_inbound_requests_shed_total{class="epoch_record",reason="serve_budget"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="epoch_record",reason="serve_budget"}rateobserver1000
tn_network_inbound_requests_shed_total{class="epoch_record",reason="unsubscribed"}ratevalidator100.4041.111.42
tn_network_inbound_requests_shed_total{class="epoch_record",reason="unsubscribed"}rateobserver10.2490.5050.632
tn_network_inbound_requests_shed_total{class="gossip",reason="admission"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="gossip",reason="admission"}rateobserver1000
tn_network_inbound_requests_shed_total{class="gossip",reason="queue_budget"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="gossip",reason="queue_budget"}rateobserver1000
tn_network_inbound_requests_shed_total{class="gossip",reason="queue_full"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="gossip",reason="queue_full"}rateobserver1000
tn_network_inbound_requests_shed_total{class="gossip",reason="serve_budget"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="gossip",reason="serve_budget"}rateobserver1000
tn_network_inbound_requests_shed_total{class="gossip",reason="unsubscribed"}ratevalidator100.009570.20.222
tn_network_inbound_requests_shed_total{class="gossip",reason="unsubscribed"}rateobserver10.07460.1490.186
tn_network_inbound_requests_shed_total{class="other",reason="admission"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="other",reason="admission"}rateobserver1000
tn_network_inbound_requests_shed_total{class="other",reason="queue_budget"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="other",reason="queue_budget"}rateobserver1000
tn_network_inbound_requests_shed_total{class="other",reason="queue_full"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="other",reason="queue_full"}rateobserver1000
tn_network_inbound_requests_shed_total{class="other",reason="serve_budget"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="other",reason="serve_budget"}rateobserver1000
tn_network_inbound_requests_shed_total{class="other",reason="unsubscribed"}ratevalidator100.04420.2270.276
tn_network_inbound_requests_shed_total{class="other",reason="unsubscribed"}rateobserver1000
tn_network_inbound_requests_shed_total{class="vote",reason="admission"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="vote",reason="admission"}rateobserver1000
tn_network_inbound_requests_shed_total{class="vote",reason="queue_budget"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="vote",reason="queue_budget"}rateobserver1000
tn_network_inbound_requests_shed_total{class="vote",reason="queue_full"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="vote",reason="queue_full"}rateobserver1000
tn_network_inbound_requests_shed_total{class="vote",reason="serve_budget"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="vote",reason="serve_budget"}rateobserver1000
tn_network_inbound_requests_shed_total{class="vote",reason="unsubscribed"}ratevalidator100.2238.648
tn_network_inbound_requests_shed_total{class="vote",reason="unsubscribed"}rateobserver1000
tn_network_inbound_request_service_seconds{class="batch"}p95validator232 ms181 ms184 ms
tn_network_inbound_request_service_seconds{class="epoch_record"}p95validator35 ms52 ms69 ms
tn_network_inbound_request_service_seconds{class="epoch_record"}p95observer15 ms17 ms52 ms
tn_network_inbound_request_service_seconds{class="vote"}p95validator640 ms47 ms47 ms
tn_network_established_connectionsmaxvalidator1019.22020
tn_network_established_connectionsmaxobserver119.720.521
tn_worker_sync_streams_shed_total{reason="denied"}ratevalidator90.01150.5330.711
tn_worker_sync_streams_shed_total{reason="denied"}rateobserver1000
tn_worker_batch_fetch_duration_secondsp95validator105 ms492 ms712 ms
tn_executor_batch_fetch_failures_totalratevalidator10000
tn_executor_batch_fetch_failures_totalrateobserver1000
tn_primary_votes_received_totalratevalidator1016.517.317.3
tn_primary_votes_received_totalrateobserver1000
tn_primary_vote_request_failures_totalratevalidator1000.150.2
tn_primary_vote_request_failures_totalrateobserver1000
tn_primary_certificates_formed_totalratevalidator102.762.892.89
tn_primary_certificates_formed_totalrateobserver1000
tn_primary_certificate_form_duration_secondsp95validator10243 ms490 ms490 ms
tn_engine_outputs_executed_totalratevalidator101.371.481.67
tn_engine_outputs_executed_totalrateobserver11.361.491.53
tn_engine_persist_failures_totalratevalidator10000
tn_engine_persist_failures_totalrateobserver1000
tn_node_consensus_sync_distancemaxvalidator1017.6506507
tn_node_consensus_sync_distancemaxobserver116.729.230
reth_db_table_size{table="AccountChangeSets"}maxvalidator10382 MiB568 MiB571 MiB
reth_db_table_size{table="AccountChangeSets"}maxobserver1383 MiB566 MiB569 MiB
reth_db_table_size{table="AccountsHistory"}maxvalidator1042 MiB76 MiB77 MiB
reth_db_table_size{table="AccountsHistory"}maxobserver142 MiB76 MiB77 MiB
reth_db_table_size{table="HashedAccounts"}maxvalidator1027 MiB44 MiB45 MiB
reth_db_table_size{table="HashedAccounts"}maxobserver127 MiB44 MiB44 MiB
reth_db_table_size{table="HashedStorages"}maxvalidator1041 MiB44 MiB44 MiB
reth_db_table_size{table="HashedStorages"}maxobserver141 MiB44 MiB44 MiB
reth_db_table_size{table="PlainAccountState"}maxvalidator1022 MiB37 MiB37 MiB
reth_db_table_size{table="PlainAccountState"}maxobserver122 MiB37 MiB37 MiB
reth_db_table_size{table="PlainStorageState"}maxvalidator1041 MiB44 MiB44 MiB
reth_db_table_size{table="PlainStorageState"}maxobserver141 MiB44 MiB44 MiB
reth_db_table_size{table="StorageChangeSets"}maxvalidator10458 MiB689 MiB694 MiB
reth_db_table_size{table="StorageChangeSets"}maxobserver1459 MiB687 MiB691 MiB
reth_db_table_size{table="StoragesHistory"}maxvalidator10100 MiB116 MiB116 MiB
reth_db_table_size{table="StoragesHistory"}maxobserver1100 MiB116 MiB116 MiB
reth_db_table_size{table="TransactionHashNumbers"}maxvalidator10826 MiB1.23 GiB1.24 GiB
reth_db_table_size{table="TransactionHashNumbers"}maxobserver1827 MiB1.23 GiB1.23 GiB
reth_db_table_size{table="TransactionSenders"}maxvalidator10425 MiB651 MiB655 MiB
reth_db_table_size{table="TransactionSenders"}maxobserver1426 MiB649 MiB653 MiB
reth_db_freelistmaxvalidator1028.1K53.9K58K
reth_db_freelistmaxobserver128.1K53K56.5K
reth_database_transaction_commit_write_duration_secondsp95validator105 ms5 ms5 ms
reth_database_transaction_commit_write_duration_secondsp95observer15 ms5 ms5 ms
reth_database_transaction_commit_sync_duration_secondsp95validator10476 ms543 ms683 ms
reth_database_transaction_commit_sync_duration_secondsp95observer1491 ms860 ms918 ms
reth_database_transaction_commit_whole_duration_secondsp95validator10476 ms543 ms683 ms
reth_database_transaction_commit_whole_duration_secondsp95observer1491 ms860 ms918 ms
tn_engine_recover_secondsp95validator10116 ms367 ms488 ms
tn_engine_recover_secondsp95observer1128 ms452 ms467 ms
tn_engine_evm_secondsp95validator10253 ms725 ms725 ms
tn_engine_evm_secondsp95observer1244 ms444 ms456 ms
tn_engine_root_secondsp95validator10358 ms901 ms1,038 ms
tn_engine_root_secondsp95observer1338 ms826 ms867 ms
tn_engine_finish_secondsp95validator10165 ms440 ms440 ms
tn_engine_finish_secondsp95observer1127 ms305 ms350 ms
tn_engine_forward_wait_secondsp95validator109,865 ms29.0 s29.0 s
tn_engine_forward_wait_secondsp95observer19,605 ms29.0 s29.0 s
tn_engine_channel_depthmaxvalidator108.3120.221
tn_engine_channel_depthmaxobserver16.918.519
tn_engine_executed_transactions_totalratevalidator105.48K13.8K13.8K
tn_engine_executed_transactions_totalrateobserver15.43K13.1K13.1K
tn_executor_execution_wait_secondsp95validator105 ms5 ms5 ms
tn_primary_commit_run_ahead_blocked_totalratevalidator10000
tn_primary_commit_run_ahead_blocked_totalrateobserver1000
tn_primary_consensus_exec_lag_roundsmaxvalidator1035.56062
tn_primary_consensus_exec_lag_roundsmaxobserver132.256.558
tn_primary_commit_lag_roundsmaxvalidator102.0933
tn_primary_commit_lag_roundsmaxobserver1000
tn_primary_rejoin_grace_holds_totalratevalidator10000
tn_primary_rejoin_grace_holds_totalrateobserver1000
tn_primary_votes_refused_inactive_totalratevalidator10033.333.4
tn_primary_votes_refused_inactive_totalrateobserver1000
tn_primary_votes_refused_execution_behind_totalratevalidator10000
tn_primary_votes_refused_execution_behind_totalrateobserver1000
tn_batch_builder_pool_age_secondsp95validator105,413 ms30.0 s30.0 s
tn_batch_builder_builds_total{reason="bytes_full"}ratevalidator10000
tn_batch_builder_builds_total{reason="bytes_full"}rateobserver1000
tn_batch_builder_builds_total{reason="drained"}ratevalidator102.053.093.09
tn_batch_builder_builds_total{reason="drained"}rateobserver1000
tn_batch_builder_builds_total{reason="gas_full"}ratevalidator101.072.833.49
tn_batch_builder_builds_total{reason="gas_full"}rateobserver1000
tn_worker_batch_report_secondsp95validator100 ms1 ms4 ms
tn_primary_digest_wait_secondsp95validator10478 ms30.0 s30.0 s
tn_primary_headers_proposed_total{reason="max_delay"}ratevalidator100.004530.1170.133
tn_primary_headers_proposed_total{reason="min_delay"}ratevalidator102.752.872.87
tn_primary_headers_proposed_total{reason="parents_and_threshold"}ratevalidator1000.10.133
tn_primary_digests_requeued_total{cause="canceled"}ratevalidator1000.1670.222
tn_primary_digests_requeued_total{cause="committed"}ratevalidator1000.9281.38
tn_executor_consensus_pack_persist_secondsp95validator1048 ms83 ms86 ms
tn_batch_builder_seal_duration_secondsp95validator10252 ms900 ms900 ms
tn_worker_quorum_wait_secondsp95validator10251 ms800 ms867 ms
tn_primary_proposer_pending_digestsmaxvalidator100.5496.2524
tn_primary_proposer_pending_digestsmaxobserver1000
reth_transaction_pool_pending_pool_transactionsmaxvalidator103084.98K9.84K
reth_transaction_pool_pending_pool_transactionsmaxobserver1000
reth_transaction_pool_queued_pool_transactionsmaxvalidator101744.41K6.4K
reth_transaction_pool_queued_pool_transactionsmaxobserver1000
reth_transaction_pool_all_transactions_by_hashmaxvalidator108399.9K10.2K
reth_transaction_pool_all_transactions_by_hashmaxobserver1000
reth_storage_providers_database_save_blocks_totalp95validator10120 ms320 ms374 ms
reth_storage_providers_database_save_blocks_totalp95observer1109 ms215 ms240 ms
reth_storage_providers_database_save_blocks_mdbxp95validator10120 ms320 ms374 ms
reth_storage_providers_database_save_blocks_mdbxp95observer1109 ms215 ms240 ms
reth_storage_providers_database_save_blocks_sfp95validator1028 ms60 ms69 ms
reth_storage_providers_database_save_blocks_sfp95observer131 ms66 ms78 ms
reth_storage_providers_database_save_blocks_commit_mdbxp95validator10363 ms493 ms525 ms
reth_storage_providers_database_save_blocks_commit_mdbxp95observer1403 ms551 ms565 ms
reth_storage_providers_database_save_blocks_commit_sfp95validator1010 ms16 ms16 ms
reth_storage_providers_database_save_blocks_commit_sfp95observer18 ms9 ms10 ms
reth_storage_providers_database_save_blocks_write_statep95validator1021 ms46 ms51 ms
reth_storage_providers_database_save_blocks_write_statep95observer119 ms38 ms44 ms
reth_storage_providers_database_save_blocks_write_hashed_statep95validator1014 ms26 ms29 ms
reth_storage_providers_database_save_blocks_write_hashed_statep95observer113 ms23 ms24 ms
reth_storage_providers_database_save_blocks_write_trie_updatesp95validator1019 ms35 ms44 ms
reth_storage_providers_database_save_blocks_write_trie_updatesp95observer117 ms30 ms33 ms
reth_database_transaction_commit_preparation_duration_secondsp95validator105 ms5 ms5 ms
reth_database_transaction_commit_preparation_duration_secondsp95observer15 ms5 ms5 ms
reth_database_transaction_commit_gc_wallclock_duration_secondsp95validator105 ms5 ms5 ms
reth_database_transaction_commit_gc_wallclock_duration_secondsp95observer15 ms5 ms5 ms
reth_database_transaction_commit_gc_cputime_duration_secondsp95validator105 ms5 ms5 ms
reth_database_transaction_commit_gc_cputime_duration_secondsp95observer15 ms5 ms5 ms
reth_database_transaction_commit_audit_duration_secondsp95validator105 ms5 ms5 ms
reth_database_transaction_commit_audit_duration_secondsp95observer15 ms5 ms5 ms
reth_database_transaction_commit_ending_duration_secondsp95validator105 ms5 ms5 ms
reth_database_transaction_commit_ending_duration_secondsp95observer15 ms5 ms5 ms
tn_node_switch{name="digest_dedupe"}maxvalidator10000
tn_node_switch{name="digest_dedupe"}maxobserver1000
tn_node_switch{name="epoch_close_task_drain"}maxvalidator10000
tn_node_switch{name="epoch_close_task_drain"}maxobserver1000
tn_node_switch{name="epoch_record_state_check"}maxvalidator10111
tn_node_switch{name="epoch_record_state_check"}maxobserver1111
tn_node_switch{name="mdbx_random_advice"}maxvalidator10111
tn_node_switch{name="mdbx_random_advice"}maxobserver1111
tn_node_switch{name="parent_grace"}maxvalidator10000
tn_node_switch{name="parent_grace"}maxobserver1000
tn_node_switch{name="pipelined_persist"}maxvalidator10000
tn_node_switch{name="pipelined_persist"}maxobserver1000
tn_node_switch{name="quorum_exec_block"}maxvalidator10111
tn_node_switch{name="quorum_exec_block"}maxobserver1111
tn_node_switch{name="quorum_exec_block_starved_rounds"}maxvalidator10000
tn_node_switch{name="quorum_exec_block_starved_rounds"}maxobserver1000
tn_node_switch{name="recovery_threads"}maxvalidator10000
tn_node_switch{name="recovery_threads"}maxobserver1000
tn_node_switch{name="rpc_admission_backlog_outputs"}maxvalidator10000
tn_node_switch{name="rpc_admission_backlog_outputs"}maxobserver1000
tn_node_switch{name="sealed_tx_keep"}maxvalidator10000
tn_node_switch{name="sealed_tx_keep"}maxobserver1000
tn_node_switch{name="vote_queue_fix"}maxvalidator10111
tn_node_switch{name="vote_queue_fix"}maxobserver1111
tn_primary_epoch_entry_rounds_behindmaxvalidator1032929
tn_primary_epoch_entry_rounds_behindmaxobserver1000
tn_reth_mdbx_random_advice_bytesmaxvalidator108192.00 GiB8192.00 GiB8192.00 GiB
tn_reth_mdbx_random_advice_bytesmaxobserver18192.00 GiB8192.00 GiB8192.00 GiB
tn_reth_mdbx_used_bytesmaxvalidator102.44 GiB3.65 GiB3.65 GiB
tn_reth_mdbx_used_bytesmaxobserver12.45 GiB3.62 GiB3.65 GiB
tn_reth_mdbx_page_ops_total{op="cow"}maxvalidator1036.8M61.2M61.6M
tn_reth_mdbx_page_ops_total{op="cow"}maxobserver129.1M54.3M54.7M
tn_reth_mdbx_page_ops_total{op="msync"}maxvalidator106.49K9.76K9.83K
tn_reth_mdbx_page_ops_total{op="msync"}maxobserver15.24K8.8K8.86K
tn_reth_mdbx_page_ops_total{op="newly"}maxvalidator10637K1.21M1.24M
tn_reth_mdbx_page_ops_total{op="newly"}maxobserver1515K850K864K
tn_reth_mdbx_page_ops_total{op="prefault"}maxvalidator10640K956K957K
tn_reth_mdbx_page_ops_total{op="prefault"}maxobserver1518K857K867K
tn_reth_mdbx_page_ops_total{op="spill"}maxvalidator10000
tn_reth_mdbx_page_ops_total{op="spill"}maxobserver1000
tn_reth_mdbx_page_ops_total{op="unspill"}maxvalidator10000
tn_reth_mdbx_page_ops_total{op="unspill"}maxobserver1000
tn_reth_mdbx_page_ops_total{op="wops"}maxvalidator10111
tn_reth_mdbx_page_ops_total{op="wops"}maxobserver1111
tn_reth_sealed_txs_markedmaxvalidator10000
tn_reth_sealed_txs_markedmaxobserver1000
tn_reth_sealed_txs_cleared_executed_totalmaxvalidator10000
tn_reth_sealed_txs_cleared_executed_totalmaxobserver1000
tn_reth_sealed_txs_cleared_repooled_totalmaxvalidator10000
tn_reth_sealed_txs_cleared_repooled_totalmaxobserver1000
tn_reth_sealed_txs_dropped_evicted_totalmaxvalidator10000
tn_reth_sealed_txs_dropped_evicted_totalmaxobserver1000
tn_reth_sealed_txs_cleared_lag_totalmaxvalidator10000
tn_reth_sealed_txs_cleared_lag_totalmaxobserver1000
tn_reth_sealed_txs_late_marks_refused_totalmaxvalidator10000
tn_reth_sealed_txs_late_marks_refused_totalmaxobserver1000
tn_batch_builder_sealed_txs_skipped_total{worker="0"}maxvalidator10000
tn_batch_builder_sealed_txs_skipped_total{worker="0"}maxobserver1000
reth_transaction_pool_pending_pool_size_bytesmaxvalidator10110.0 KiB1.7 MiB3.4 MiB
reth_transaction_pool_pending_pool_size_bytesmaxobserver10 B0 B0 B
reth_transaction_pool_queued_pool_size_bytesmaxvalidator1061.9 KiB1.5 MiB1.8 MiB
reth_transaction_pool_queued_pool_size_bytesmaxobserver10 B0 B0 B
reth_transaction_pool_pending_transactions_evictedmaxvalidator10000
reth_transaction_pool_pending_transactions_evictedmaxobserver1000
reth_transaction_pool_queued_transactions_evictedmaxvalidator10000
reth_transaction_pool_queued_transactions_evictedmaxobserver1000
reth_transaction_pool_basefee_transactions_evictedmaxvalidator10000
reth_transaction_pool_basefee_transactions_evictedmaxobserver1000
reth_transaction_pool_removed_transactionsmaxvalidator101.4M2.39M2.4M
reth_transaction_pool_removed_transactionsmaxobserver1000
reth_transaction_pool_invalid_transactionsmaxvalidator10000
reth_transaction_pool_invalid_transactionsmaxobserver1000
tn_worker_duplicate_digest_suppressed_total{worker="0"}maxvalidator10000
tn_worker_duplicate_digest_suppressed_total{worker="0"}maxobserver1000
tn_primary_duplicate_digest_suppressed_totalmaxvalidator10000
tn_primary_duplicate_digest_suppressed_totalmaxobserver1000
tn_engine_recovery_fallback_totalmaxvalidator10000
tn_engine_recovery_fallback_totalmaxobserver1000
tn_node_epoch_close_backlog_outputsmaxvalidator10152626
tn_node_epoch_close_backlog_outputsmaxobserver17.6899
tn_node_epoch_close_detect_lag_secondsmaxvalidator10637 ms343.4 s343.4 s
tn_node_epoch_close_detect_lag_secondsmaxobserver11,128 ms1,204 ms1,204 ms
tn_node_epoch_close_drain_secondsmaxvalidator1010.2 s20.5 s20.5 s
tn_node_epoch_close_drain_secondsmaxobserver15,097 ms6,386 ms6,386 ms
tn_node_epoch_record_vote_skipped_total{reason="certified_before_close"}maxvalidator1111
tn_node_epoch_record_vote_skipped_total{reason="not_in_committee"}maxobserver14.0955
tn_primary_parent_grace_total{outcome="complete"}maxvalidator10000
tn_primary_parent_grace_total{outcome="expired"}maxvalidator10000
tn_primary_parent_grace_total{outcome="preempted"}maxvalidator10000
tn_primary_late_parents_total{origin="older_round"}maxvalidator105.28K10.3K10.4K
tn_primary_late_parents_total{origin="own_round_behind"}maxvalidator1031.2K41K41K
tn_node_execution_backlog_outputsmaxvalidator1018.230.531
tn_node_execution_backlog_outputsmaxobserver116.828.529
tn_engine_unpersisted_outputsmaxvalidator10000
tn_engine_unpersisted_outputsmaxobserver1000
tn_engine_built_heightmaxvalidator10000
tn_engine_built_heightmaxobserver1000
tn_engine_close_barrier_holds_totalmaxvalidator10000
tn_engine_close_barrier_holds_totalmaxobserver1000

Commit latency p95

tn_primary_commit_latency_seconds

Round lag

tn_primary_round − tn_primary_committed_round

Engine queued outputs

tn_engine_queued_outputs, max by node

Certification by node

The share of each validator's own headers that formed a certificate, per window (tools/cert-rate.py on the node logs). Every validator certified at least 90 % of its own headers in every window where it proposed at least 50. Own leader % is the validator's share of the committed leaders and not committed % the share of its certificates no committed leader reached, both over the run window; the fair share is 100 / 10 validators = 10.0 %. Greyed: fewer than 50 headers proposed, not judged.

ValidatorRegionblast 0 preambleblast 0 warmupblast 0 mixedblast 0 cooldownblast 1 preambleblast 1 warmupblast 1 mixedblast 1 cooldownblast 2 preambleblast 2 warmupblast 2 mixedblast 2 cooldownrunOwn leader %Not committed %
bench-validator-01us-west2 (Los Angeles)100.0100.4100.1100.0100.0100.099.999.6101.0100.0100.0100.0100.013.60.0
bench-validator-02europe-west2 (London)100.0100.099.399.4100.699.799.8100.4100.0100.099.999.599.70.572.0
bench-validator-03australia-southeast1 (Sydney)100.0100.0100.1100.699.4100.0100.1100.099.0100.0100.0100.0100.0––
bench-validator-04asia-northeast1 (Tokyo)100.0100.0100.1100.699.4100.0100.099.6100.0100.0100.099.5100.0––
bench-validator-05us-east4 (Ashburn)100.0100.099.8100.0100.0100.099.6100.099.0100.099.9–99.88.512.1
bench-validator-06europe-west4 (Netherlands)98.699.299.299.4100.0100.099.4100.0100.0100.099.799.599.60.375.0
bench-validator-07asia-southeast1 (Singapore)100.0100.4100.199.4100.099.7100.0100.0100.0100.2100.099.5100.013.70.0
bench-validator-08asia-east2 (Hong Kong)100.099.6100.1100.699.4100.3100.0100.0100.0100.2100.099.5100.0––
bench-validator-09asia-northeast3 (Seoul)100.099.6100.1100.6100.0100.099.9100.0101.0100.2100.099.5100.0––
bench-validator-10asia-south1 (Mumbai)98.699.6100.0100.099.499.799.9100.0100.0100.099.999.599.911.74.8

On-node sampler

NodeClassPeak RssAnonPeak RssFilePeak cachePSI memPSI ioPSI cpudb growthstatic_files growthRoot usedOOM Δ
bench-observer-11observer1010 MiB3.55 GiB– 0.0 %13.7 %1.0 %0 B4.08 GiB16.0 %0
bench-validator-01validator2.36 GiB3.23 GiB– 0.0 %13.6 %8.7 %0 B4.08 GiB18.0 %0
bench-validator-02validator2.47 GiB3.65 GiB– 0.0 %14.3 %8.1 %0 B4.08 GiB18.0 %0
bench-validator-03validator2.47 GiB3.82 GiB– 0.0 %12.9 %9.3 %0 B4.08 GiB18.0 %0
bench-validator-04validator2.53 GiB3.51 GiB– 0.0 %14.2 %8.8 %0 B4.08 GiB18.0 %0
bench-validator-05validator2.27 GiB3.93 GiB– 0.0 %11.9 %6.0 %0 B4.08 GiB18.0 %0
bench-validator-06validator2.70 GiB3.80 GiB– 0.0 %14.8 %6.1 %0 B4.08 GiB18.0 %0
bench-validator-07validator1.98 GiB3.82 GiB– 0.0 %14.4 %6.4 %0 B4.08 GiB18.0 %0
bench-validator-08validator2.54 GiB3.63 GiB– 0.0 %14.9 %6.9 %0 B4.08 GiB18.0 %0
bench-validator-09validator2.00 GiB3.78 GiB– 0.0 %14.3 %5.6 %0 B4.08 GiB18.0 %0
bench-validator-10validator2.53 GiB3.90 GiB– 0.0 %13.4 %8.3 %0 B4.08 GiB18.0 %0

RssAnon vs RssFile

/proc/<pid>/status of the telcoin process on the node with the largest RSS

consensus-db/cache size

Not collected: consensus-db/cache size.

Pressure stall (some, avg10)

/proc/pressure on the node with the highest memory PSI

Headroom

Per role and per node, from sizing.json (p95 over the landed window; PSI and RssAnon max over the whole sampler run). The lines follow the hardware doc's tier rules: CPU p95 over 70 % of vCPUs; memory used p95 over 75 % of RAM, or telcoin RSS p95 over 85 % of RAM; disk write p95 over 60 % of the PD MiB/s limit, or write IOPS p95 over 60 % of the IOPS limit; IO pressure over 20 % or memory pressure over 5 % (PSI some avg10, max); network p95 over 50 % of 1 Gbps; root disk over 80 %. A line crossed on any node puts the role at the limit; a nonzero reading within 15 points of its line is tight; otherwise the role has headroom. OOM kills, disk throttling and degraded disk performance are flagged too.

SignalClassWorst nodeValueShare of capacityLineNodes over / tightStatus
CPU p95validatorbench-validator-0336.0 %36.0 %> 70 %0 / 0 headroom
Memory used p95validatorbench-validator-0622.7 %22.7 %> 75 %0 / 0 headroom
telcoin RSS p95validatorbench-validator-066.87 GB40.0 %> 85 %0 / 0 headroom
PSI memory maxvalidatorbench-validator-010.0 %0.0 %> 5 %0 / 0 headroom
RssAnon peakvalidatorbench-validator-062.90 GB16.9 %info– info
Disk write p95validatorbench-validator-03110 MB/s13.1 %> 60 %0 / 0 headroom
Write IOPS p95validatorbench-validator-0620,146100.7 %> 60 %10 / 0 at the limit
PSI io maxvalidatorbench-validator-0814.9 %14.9 %> 20 %0 / 10 tight
Root disk maxvalidatorbench-validator-0317.5 %17.5 %> 80 %0 / 0 headroom
Network p95validatorbench-validator-07130 Mbps13.0 %> 50 %0 / 0 headroom
Consensus cache peakvalidator–––info– not collected
OOM killsvalidator––anyany– none
Disk throttlingvalidator––metric type unavailable in projectany– not collected
Disk performance degradedvalidator––any sampleany– none
CPU p95observerbench-observer-1120.4 %20.4 %> 70 %0 / 0 headroom
Memory used p95observerbench-observer-1112.4 %12.4 %> 75 %0 / 0 headroom
telcoin RSS p95observerbench-observer-114.54 GB26.4 %> 85 %0 / 0 headroom
PSI memory maxobserverbench-observer-110.0 %0.0 %> 5 %0 / 0 headroom
RssAnon peakobserverbench-observer-111.06 GB6.2 %info– info
Disk write p95observerbench-observer-1197 MB/s11.5 %> 60 %0 / 0 headroom
Write IOPS p95observerbench-observer-1120,071100.4 %> 60 %1 / 0 at the limit
PSI io maxobserverbench-observer-1113.7 %13.7 %> 20 %0 / 1 tight
Root disk maxobserverbench-observer-1115.9 %15.9 %> 80 %0 / 0 headroom
Network p95observerbench-observer-1112 Mbps1.2 %> 50 %0 / 0 headroom
Consensus cache peakobserver–––info– not collected
OOM killsobserver––anyany– none
Disk throttlingobserver––metric type unavailable in projectany– not collected
Disk performance degradedobserver––any sampleany– none

Per node

NodeClassCPU p95Memory used p95telcoin RSS p95PSI memory maxRssAnon peakDisk write p95Write IOPS p95PSI io maxRoot disk maxNetwork p95Consensus cache peak
Line> 70 %> 75 %> 85 %> 5 %info> 60 %> 60 %> 20 %> 80 %> 50 %info
bench-validator-01validator35.3 %20.7 %5.89 GB 34 %0.0 %2.53 GB 15 %109 MB/s 13 %20,103 101 %13.6 %17.5 %111 Mbps 11 %–
bench-validator-02validator33.6 %21.3 %6.35 GB 37 %0.0 %2.66 GB 15 %109 MB/s 13 %20,123 101 %14.3 %17.5 %102 Mbps 10 %–
bench-validator-03validator36.0 %21.3 %6.52 GB 38 %0.0 %2.65 GB 15 %110 MB/s 13 %20,144 101 %12.9 %17.5 %109 Mbps 11 %–
bench-validator-04validator35.3 %21.7 %6.26 GB 36 %0.0 %2.72 GB 16 %109 MB/s 13 %20,093 100 %14.2 %17.5 %107 Mbps 11 %–
bench-validator-05validator34.9 %20.2 %5.67 GB 33 %0.0 %2.43 GB 14 %108 MB/s 13 %20,113 101 %11.9 %16.9 %107 Mbps 11 %–
bench-validator-06validator34.8 %22.7 %6.87 GB 40 %0.0 %2.90 GB 17 %109 MB/s 13 %20,146 101 %14.8 %17.3 %104 Mbps 10 %–
bench-validator-07validator35.1 %18.3 %5.92 GB 34 %0.0 %2.13 GB 12 %108 MB/s 13 %20,101 101 %14.4 %17.5 %130 Mbps 13 %–
bench-validator-08validator35.0 %21.6 %6.35 GB 37 %0.0 %2.73 GB 16 %108 MB/s 13 %20,084 100 %14.9 %17.5 %112 Mbps 11 %–
bench-validator-09validator34.1 %18.5 %5.96 GB 35 %0.0 %2.15 GB 12 %108 MB/s 13 %20,050 100 %14.3 %17.5 %111 Mbps 11 %–
bench-validator-10validator35.4 %21.6 %6.72 GB 39 %0.0 %2.72 GB 16 %108 MB/s 13 %20,096 100 %13.4 %17.5 %111 Mbps 11 %–
bench-observer-11observer20.4 %12.4 %4.54 GB 26 %0.0 %1.06 GB 6 %97 MB/s 12 %20,071 100 %13.7 %15.9 %12 Mbps 1 %–

Small figures are the share of capacity: RAM for RSS, the PD limit for disk write (VM cap applied), 1 Gbps for network. Hover a cell for its JSON path.

Verdict

Fleet A vs Fleet B

A = Fleet c3-hdb · c3-highcpu-8 · r20261113-1351
B = Fleet c3-hdb · c3-highcpu-8 · r20261107-1504

Verdict

at the limit c3-highcpu-8 validators at 8,065 sustained TPS: at the limit on disk. Disk at the limit: write IOPS p95 20,050–20,146, up to 100.7 % of the 20,000 IOPS limit, over the 60 % line on all 10 validators (bench-validator-01, 02, 03, 04, 05, 06, 07, 08, 09, 10); IO pressure (PSI some avg10) max 14.9 % (bench-validator-08), tight against the 20 % line. CPU has headroom: CPU p95 36.0 % of 8 vCPUs (bench-validator-03). Memory has headroom: memory used p95 22.7 % of RAM (bench-validator-06); telcoin RSS p95 6.87 GB, 40.0 % of 16 GiB RAM (bench-validator-06); memory pressure (PSI some avg10) max 0.0 % (bench-validator-01). Network has headroom: network p95 130 Mbps, 13.0 % of 1 Gbps (bench-validator-07). Next step: a faster volume (pd-ssd adds 30 IOPS and 0.48 MiB/s per GB up to the per-VM cap; past the cap, a larger VM).

at the limit c3-highcpu-8 observers at 8,065 sustained TPS: at the limit on disk. Disk at the limit: write IOPS p95 20,071, 100.4 % of the 20,000 IOPS limit on bench-observer-11, over the 60 % line; IO pressure (PSI some avg10) max 13.7 % (bench-observer-11), tight against the 20 % line. CPU has headroom: CPU p95 20.4 % of 8 vCPUs (bench-observer-11). Memory has headroom: memory used p95 12.4 % of RAM (bench-observer-11); telcoin RSS p95 4.54 GB, 26.4 % of 16 GiB RAM (bench-observer-11); memory pressure (PSI some avg10) max 0.0 % (bench-observer-11). Network has headroom: network p95 12 Mbps, 1.2 % of 1 Gbps (bench-observer-11). Next step: a faster volume (pd-ssd adds 30 IOPS and 0.48 MiB/s per GB up to the per-VM cap; past the cap, a larger VM).

at the limit c3-highcpu-8 validators at 4,939 sustained TPS: at the limit on disk. Disk at the limit: write p95 547–595 MB/s, up to 70.9 % of the 800 MiB/s limit, over the 60 % line on all 10 validators (bench-validator-01, 02, 03, 04, 05, 06, 07, 08, 09, 10); IO pressure (PSI some avg10) max 29.4–34.2 %, over the 20 % line on all 10 validators (bench-validator-01, 02, 03, 04, 05, 06, 07, 08, 09, 10). CPU has headroom: CPU p95 22.2 % of 8 vCPUs (bench-validator-08). Memory has headroom: memory used p95 34.2 % of RAM (bench-validator-02); telcoin RSS p95 8.11 GB, 47.2 % of 16 GiB RAM (bench-validator-02); memory pressure (PSI some avg10) max 0.0 % (bench-validator-01). Network has headroom: network p95 82 Mbps, 8.2 % of 1 Gbps (bench-validator-08). Next step: a faster volume (pd-ssd adds 30 IOPS and 0.48 MiB/s per GB up to the per-VM cap; past the cap, a larger VM).

at the limit c3-highcpu-8 observers at 4,939 sustained TPS: at the limit on disk. Disk at the limit: write p95 588 MB/s, 70.1 % of the 800 MiB/s limit on bench-observer-11, over the 60 % line; IO pressure (PSI some avg10) max 32.8 % on bench-observer-11, over the 20 % line. CPU has headroom: CPU p95 13.2 % of 8 vCPUs (bench-observer-11). Memory has headroom: memory used p95 11.7 % of RAM (bench-observer-11); telcoin RSS p95 3.79 GB, 22.1 % of 16 GiB RAM (bench-observer-11); memory pressure (PSI some avg10) max 0.0 % (bench-observer-11). Network has headroom: network p95 4 Mbps, 0.4 % of 1 Gbps (bench-observer-11). Next step: a faster volume (pd-ssd adds 30 IOPS and 0.48 MiB/s per GB up to the per-VM cap; past the cap, a larger VM).

Validators: Fleet A (c3-highcpu-8) is at the limit on disk, with headroom on CPU, memory and network; Fleet B (c3-highcpu-8) is at the limit on disk, with headroom on CPU, memory and network. The same disk limit binds both fleets: every validator runs a 100 GB hyperdisk-balanced, which allows 20,000 IOPS on both; its throughput limit is 800 MiB/s on both. Disk is at the limit on both (Fleet A (c3-highcpu-8): write IOPS p95 20,146, 100.7 % of the 20,000 IOPS limit; Fleet B (c3-highcpu-8): write p95 595 MB/s, 70.9 % of the 800 MiB/s limit, IO pressure (PSI some avg10) max 34.2 %). Changing the machine shape did not clear it; the volume is the binding resource.

Observers: Fleet A (c3-highcpu-8) is at the limit on disk, with headroom on CPU, memory and network; Fleet B (c3-highcpu-8) is at the limit on disk, with headroom on CPU, memory and network. The same disk limit binds both fleets: every observer runs a 100 GB hyperdisk-balanced, which allows 20,000 IOPS on both; its throughput limit is 800 MiB/s on both. Disk is at the limit on both (Fleet A (c3-highcpu-8): write IOPS p95 20,071, 100.4 % of the 20,000 IOPS limit; Fleet B (c3-highcpu-8): write p95 588 MB/s, 70.1 % of the 800 MiB/s limit, IO pressure (PSI some avg10) max 32.8 %). Changing the machine shape did not clear it; the volume is the binding resource.

KPIFleet AFleet BB ÷ A
Sustained mixed TPS (own)8,065.4 tx/s4,939.2 tx/s0.61×
Sustained mixed TPS (chain)8,065.4 tx/s4,939.2 tx/s0.61×
Landed / accepted100.0 %99.6 %1.00×
Reverted (own)00–
Latency p50679 ms2,809 ms4.13×
Latency p9514,662 ms14,404 ms0.98×
Gas / s (own)269M gas/s174M gas/s0.65×
Capacity at mix14,103.0 tx/s7,705.4 tx/s0.55×

Headroom, worst node per class

SignalClassFleet AFleet BLine
CPU p95validator36.0 % headroom22.2 % headroom> 70 %
Memory used p95validator22.7 % headroom34.2 % headroom> 75 %
telcoin RSS p95validator6.87 GB 40.0 % headroom8.11 GB 47.2 % headroom> 85 %
PSI memory maxvalidator0.0 % headroom0.0 % headroom> 5 %
Disk write p95validator110 MB/s 13.1 % headroom595 MB/s 70.9 % at the limit> 60 %
Write IOPS p95validator20,146 100.7 % at the limit8,909 44.5 % headroom> 60 %
PSI io maxvalidator14.9 % tight34.2 % at the limit> 20 %
Root disk maxvalidator17.5 % headroom15.6 % headroom> 80 %
Network p95validator130 Mbps 13.0 % headroom82 Mbps 8.2 % headroom> 50 %
CPU p95observer20.4 % headroom13.2 % headroom> 70 %
Memory used p95observer12.4 % headroom11.7 % headroom> 75 %
telcoin RSS p95observer4.54 GB 26.4 % headroom3.79 GB 22.1 % headroom> 85 %
PSI memory maxobserver0.0 % headroom0.0 % headroom> 5 %
Disk write p95observer97 MB/s 11.5 % headroom588 MB/s 70.1 % at the limit> 60 %
Write IOPS p95observer20,071 100.4 % at the limit5,206 26.0 % headroom> 60 %
PSI io maxobserver13.7 % tight32.8 % at the limit> 20 %
Root disk maxobserver15.9 % headroom13.2 % headroom> 80 %
Network p95observer12 Mbps 1.2 % headroom4 Mbps 0.4 % headroom> 50 %

Own TPS, trailing 60-s mean

aligned on each fleet's mixed-phase start

CPU, hottest validator

aligned on each fleet's mixed-phase start

Memory used, hottest validator

aligned on each fleet's mixed-phase start

telcoin RSS, hottest validator

aligned on each fleet's mixed-phase start

Disk MiB/s, hottest validator

aligned on each fleet's mixed-phase start

Network sent, hottest validator

aligned on each fleet's mixed-phase start

Reference chains

Not collected: reference chains.