This is the full report of the run “The adaptive vote fix on Hyperdisk Balanced at 20,000 IOPS”. Back to the run page.
r20261113-2118 · Fleet c3-hdb · c3-highcpu-8

Telcoin Network benchmark: mixed workload

Throughput is measured from blocks the chain produced (own senders vs every sender), resource use from Cloud Monitoring, Prometheus and the on-node sampler. Summaries cover the mixed phase. Hover any number for the JSON field it came from.

10 blaster(s), 6K senders, chain 32286, schema tn-blast-combined/v2 · mixed phase 2026-11-13 22:45–23:25 UTC · 1 observers, 10 validators

Preflight: WTEL from genesis (0x0000…37e1), v3 fee tier 3000, v4 available, Multicall3 present · Cooldown ended: all_landed ×3, idle_30s ×7

Headline numbers

Sustained mixed TPS (own)
7,637.0 tx/s
best 60-s mean of our own landed tx in the mixed phase
Sustained mixed TPS (chain)
7,637.0 tx/s
all tx in blocks, same window
Landed / accepted
99.9 %
14.7M of 14.7M accepted
Reverted (own)
0
0.00 % of landed
Latency p50
745 ms
submit → block timestamp; seen p50 12,895 ms
Latency p95
6,788 ms
seen p95 28,759 ms
Gas / s (own)
273M gas/s
chain 273M gas/s
Capacity at mix
14,188.5 tx/s
gas-limit basis 9,310.6 tx/s; block gas limit 30M

Per transaction type

Landed counts only our own senders; latency is submit → block timestamp.

TypeSubmittedAcceptedLandedRevertedUnlandedSkipsGas limitGas used avgp50p95
native_transfer4.53M4.45M4.45M02.22K024.1K21K739 ms6,617 ms
batched_self_send2.24M2.18M2.17M01.98K024.1K21K763 ms6,892 ms
uniswap_v2_swap2.01M1.95M1.95M01.63K0180K106K741 ms6,650 ms
stablecoin_transfer1.67M1.62M1.62M01.38K075K43.9K740 ms6,608 ms
wtel_wrap1.34M1.3M1.3M01.06K050K27.9K739 ms6,623 ms
wtel_unwrap898K873K872K0744045K35.2K740 ms6,669 ms
stablecoin_approve670K651K651K0531060K31.8K739 ms6,579 ms
uniswap_v2_add_liquidity450K437K437K03480250K128K743 ms6,684 ms
contract_deploy335K326K326K02490140K74.2K741 ms6,654 ms
uniswap_v4_swap335K326K326K02600250K124K742 ms6,745 ms
uniswap_v3_swap225K219K219K01620220K120K740 ms6,661 ms
uniswap_v2_remove_liquidity225K219K219K02040220K129K739 ms6,558 ms
faucet_drip126K126K126K01428K150K107K886 ms14,804 ms

Submission by RPC node

NodeSubmittedAcceptedAccepted %Top errorsSend p50Send p95
bench-validator-01
bench-validator-01:8545
1.82M1.82M99.9 %known 1.43K3 ms14 ms
bench-validator-02
bench-validator-02:8545
955K930K97.4 %known 12.7K8 ms34 ms
bench-validator-03
bench-validator-03:8545
1.83M1.83M100.0 %known 3484 ms19 ms
bench-validator-04
bench-validator-04:8545
1.84M1.84M100.0 %none6 ms23 ms
bench-validator-05
bench-validator-05:8545
2.19M1.1M50.4 %other 1.09M, known 641 ms15 ms
bench-validator-06
bench-validator-06:8545
738K701K94.9 %known 20K8 ms34 ms
bench-validator-07
bench-validator-07:8545
1.58M1.57M99.2 %known 5.71K5 ms19 ms
bench-validator-08
bench-validator-08:8545
1.73M1.73M99.6 %known 2.88K4 ms19 ms
bench-validator-09
bench-validator-09:8545
1.78M1.77M99.7 %known 3.22K3 ms16 ms
bench-validator-10
bench-validator-10:8545
1.48M1.46M99.0 %known 9.46K4 ms15 ms

Throughput over time

Shaded: phases outside the mixed window. Hairlines: epoch closes. Dashed orange: chaos events.

Landed transactions per second

own senders vs chain total, per block-timestamp second

Gas per second

own vs chain gas used, per block-timestamp second

CPU and memory per node

CPU utilization

compute.googleapis.com/instance/cpu/utilization, 60-s mean, % of all vCPUs; summaries over the mixed phase

Table view
NodeClassMeanp95Peak 60-s meanMax
bench-observer-11observer18.4 %20.2 %26.6 %26.6 %
bench-validator-01validator31.7 %34.5 %49.9 %49.9 %
bench-validator-02validator30.3 %33.0 %45.8 %45.8 %
bench-validator-03validator32.0 %35.2 %50.2 %50.2 %
bench-validator-04validator31.8 %35.0 %48.6 %48.6 %
bench-validator-05validator27.3 %35.4 %48.0 %48.0 %
bench-validator-06validator30.2 %32.7 %45.4 %45.4 %
bench-validator-07validator31.3 %33.2 %45.9 %45.9 %
bench-validator-08validator31.4 %34.1 %48.6 %48.6 %
bench-validator-09validator31.2 %33.5 %47.8 %47.8 %
bench-validator-10validator31.2 %33.1 %45.2 %45.2 %

CPU: mean vs peak 60-s mean

One row per node, over the mixed phase.

telcoin resident memory

agent processes/rss_usage summed over telcoin pids; summaries over the mixed phase

Table view
NodeClassMeanp95Peak 60-s meanMax
bench-observer-11observer3.00 GiB3.93 GiB4.11 GiB4.11 GiB
bench-validator-01validator4.80 GiB5.79 GiB5.90 GiB5.90 GiB
bench-validator-02validator5.24 GiB6.14 GiB6.18 GiB6.18 GiB
bench-validator-03validator4.92 GiB5.51 GiB5.55 GiB5.55 GiB
bench-validator-04validator4.64 GiB5.54 GiB5.62 GiB5.62 GiB
bench-validator-05validator4.48 GiB5.38 GiB5.57 GiB5.57 GiB
bench-validator-06validator4.86 GiB5.68 GiB5.69 GiB5.69 GiB
bench-validator-07validator4.83 GiB5.51 GiB5.55 GiB5.55 GiB
bench-validator-08validator5.02 GiB6.12 GiB6.17 GiB6.17 GiB
bench-validator-09validator5.01 GiB6.11 GiB6.14 GiB6.14 GiB
bench-validator-10validator4.90 GiB5.54 GiB5.59 GiB5.59 GiB

telcoin RSS: mean vs peak 60-s mean

One row per node, over the mixed phase.

Memory used

agent memory/percent_used, state=used; summaries over the mixed phase

Table view
NodeClassMeanp95Peak 60-s meanMax
bench-observer-11observer9.5 %10.1 %10.2 %10.2 %
bench-validator-01validator19.6 %19.8 %19.8 %19.8 %
bench-validator-02validator22.4 %22.8 %22.8 %22.8 %
bench-validator-03validator20.9 %21.2 %21.3 %21.3 %
bench-validator-04validator18.7 %19.0 %19.3 %19.3 %
bench-validator-05validator16.5 %20.2 %20.3 %20.3 %
bench-validator-06validator20.1 %20.6 %20.8 %20.8 %
bench-validator-07validator20.1 %20.6 %20.7 %20.7 %
bench-validator-08validator20.6 %21.1 %21.1 %21.1 %
bench-validator-09validator20.5 %20.8 %20.9 %20.9 %
bench-validator-10validator20.5 %20.9 %20.9 %20.9 %

Memory used: mean vs peak 60-s mean

One row per node, over the mixed phase.

Disk

Disk throttling (validator)
not collected –
metric type unavailable in project
Disk performance degraded (validator)
none –
any sample

Disk read + write throughput

instance/disk/{read,write}_bytes_count rate, all devices; summaries over the mixed phase

Table view
NodeClassMeanp95Peak 60-s meanMaxLimit
bench-observer-11observer87.9 MiB/s92.2 MiB/s93.6 MiB/s93.6 MiB/s800.0 MiB/s
bench-validator-01validator98.7 MiB/s103.1 MiB/s107.3 MiB/s107.3 MiB/s800.0 MiB/s
bench-validator-02validator98.5 MiB/s105.0 MiB/s105.3 MiB/s105.3 MiB/s800.0 MiB/s
bench-validator-03validator98.6 MiB/s104.7 MiB/s105.5 MiB/s105.5 MiB/s800.0 MiB/s
bench-validator-04validator98.7 MiB/s104.5 MiB/s105.2 MiB/s105.2 MiB/s800.0 MiB/s
bench-validator-05validator85.1 MiB/s103.8 MiB/s105.3 MiB/s105.3 MiB/s800.0 MiB/s
bench-validator-06validator98.5 MiB/s104.3 MiB/s108.2 MiB/s108.2 MiB/s800.0 MiB/s
bench-validator-07validator98.3 MiB/s104.0 MiB/s107.8 MiB/s107.8 MiB/s800.0 MiB/s
bench-validator-08validator98.5 MiB/s104.0 MiB/s107.4 MiB/s107.4 MiB/s800.0 MiB/s
bench-validator-09validator98.5 MiB/s104.2 MiB/s106.9 MiB/s106.9 MiB/s800.0 MiB/s
bench-validator-10validator98.4 MiB/s104.2 MiB/s106.3 MiB/s106.3 MiB/s800.0 MiB/s

Disk read + write IOPS

instance/disk/{read,write}_ops_count rate, all devices; summaries over the mixed phase

Table view
NodeClassMeanp95Peak 60-s meanMaxLimit
bench-observer-11observer19.7K IOPS20.1K IOPS20.1K IOPS20.1K IOPS20K IOPS
bench-validator-01validator19.9K IOPS20.1K IOPS20.1K IOPS20.1K IOPS20K IOPS
bench-validator-02validator19.9K IOPS20.1K IOPS20.1K IOPS20.1K IOPS20K IOPS
bench-validator-03validator19.9K IOPS20.1K IOPS20.1K IOPS20.1K IOPS20K IOPS
bench-validator-04validator19.9K IOPS20.1K IOPS20.1K IOPS20.1K IOPS20K IOPS
bench-validator-05validator17.1K IOPS20.1K IOPS20.2K IOPS20.2K IOPS20K IOPS
bench-validator-06validator19.9K IOPS20.1K IOPS20.2K IOPS20.2K IOPS20K IOPS
bench-validator-07validator19.9K IOPS20.1K IOPS20.2K IOPS20.2K IOPS20K IOPS
bench-validator-08validator19.9K IOPS20.1K IOPS20.1K IOPS20.1K IOPS20K IOPS
bench-validator-09validator19.9K IOPS20.1K IOPS20.1K IOPS20.1K IOPS20K IOPS
bench-validator-10validator19.9K IOPS20.1K IOPS20.1K IOPS20.1K IOPS20K IOPS
Disk limits used
NodeMachinevCPUDiskIOPS limitMiB/s limitBasis
bench-observer-11c3-highcpu-88hyperdisk-balanced 100 GB20K IOPS800.0 MiB/shyperdisk-balanced 100 GB: provisioned 20,000 IOPS, provisioned 800 MiB/s; c3 8 vCPU cap 50,000 IOPS / 800 MiB/s; reads and writes share this budget
bench-validator-01c3-highcpu-88hyperdisk-balanced 100 GB20K IOPS800.0 MiB/shyperdisk-balanced 100 GB: provisioned 20,000 IOPS, provisioned 800 MiB/s; c3 8 vCPU cap 50,000 IOPS / 800 MiB/s; reads and writes share this budget
bench-validator-02c3-highcpu-88hyperdisk-balanced 100 GB20K IOPS800.0 MiB/shyperdisk-balanced 100 GB: provisioned 20,000 IOPS, provisioned 800 MiB/s; c3 8 vCPU cap 50,000 IOPS / 800 MiB/s; reads and writes share this budget
bench-validator-03c3-highcpu-88hyperdisk-balanced 100 GB20K IOPS800.0 MiB/shyperdisk-balanced 100 GB: provisioned 20,000 IOPS, provisioned 800 MiB/s; c3 8 vCPU cap 50,000 IOPS / 800 MiB/s; reads and writes share this budget
bench-validator-04c3-highcpu-88hyperdisk-balanced 100 GB20K IOPS800.0 MiB/shyperdisk-balanced 100 GB: provisioned 20,000 IOPS, provisioned 800 MiB/s; c3 8 vCPU cap 50,000 IOPS / 800 MiB/s; reads and writes share this budget
bench-validator-05c3-highcpu-88hyperdisk-balanced 100 GB20K IOPS800.0 MiB/shyperdisk-balanced 100 GB: provisioned 20,000 IOPS, provisioned 800 MiB/s; c3 8 vCPU cap 50,000 IOPS / 800 MiB/s; reads and writes share this budget
bench-validator-06c3-highcpu-88hyperdisk-balanced 100 GB20K IOPS800.0 MiB/shyperdisk-balanced 100 GB: provisioned 20,000 IOPS, provisioned 800 MiB/s; c3 8 vCPU cap 50,000 IOPS / 800 MiB/s; reads and writes share this budget
bench-validator-07c3-highcpu-88hyperdisk-balanced 100 GB20K IOPS800.0 MiB/shyperdisk-balanced 100 GB: provisioned 20,000 IOPS, provisioned 800 MiB/s; c3 8 vCPU cap 50,000 IOPS / 800 MiB/s; reads and writes share this budget
bench-validator-08c3-highcpu-88hyperdisk-balanced 100 GB20K IOPS800.0 MiB/shyperdisk-balanced 100 GB: provisioned 20,000 IOPS, provisioned 800 MiB/s; c3 8 vCPU cap 50,000 IOPS / 800 MiB/s; reads and writes share this budget
bench-validator-09c3-highcpu-88hyperdisk-balanced 100 GB20K IOPS800.0 MiB/shyperdisk-balanced 100 GB: provisioned 20,000 IOPS, provisioned 800 MiB/s; c3 8 vCPU cap 50,000 IOPS / 800 MiB/s; reads and writes share this budget
bench-validator-10c3-highcpu-88hyperdisk-balanced 100 GB20K IOPS800.0 MiB/shyperdisk-balanced 100 GB: provisioned 20,000 IOPS, provisioned 800 MiB/s; c3 8 vCPU cap 50,000 IOPS / 800 MiB/s; reads and writes share this budget

Network

Network sent

instance/network/sent_bytes_count rate; summaries over the mixed phase

Table view
NodeClassMeanp95Peak 60-s meanMax
bench-observer-11observer4.0 Mbps7.8 Mbps61.9 Mbps61.9 Mbps
bench-validator-01validator98.0 Mbps107.3 Mbps136.0 Mbps136.0 Mbps
bench-validator-02validator94.7 Mbps104.8 Mbps129.4 Mbps129.4 Mbps
bench-validator-03validator98.8 Mbps109.5 Mbps133.3 Mbps133.3 Mbps
bench-validator-04validator98.1 Mbps108.4 Mbps133.0 Mbps133.0 Mbps
bench-validator-05validator84.1 Mbps107.5 Mbps131.4 Mbps131.4 Mbps
bench-validator-06validator93.7 Mbps102.5 Mbps128.8 Mbps128.8 Mbps
bench-validator-07validator97.3 Mbps104.3 Mbps129.3 Mbps129.3 Mbps
bench-validator-08validator97.9 Mbps105.2 Mbps135.7 Mbps135.7 Mbps
bench-validator-09validator97.8 Mbps104.8 Mbps137.7 Mbps137.7 Mbps
bench-validator-10validator96.7 Mbps103.3 Mbps127.9 Mbps127.9 Mbps

Network received

instance/network/received_bytes_count rate; summaries over the mixed phase

Table view
NodeClassMeanp95Peak 60-s meanMax
bench-observer-11observer6.8 Mbps11.1 Mbps33.4 Mbps33.4 Mbps
bench-validator-01validator10.3 Mbps11.3 Mbps18.0 Mbps18.0 Mbps
bench-validator-02validator8.9 Mbps10.0 Mbps15.6 Mbps15.6 Mbps
bench-validator-03validator10.6 Mbps12.1 Mbps18.2 Mbps18.2 Mbps
bench-validator-04validator10.6 Mbps12.1 Mbps18.0 Mbps18.0 Mbps
bench-validator-05validator14.4 Mbps31.7 Mbps70.9 Mbps70.9 Mbps
bench-validator-06validator8.6 Mbps9.5 Mbps14.8 Mbps14.8 Mbps
bench-validator-07validator10.1 Mbps10.5 Mbps16.2 Mbps16.2 Mbps
bench-validator-08validator10.3 Mbps10.5 Mbps18.5 Mbps18.5 Mbps
bench-validator-09validator10.4 Mbps10.6 Mbps20.1 Mbps20.1 Mbps
bench-validator-10validator10.0 Mbps10.6 Mbps14.8 Mbps14.8 Mbps

Consensus health

MetricQueryClassNodesMedian of node meansWorst peak 60-s meanMax
tn_primary_commit_latency_secondsp95validator9975 ms983 ms986 ms
tn_engine_execution_duration_secondsp95validator91,370 ms2,393 ms2,500 ms
tn_engine_execution_duration_secondsp95observer11,325 ms2,380 ms2,444 ms
tn_worker_batch_size_bytesp95validator9230.6 KiB245.6 KiB245.9 KiB
tn_worker_batch_transactionsp95validator74941.48K1.48K
tn_executor_output_transactionsp95validator109.34K23.7K23.7K
tn_primary_consensus_output_bytesmaxvalidator10358.6 KiB1.5 MiB1.5 MiB
tn_primary_consensus_output_bytesmaxobserver1357.0 KiB1.0 MiB1.4 MiB
tn_engine_queued_outputsmaxvalidator107.8288
tn_engine_queued_outputsmaxobserver17.688
tn_primary_roundmaxvalidator101.66K3.35K3.39K
tn_primary_roundmaxobserver11.64K3.35K3.39K
tn_primary_committed_roundmaxvalidator101.66K3.35K3.39K
tn_primary_committed_roundmaxobserver11.64K3.35K3.39K
tn_epoch_currentmaxvalidator104.2855
tn_epoch_currentmaxobserver14.2755
tn_batch_builder_pending_pool_transactionsmaxvalidator103564.59K9.85K
tn_batch_builder_pending_pool_transactionsmaxobserver1000
reth_process_resident_memory_bytesmaxvalidator104.88 GiB6.22 GiB6.26 GiB
reth_process_resident_memory_bytesmaxobserver13.03 GiB4.06 GiB4.12 GiB
tn_primary_subdags_committed_totalratevalidator101.331.421.42
tn_primary_subdags_committed_totalrateobserver1000
tn_primary_round_laglagvalidator102.163.55
tn_primary_round_laglagobserver1000
tn_batch_builder_seal_failures_totalratevalidator800.01670.0222
tn_node_modemaxvalidator10011
tn_node_modemaxobserver1000
tn_primary_vote_requests_totalratevalidator1023.133.133.2
tn_primary_vote_requests_totalrateobserver1000
tn_primary_votes_created_totalratevalidator102325.225.3
tn_primary_votes_created_totalrateobserver1000
tn_primary_votes_recast_totalratevalidator10000
tn_primary_votes_recast_totalrateobserver1000
tn_primary_vote_requests_too_new_totalratevalidator10000
tn_primary_vote_requests_too_new_totalrateobserver1000
tn_primary_vote_timeouts_total{stage="exec_wait"}ratevalidator10000
tn_primary_vote_timeouts_total{stage="exec_wait"}rateobserver1000
tn_primary_vote_timeouts_total{stage="lead_sleep"}ratevalidator10000
tn_primary_vote_timeouts_total{stage="lead_sleep"}rateobserver1000
tn_primary_vote_timeouts_total{stage="parents"}ratevalidator10000
tn_primary_vote_timeouts_total{stage="parents"}rateobserver1000
tn_primary_vote_timeouts_total{stage="persist"}ratevalidator10000
tn_primary_vote_timeouts_total{stage="persist"}rateobserver1000
tn_primary_vote_timeouts_total{stage="prelock_wait"}ratevalidator10000
tn_primary_vote_timeouts_total{stage="prelock_wait"}rateobserver1000
tn_primary_vote_timeouts_total{stage="sync_batches"}ratevalidator10000
tn_primary_vote_timeouts_total{stage="sync_batches"}rateobserver1000
tn_primary_vote_timeouts_total{stage="validate"}ratevalidator10000
tn_primary_vote_timeouts_total{stage="validate"}rateobserver1000
tn_primary_vote_timeouts_total{stage="vote_store"}ratevalidator10000
tn_primary_vote_timeouts_total{stage="vote_store"}rateobserver1000
tn_primary_vote_stage_seconds{stage="exec_wait"}p95validator80 ms0 ms0 ms
tn_primary_vote_stage_seconds{stage="lead_sleep"}p95validator80 ms0 ms0 ms
tn_primary_vote_stage_seconds{stage="lock"}p95validator80 ms0 ms0 ms
tn_primary_vote_stage_seconds{stage="parents"}p95validator81 ms2 ms2 ms
tn_primary_vote_stage_seconds{stage="persist"}p95validator836 ms43 ms44 ms
tn_primary_vote_stage_seconds{stage="prelock_wait"}p95validator883 ms1,871 ms1,888 ms
tn_primary_vote_stage_seconds{stage="sync_batches"}p95validator80 ms0 ms0 ms
tn_primary_vote_stage_seconds{stage="validate"}p95validator83 ms4 ms4 ms
tn_primary_vote_stage_seconds{stage="vote_store"}p95validator81 ms3 ms5 ms
tn_primary_leader_rounds_total{leader="27D3ei95A1CBzPTdUJ6FGK4vFJ4uFiT1e1GiWgYwmw6C",swapped="false",found="false"}ratevalidator10000
tn_primary_leader_rounds_total{leader="27D3ei95A1CBzPTdUJ6FGK4vFJ4uFiT1e1GiWgYwmw6C",swapped="false",found="true"}ratevalidator100.1340.1560.156
tn_primary_leader_rounds_total{leader="27D3ei95A1CBzPTdUJ6FGK4vFJ4uFiT1e1GiWgYwmw6C",swapped="true",found="false"}ratevalidator10000
tn_primary_leader_rounds_total{leader="27D3ei95A1CBzPTdUJ6FGK4vFJ4uFiT1e1GiWgYwmw6C",swapped="true",found="true"}ratevalidator100.03770.1170.156
tn_primary_leader_rounds_total{leader="9D2UZkhAZiGL5H1U2B2e3wVJ9M6MuoetaNbjmKDmADQj",swapped="false",found="false"}ratevalidator100.0005520.02220.0222
tn_primary_leader_rounds_total{leader="9D2UZkhAZiGL5H1U2B2e3wVJ9M6MuoetaNbjmKDmADQj",swapped="false",found="true"}ratevalidator100.1330.150.156
tn_primary_leader_rounds_total{leader="9D2UZkhAZiGL5H1U2B2e3wVJ9M6MuoetaNbjmKDmADQj",swapped="true",found="false"}ratevalidator10000
tn_primary_leader_rounds_total{leader="9D2UZkhAZiGL5H1U2B2e3wVJ9M6MuoetaNbjmKDmADQj",swapped="true",found="true"}ratevalidator100.04330.1170.156
tn_primary_leader_rounds_total{leader="APagdtGMZbPoLsZ19JL8rdEViw34MudsodHkKzSH95Gb",swapped="false",found="false"}ratevalidator10000
tn_primary_leader_rounds_total{leader="APagdtGMZbPoLsZ19JL8rdEViw34MudsodHkKzSH95Gb",swapped="false",found="true"}ratevalidator100.1340.1560.156
tn_primary_leader_rounds_total{leader="APagdtGMZbPoLsZ19JL8rdEViw34MudsodHkKzSH95Gb",swapped="true",found="false"}ratevalidator10000
tn_primary_leader_rounds_total{leader="APagdtGMZbPoLsZ19JL8rdEViw34MudsodHkKzSH95Gb",swapped="true",found="true"}ratevalidator100.04620.1110.156
tn_primary_leader_rounds_total{leader="AhfMY1MVf2pK2MTyfQz5rxssor6sDQVXMBmnmkhszqnF",swapped="false",found="false"}ratevalidator100.003730.1060.133
tn_primary_leader_rounds_total{leader="AhfMY1MVf2pK2MTyfQz5rxssor6sDQVXMBmnmkhszqnF",swapped="false",found="true"}ratevalidator100.08160.1560.156
tn_primary_leader_rounds_total{leader="AhfMY1MVf2pK2MTyfQz5rxssor6sDQVXMBmnmkhszqnF",swapped="true",found="false"}ratevalidator10000
tn_primary_leader_rounds_total{leader="AhfMY1MVf2pK2MTyfQz5rxssor6sDQVXMBmnmkhszqnF",swapped="true",found="true"}ratevalidator100.004140.06670.0889
tn_primary_leader_rounds_total{leader="E5jhK1sYMQVhD3W5kvyGx8ih3zZ8z1KtexypkdgXrdMW",swapped="false",found="false"}ratevalidator10000
tn_primary_leader_rounds_total{leader="E5jhK1sYMQVhD3W5kvyGx8ih3zZ8z1KtexypkdgXrdMW",swapped="false",found="true"}ratevalidator100.1340.1560.156
tn_primary_leader_rounds_total{leader="E5jhK1sYMQVhD3W5kvyGx8ih3zZ8z1KtexypkdgXrdMW",swapped="true",found="false"}ratevalidator10000
tn_primary_leader_rounds_total{leader="E5jhK1sYMQVhD3W5kvyGx8ih3zZ8z1KtexypkdgXrdMW",swapped="true",found="true"}ratevalidator100.04210.10.133
tn_primary_leader_rounds_total{leader="Ev7A35YTdxtWkdgRVs7zLUxAdeVaDHbjb6bf7qp8QJBT",swapped="false",found="false"}ratevalidator10000
tn_primary_leader_rounds_total{leader="Ev7A35YTdxtWkdgRVs7zLUxAdeVaDHbjb6bf7qp8QJBT",swapped="false",found="true"}ratevalidator100.1340.1560.156
tn_primary_leader_rounds_total{leader="Ev7A35YTdxtWkdgRVs7zLUxAdeVaDHbjb6bf7qp8QJBT",swapped="true",found="false"}ratevalidator10000
tn_primary_leader_rounds_total{leader="Ev7A35YTdxtWkdgRVs7zLUxAdeVaDHbjb6bf7qp8QJBT",swapped="true",found="true"}ratevalidator100.04380.1110.133
tn_primary_leader_rounds_total{leader="FAfheD4QBfkVvGtVYumXTkK8Rd98dQkU59veFXZpANH6",swapped="false",found="false"}ratevalidator10000
tn_primary_leader_rounds_total{leader="FAfheD4QBfkVvGtVYumXTkK8Rd98dQkU59veFXZpANH6",swapped="false",found="true"}ratevalidator100.1340.1560.156
tn_primary_leader_rounds_total{leader="FAfheD4QBfkVvGtVYumXTkK8Rd98dQkU59veFXZpANH6",swapped="true",found="false"}ratevalidator10000
tn_primary_leader_rounds_total{leader="FAfheD4QBfkVvGtVYumXTkK8Rd98dQkU59veFXZpANH6",swapped="true",found="true"}ratevalidator100.0480.1280.178
tn_primary_leader_rounds_total{leader="G8c2xYD3JucykAs8f6jZ2VRF3YenbTF9qyYuXaamKaYN",swapped="false",found="false"}ratevalidator10000
tn_primary_leader_rounds_total{leader="G8c2xYD3JucykAs8f6jZ2VRF3YenbTF9qyYuXaamKaYN",swapped="false",found="true"}ratevalidator100.001660.03330.0444
tn_primary_leader_rounds_total{leader="G8c2xYD3JucykAs8f6jZ2VRF3YenbTF9qyYuXaamKaYN",swapped="true",found="false"}ratevalidator10000
tn_primary_leader_rounds_total{leader="G8c2xYD3JucykAs8f6jZ2VRF3YenbTF9qyYuXaamKaYN",swapped="true",found="true"}ratevalidator10000
tn_primary_leader_rounds_total{leader="H8WP22xMV66i5qYKQd8EHh5pvjWC7XwvnjGtknDP4PVL",swapped="false",found="false"}ratevalidator100.0003560.01630.0217
tn_primary_leader_rounds_total{leader="H8WP22xMV66i5qYKQd8EHh5pvjWC7XwvnjGtknDP4PVL",swapped="false",found="true"}ratevalidator100.1330.1560.156
tn_primary_leader_rounds_total{leader="H8WP22xMV66i5qYKQd8EHh5pvjWC7XwvnjGtknDP4PVL",swapped="true",found="false"}ratevalidator10000
tn_primary_leader_rounds_total{leader="H8WP22xMV66i5qYKQd8EHh5pvjWC7XwvnjGtknDP4PVL",swapped="true",found="true"}ratevalidator100.04330.1060.133
tn_primary_leader_rounds_total{leader="HiHSAySa8N26j3xdyhXZ8TgfU6LJoTr4TnBoQHA8jViw",swapped="false",found="false"}ratevalidator10000
tn_primary_leader_rounds_total{leader="HiHSAySa8N26j3xdyhXZ8TgfU6LJoTr4TnBoQHA8jViw",swapped="false",found="true"}ratevalidator100.005380.1060.156
tn_primary_leader_rounds_total{leader="HiHSAySa8N26j3xdyhXZ8TgfU6LJoTr4TnBoQHA8jViw",swapped="true",found="false"}ratevalidator10000
tn_primary_leader_rounds_total{leader="HiHSAySa8N26j3xdyhXZ8TgfU6LJoTr4TnBoQHA8jViw",swapped="true",found="true"}ratevalidator10000
tn_primary_leader_commits_total{leader="27D3ei95A1CBzPTdUJ6FGK4vFJ4uFiT1e1GiWgYwmw6C"}ratevalidator100.1720.2610.289
tn_primary_leader_commits_total{leader="9D2UZkhAZiGL5H1U2B2e3wVJ9M6MuoetaNbjmKDmADQj"}ratevalidator100.1770.250.289
tn_primary_leader_commits_total{leader="APagdtGMZbPoLsZ19JL8rdEViw34MudsodHkKzSH95Gb"}ratevalidator100.180.2440.289
tn_primary_leader_commits_total{leader="AhfMY1MVf2pK2MTyfQz5rxssor6sDQVXMBmnmkhszqnF"}ratevalidator100.08570.20.222
tn_primary_leader_commits_total{leader="E5jhK1sYMQVhD3W5kvyGx8ih3zZ8z1KtexypkdgXrdMW"}ratevalidator100.1760.2390.289
tn_primary_leader_commits_total{leader="Ev7A35YTdxtWkdgRVs7zLUxAdeVaDHbjb6bf7qp8QJBT"}ratevalidator100.1770.2610.289
tn_primary_leader_commits_total{leader="FAfheD4QBfkVvGtVYumXTkK8Rd98dQkU59veFXZpANH6"}ratevalidator100.1820.2610.311
tn_primary_leader_commits_total{leader="G8c2xYD3JucykAs8f6jZ2VRF3YenbTF9qyYuXaamKaYN"}ratevalidator100.001660.03330.0444
tn_primary_leader_commits_total{leader="H8WP22xMV66i5qYKQd8EHh5pvjWC7XwvnjGtknDP4PVL"}ratevalidator100.1760.2390.289
tn_primary_leader_commits_total{leader="HiHSAySa8N26j3xdyhXZ8TgfU6LJoTr4TnBoQHA8jViw"}ratevalidator100.005380.1060.156
tn_engine_queue_wait_secondsp95validator89,667 ms12.5 s15.7 s
tn_engine_queue_wait_secondsp95observer19,505 ms9,750 ms9,750 ms
tn_engine_persist_duration_secondsp95validator10620 ms945 ms954 ms
tn_engine_persist_duration_secondsp95observer1679 ms946 ms957 ms
tn_engine_busy_milliseconds_totalratevalidator109961.01K1.03K
tn_engine_busy_milliseconds_totalrateobserver19831.01K1.02K
tn_reth_persist_save_blocks_secondsp95validator8200 ms308 ms342 ms
tn_reth_persist_save_blocks_secondsp95observer1170 ms241 ms245 ms
tn_reth_persist_commit_secondsp95validator7476 ms501 ms525 ms
tn_reth_persist_commit_secondsp95observer1489 ms648 ms806 ms
tn_reth_persisted_accounts_totalratevalidator102.28K7.52K7.52K
tn_reth_persisted_accounts_totalrateobserver12.27K7.46K7.46K
tn_reth_persisted_storage_slots_totalratevalidator102.97K3.98K4.16K
tn_reth_persisted_storage_slots_totalrateobserver12.95K3.98K4.16K
tn_executor_rejoin_blocked_total{reason="attempts_exhausted"}ratevalidator1000
tn_executor_rejoin_blocked_total{reason="epoch_closed"}ratevalidator10.09710.9281.24
tn_executor_rejoin_blocked_total{reason="epoch_mismatch"}ratevalidator1000
tn_executor_rejoin_blocked_total{reason="no_known_header"}ratevalidator1000
tn_executor_rejoin_blocked_total{reason="round_gap"}ratevalidator11.151.381.38
tn_executor_rejoin_blocked_total{reason="stale_head"}ratevalidator10.004010.04510.0602
tn_worker_sync_serves_denied_total{worker="0",tier="committee"}ratevalidator10000
tn_worker_sync_serves_denied_total{worker="0",tier="committee"}rateobserver1000
tn_worker_sync_serves_denied_total{worker="0",tier="public"}ratevalidator10000
tn_worker_sync_serves_denied_total{worker="0",tier="public"}rateobserver1000
tn_worker_batch_requests_denied_total{worker="0",peer="21zYhHLWXWT2CY9W7bZ6X9V3uj38gUu1XajTpHasXCre5Z16EreyvoyD64wRUcNETnWaKUM13QSSz5R8EAt5tCbmQE88sRuPXRpLS7ZPU9BcQFTDxTC7tRFj6YE6oiFskgGc"}ratevalidator40.001080.0620.062
tn_worker_batch_requests_denied_total{worker="0",peer="22qKoFFaQjfVtdPKTwt5weZibVA7gB2YkN4VLJLUnnKsAuz1JAtaRDVYjKVrc1VpUgFTjuX5sdvPzr2Aat32TZmua4grMWLXQE6Be3aDSD8bUzH4YVFtQLLSzwWUBWLCydwr"}ratevalidator20.002080.050.0667
tn_worker_batch_requests_denied_total{worker="0",peer="2337m4n6HoB5tcF23aR6m9VUHivHCyR8AuukU3c8ARff6hpc48xrMyj3JYA3ou2AkW6zv3Yw3W3YLEfGtPkJ2phnGgVkZXLgaW9fRvRwzJGfTwo6epWmXjqDhcwDK26uXzv5"}ratevalidator20.0006940.01670.0222
tn_worker_batch_requests_denied_total{worker="0",peer="23GgA1MoGNnQaNMCPqyUg9K1i9ZXtHY5RLEP1HriPvtQsTyoTdvkYFxHFTaUFWozNDsHQ6F4DSSGSLyCJHqiU9EumYMCvz75ZSNz6rfrKPvdGnJdHqsJUKMn11Q31RaB1Cf1"}ratevalidator20.0006940.01670.0222
tn_worker_batch_requests_denied_total{worker="0",peer="24CvwiTuMvwQ6ZGnYqa1WqEbnieeig65Mm3eR2Nn29fVX93yYqmbwTcHQPscSeNv9UL4nBQTvE9pdcr8vnbLfx7nwVMxLVR8e7QBjkzRuovArqKo8edo5DVPYxoChFXa5wZZ"}ratevalidator400.03330.0444
tn_worker_batch_requests_denied_total{worker="0",peer="26t6sfMhKV4vPh5GGSjsENTYwE7MfoQ2jT9yBgHnCMQ2FXgv6n5QWT1jQN8A8FS3nAhmVtwR4nbxHCTSg8yWdgkV9Tyy6vojpMW3H6tPD6LE75cM8KhvAkhMmdzFUjxyAApv"}ratevalidator20.0004940.01670.0222
tn_worker_batch_requests_denied_total{worker="0",peer="tbB9ra6HZHyxHqFeLU9ztG7SFfoF1hdtBKgPhZUwRJstH2DGuNX5CoN5pmSYnAySsumMujLZauaCQWKH1fyHHtH51tzxBGHbkWx4kJQu8bfmb9rLjvhvYkBDzRFu3MdjK1w"}ratevalidator300.03330.0444
tn_worker_batch_requests_denied_total{worker="0",peer="urDs8966E38yGW6zUrqC69XwUAcoCHCE1XkfXJsCEa3r5dJS1sXN4Sw69djMQhqPjRjmgs3puXVR8pQ7UUjuo7fyRTjEmwi6MRwHHGci6Xg5j4RVHQX9aHWDw3pmPp8p5Ky"}ratevalidator20.001390.03330.0444
tn_worker_batch_requests_denied_total{worker="0",peer="yquuYRbvCft4NiVr8iqER9ZGUAZ7vdQFTdrkSkcebAiSBZGaq7eF53BwfEa92g9u9XfK6WjpHAZ9EhHCge58cNH6bpY4s7ZUd7dp3sJdFEzYDgtvJqt7yvP5RNMbLrRFjgj"}ratevalidator20.0005830.01620.0216
tn_worker_batch_requests_denied_total{worker="0",peer="zEdqE6sGab3yUPEvrkJFrAhHqRAjuaKfqmcvY1E5k1mcVdVuNVf3hrWza1UZvKzDqzqc1jfhBKwih9rAKVHwQnQSTTbbYhhnQUN3DZMP3hs7Z34iHdt5nTrcQpsoe8WLiGM"}ratevalidator30.0004290.03330.0444
tn_worker_batch_requests_failed_total{worker="0",peer="21zYhHLWXWT2CY9W7bZ6X9V3uj38gUu1XajTpHasXCre5Z16EreyvoyD64wRUcNETnWaKUM13QSSz5R8EAt5tCbmQE88sRuPXRpLS7ZPU9BcQFTDxTC7tRFj6YE6oiFskgGc"}ratevalidator400.02220.0222
tn_worker_batch_requests_failed_total{worker="0",peer="22qKoFFaQjfVtdPKTwt5weZibVA7gB2YkN4VLJLUnnKsAuz1JAtaRDVYjKVrc1VpUgFTjuX5sdvPzr2Aat32TZmua4grMWLXQE6Be3aDSD8bUzH4YVFtQLLSzwWUBWLCydwr"}ratevalidator2000
tn_worker_batch_requests_failed_total{worker="0",peer="2337m4n6HoB5tcF23aR6m9VUHivHCyR8AuukU3c8ARff6hpc48xrMyj3JYA3ou2AkW6zv3Yw3W3YLEfGtPkJ2phnGgVkZXLgaW9fRvRwzJGfTwo6epWmXjqDhcwDK26uXzv5"}ratevalidator2000
tn_worker_batch_requests_failed_total{worker="0",peer="23GgA1MoGNnQaNMCPqyUg9K1i9ZXtHY5RLEP1HriPvtQsTyoTdvkYFxHFTaUFWozNDsHQ6F4DSSGSLyCJHqiU9EumYMCvz75ZSNz6rfrKPvdGnJdHqsJUKMn11Q31RaB1Cf1"}ratevalidator2000
tn_worker_batch_requests_failed_total{worker="0",peer="24CvwiTuMvwQ6ZGnYqa1WqEbnieeig65Mm3eR2Nn29fVX93yYqmbwTcHQPscSeNv9UL4nBQTvE9pdcr8vnbLfx7nwVMxLVR8e7QBjkzRuovArqKo8edo5DVPYxoChFXa5wZZ"}ratevalidator4000
tn_worker_batch_requests_failed_total{worker="0",peer="26t6sfMhKV4vPh5GGSjsENTYwE7MfoQ2jT9yBgHnCMQ2FXgv6n5QWT1jQN8A8FS3nAhmVtwR4nbxHCTSg8yWdgkV9Tyy6vojpMW3H6tPD6LE75cM8KhvAkhMmdzFUjxyAApv"}ratevalidator2000
tn_worker_batch_requests_failed_total{worker="0",peer="tbB9ra6HZHyxHqFeLU9ztG7SFfoF1hdtBKgPhZUwRJstH2DGuNX5CoN5pmSYnAySsumMujLZauaCQWKH1fyHHtH51tzxBGHbkWx4kJQu8bfmb9rLjvhvYkBDzRFu3MdjK1w"}ratevalidator3000
tn_worker_batch_requests_failed_total{worker="0",peer="urDs8966E38yGW6zUrqC69XwUAcoCHCE1XkfXJsCEa3r5dJS1sXN4Sw69djMQhqPjRjmgs3puXVR8pQ7UUjuo7fyRTjEmwi6MRwHHGci6Xg5j4RVHQX9aHWDw3pmPp8p5Ky"}ratevalidator20.0006940.01670.0222
tn_worker_batch_requests_failed_total{worker="0",peer="yquuYRbvCft4NiVr8iqER9ZGUAZ7vdQFTdrkSkcebAiSBZGaq7eF53BwfEa92g9u9XfK6WjpHAZ9EhHCge58cNH6bpY4s7ZUd7dp3sJdFEzYDgtvJqt7yvP5RNMbLrRFjgj"}ratevalidator20.00240.06110.0889
tn_worker_batch_requests_failed_total{worker="0",peer="zEdqE6sGab3yUPEvrkJFrAhHqRAjuaKfqmcvY1E5k1mcVdVuNVf3hrWza1UZvKzDqzqc1jfhBKwih9rAKVHwQnQSTTbbYhhnQUN3DZMP3hs7Z34iHdt5nTrcQpsoe8WLiGM"}ratevalidator3000
tn_node_epoch_record_signersmaxvalidator108.011010
tn_node_epoch_record_signersmaxobserver17.8199
tn_node_epoch_record_missing_signers_total{signer="27D3ei95A1CBzPTdUJ6FGK4vFJ4uFiT1e1GiWgYwmw6C"}ratevalidator10000
tn_node_epoch_record_missing_signers_total{signer="27D3ei95A1CBzPTdUJ6FGK4vFJ4uFiT1e1GiWgYwmw6C"}rateobserver1000
tn_node_epoch_record_missing_signers_total{signer="9D2UZkhAZiGL5H1U2B2e3wVJ9M6MuoetaNbjmKDmADQj"}ratevalidator1000.01630.0217
tn_node_epoch_record_missing_signers_total{signer="9D2UZkhAZiGL5H1U2B2e3wVJ9M6MuoetaNbjmKDmADQj"}rateobserver1000
tn_node_epoch_record_missing_signers_total{signer="APagdtGMZbPoLsZ19JL8rdEViw34MudsodHkKzSH95Gb"}ratevalidator10000
tn_node_epoch_record_missing_signers_total{signer="APagdtGMZbPoLsZ19JL8rdEViw34MudsodHkKzSH95Gb"}rateobserver1000
tn_node_epoch_record_missing_signers_total{signer="AhfMY1MVf2pK2MTyfQz5rxssor6sDQVXMBmnmkhszqnF"}ratevalidator100.0004140.01670.0222
tn_node_epoch_record_missing_signers_total{signer="AhfMY1MVf2pK2MTyfQz5rxssor6sDQVXMBmnmkhszqnF"}rateobserver10.000980.01670.0222
tn_node_epoch_record_missing_signers_total{signer="E5jhK1sYMQVhD3W5kvyGx8ih3zZ8z1KtexypkdgXrdMW"}ratevalidator1000.01630.0217
tn_node_epoch_record_missing_signers_total{signer="E5jhK1sYMQVhD3W5kvyGx8ih3zZ8z1KtexypkdgXrdMW"}rateobserver10.0004510.01530.0204
tn_node_epoch_record_missing_signers_total{signer="Ev7A35YTdxtWkdgRVs7zLUxAdeVaDHbjb6bf7qp8QJBT"}ratevalidator10000
tn_node_epoch_record_missing_signers_total{signer="Ev7A35YTdxtWkdgRVs7zLUxAdeVaDHbjb6bf7qp8QJBT"}rateobserver1000
tn_node_epoch_record_missing_signers_total{signer="FAfheD4QBfkVvGtVYumXTkK8Rd98dQkU59veFXZpANH6"}ratevalidator1000.01630.0217
tn_node_epoch_record_missing_signers_total{signer="FAfheD4QBfkVvGtVYumXTkK8Rd98dQkU59veFXZpANH6"}rateobserver10.0004510.01530.0204
tn_node_epoch_record_missing_signers_total{signer="G8c2xYD3JucykAs8f6jZ2VRF3YenbTF9qyYuXaamKaYN"}ratevalidator10000
tn_node_epoch_record_missing_signers_total{signer="G8c2xYD3JucykAs8f6jZ2VRF3YenbTF9qyYuXaamKaYN"}rateobserver1000
tn_node_epoch_record_missing_signers_total{signer="H8WP22xMV66i5qYKQd8EHh5pvjWC7XwvnjGtknDP4PVL"}ratevalidator10000
tn_node_epoch_record_missing_signers_total{signer="H8WP22xMV66i5qYKQd8EHh5pvjWC7XwvnjGtknDP4PVL"}rateobserver1000
tn_node_epoch_record_missing_signers_total{signer="HiHSAySa8N26j3xdyhXZ8TgfU6LJoTr4TnBoQHA8jViw"}ratevalidator10000
tn_node_epoch_record_missing_signers_total{signer="HiHSAySa8N26j3xdyhXZ8TgfU6LJoTr4TnBoQHA8jViw"}rateobserver1000
tn_network_outbound_request_duration_seconds{outcome="failure",class="batch"}p95validator716.4 s29.0 s29.0 s
tn_network_outbound_request_duration_seconds{outcome="failure",class="epoch_record"}p95validator214.7 s29.0 s29.0 s
tn_network_outbound_request_duration_seconds{outcome="failure",class="vote"}p95validator412.2 s29.0 s29.0 s
tn_network_outbound_request_duration_seconds{outcome="response",class="batch"}p95validator1409 ms475 ms476 ms
tn_network_outbound_request_duration_seconds{outcome="response",class="epoch_record"}p95validator6437 ms450 ms457 ms
tn_network_outbound_request_duration_seconds{outcome="response",class="epoch_record"}p95observer1240 ms245 ms247 ms
tn_network_outbound_request_duration_seconds{outcome="response",class="vote"}p95validator3475 ms1,090 ms1,168 ms
tn_network_closed_admission_denied_total{network="bench",direction="in"}ratevalidator10000
tn_network_closed_admission_denied_total{network="bench",direction="in"}rateobserver1000
tn_network_closed_admission_denied_total{network="bench",direction="out"}ratevalidator10000
tn_network_closed_admission_denied_total{network="bench",direction="out"}rateobserver1000
tn_network_source_admission_denied_total{network="bench",direction="in",reason="address_full"}ratevalidator10000
tn_network_source_admission_denied_total{network="bench",direction="in",reason="address_full"}rateobserver1000
tn_network_source_admission_denied_total{network="bench",direction="in",reason="duplicate_connection"}ratevalidator10000
tn_network_source_admission_denied_total{network="bench",direction="in",reason="duplicate_connection"}rateobserver1000
tn_network_source_admission_denied_total{network="bench",direction="in",reason="invalid_limits"}ratevalidator10000
tn_network_source_admission_denied_total{network="bench",direction="in",reason="invalid_limits"}rateobserver1000
tn_network_source_admission_denied_total{network="bench",direction="in",reason="peer_full"}ratevalidator10000
tn_network_source_admission_denied_total{network="bench",direction="in",reason="peer_full"}rateobserver1000
tn_network_source_admission_denied_total{network="bench",direction="in",reason="poisoned"}ratevalidator10000
tn_network_source_admission_denied_total{network="bench",direction="in",reason="poisoned"}rateobserver1000
tn_network_source_admission_denied_total{network="bench",direction="in",reason="prefix_full"}ratevalidator10000
tn_network_source_admission_denied_total{network="bench",direction="in",reason="prefix_full"}rateobserver1000
tn_network_source_admission_denied_total{network="bench",direction="in",reason="process_full"}ratevalidator10000
tn_network_source_admission_denied_total{network="bench",direction="in",reason="process_full"}rateobserver1000
tn_network_source_admission_denied_total{network="bench",direction="in",reason="sources_full"}ratevalidator10000
tn_network_source_admission_denied_total{network="bench",direction="in",reason="sources_full"}rateobserver1000
tn_network_source_admission_denied_total{network="bench",direction="in",reason="unsupported_address"}ratevalidator10000
tn_network_source_admission_denied_total{network="bench",direction="in",reason="unsupported_address"}rateobserver1000
tn_network_source_admission_denied_total{network="bench",direction="out",reason="address_full"}ratevalidator10000
tn_network_source_admission_denied_total{network="bench",direction="out",reason="address_full"}rateobserver1000
tn_network_source_admission_denied_total{network="bench",direction="out",reason="duplicate_connection"}ratevalidator10000
tn_network_source_admission_denied_total{network="bench",direction="out",reason="duplicate_connection"}rateobserver1000
tn_network_source_admission_denied_total{network="bench",direction="out",reason="invalid_limits"}ratevalidator10000
tn_network_source_admission_denied_total{network="bench",direction="out",reason="invalid_limits"}rateobserver1000
tn_network_source_admission_denied_total{network="bench",direction="out",reason="peer_full"}ratevalidator10000
tn_network_source_admission_denied_total{network="bench",direction="out",reason="peer_full"}rateobserver1000
tn_network_source_admission_denied_total{network="bench",direction="out",reason="poisoned"}ratevalidator10000
tn_network_source_admission_denied_total{network="bench",direction="out",reason="poisoned"}rateobserver1000
tn_network_source_admission_denied_total{network="bench",direction="out",reason="prefix_full"}ratevalidator10000
tn_network_source_admission_denied_total{network="bench",direction="out",reason="prefix_full"}rateobserver1000
tn_network_source_admission_denied_total{network="bench",direction="out",reason="process_full"}ratevalidator10000
tn_network_source_admission_denied_total{network="bench",direction="out",reason="process_full"}rateobserver1000
tn_network_source_admission_denied_total{network="bench",direction="out",reason="sources_full"}ratevalidator10000
tn_network_source_admission_denied_total{network="bench",direction="out",reason="sources_full"}rateobserver1000
tn_network_source_admission_denied_total{network="bench",direction="out",reason="unsupported_address"}ratevalidator10000
tn_network_source_admission_denied_total{network="bench",direction="out",reason="unsupported_address"}rateobserver1000
tn_network_admission_modemaxvalidator10222
tn_network_admission_modemaxobserver11.9922
tn_network_admission_fallbackmaxvalidator10011
tn_network_admission_fallbackmaxobserver10.006210.251
tn_network_outbound_request_failures_total{kind="connection"}ratevalidator1000.02830.0399
tn_network_outbound_request_failures_total{kind="connection"}rateobserver1000
tn_network_outbound_request_failures_total{kind="dial"}ratevalidator1012.780.598.5
tn_network_outbound_request_failures_total{kind="dial"}rateobserver1000
tn_network_outbound_request_failures_total{kind="io"}ratevalidator102.5312.414.5
tn_network_outbound_request_failures_total{kind="io"}rateobserver10.4791.171.55
tn_network_outbound_request_failures_total{kind="timeout"}ratevalidator100.5322.93.2
tn_network_outbound_request_failures_total{kind="timeout"}rateobserver10.004310.02690.0409
tn_network_outbound_request_failures_total{kind="unsupported"}ratevalidator10000
tn_network_outbound_request_failures_total{kind="unsupported"}rateobserver1000
tn_network_inbound_requests_shed_total{class="batch",reason="admission"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="batch",reason="admission"}rateobserver1000
tn_network_inbound_requests_shed_total{class="batch",reason="queue_budget"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="batch",reason="queue_budget"}rateobserver1000
tn_network_inbound_requests_shed_total{class="batch",reason="queue_full"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="batch",reason="queue_full"}rateobserver1000
tn_network_inbound_requests_shed_total{class="batch",reason="serve_budget"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="batch",reason="serve_budget"}rateobserver1000
tn_network_inbound_requests_shed_total{class="batch",reason="unsubscribed"}ratevalidator100.020656.286.1
tn_network_inbound_requests_shed_total{class="batch",reason="unsubscribed"}rateobserver1000
tn_network_inbound_requests_shed_total{class="certificate_sync",reason="admission"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="certificate_sync",reason="admission"}rateobserver1000
tn_network_inbound_requests_shed_total{class="certificate_sync",reason="queue_budget"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="certificate_sync",reason="queue_budget"}rateobserver1000
tn_network_inbound_requests_shed_total{class="certificate_sync",reason="queue_full"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="certificate_sync",reason="queue_full"}rateobserver1000
tn_network_inbound_requests_shed_total{class="certificate_sync",reason="serve_budget"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="certificate_sync",reason="serve_budget"}rateobserver1000
tn_network_inbound_requests_shed_total{class="certificate_sync",reason="unsubscribed"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="certificate_sync",reason="unsubscribed"}rateobserver1000
tn_network_inbound_requests_shed_total{class="epoch_record",reason="admission"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="epoch_record",reason="admission"}rateobserver1000
tn_network_inbound_requests_shed_total{class="epoch_record",reason="queue_budget"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="epoch_record",reason="queue_budget"}rateobserver1000
tn_network_inbound_requests_shed_total{class="epoch_record",reason="queue_full"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="epoch_record",reason="queue_full"}rateobserver1000
tn_network_inbound_requests_shed_total{class="epoch_record",reason="serve_budget"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="epoch_record",reason="serve_budget"}rateobserver1000
tn_network_inbound_requests_shed_total{class="epoch_record",reason="unsubscribed"}ratevalidator100.3861.011.34
tn_network_inbound_requests_shed_total{class="epoch_record",reason="unsubscribed"}rateobserver10.2380.430.654
tn_network_inbound_requests_shed_total{class="gossip",reason="admission"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="gossip",reason="admission"}rateobserver1000
tn_network_inbound_requests_shed_total{class="gossip",reason="queue_budget"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="gossip",reason="queue_budget"}rateobserver1000
tn_network_inbound_requests_shed_total{class="gossip",reason="queue_full"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="gossip",reason="queue_full"}rateobserver1000
tn_network_inbound_requests_shed_total{class="gossip",reason="serve_budget"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="gossip",reason="serve_budget"}rateobserver1000
tn_network_inbound_requests_shed_total{class="gossip",reason="unsubscribed"}ratevalidator1000.08860.11
tn_network_inbound_requests_shed_total{class="gossip",reason="unsubscribed"}rateobserver10.0450.1080.164
tn_network_inbound_requests_shed_total{class="other",reason="admission"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="other",reason="admission"}rateobserver1000
tn_network_inbound_requests_shed_total{class="other",reason="queue_budget"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="other",reason="queue_budget"}rateobserver1000
tn_network_inbound_requests_shed_total{class="other",reason="queue_full"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="other",reason="queue_full"}rateobserver1000
tn_network_inbound_requests_shed_total{class="other",reason="serve_budget"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="other",reason="serve_budget"}rateobserver1000
tn_network_inbound_requests_shed_total{class="other",reason="unsubscribed"}ratevalidator100.0460.2120.276
tn_network_inbound_requests_shed_total{class="other",reason="unsubscribed"}rateobserver1000
tn_network_inbound_requests_shed_total{class="vote",reason="admission"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="vote",reason="admission"}rateobserver1000
tn_network_inbound_requests_shed_total{class="vote",reason="queue_budget"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="vote",reason="queue_budget"}rateobserver1000
tn_network_inbound_requests_shed_total{class="vote",reason="queue_full"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="vote",reason="queue_full"}rateobserver1000
tn_network_inbound_requests_shed_total{class="vote",reason="serve_budget"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="vote",reason="serve_budget"}rateobserver1000
tn_network_inbound_requests_shed_total{class="vote",reason="unsubscribed"}ratevalidator100.31830.145.2
tn_network_inbound_requests_shed_total{class="vote",reason="unsubscribed"}rateobserver1000
tn_network_inbound_request_service_seconds{class="batch"}p95validator630 ms194 ms194 ms
tn_network_inbound_request_service_seconds{class="epoch_record"}p95validator45 ms5 ms5 ms
tn_network_inbound_request_service_seconds{class="epoch_record"}p95observer15 ms13 ms23 ms
tn_network_inbound_request_service_seconds{class="vote"}p95validator543 ms1,626 ms1,681 ms
tn_network_established_connectionsmaxvalidator1019.22020
tn_network_established_connectionsmaxobserver118.32020
tn_worker_sync_streams_shed_total{reason="denied"}ratevalidator100.001660.150.2
tn_worker_batch_fetch_duration_secondsp95validator95 ms7 ms8 ms
tn_executor_batch_fetch_failures_totalratevalidator10000
tn_executor_batch_fetch_failures_totalrateobserver1000
tn_primary_votes_received_totalratevalidator101616.816.9
tn_primary_votes_received_totalrateobserver1000
tn_primary_vote_request_failures_totalratevalidator100.001790.1670.222
tn_primary_vote_request_failures_totalrateobserver1000
tn_primary_certificates_formed_totalratevalidator102.672.812.82
tn_primary_certificates_formed_totalrateobserver1000
tn_primary_certificate_form_duration_secondsp95validator8400 ms490 ms499 ms
tn_engine_outputs_executed_totalratevalidator101.331.441.49
tn_engine_outputs_executed_totalrateobserver11.321.441.47
tn_engine_persist_failures_totalratevalidator10000
tn_engine_persist_failures_totalrateobserver1000
tn_node_consensus_sync_distancemaxvalidator1016.6468468
tn_node_consensus_sync_distancemaxobserver11527.228
reth_db_table_size{table="AccountChangeSets"}maxvalidator10377 MiB566 MiB569 MiB
reth_db_table_size{table="AccountChangeSets"}maxobserver1377 MiB564 MiB567 MiB
reth_db_table_size{table="AccountsHistory"}maxvalidator1041 MiB73 MiB74 MiB
reth_db_table_size{table="AccountsHistory"}maxobserver141 MiB72 MiB73 MiB
reth_db_table_size{table="HashedAccounts"}maxvalidator1026 MiB44 MiB44 MiB
reth_db_table_size{table="HashedAccounts"}maxobserver126 MiB44 MiB44 MiB
reth_db_table_size{table="HashedStorages"}maxvalidator1041 MiB44 MiB44 MiB
reth_db_table_size{table="HashedStorages"}maxobserver141 MiB44 MiB44 MiB
reth_db_table_size{table="PlainAccountState"}maxvalidator1022 MiB37 MiB37 MiB
reth_db_table_size{table="PlainAccountState"}maxobserver122 MiB37 MiB37 MiB
reth_db_table_size{table="PlainStorageState"}maxvalidator1041 MiB44 MiB44 MiB
reth_db_table_size{table="PlainStorageState"}maxobserver141 MiB44 MiB44 MiB
reth_db_table_size{table="StorageChangeSets"}maxvalidator10451 MiB684 MiB688 MiB
reth_db_table_size{table="StorageChangeSets"}maxobserver1451 MiB681 MiB685 MiB
reth_db_table_size{table="StoragesHistory"}maxvalidator1098 MiB115 MiB115 MiB
reth_db_table_size{table="StoragesHistory"}maxobserver198 MiB115 MiB115 MiB
reth_db_table_size{table="TransactionHashNumbers"}maxvalidator10825 MiB1.23 GiB1.24 GiB
reth_db_table_size{table="TransactionHashNumbers"}maxobserver1824 MiB1.23 GiB1.24 GiB
reth_db_table_size{table="TransactionSenders"}maxvalidator10425 MiB653 MiB657 MiB
reth_db_table_size{table="TransactionSenders"}maxobserver1424 MiB650 MiB654 MiB
reth_db_freelistmaxvalidator1025.9K50.6K54.3K
reth_db_freelistmaxobserver125.8K48.2K50.7K
reth_database_transaction_commit_write_duration_secondsp95validator95 ms5 ms5 ms
reth_database_transaction_commit_write_duration_secondsp95observer15 ms5 ms5 ms
reth_database_transaction_commit_sync_duration_secondsp95validator9477 ms630 ms756 ms
reth_database_transaction_commit_sync_duration_secondsp95observer1485 ms522 ms612 ms
reth_database_transaction_commit_whole_duration_secondsp95validator9477 ms630 ms756 ms
reth_database_transaction_commit_whole_duration_secondsp95observer1486 ms564 ms742 ms
tn_engine_recover_secondsp95validator9115 ms864 ms864 ms
tn_engine_recover_secondsp95observer1120 ms451 ms469 ms
tn_engine_evm_secondsp95validator9249 ms454 ms500 ms
tn_engine_evm_secondsp95observer1246 ms443 ms475 ms
tn_engine_root_secondsp95validator8368 ms839 ms1,000 ms
tn_engine_root_secondsp95observer1337 ms810 ms858 ms
tn_engine_finish_secondsp95validator10173 ms469 ms469 ms
tn_engine_finish_secondsp95observer1136 ms321 ms366 ms
tn_engine_forward_wait_secondsp95validator98,669 ms9,819 ms9,845 ms
tn_engine_forward_wait_secondsp95observer15,784 ms9,820 ms9,936 ms
tn_engine_channel_depthmaxvalidator107.4211.812
tn_engine_channel_depthmaxobserver14.599.2510
tn_engine_executed_transactions_totalratevalidator105.54K13.9K13.9K
tn_engine_executed_transactions_totalrateobserver15.52K13.9K13.9K
tn_executor_execution_wait_secondsp95validator105 ms2,034 ms2,106 ms
tn_primary_commit_run_ahead_blocked_totalratevalidator10000
tn_primary_commit_run_ahead_blocked_totalrateobserver1000
tn_primary_consensus_exec_lag_roundsmaxvalidator1033.543.544
tn_primary_consensus_exec_lag_roundsmaxobserver127.23940
tn_primary_commit_lag_roundsmaxvalidator102.163.55
tn_primary_commit_lag_roundsmaxobserver1000
tn_primary_rejoin_grace_holds_totalratevalidator10000
tn_primary_rejoin_grace_holds_totalrateobserver1000
tn_primary_votes_refused_inactive_totalratevalidator10033.133.2
tn_primary_votes_refused_inactive_totalrateobserver1000
tn_primary_votes_refused_execution_behind_totalratevalidator1000.03890.0444
tn_primary_votes_refused_execution_behind_totalrateobserver1000
tn_batch_builder_pool_age_secondsp95validator75,563 ms30.0 s30.0 s
tn_batch_builder_builds_total{reason="bytes_full"}ratevalidator10000
tn_batch_builder_builds_total{reason="bytes_full"}rateobserver1000
tn_batch_builder_builds_total{reason="drained"}ratevalidator101.793.363.36
tn_batch_builder_builds_total{reason="drained"}rateobserver1000
tn_batch_builder_builds_total{reason="gas_full"}ratevalidator101.13.223.98
tn_batch_builder_builds_total{reason="gas_full"}rateobserver1000
tn_worker_batch_report_secondsp95validator80 ms1 ms1 ms
tn_primary_digest_wait_secondsp95validator10490 ms30.0 s30.0 s
tn_primary_headers_proposed_total{reason="max_delay"}ratevalidator100.005940.1170.133
tn_primary_headers_proposed_total{reason="min_delay"}ratevalidator102.672.812.82
tn_primary_headers_proposed_total{reason="parents_and_threshold"}ratevalidator100.0008280.10.156
tn_primary_digests_requeued_total{cause="canceled"}ratevalidator100.008420.1110.133
tn_primary_digests_requeued_total{cause="committed"}ratevalidator100.004970.9941.53
tn_executor_consensus_pack_persist_secondsp95validator847 ms81 ms84 ms
tn_batch_builder_seal_duration_secondsp95validator9252 ms731 ms731 ms
tn_worker_quorum_wait_secondsp95validator9252 ms854 ms854 ms
tn_primary_proposer_pending_digestsmaxvalidator100.5712.255
tn_primary_proposer_pending_digestsmaxobserver1000
reth_transaction_pool_pending_pool_transactionsmaxvalidator103564.59K9.85K
reth_transaction_pool_pending_pool_transactionsmaxobserver1000
reth_transaction_pool_queued_pool_transactionsmaxvalidator101.15K6.22K6.61K
reth_transaction_pool_queued_pool_transactionsmaxobserver1000
reth_transaction_pool_all_transactions_by_hashmaxvalidator101.7K9.43K10.5K
reth_transaction_pool_all_transactions_by_hashmaxobserver1000
reth_storage_providers_database_save_blocks_totalp95validator9121 ms230 ms272 ms
reth_storage_providers_database_save_blocks_totalp95observer1112 ms188 ms197 ms
reth_storage_providers_database_save_blocks_mdbxp95validator8124 ms230 ms272 ms
reth_storage_providers_database_save_blocks_mdbxp95observer1112 ms188 ms197 ms
reth_storage_providers_database_save_blocks_sfp95validator929 ms60 ms73 ms
reth_storage_providers_database_save_blocks_sfp95observer122 ms38 ms42 ms
reth_storage_providers_database_save_blocks_commit_mdbxp95validator9372 ms496 ms504 ms
reth_storage_providers_database_save_blocks_commit_mdbxp95observer1395 ms491 ms495 ms
reth_storage_providers_database_save_blocks_commit_sfp95validator910 ms15 ms15 ms
reth_storage_providers_database_save_blocks_commit_sfp95observer19 ms11 ms12 ms
reth_storage_providers_database_save_blocks_write_statep95validator921 ms39 ms44 ms
reth_storage_providers_database_save_blocks_write_statep95observer120 ms33 ms35 ms
reth_storage_providers_database_save_blocks_write_hashed_statep95validator815 ms25 ms29 ms
reth_storage_providers_database_save_blocks_write_hashed_statep95observer114 ms21 ms22 ms
reth_storage_providers_database_save_blocks_write_trie_updatesp95validator1020 ms30 ms33 ms
reth_storage_providers_database_save_blocks_write_trie_updatesp95observer118 ms27 ms29 ms
reth_database_transaction_commit_preparation_duration_secondsp95validator85 ms5 ms5 ms
reth_database_transaction_commit_preparation_duration_secondsp95observer15 ms5 ms5 ms
reth_database_transaction_commit_gc_wallclock_duration_secondsp95validator95 ms5 ms5 ms
reth_database_transaction_commit_gc_wallclock_duration_secondsp95observer15 ms5 ms5 ms
reth_database_transaction_commit_gc_cputime_duration_secondsp95validator75 ms5 ms5 ms
reth_database_transaction_commit_gc_cputime_duration_secondsp95observer15 ms5 ms5 ms
reth_database_transaction_commit_audit_duration_secondsp95validator75 ms5 ms5 ms
reth_database_transaction_commit_audit_duration_secondsp95observer15 ms5 ms5 ms
reth_database_transaction_commit_ending_duration_secondsp95validator75 ms5 ms5 ms
reth_database_transaction_commit_ending_duration_secondsp95observer15 ms5 ms5 ms
tn_node_switch{name="digest_dedupe"}maxvalidator10000
tn_node_switch{name="digest_dedupe"}maxobserver1000
tn_node_switch{name="epoch_close_task_drain"}maxvalidator10000
tn_node_switch{name="epoch_close_task_drain"}maxobserver1000
tn_node_switch{name="epoch_record_state_check"}maxvalidator10111
tn_node_switch{name="epoch_record_state_check"}maxobserver1111
tn_node_switch{name="mdbx_random_advice"}maxvalidator10111
tn_node_switch{name="mdbx_random_advice"}maxobserver1111
tn_node_switch{name="parent_grace"}maxvalidator10000
tn_node_switch{name="parent_grace"}maxobserver1000
tn_node_switch{name="pipelined_persist"}maxvalidator10000
tn_node_switch{name="pipelined_persist"}maxobserver1000
tn_node_switch{name="quorum_exec_block"}maxvalidator10111
tn_node_switch{name="quorum_exec_block"}maxobserver1111
tn_node_switch{name="quorum_exec_block_starved_rounds"}maxvalidator10444
tn_node_switch{name="quorum_exec_block_starved_rounds"}maxobserver1444
tn_node_switch{name="recovery_threads"}maxvalidator10000
tn_node_switch{name="recovery_threads"}maxobserver1000
tn_node_switch{name="rpc_admission_backlog_outputs"}maxvalidator10000
tn_node_switch{name="rpc_admission_backlog_outputs"}maxobserver1000
tn_node_switch{name="sealed_tx_keep"}maxvalidator10000
tn_node_switch{name="sealed_tx_keep"}maxobserver1000
tn_node_switch{name="vote_queue_fix"}maxvalidator10111
tn_node_switch{name="vote_queue_fix"}maxobserver1111
tn_primary_epoch_entry_rounds_behindmaxvalidator1031616
tn_primary_epoch_entry_rounds_behindmaxobserver1000
tn_reth_mdbx_random_advice_bytesmaxvalidator108192.00 GiB8192.00 GiB8192.00 GiB
tn_reth_mdbx_random_advice_bytesmaxobserver18192.00 GiB8192.00 GiB8192.00 GiB
tn_reth_mdbx_used_bytesmaxvalidator102.42 GiB3.57 GiB3.58 GiB
tn_reth_mdbx_used_bytesmaxobserver12.42 GiB3.56 GiB3.58 GiB
tn_reth_mdbx_page_ops_total{op="cow"}maxvalidator1036.6M61M61.5M
tn_reth_mdbx_page_ops_total{op="cow"}maxobserver127.6M52.6M53.1M
tn_reth_mdbx_page_ops_total{op="msync"}maxvalidator106.39K9.49K9.54K
tn_reth_mdbx_page_ops_total{op="msync"}maxobserver14.95K8.46K8.52K
tn_reth_mdbx_page_ops_total{op="newly"}maxvalidator10620K1.05M1.07M
tn_reth_mdbx_page_ops_total{op="newly"}maxobserver1481K815K821K
tn_reth_mdbx_page_ops_total{op="prefault"}maxvalidator10634K936K940K
tn_reth_mdbx_page_ops_total{op="prefault"}maxobserver1493K832K835K
tn_reth_mdbx_page_ops_total{op="spill"}maxvalidator10000
tn_reth_mdbx_page_ops_total{op="spill"}maxobserver1000
tn_reth_mdbx_page_ops_total{op="unspill"}maxvalidator10000
tn_reth_mdbx_page_ops_total{op="unspill"}maxobserver1000
tn_reth_mdbx_page_ops_total{op="wops"}maxvalidator10111
tn_reth_mdbx_page_ops_total{op="wops"}maxobserver1111
tn_reth_sealed_txs_markedmaxvalidator10000
tn_reth_sealed_txs_markedmaxobserver1000
tn_reth_sealed_txs_cleared_executed_totalmaxvalidator10000
tn_reth_sealed_txs_cleared_executed_totalmaxobserver1000
tn_reth_sealed_txs_cleared_repooled_totalmaxvalidator10000
tn_reth_sealed_txs_cleared_repooled_totalmaxobserver1000
tn_reth_sealed_txs_dropped_evicted_totalmaxvalidator10000
tn_reth_sealed_txs_dropped_evicted_totalmaxobserver1000
tn_reth_sealed_txs_cleared_lag_totalmaxvalidator10000
tn_reth_sealed_txs_cleared_lag_totalmaxobserver1000
tn_reth_sealed_txs_late_marks_refused_totalmaxvalidator10000
tn_reth_sealed_txs_late_marks_refused_totalmaxobserver1000
tn_batch_builder_sealed_txs_skipped_total{worker="0"}maxvalidator10000
tn_batch_builder_sealed_txs_skipped_total{worker="0"}maxobserver1000
reth_transaction_pool_pending_pool_size_bytesmaxvalidator10125.7 KiB1.6 MiB3.4 MiB
reth_transaction_pool_pending_pool_size_bytesmaxobserver10 B0 B0 B
reth_transaction_pool_queued_pool_size_bytesmaxvalidator10388.9 KiB1.7 MiB2.0 MiB
reth_transaction_pool_queued_pool_size_bytesmaxobserver10 B0 B0 B
reth_transaction_pool_pending_transactions_evictedmaxvalidator10000
reth_transaction_pool_pending_transactions_evictedmaxobserver1000
reth_transaction_pool_queued_transactions_evictedmaxvalidator10000
reth_transaction_pool_queued_transactions_evictedmaxobserver1000
reth_transaction_pool_basefee_transactions_evictedmaxvalidator10000
reth_transaction_pool_basefee_transactions_evictedmaxobserver1000
reth_transaction_pool_removed_transactionsmaxvalidator101.34M2.26M2.28M
reth_transaction_pool_removed_transactionsmaxobserver1000
reth_transaction_pool_invalid_transactionsmaxvalidator10000
reth_transaction_pool_invalid_transactionsmaxobserver1000
tn_worker_duplicate_digest_suppressed_total{worker="0"}maxvalidator10000
tn_worker_duplicate_digest_suppressed_total{worker="0"}maxobserver1000
tn_primary_duplicate_digest_suppressed_totalmaxvalidator10000
tn_primary_duplicate_digest_suppressed_totalmaxobserver1000
tn_engine_recovery_fallback_totalmaxvalidator10000
tn_engine_recovery_fallback_totalmaxobserver1000
tn_node_epoch_close_backlog_outputsmaxvalidator1012.51919
tn_node_epoch_close_backlog_outputsmaxobserver19.241313
tn_node_epoch_close_detect_lag_secondsmaxvalidator10718 ms337.2 s337.2 s
tn_node_epoch_close_detect_lag_secondsmaxobserver11,153 ms1,401 ms1,401 ms
tn_node_epoch_close_drain_secondsmaxvalidator1010.4 s14.5 s14.5 s
tn_node_epoch_close_drain_secondsmaxobserver17,109 ms10.0 s10.0 s
tn_node_epoch_record_vote_skipped_total{reason="certified_before_close"}maxvalidator1111
tn_node_epoch_record_vote_skipped_total{reason="not_in_committee"}maxobserver14.1355
tn_primary_parent_grace_total{outcome="complete"}maxvalidator10000
tn_primary_parent_grace_total{outcome="expired"}maxvalidator10000
tn_primary_parent_grace_total{outcome="preempted"}maxvalidator10000
tn_primary_late_parents_total{origin="older_round"}maxvalidator104.12K7.11K7.11K
tn_primary_late_parents_total{origin="own_round_behind"}maxvalidator1033.6K42.6K42.7K
tn_node_execution_backlog_outputsmaxvalidator1017.321.822
tn_node_execution_backlog_outputsmaxobserver114.319.820
tn_engine_unpersisted_outputsmaxvalidator10000
tn_engine_unpersisted_outputsmaxobserver1000
tn_engine_built_heightmaxvalidator10000
tn_engine_built_heightmaxobserver1000
tn_engine_close_barrier_holds_totalmaxvalidator10000
tn_engine_close_barrier_holds_totalmaxobserver1000

Commit latency p95

tn_primary_commit_latency_seconds

Round lag

tn_primary_round − tn_primary_committed_round

Engine queued outputs

tn_engine_queued_outputs, max by node

Certification by node

The share of each validator's own headers that formed a certificate, per window (tools/cert-rate.py on the node logs). Every validator certified at least 90 % of its own headers in every window where it proposed at least 50. Own leader % is the validator's share of the committed leaders and not committed % the share of its certificates no committed leader reached, both over the run window; the fair share is 100 / 10 validators = 10.0 %. Greyed: fewer than 50 headers proposed, not judged.

ValidatorRegionblast 0 preambleblast 0 warmupblast 0 mixedblast 0 cooldownblast 1 preambleblast 1 warmupblast 1 mixedblast 1 cooldownblast 2 preambleblast 2 warmupblast 2 mixedblast 2 cooldownrunOwn leader %Not committed %
bench-validator-01us-west2 (Los Angeles)100.0100.3100.0100.099.699.8100.1100.098.099.199.9100.099.913.20.1
bench-validator-02europe-west2 (London)100.699.7100.099.5100.099.098.899.3100.096.699.0100.899.20.441.4
bench-validator-03australia-southeast1 (Sydney)100.699.7100.099.5100.0100.2100.1100.0100.0100.3100.0100.8100.0––
bench-validator-04asia-northeast1 (Tokyo)100.0100.0100.0100.099.699.8100.1100.0100.0100.0100.0100.8100.0––
bench-validator-05us-east4 (Ashburn)100.0100.3100.199.5100.0100.099.9100.0100.0100.099.9–100.010.60.2
bench-validator-06europe-west4 (Netherlands)100.699.7100.099.599.398.398.398.5100.097.199.2100.099.20.145.8
bench-validator-07asia-southeast1 (Singapore)100.0100.0100.1100.099.698.698.7100.0100.096.899.7100.099.613.30.2
bench-validator-08asia-east2 (Hong Kong)100.6100.0100.099.5100.099.898.9100.2100.097.199.8100.099.713.10.1
bench-validator-09asia-northeast3 (Seoul)100.6100.0100.0100.098.998.698.8100.2102.097.199.799.299.613.10.1
bench-validator-10asia-south1 (Mumbai)100.0100.3100.1100.0100.098.398.1100.0100.096.899.2100.099.312.10.3

On-node sampler

NodeClassPeak RssAnonPeak RssFilePeak cachePSI memPSI ioPSI cpudb growthstatic_files growthRoot usedOOM Δ
bench-observer-11observer621 MiB3.65 GiB– 0.0 %10.8 %1.2 %0 B4.09 GiB16.0 %0
bench-validator-01validator2.21 GiB3.80 GiB– 0.0 %12.4 %5.5 %0 B4.09 GiB18.0 %0
bench-validator-02validator2.70 GiB3.58 GiB– 0.0 %13.0 %6.9 %0 B4.09 GiB18.0 %0
bench-validator-03validator2.45 GiB3.20 GiB– 0.0 %12.0 %9.0 %0 B4.09 GiB18.0 %0
bench-validator-04validator2.05 GiB3.62 GiB– 0.0 %12.4 %8.5 %0 B4.09 GiB18.0 %0
bench-validator-05validator2.26 GiB4.22 GiB– 0.0 %11.5 %6.9 %0 B4.09 GiB18.0 %0
bench-validator-06validator2.37 GiB3.39 GiB– 0.0 %13.4 %6.5 %0 B4.09 GiB18.0 %0
bench-validator-07validator2.37 GiB3.22 GiB– 0.0 %14.1 %6.1 %0 B4.09 GiB18.0 %0
bench-validator-08validator2.44 GiB3.76 GiB– 0.0 %12.9 %7.0 %0 B4.09 GiB18.0 %0
bench-validator-09validator2.40 GiB3.78 GiB– 0.0 %13.1 %5.7 %0 B4.09 GiB18.0 %0
bench-validator-10validator2.39 GiB3.23 GiB– 0.0 %12.7 %5.3 %0 B4.09 GiB18.0 %0

RssAnon vs RssFile

/proc/<pid>/status of the telcoin process on the node with the largest RSS

consensus-db/cache size

Not collected: consensus-db/cache size.

Pressure stall (some, avg10)

/proc/pressure on the node with the highest memory PSI

Headroom

Per role and per node, from sizing.json (p95 over the landed window; PSI and RssAnon max over the whole sampler run). The lines follow the hardware doc's tier rules: CPU p95 over 70 % of vCPUs; memory used p95 over 75 % of RAM, or telcoin RSS p95 over 85 % of RAM; disk write p95 over 60 % of the PD MiB/s limit, or write IOPS p95 over 60 % of the IOPS limit; IO pressure over 20 % or memory pressure over 5 % (PSI some avg10, max); network p95 over 50 % of 1 Gbps; root disk over 80 %. A line crossed on any node puts the role at the limit; a nonzero reading within 15 points of its line is tight; otherwise the role has headroom. OOM kills, disk throttling and degraded disk performance are flagged too.

SignalClassWorst nodeValueShare of capacityLineNodes over / tightStatus
CPU p95validatorbench-validator-0535.4 %35.4 %> 70 %0 / 0 headroom
Memory used p95validatorbench-validator-0222.8 %22.8 %> 75 %0 / 0 headroom
telcoin RSS p95validatorbench-validator-026.59 GB38.4 %> 85 %0 / 0 headroom
PSI memory maxvalidatorbench-validator-010.0 %0.0 %> 5 %0 / 0 headroom
RssAnon peakvalidatorbench-validator-022.89 GB16.8 %info– info
Disk write p95validatorbench-validator-02110 MB/s13.1 %> 60 %0 / 0 headroom
Write IOPS p95validatorbench-validator-0420,098100.5 %> 60 %10 / 0 at the limit
PSI io maxvalidatorbench-validator-0714.1 %14.1 %> 20 %0 / 10 tight
Root disk maxvalidatorbench-validator-0817.5 %17.5 %> 80 %0 / 0 headroom
Network p95validatorbench-validator-03110 Mbps11.0 %> 50 %0 / 0 headroom
Consensus cache peakvalidator–––info– not collected
OOM killsvalidator––anyany– none
Disk throttlingvalidator––metric type unavailable in projectany– not collected
Disk performance degradedvalidator––any sampleany– none
CPU p95observerbench-observer-1120.2 %20.2 %> 70 %0 / 0 headroom
Memory used p95observerbench-observer-1110.1 %10.1 %> 75 %0 / 0 headroom
telcoin RSS p95observerbench-observer-114.22 GB24.6 %> 85 %0 / 0 headroom
PSI memory maxobserverbench-observer-110.0 %0.0 %> 5 %0 / 0 headroom
RssAnon peakobserverbench-observer-110.65 GB3.8 %info– info
Disk write p95observerbench-observer-1197 MB/s11.5 %> 60 %0 / 0 headroom
Write IOPS p95observerbench-observer-1120,087100.4 %> 60 %1 / 0 at the limit
PSI io maxobserverbench-observer-1110.8 %10.8 %> 20 %0 / 1 tight
Root disk maxobserverbench-observer-1115.8 %15.8 %> 80 %0 / 0 headroom
Network p95observerbench-observer-1111 Mbps1.1 %> 50 %0 / 0 headroom
Consensus cache peakobserver–––info– not collected
OOM killsobserver––anyany– none
Disk throttlingobserver––metric type unavailable in projectany– not collected
Disk performance degradedobserver––any sampleany– none

Per node

NodeClassCPU p95Memory used p95telcoin RSS p95PSI memory maxRssAnon peakDisk write p95Write IOPS p95PSI io maxRoot disk maxNetwork p95Consensus cache peak
Line> 70 %> 75 %> 85 %> 5 %info> 60 %> 60 %> 20 %> 80 %> 50 %info
bench-validator-01validator34.5 %19.8 %6.21 GB 36 %0.0 %2.37 GB 14 %108 MB/s 13 %20,057 100 %12.4 %17.5 %107 Mbps 11 %–
bench-validator-02validator33.0 %22.8 %6.59 GB 38 %0.0 %2.89 GB 17 %110 MB/s 13 %20,055 100 %13.0 %17.4 %105 Mbps 10 %–
bench-validator-03validator35.2 %21.2 %5.91 GB 34 %0.0 %2.63 GB 15 %110 MB/s 13 %20,041 100 %12.0 %17.4 %110 Mbps 11 %–
bench-validator-04validator35.0 %19.0 %5.95 GB 35 %0.0 %2.21 GB 13 %110 MB/s 13 %20,098 100 %12.4 %17.5 %108 Mbps 11 %–
bench-validator-05validator35.4 %20.2 %5.67 GB 33 %0.0 %2.43 GB 14 %109 MB/s 13 %20,071 100 %11.5 %17.1 %108 Mbps 11 %–
bench-validator-06validator32.7 %20.6 %6.10 GB 35 %0.0 %2.54 GB 15 %109 MB/s 13 %20,078 100 %13.4 %17.4 %103 Mbps 10 %–
bench-validator-07validator33.2 %20.6 %5.92 GB 34 %0.0 %2.54 GB 15 %109 MB/s 13 %20,074 100 %14.1 %17.4 %104 Mbps 10 %–
bench-validator-08validator34.1 %21.1 %6.58 GB 38 %0.0 %2.62 GB 15 %109 MB/s 13 %20,076 100 %12.9 %17.5 %105 Mbps 11 %–
bench-validator-09validator33.5 %20.8 %6.56 GB 38 %0.0 %2.58 GB 15 %109 MB/s 13 %20,075 100 %13.1 %17.5 %105 Mbps 10 %–
bench-validator-10validator33.1 %20.9 %5.95 GB 35 %0.0 %2.57 GB 15 %109 MB/s 13 %20,086 100 %12.7 %17.2 %103 Mbps 10 %–
bench-observer-11observer20.2 %10.1 %4.22 GB 25 %0.0 %0.65 GB 4 %97 MB/s 12 %20,087 100 %10.8 %15.8 %11 Mbps 1 %–

Small figures are the share of capacity: RAM for RSS, the PD limit for disk write (VM cap applied), 1 Gbps for network. Hover a cell for its JSON path.

Verdict

Fleet A vs Fleet B

A = Fleet c3-hdb · c3-highcpu-8 · r20261113-2118
B = Fleet c3-hdb · c3-highcpu-8 · r20261113-1351

Verdict

at the limit c3-highcpu-8 validators at 7,637 sustained TPS: at the limit on disk. Disk at the limit: write IOPS p95 20,041–20,098, up to 100.5 % of the 20,000 IOPS limit, over the 60 % line on all 10 validators (bench-validator-01, 02, 03, 04, 05, 06, 07, 08, 09, 10); IO pressure (PSI some avg10) max 14.1 % (bench-validator-07), tight against the 20 % line. CPU has headroom: CPU p95 35.4 % of 8 vCPUs (bench-validator-05). Memory has headroom: memory used p95 22.8 % of RAM (bench-validator-02); telcoin RSS p95 6.59 GB, 38.4 % of 16 GiB RAM (bench-validator-02); memory pressure (PSI some avg10) max 0.0 % (bench-validator-01). Network has headroom: network p95 110 Mbps, 11.0 % of 1 Gbps (bench-validator-03). Next step: a faster volume (pd-ssd adds 30 IOPS and 0.48 MiB/s per GB up to the per-VM cap; past the cap, a larger VM).

at the limit c3-highcpu-8 observers at 7,637 sustained TPS: at the limit on disk. Disk at the limit: write IOPS p95 20,087, 100.4 % of the 20,000 IOPS limit on bench-observer-11, over the 60 % line; IO pressure (PSI some avg10) max 10.8 % (bench-observer-11), tight against the 20 % line. CPU has headroom: CPU p95 20.2 % of 8 vCPUs (bench-observer-11). Memory has headroom: memory used p95 10.1 % of RAM (bench-observer-11); telcoin RSS p95 4.22 GB, 24.6 % of 16 GiB RAM (bench-observer-11); memory pressure (PSI some avg10) max 0.0 % (bench-observer-11). Network has headroom: network p95 11 Mbps, 1.1 % of 1 Gbps (bench-observer-11). Next step: a faster volume (pd-ssd adds 30 IOPS and 0.48 MiB/s per GB up to the per-VM cap; past the cap, a larger VM).

at the limit c3-highcpu-8 validators at 8,065 sustained TPS: at the limit on disk. Disk at the limit: write IOPS p95 20,050–20,146, up to 100.7 % of the 20,000 IOPS limit, over the 60 % line on all 10 validators (bench-validator-01, 02, 03, 04, 05, 06, 07, 08, 09, 10); IO pressure (PSI some avg10) max 14.9 % (bench-validator-08), tight against the 20 % line. CPU has headroom: CPU p95 36.0 % of 8 vCPUs (bench-validator-03). Memory has headroom: memory used p95 22.7 % of RAM (bench-validator-06); telcoin RSS p95 6.87 GB, 40.0 % of 16 GiB RAM (bench-validator-06); memory pressure (PSI some avg10) max 0.0 % (bench-validator-01). Network has headroom: network p95 130 Mbps, 13.0 % of 1 Gbps (bench-validator-07). Next step: a faster volume (pd-ssd adds 30 IOPS and 0.48 MiB/s per GB up to the per-VM cap; past the cap, a larger VM).

at the limit c3-highcpu-8 observers at 8,065 sustained TPS: at the limit on disk. Disk at the limit: write IOPS p95 20,071, 100.4 % of the 20,000 IOPS limit on bench-observer-11, over the 60 % line; IO pressure (PSI some avg10) max 13.7 % (bench-observer-11), tight against the 20 % line. CPU has headroom: CPU p95 20.4 % of 8 vCPUs (bench-observer-11). Memory has headroom: memory used p95 12.4 % of RAM (bench-observer-11); telcoin RSS p95 4.54 GB, 26.4 % of 16 GiB RAM (bench-observer-11); memory pressure (PSI some avg10) max 0.0 % (bench-observer-11). Network has headroom: network p95 12 Mbps, 1.2 % of 1 Gbps (bench-observer-11). Next step: a faster volume (pd-ssd adds 30 IOPS and 0.48 MiB/s per GB up to the per-VM cap; past the cap, a larger VM).

Validators: Fleet A (c3-highcpu-8) is at the limit on disk, with headroom on CPU, memory and network; Fleet B (c3-highcpu-8) is at the limit on disk, with headroom on CPU, memory and network. The same disk limit binds both fleets: every validator runs a 100 GB hyperdisk-balanced, which allows 20,000 IOPS on both; its throughput limit is 800 MiB/s on both. Disk is at the limit on both (Fleet A (c3-highcpu-8): write IOPS p95 20,098, 100.5 % of the 20,000 IOPS limit; Fleet B (c3-highcpu-8): write IOPS p95 20,146, 100.7 % of the 20,000 IOPS limit). Changing the machine shape did not clear it; the volume is the binding resource.

Observers: Fleet A (c3-highcpu-8) is at the limit on disk, with headroom on CPU, memory and network; Fleet B (c3-highcpu-8) is at the limit on disk, with headroom on CPU, memory and network. The same disk limit binds both fleets: every observer runs a 100 GB hyperdisk-balanced, which allows 20,000 IOPS on both; its throughput limit is 800 MiB/s on both. Disk is at the limit on both (Fleet A (c3-highcpu-8): write IOPS p95 20,087, 100.4 % of the 20,000 IOPS limit; Fleet B (c3-highcpu-8): write IOPS p95 20,071, 100.4 % of the 20,000 IOPS limit). Changing the machine shape did not clear it; the volume is the binding resource.

KPIFleet AFleet BB ÷ A
Sustained mixed TPS (own)7,637.0 tx/s8,065.4 tx/s1.06×
Sustained mixed TPS (chain)7,637.0 tx/s8,065.4 tx/s1.06×
Landed / accepted99.9 %100.0 %1.00×
Reverted (own)00–
Latency p50745 ms679 ms0.91×
Latency p956,788 ms14,662 ms2.16×
Gas / s (own)273M gas/s269M gas/s0.98×
Capacity at mix14,188.5 tx/s14,103.0 tx/s0.99×

Headroom, worst node per class

SignalClassFleet AFleet BLine
CPU p95validator35.4 % headroom36.0 % headroom> 70 %
Memory used p95validator22.8 % headroom22.7 % headroom> 75 %
telcoin RSS p95validator6.59 GB 38.4 % headroom6.87 GB 40.0 % headroom> 85 %
PSI memory maxvalidator0.0 % headroom0.0 % headroom> 5 %
Disk write p95validator110 MB/s 13.1 % headroom110 MB/s 13.1 % headroom> 60 %
Write IOPS p95validator20,098 100.5 % at the limit20,146 100.7 % at the limit> 60 %
PSI io maxvalidator14.1 % tight14.9 % tight> 20 %
Root disk maxvalidator17.5 % headroom17.5 % headroom> 80 %
Network p95validator110 Mbps 11.0 % headroom130 Mbps 13.0 % headroom> 50 %
CPU p95observer20.2 % headroom20.4 % headroom> 70 %
Memory used p95observer10.1 % headroom12.4 % headroom> 75 %
telcoin RSS p95observer4.22 GB 24.6 % headroom4.54 GB 26.4 % headroom> 85 %
PSI memory maxobserver0.0 % headroom0.0 % headroom> 5 %
Disk write p95observer97 MB/s 11.5 % headroom97 MB/s 11.5 % headroom> 60 %
Write IOPS p95observer20,087 100.4 % at the limit20,071 100.4 % at the limit> 60 %
PSI io maxobserver10.8 % tight13.7 % tight> 20 %
Root disk maxobserver15.8 % headroom15.9 % headroom> 80 %
Network p95observer11 Mbps 1.1 % headroom12 Mbps 1.2 % headroom> 50 %

Own TPS, trailing 60-s mean

aligned on each fleet's mixed-phase start

CPU, hottest validator

aligned on each fleet's mixed-phase start

Memory used, hottest validator

aligned on each fleet's mixed-phase start

telcoin RSS, hottest validator

aligned on each fleet's mixed-phase start

Disk MiB/s, hottest validator

aligned on each fleet's mixed-phase start

Network sent, hottest validator

aligned on each fleet's mixed-phase start

Reference chains

Not collected: reference chains.