This is the full report of the run “The always-on vote fix on Hyperdisk Balanced at 50,000 IOPS”. Back to the run page.
r20261113-1630 · Fleet c3-hdb · c3-highcpu-8

Telcoin Network benchmark: mixed workload

Throughput is measured from blocks the chain produced (own senders vs every sender), resource use from Cloud Monitoring, Prometheus and the on-node sampler. Summaries cover the mixed phase. Hover any number for the JSON field it came from.

10 blaster(s), 6K senders, chain 32286, schema tn-blast-combined/v2 · mixed phase 2026-11-13 17:55–18:36 UTC · 1 observers, 10 validators

Preflight: WTEL from genesis (0x0000…37e1), v3 fee tier 3000, v4 available, Multicall3 present · Cooldown ended: all_landed ×6, idle_30s ×4

Headline numbers

Sustained mixed TPS (own)
8,539.0 tx/s
best 60-s mean of our own landed tx in the mixed phase
Sustained mixed TPS (chain)
8,539.0 tx/s
all tx in blocks, same window
Landed / accepted
99.9 %
19.4M of 19.4M accepted
Reverted (own)
0
0.00 % of landed
Latency p50
858 ms
submit → block timestamp; seen p50 8,322 ms
Latency p95
3,528 ms
seen p95 25,736 ms
Gas / s (own)
365M gas/s
chain 365M gas/s
Capacity at mix
16,530.3 tx/s
gas-limit basis 10,624.0 tx/s; block gas limit 30M

Per transaction type

Landed counts only our own senders; latency is submit → block timestamp.

TypeSubmittedAcceptedLandedRevertedUnlandedSkipsGas limitGas used avgp50p95
native_transfer5.56M5.49M5.48M03.15K024.1K21K853 ms3,507 ms
batched_self_send3.03M2.98M2.97M02.74K024.1K21K881 ms3,577 ms
uniswap_v2_swap2.72M2.68M2.67M02.19K0180K106K854 ms3,504 ms
stablecoin_transfer2.27M2.23M2.23M01.85K075K43.9K853 ms3,500 ms
wtel_wrap1.81M1.78M1.78M01.45K050K27.9K853 ms3,506 ms
wtel_unwrap1.21M1.19M1.18M0955045K35.2K853 ms3,503 ms
stablecoin_approve908K892K891K0731060K31.8K853 ms3,502 ms
uniswap_v2_add_liquidity606K595K595K05020250K128K854 ms3,510 ms
uniswap_v4_swap454K446K446K03970250K124K855 ms3,507 ms
contract_deploy453K445K445K03810140K74.2K853 ms3,503 ms
uniswap_v3_swap303K298K298K02380220K120K855 ms3,499 ms
uniswap_v2_remove_liquidity302K296K296K02250220K129K854 ms3,494 ms
faucet_drip124K124K124K0114633K150K107K832 ms13,729 ms

Submission by RPC node

NodeSubmittedAcceptedAccepted %Top errorsSend p50Send p95
bench-validator-01
bench-validator-01:8545
1.44M1.44M100.0 %known 1093 ms17 ms
bench-validator-02
bench-validator-02:8545
1.33M1.31M98.6 %known 9.86K5 ms25 ms
bench-validator-03
bench-validator-03:8545
3.46M3.46M100.0 %none6 ms18 ms
bench-validator-04
bench-validator-04:8545
2.41M2.41M100.0 %known 96 ms19 ms
bench-validator-05
bench-validator-05:8545
1.39M440K31.7 %other 948K, known 831 ms24 ms
bench-validator-06
bench-validator-06:8545
502K457K91.1 %known 22.4K6 ms39 ms
bench-validator-07
bench-validator-07:8545
2.74M2.74M100.0 %none5 ms17 ms
bench-validator-08
bench-validator-08:8545
3.06M3.06M99.9 %known 1.35K6 ms20 ms
bench-validator-09
bench-validator-09:8545
2.68M2.65M98.8 %known 16.7K4 ms15 ms
bench-validator-10
bench-validator-10:8545
1.55M1.52M98.0 %known 11.8K6 ms21 ms

Throughput over time

Shaded: phases outside the mixed window. Hairlines: epoch closes. Dashed orange: chaos events.

Landed transactions per second

own senders vs chain total, per block-timestamp second

Gas per second

own vs chain gas used, per block-timestamp second

CPU and memory per node

CPU utilization

compute.googleapis.com/instance/cpu/utilization, 60-s mean, % of all vCPUs; summaries over the mixed phase

Table view
NodeClassMeanp95Peak 60-s meanMax
bench-observer-11observer22.3 %23.9 %26.4 %26.4 %
bench-validator-01validator37.1 %40.1 %48.3 %48.3 %
bench-validator-02validator37.2 %40.0 %45.3 %45.3 %
bench-validator-03validator40.5 %43.4 %48.4 %48.4 %
bench-validator-04validator38.6 %41.2 %48.3 %48.3 %
bench-validator-05validator31.2 %38.2 %45.7 %45.7 %
bench-validator-06validator36.8 %39.3 %46.0 %46.0 %
bench-validator-07validator38.9 %42.8 %47.2 %47.2 %
bench-validator-08validator39.6 %43.0 %47.2 %47.2 %
bench-validator-09validator39.0 %43.6 %49.1 %49.1 %
bench-validator-10validator38.1 %40.2 %48.7 %48.7 %

CPU: mean vs peak 60-s mean

One row per node, over the mixed phase.

telcoin resident memory

agent processes/rss_usage summed over telcoin pids; summaries over the mixed phase

Table view
NodeClassMeanp95Peak 60-s meanMax
bench-observer-11observer3.43 GiB4.76 GiB4.87 GiB4.87 GiB
bench-validator-01validator5.55 GiB6.37 GiB6.41 GiB6.41 GiB
bench-validator-02validator5.68 GiB6.51 GiB6.61 GiB6.61 GiB
bench-validator-03validator5.01 GiB5.95 GiB6.00 GiB6.00 GiB
bench-validator-04validator5.03 GiB5.90 GiB5.99 GiB5.99 GiB
bench-validator-05validator5.20 GiB6.36 GiB6.40 GiB6.40 GiB
bench-validator-06validator5.30 GiB6.25 GiB6.29 GiB6.29 GiB
bench-validator-07validator4.93 GiB5.64 GiB5.71 GiB5.71 GiB
bench-validator-08validator5.15 GiB5.98 GiB6.11 GiB6.11 GiB
bench-validator-09validator5.00 GiB5.70 GiB5.80 GiB5.80 GiB
bench-validator-10validator5.67 GiB6.55 GiB6.70 GiB6.70 GiB

telcoin RSS: mean vs peak 60-s mean

One row per node, over the mixed phase.

Memory used

agent memory/percent_used, state=used; summaries over the mixed phase

Table view
NodeClassMeanp95Peak 60-s meanMax
bench-observer-11observer9.3 %10.4 %10.4 %10.4 %
bench-validator-01validator23.5 %24.5 %24.6 %24.6 %
bench-validator-02validator23.5 %24.5 %24.6 %24.6 %
bench-validator-03validator19.1 %19.7 %19.7 %19.7 %
bench-validator-04validator19.6 %20.3 %20.3 %20.3 %
bench-validator-05validator19.8 %27.3 %27.5 %27.5 %
bench-validator-06validator20.6 %21.1 %21.2 %21.2 %
bench-validator-07validator19.6 %19.8 %19.8 %19.8 %
bench-validator-08validator20.6 %21.2 %21.3 %21.3 %
bench-validator-09validator19.9 %20.5 %20.6 %20.6 %
bench-validator-10validator23.1 %24.1 %24.1 %24.1 %

Memory used: mean vs peak 60-s mean

One row per node, over the mixed phase.

Disk

Disk throttling (validator)
not collected –
metric type unavailable in project
Disk performance degraded (validator)
none –
any sample

Disk read + write throughput

instance/disk/{read,write}_bytes_count rate, all devices; summaries over the mixed phase

Table view
NodeClassMeanp95Peak 60-s meanMaxLimit
bench-observer-11observer111.4 MiB/s118.9 MiB/s125.8 MiB/s125.8 MiB/s800.0 MiB/s
bench-validator-01validator120.6 MiB/s124.7 MiB/s126.0 MiB/s126.0 MiB/s800.0 MiB/s
bench-validator-02validator122.3 MiB/s129.2 MiB/s130.1 MiB/s130.1 MiB/s800.0 MiB/s
bench-validator-03validator122.9 MiB/s129.0 MiB/s131.5 MiB/s131.5 MiB/s800.0 MiB/s
bench-validator-04validator122.5 MiB/s128.0 MiB/s129.3 MiB/s129.3 MiB/s800.0 MiB/s
bench-validator-05validator101.2 MiB/s123.2 MiB/s125.9 MiB/s125.9 MiB/s800.0 MiB/s
bench-validator-06validator122.5 MiB/s129.4 MiB/s129.7 MiB/s129.7 MiB/s800.0 MiB/s
bench-validator-07validator122.6 MiB/s128.1 MiB/s128.9 MiB/s128.9 MiB/s800.0 MiB/s
bench-validator-08validator122.8 MiB/s128.2 MiB/s129.4 MiB/s129.4 MiB/s800.0 MiB/s
bench-validator-09validator122.8 MiB/s130.1 MiB/s131.7 MiB/s131.7 MiB/s800.0 MiB/s
bench-validator-10validator122.6 MiB/s129.4 MiB/s131.6 MiB/s131.6 MiB/s800.0 MiB/s

Disk read + write IOPS

instance/disk/{read,write}_ops_count rate, all devices; summaries over the mixed phase

Table view
NodeClassMeanp95Peak 60-s meanMaxLimit
bench-observer-11observer24.6K IOPS27.1K IOPS28.1K IOPS28.1K IOPS50K IOPS
bench-validator-01validator24.5K IOPS26.2K IOPS26.4K IOPS26.4K IOPS50K IOPS
bench-validator-02validator24.7K IOPS26.9K IOPS27.2K IOPS27.2K IOPS50K IOPS
bench-validator-03validator24.7K IOPS26.7K IOPS27.6K IOPS27.6K IOPS50K IOPS
bench-validator-04validator24.7K IOPS26.3K IOPS26.9K IOPS26.9K IOPS50K IOPS
bench-validator-05validator20.4K IOPS26.2K IOPS26.6K IOPS26.6K IOPS50K IOPS
bench-validator-06validator24.7K IOPS26.8K IOPS27.2K IOPS27.2K IOPS50K IOPS
bench-validator-07validator24.7K IOPS26.4K IOPS27.3K IOPS27.3K IOPS50K IOPS
bench-validator-08validator24.7K IOPS26.4K IOPS27K IOPS27K IOPS50K IOPS
bench-validator-09validator24.7K IOPS27.1K IOPS27.4K IOPS27.4K IOPS50K IOPS
bench-validator-10validator24.7K IOPS26.8K IOPS27.6K IOPS27.6K IOPS50K IOPS
Disk limits used
NodeMachinevCPUDiskIOPS limitMiB/s limitBasis
bench-observer-11c3-highcpu-88hyperdisk-balanced 100 GB50K IOPS800.0 MiB/shyperdisk-balanced 100 GB: provisioned 50,000 IOPS, provisioned 800 MiB/s; c3 8 vCPU cap 50,000 IOPS / 800 MiB/s; reads and writes share this budget
bench-validator-01c3-highcpu-88hyperdisk-balanced 100 GB50K IOPS800.0 MiB/shyperdisk-balanced 100 GB: provisioned 50,000 IOPS, provisioned 800 MiB/s; c3 8 vCPU cap 50,000 IOPS / 800 MiB/s; reads and writes share this budget
bench-validator-02c3-highcpu-88hyperdisk-balanced 100 GB50K IOPS800.0 MiB/shyperdisk-balanced 100 GB: provisioned 50,000 IOPS, provisioned 800 MiB/s; c3 8 vCPU cap 50,000 IOPS / 800 MiB/s; reads and writes share this budget
bench-validator-03c3-highcpu-88hyperdisk-balanced 100 GB50K IOPS800.0 MiB/shyperdisk-balanced 100 GB: provisioned 50,000 IOPS, provisioned 800 MiB/s; c3 8 vCPU cap 50,000 IOPS / 800 MiB/s; reads and writes share this budget
bench-validator-04c3-highcpu-88hyperdisk-balanced 100 GB50K IOPS800.0 MiB/shyperdisk-balanced 100 GB: provisioned 50,000 IOPS, provisioned 800 MiB/s; c3 8 vCPU cap 50,000 IOPS / 800 MiB/s; reads and writes share this budget
bench-validator-05c3-highcpu-88hyperdisk-balanced 100 GB50K IOPS800.0 MiB/shyperdisk-balanced 100 GB: provisioned 50,000 IOPS, provisioned 800 MiB/s; c3 8 vCPU cap 50,000 IOPS / 800 MiB/s; reads and writes share this budget
bench-validator-06c3-highcpu-88hyperdisk-balanced 100 GB50K IOPS800.0 MiB/shyperdisk-balanced 100 GB: provisioned 50,000 IOPS, provisioned 800 MiB/s; c3 8 vCPU cap 50,000 IOPS / 800 MiB/s; reads and writes share this budget
bench-validator-07c3-highcpu-88hyperdisk-balanced 100 GB50K IOPS800.0 MiB/shyperdisk-balanced 100 GB: provisioned 50,000 IOPS, provisioned 800 MiB/s; c3 8 vCPU cap 50,000 IOPS / 800 MiB/s; reads and writes share this budget
bench-validator-08c3-highcpu-88hyperdisk-balanced 100 GB50K IOPS800.0 MiB/shyperdisk-balanced 100 GB: provisioned 50,000 IOPS, provisioned 800 MiB/s; c3 8 vCPU cap 50,000 IOPS / 800 MiB/s; reads and writes share this budget
bench-validator-09c3-highcpu-88hyperdisk-balanced 100 GB50K IOPS800.0 MiB/shyperdisk-balanced 100 GB: provisioned 50,000 IOPS, provisioned 800 MiB/s; c3 8 vCPU cap 50,000 IOPS / 800 MiB/s; reads and writes share this budget
bench-validator-10c3-highcpu-88hyperdisk-balanced 100 GB50K IOPS800.0 MiB/shyperdisk-balanced 100 GB: provisioned 50,000 IOPS, provisioned 800 MiB/s; c3 8 vCPU cap 50,000 IOPS / 800 MiB/s; reads and writes share this budget

Network

Network sent

instance/network/sent_bytes_count rate; summaries over the mixed phase

Table view
NodeClassMeanp95Peak 60-s meanMax
bench-observer-11observer1.9 Mbps2.2 Mbps3.5 Mbps3.5 Mbps
bench-validator-01validator125.1 Mbps133.4 Mbps136.0 Mbps136.0 Mbps
bench-validator-02validator126.1 Mbps135.6 Mbps136.2 Mbps136.2 Mbps
bench-validator-03validator134.9 Mbps143.9 Mbps144.6 Mbps144.6 Mbps
bench-validator-04validator129.8 Mbps136.2 Mbps138.1 Mbps138.1 Mbps
bench-validator-05validator104.2 Mbps125.8 Mbps128.5 Mbps128.5 Mbps
bench-validator-06validator124.1 Mbps132.9 Mbps150.0 Mbps150.0 Mbps
bench-validator-07validator134.4 Mbps152.6 Mbps191.9 Mbps191.9 Mbps
bench-validator-08validator132.8 Mbps142.7 Mbps145.1 Mbps145.1 Mbps
bench-validator-09validator131.1 Mbps141.3 Mbps150.8 Mbps150.8 Mbps
bench-validator-10validator126.7 Mbps135.1 Mbps142.7 Mbps142.7 Mbps

Network received

instance/network/received_bytes_count rate; summaries over the mixed phase

Table view
NodeClassMeanp95Peak 60-s meanMax
bench-observer-11observer8.3 Mbps10.5 Mbps30.9 Mbps30.9 Mbps
bench-validator-01validator11.2 Mbps13.4 Mbps17.9 Mbps17.9 Mbps
bench-validator-02validator11.1 Mbps13.8 Mbps14.4 Mbps14.4 Mbps
bench-validator-03validator15.1 Mbps16.9 Mbps18.0 Mbps18.0 Mbps
bench-validator-04validator13.0 Mbps14.9 Mbps17.5 Mbps17.5 Mbps
bench-validator-05validator15.5 Mbps45.7 Mbps90.4 Mbps90.4 Mbps
bench-validator-06validator9.8 Mbps11.6 Mbps14.5 Mbps14.5 Mbps
bench-validator-07validator13.6 Mbps16.2 Mbps17.8 Mbps17.8 Mbps
bench-validator-08validator14.3 Mbps17.5 Mbps17.7 Mbps17.7 Mbps
bench-validator-09validator13.5 Mbps16.7 Mbps19.3 Mbps19.3 Mbps
bench-validator-10validator11.4 Mbps14.9 Mbps19.0 Mbps19.0 Mbps

Consensus health

MetricQueryClassNodesMedian of node meansWorst peak 60-s meanMax
tn_primary_commit_latency_secondsp95validator101,000 ms17.3 s23.8 s
tn_engine_execution_duration_secondsp95validator101,540 ms2,687 ms3,250 ms
tn_engine_execution_duration_secondsp95observer11,274 ms2,616 ms3,188 ms
tn_worker_batch_size_bytesp95validator10235.6 KiB246.4 KiB246.4 KiB
tn_worker_batch_transactionsp95validator94911.48K1.48K
tn_executor_output_transactionsp95validator1010.7K23.8K23.8K
tn_primary_consensus_output_bytesmaxvalidator10513.9 KiB1.5 MiB1.6 MiB
tn_primary_consensus_output_bytesmaxobserver1516.1 KiB1.3 MiB1.9 MiB
tn_engine_queued_outputsmaxvalidator106.5688
tn_engine_queued_outputsmaxobserver11.497.758
tn_primary_roundmaxvalidator101.6K3.39K3.41K
tn_primary_roundmaxobserver11.59K3.32K3.38K
tn_primary_committed_roundmaxvalidator101.6K3.39K3.41K
tn_primary_committed_roundmaxobserver11.59K3.32K3.38K
tn_epoch_currentmaxvalidator104.1755
tn_epoch_currentmaxobserver14.1755
tn_batch_builder_pending_pool_transactionsmaxvalidator105056.69K9.81K
tn_batch_builder_pending_pool_transactionsmaxobserver1000
reth_process_resident_memory_bytesmaxvalidator105.28 GiB6.63 GiB6.75 GiB
reth_process_resident_memory_bytesmaxobserver13.42 GiB4.86 GiB4.90 GiB
tn_primary_subdags_committed_totalratevalidator101.261.441.44
tn_primary_subdags_committed_totalrateobserver1000
tn_primary_round_laglagvalidator102.233.256
tn_primary_round_laglagobserver1000
tn_batch_builder_seal_failures_totalratevalidator80.0004090.01670.0222
tn_node_modemaxvalidator10011
tn_node_modemaxobserver1000
tn_primary_vote_requests_totalratevalidator1022.131.131.4
tn_primary_vote_requests_totalrateobserver1000
tn_primary_votes_created_totalratevalidator102125.925.9
tn_primary_votes_created_totalrateobserver1000
tn_primary_votes_recast_totalratevalidator1000.1670.222
tn_primary_votes_recast_totalrateobserver1000
tn_primary_vote_requests_too_new_totalratevalidator10000
tn_primary_vote_requests_too_new_totalrateobserver1000
tn_primary_vote_timeouts_total{stage="exec_wait"}ratevalidator10000
tn_primary_vote_timeouts_total{stage="exec_wait"}rateobserver1000
tn_primary_vote_timeouts_total{stage="lead_sleep"}ratevalidator10000
tn_primary_vote_timeouts_total{stage="lead_sleep"}rateobserver1000
tn_primary_vote_timeouts_total{stage="parents"}ratevalidator10000
tn_primary_vote_timeouts_total{stage="parents"}rateobserver1000
tn_primary_vote_timeouts_total{stage="persist"}ratevalidator10000
tn_primary_vote_timeouts_total{stage="persist"}rateobserver1000
tn_primary_vote_timeouts_total{stage="prelock_wait"}ratevalidator10000
tn_primary_vote_timeouts_total{stage="prelock_wait"}rateobserver1000
tn_primary_vote_timeouts_total{stage="sync_batches"}ratevalidator10000
tn_primary_vote_timeouts_total{stage="sync_batches"}rateobserver1000
tn_primary_vote_timeouts_total{stage="validate"}ratevalidator10000
tn_primary_vote_timeouts_total{stage="validate"}rateobserver1000
tn_primary_vote_timeouts_total{stage="vote_store"}ratevalidator10000
tn_primary_vote_timeouts_total{stage="vote_store"}rateobserver1000
tn_primary_vote_stage_seconds{stage="exec_wait"}p95validator100 ms0 ms0 ms
tn_primary_vote_stage_seconds{stage="lead_sleep"}p95validator100 ms0 ms0 ms
tn_primary_vote_stage_seconds{stage="lock"}p95validator100 ms0 ms0 ms
tn_primary_vote_stage_seconds{stage="parents"}p95validator101 ms2 ms2 ms
tn_primary_vote_stage_seconds{stage="persist"}p95validator103 ms5 ms5 ms
tn_primary_vote_stage_seconds{stage="prelock_wait"}p95validator10610 ms2,425 ms2,425 ms
tn_primary_vote_stage_seconds{stage="sync_batches"}p95validator100 ms0 ms0 ms
tn_primary_vote_stage_seconds{stage="validate"}p95validator103 ms19 ms19 ms
tn_primary_vote_stage_seconds{stage="vote_store"}p95validator101 ms2 ms2 ms
tn_primary_leader_rounds_total{leader="47oQDzXUa9Kv8fzSat8zoggtm4MsQLWEiS8FG8u1HPff",swapped="false",found="false"}ratevalidator10000
tn_primary_leader_rounds_total{leader="47oQDzXUa9Kv8fzSat8zoggtm4MsQLWEiS8FG8u1HPff",swapped="false",found="true"}ratevalidator100.1280.1560.156
tn_primary_leader_rounds_total{leader="47oQDzXUa9Kv8fzSat8zoggtm4MsQLWEiS8FG8u1HPff",swapped="true",found="false"}ratevalidator10000
tn_primary_leader_rounds_total{leader="47oQDzXUa9Kv8fzSat8zoggtm4MsQLWEiS8FG8u1HPff",swapped="true",found="true"}ratevalidator100.04890.1060.156
tn_primary_leader_rounds_total{leader="4W4ApMvFcM9ipf73ecsU25MbSrzHu5m9kqJE5A17jT5e",swapped="false",found="false"}ratevalidator100.001190.02150.0222
tn_primary_leader_rounds_total{leader="4W4ApMvFcM9ipf73ecsU25MbSrzHu5m9kqJE5A17jT5e",swapped="false",found="true"}ratevalidator100.1260.1560.156
tn_primary_leader_rounds_total{leader="4W4ApMvFcM9ipf73ecsU25MbSrzHu5m9kqJE5A17jT5e",swapped="true",found="false"}ratevalidator100.0007830.01670.0222
tn_primary_leader_rounds_total{leader="4W4ApMvFcM9ipf73ecsU25MbSrzHu5m9kqJE5A17jT5e",swapped="true",found="true"}ratevalidator100.03930.10.133
tn_primary_leader_rounds_total{leader="58pEnTaq2dYqsp6y1gGHu4wf5S9Ux6sJQepAbKcqR4qz",swapped="false",found="false"}ratevalidator10000
tn_primary_leader_rounds_total{leader="58pEnTaq2dYqsp6y1gGHu4wf5S9Ux6sJQepAbKcqR4qz",swapped="false",found="true"}ratevalidator100.06380.1330.156
tn_primary_leader_rounds_total{leader="58pEnTaq2dYqsp6y1gGHu4wf5S9Ux6sJQepAbKcqR4qz",swapped="true",found="false"}ratevalidator10000
tn_primary_leader_rounds_total{leader="58pEnTaq2dYqsp6y1gGHu4wf5S9Ux6sJQepAbKcqR4qz",swapped="true",found="true"}ratevalidator100.001160.04420.066
tn_primary_leader_rounds_total{leader="5RAEhJabcANXvwckXuhzdDJXWnvTQjwCZ7fESk2FfB1r",swapped="false",found="false"}ratevalidator10000
tn_primary_leader_rounds_total{leader="5RAEhJabcANXvwckXuhzdDJXWnvTQjwCZ7fESk2FfB1r",swapped="false",found="true"}ratevalidator100.00450.1060.133
tn_primary_leader_rounds_total{leader="5RAEhJabcANXvwckXuhzdDJXWnvTQjwCZ7fESk2FfB1r",swapped="true",found="false"}ratevalidator10000
tn_primary_leader_rounds_total{leader="5RAEhJabcANXvwckXuhzdDJXWnvTQjwCZ7fESk2FfB1r",swapped="true",found="true"}ratevalidator100.0003580.01650.022
tn_primary_leader_rounds_total{leader="5d3ZtLyzWxL75opPW68CCqBm1sMEY1KdwV26FRuGfDUw",swapped="false",found="false"}ratevalidator10000
tn_primary_leader_rounds_total{leader="5d3ZtLyzWxL75opPW68CCqBm1sMEY1KdwV26FRuGfDUw",swapped="false",found="true"}ratevalidator100.1270.1560.156
tn_primary_leader_rounds_total{leader="5d3ZtLyzWxL75opPW68CCqBm1sMEY1KdwV26FRuGfDUw",swapped="true",found="false"}ratevalidator10000
tn_primary_leader_rounds_total{leader="5d3ZtLyzWxL75opPW68CCqBm1sMEY1KdwV26FRuGfDUw",swapped="true",found="true"}ratevalidator100.04010.1110.133
tn_primary_leader_rounds_total{leader="5txV47P6jAHi1Uvm6sfy7osyiYRzGo3uJoX1SfaVq3n6",swapped="false",found="false"}ratevalidator100.004090.07780.111
tn_primary_leader_rounds_total{leader="5txV47P6jAHi1Uvm6sfy7osyiYRzGo3uJoX1SfaVq3n6",swapped="false",found="true"}ratevalidator100.06380.150.156
tn_primary_leader_rounds_total{leader="5txV47P6jAHi1Uvm6sfy7osyiYRzGo3uJoX1SfaVq3n6",swapped="true",found="false"}ratevalidator100.0007340.0330.044
tn_primary_leader_rounds_total{leader="5txV47P6jAHi1Uvm6sfy7osyiYRzGo3uJoX1SfaVq3n6",swapped="true",found="true"}ratevalidator100.01510.1220.133
tn_primary_leader_rounds_total{leader="A7xHY55JTpwRz2z2wPD4V6axQajsVkGJA1oG9AUjo5W1",swapped="false",found="false"}ratevalidator100.0003580.01650.022
tn_primary_leader_rounds_total{leader="A7xHY55JTpwRz2z2wPD4V6axQajsVkGJA1oG9AUjo5W1",swapped="false",found="true"}ratevalidator100.1240.1560.156
tn_primary_leader_rounds_total{leader="A7xHY55JTpwRz2z2wPD4V6axQajsVkGJA1oG9AUjo5W1",swapped="true",found="false"}ratevalidator10000
tn_primary_leader_rounds_total{leader="A7xHY55JTpwRz2z2wPD4V6axQajsVkGJA1oG9AUjo5W1",swapped="true",found="true"}ratevalidator100.04650.1390.178
tn_primary_leader_rounds_total{leader="Ab1dyJbmy75Q7dYxNn9RtQW2maYpA8FYz6i8MaX8RKfq",swapped="false",found="false"}ratevalidator10000
tn_primary_leader_rounds_total{leader="Ab1dyJbmy75Q7dYxNn9RtQW2maYpA8FYz6i8MaX8RKfq",swapped="false",found="true"}ratevalidator100.1270.1560.156
tn_primary_leader_rounds_total{leader="Ab1dyJbmy75Q7dYxNn9RtQW2maYpA8FYz6i8MaX8RKfq",swapped="true",found="false"}ratevalidator10000
tn_primary_leader_rounds_total{leader="Ab1dyJbmy75Q7dYxNn9RtQW2maYpA8FYz6i8MaX8RKfq",swapped="true",found="true"}ratevalidator100.0390.1170.156
tn_primary_leader_rounds_total{leader="CaobP5Gbf4YAuKj3qEbhtJu4FQ6MfmMuGJKPUocjqwZZ",swapped="false",found="false"}ratevalidator10000
tn_primary_leader_rounds_total{leader="CaobP5Gbf4YAuKj3qEbhtJu4FQ6MfmMuGJKPUocjqwZZ",swapped="false",found="true"}ratevalidator100.1270.1560.156
tn_primary_leader_rounds_total{leader="CaobP5Gbf4YAuKj3qEbhtJu4FQ6MfmMuGJKPUocjqwZZ",swapped="true",found="false"}ratevalidator10000
tn_primary_leader_rounds_total{leader="CaobP5Gbf4YAuKj3qEbhtJu4FQ6MfmMuGJKPUocjqwZZ",swapped="true",found="true"}ratevalidator100.04810.1330.178
tn_primary_leader_rounds_total{leader="CioqH7gHDTeNpghq5o2w8AZyRjNpheFWsRQqEQCMy6J3",swapped="false",found="false"}ratevalidator10000
tn_primary_leader_rounds_total{leader="CioqH7gHDTeNpghq5o2w8AZyRjNpheFWsRQqEQCMy6J3",swapped="false",found="true"}ratevalidator100.07280.1560.156
tn_primary_leader_rounds_total{leader="CioqH7gHDTeNpghq5o2w8AZyRjNpheFWsRQqEQCMy6J3",swapped="true",found="false"}ratevalidator10000
tn_primary_leader_rounds_total{leader="CioqH7gHDTeNpghq5o2w8AZyRjNpheFWsRQqEQCMy6J3",swapped="true",found="true"}ratevalidator100.0210.08890.111
tn_primary_leader_commits_total{leader="47oQDzXUa9Kv8fzSat8zoggtm4MsQLWEiS8FG8u1HPff"}ratevalidator100.1770.2560.311
tn_primary_leader_commits_total{leader="4W4ApMvFcM9ipf73ecsU25MbSrzHu5m9kqJE5A17jT5e"}ratevalidator100.1650.2440.289
tn_primary_leader_commits_total{leader="58pEnTaq2dYqsp6y1gGHu4wf5S9Ux6sJQepAbKcqR4qz"}ratevalidator100.0650.1670.2
tn_primary_leader_commits_total{leader="5RAEhJabcANXvwckXuhzdDJXWnvTQjwCZ7fESk2FfB1r"}ratevalidator100.004910.1220.156
tn_primary_leader_commits_total{leader="5d3ZtLyzWxL75opPW68CCqBm1sMEY1KdwV26FRuGfDUw"}ratevalidator100.1670.2670.267
tn_primary_leader_commits_total{leader="5txV47P6jAHi1Uvm6sfy7osyiYRzGo3uJoX1SfaVq3n6"}ratevalidator100.07890.250.267
tn_primary_leader_commits_total{leader="A7xHY55JTpwRz2z2wPD4V6axQajsVkGJA1oG9AUjo5W1"}ratevalidator100.170.2720.311
tn_primary_leader_commits_total{leader="Ab1dyJbmy75Q7dYxNn9RtQW2maYpA8FYz6i8MaX8RKfq"}ratevalidator100.1660.250.267
tn_primary_leader_commits_total{leader="CaobP5Gbf4YAuKj3qEbhtJu4FQ6MfmMuGJKPUocjqwZZ"}ratevalidator100.1750.2590.289
tn_primary_leader_commits_total{leader="CioqH7gHDTeNpghq5o2w8AZyRjNpheFWsRQqEQCMy6J3"}ratevalidator100.09380.2220.244
tn_engine_queue_wait_secondsp95validator108,008 ms21.5 s25.2 s
tn_engine_queue_wait_secondsp95observer12,128 ms9,697 ms9,750 ms
tn_engine_persist_duration_secondsp95validator10474 ms914 ms927 ms
tn_engine_persist_duration_secondsp95observer1465 ms677 ms750 ms
tn_engine_busy_milliseconds_totalratevalidator109861.01K1.05K
tn_engine_busy_milliseconds_totalrateobserver19041K1.01K
tn_reth_persist_save_blocks_secondsp95validator10250 ms476 ms650 ms
tn_reth_persist_save_blocks_secondsp95observer1236 ms346 ms400 ms
tn_reth_persist_commit_secondsp95validator10250 ms467 ms472 ms
tn_reth_persist_commit_secondsp95observer1251 ms370 ms425 ms
tn_reth_persisted_accounts_totalratevalidator102.99K7.47K7.47K
tn_reth_persisted_accounts_totalrateobserver12.96K7.35K7.35K
tn_reth_persisted_storage_slots_totalratevalidator103.97K4.6K4.68K
tn_reth_persisted_storage_slots_totalrateobserver13.96K4.57K4.68K
tn_node_mode_transitions_total{to="cvv_active",reason="rejoin"}ratevalidator1000
tn_node_mode_transitions_total{to="cvv_inactive",reason="epoch_behind"}ratevalidator1000
tn_executor_rejoin_blocked_total{reason="attempts_exhausted"}ratevalidator1000
tn_executor_rejoin_blocked_total{reason="epoch_closed"}ratevalidator10.06791.111.22
tn_executor_rejoin_blocked_total{reason="epoch_mismatch"}ratevalidator1000
tn_executor_rejoin_blocked_total{reason="no_known_header"}ratevalidator1000
tn_executor_rejoin_blocked_total{reason="round_gap"}ratevalidator10.2671.231.29
tn_executor_rejoin_blocked_total{reason="stale_head"}ratevalidator10.006690.1870.25
tn_worker_sync_serves_denied_total{worker="0",tier="committee"}ratevalidator10000
tn_worker_sync_serves_denied_total{worker="0",tier="committee"}rateobserver1000
tn_worker_sync_serves_denied_total{worker="0",tier="public"}ratevalidator10000
tn_worker_sync_serves_denied_total{worker="0",tier="public"}rateobserver1000
tn_worker_batch_requests_denied_total{worker="0",peer="n2PTTgCQ5BpVPV1scTWG7eH97rTajbHamJHqwT3sEVD78UHvnZWQh6CdJVdxS4XYWNCPNh3WX3k35bZ8Z4auPmiXSw8jcJJCC6zmzFCyD8qD7AnsDvCsayweBC1zHvusPmr"}ratevalidator30.009521.52
tn_worker_batch_requests_denied_total{worker="0",peer="ojnjgb2wFyUjzTxydgjVuYz9NZudiyZ42HAYhHfzh7xRwtafTJu6LGAPypt9rS7KzL3r3eApuCdrwd5HL9z3xxV2EFojLfXH4rFBy47GLnE3z4wP39whwX1D7bvqMw12skN"}ratevalidator30.01111.632.18
tn_worker_batch_requests_denied_total{worker="0",peer="other"}ratevalidator2000
tn_worker_batch_requests_denied_total{worker="0",peer="pxKG6xH2cxvStH15MW9fhJYABPBvMw4Mzv5WdARz8DrhLL5fFc9E5yZWx2sAJv9cYY4uw5jLYSScFRbeqUd1R6BTXEnuRAqNHaLQM61WYKnviDtsYgb3UvRYh1JJuFvzJ2W"}ratevalidator300.150.2
tn_worker_batch_requests_denied_total{worker="0",peer="s7bPgoys5Lf5DKaLhZiybr6SiogH78hCRgsR5fDdrYxunDjimFR7zJDcVcNcg2LiwQGGRxt8sNzLksasm5pZY1bhEQsCCagXrvm7wC4faTiy26fkNR1h7tM2Fd7tgqHMmCX"}ratevalidator30.01461.121.49
tn_worker_batch_requests_denied_total{worker="0",peer="sPUdopQ9x1BRbEWwNBTFKiXt585Spgmm14qGd1SSsydpo6KvhoqQJ5ekYsbcu7oH6sXNkwanteVtSWHRtnvzayU46qdeTXPZEaCCkbpiNYvjrFHNrKchLYNtZ1ruGoy4CbG"}ratevalidator40.006981.72.27
tn_worker_batch_requests_denied_total{worker="0",peer="sbQYUVCPr4Hv1MMVSNqzBjRPnpVkeu4TgdJubvXCKA5wND5Gia4SDQafJfyqU2tiWotf4wLKZEkvtnPasZbcyc47SNfYz5Pv4y2HieAtHRfNrsLmQMjXKn75cX99jg6t9us"}ratevalidator40.00961.532.04
tn_worker_batch_requests_denied_total{worker="0",peer="tMRJcY2rYHeGbizi7Mx7VfksKVpzpPojBcxjX46rbZChTHiDHmk9URo6uCcvVid1gnGmnDhHo1CffUxeBcpmDWsNGnVz6u77CtToKv3SFhKNS2vdj2L7wDE6uJcgtQoDWSg"}ratevalidator30.01141.572.09
tn_worker_batch_requests_denied_total{worker="0",peer="uV5rFtrqpQUa9JxdCDXjDgterkWBakX6JARNqcwKZ5H4anwJ2TbhVTs3JPaCvKqckRecBvx3c3xFWd98SbzHTpBXUk1whCVDFPpZYB7CFCukGD15EtnVQW3XmN5Jvo4kd6X"}ratevalidator30.008251.151.53
tn_worker_batch_requests_denied_total{worker="0",peer="uWmSKMfEyTpK6gTwGDiPEFqpSUA77vbPFzed26SjRnVTiLvvZhMHD8XdviFYpmKAeaSYk2sbG3SGyZxLmA9bEyWTS5zVFhjhrkPfQzzKDbTY8oyUc8LfSZ7bujYACmuty5T"}ratevalidator40.01840.6670.889
tn_worker_batch_requests_denied_total{worker="0",peer="ye6GqWBvRQ7dJr6rdJRTo3WrcBbkW696U7dx3AngZLgz8EZmqkbWbQv5fUxKKy7B42pveckCDzbFTxuw6RqhckyGRP87R7L1yH49BV1Krd3mfU7z6p6to6wkBUqmRCxdeiT"}ratevalidator300.2830.378
tn_worker_batch_requests_failed_total{worker="0",peer="n2PTTgCQ5BpVPV1scTWG7eH97rTajbHamJHqwT3sEVD78UHvnZWQh6CdJVdxS4XYWNCPNh3WX3k35bZ8Z4auPmiXSw8jcJJCC6zmzFCyD8qD7AnsDvCsayweBC1zHvusPmr"}ratevalidator30.0006350.4170.556
tn_worker_batch_requests_failed_total{worker="0",peer="ojnjgb2wFyUjzTxydgjVuYz9NZudiyZ42HAYhHfzh7xRwtafTJu6LGAPypt9rS7KzL3r3eApuCdrwd5HL9z3xxV2EFojLfXH4rFBy47GLnE3z4wP39whwX1D7bvqMw12skN"}ratevalidator30.004040.5330.711
tn_worker_batch_requests_failed_total{worker="0",peer="other"}ratevalidator2000
tn_worker_batch_requests_failed_total{worker="0",peer="pxKG6xH2cxvStH15MW9fhJYABPBvMw4Mzv5WdARz8DrhLL5fFc9E5yZWx2sAJv9cYY4uw5jLYSScFRbeqUd1R6BTXEnuRAqNHaLQM61WYKnviDtsYgb3UvRYh1JJuFvzJ2W"}ratevalidator300.150.2
tn_worker_batch_requests_failed_total{worker="0",peer="s7bPgoys5Lf5DKaLhZiybr6SiogH78hCRgsR5fDdrYxunDjimFR7zJDcVcNcg2LiwQGGRxt8sNzLksasm5pZY1bhEQsCCagXrvm7wC4faTiy26fkNR1h7tM2Fd7tgqHMmCX"}ratevalidator30.001270.751
tn_worker_batch_requests_failed_total{worker="0",peer="sPUdopQ9x1BRbEWwNBTFKiXt585Spgmm14qGd1SSsydpo6KvhoqQJ5ekYsbcu7oH6sXNkwanteVtSWHRtnvzayU46qdeTXPZEaCCkbpiNYvjrFHNrKchLYNtZ1ruGoy4CbG"}ratevalidator40.002540.4830.644
tn_worker_batch_requests_failed_total{worker="0",peer="sbQYUVCPr4Hv1MMVSNqzBjRPnpVkeu4TgdJubvXCKA5wND5Gia4SDQafJfyqU2tiWotf4wLKZEkvtnPasZbcyc47SNfYz5Pv4y2HieAtHRfNrsLmQMjXKn75cX99jg6t9us"}ratevalidator40.005560.350.467
tn_worker_batch_requests_failed_total{worker="0",peer="tMRJcY2rYHeGbizi7Mx7VfksKVpzpPojBcxjX46rbZChTHiDHmk9URo6uCcvVid1gnGmnDhHo1CffUxeBcpmDWsNGnVz6u77CtToKv3SFhKNS2vdj2L7wDE6uJcgtQoDWSg"}ratevalidator30.004440.3330.444
tn_worker_batch_requests_failed_total{worker="0",peer="uV5rFtrqpQUa9JxdCDXjDgterkWBakX6JARNqcwKZ5H4anwJ2TbhVTs3JPaCvKqckRecBvx3c3xFWd98SbzHTpBXUk1whCVDFPpZYB7CFCukGD15EtnVQW3XmN5Jvo4kd6X"}ratevalidator30.00190.7671.02
tn_worker_batch_requests_failed_total{worker="0",peer="uWmSKMfEyTpK6gTwGDiPEFqpSUA77vbPFzed26SjRnVTiLvvZhMHD8XdviFYpmKAeaSYk2sbG3SGyZxLmA9bEyWTS5zVFhjhrkPfQzzKDbTY8oyUc8LfSZ7bujYACmuty5T"}ratevalidator40.003171.091.44
tn_worker_batch_requests_failed_total{worker="0",peer="ye6GqWBvRQ7dJr6rdJRTo3WrcBbkW696U7dx3AngZLgz8EZmqkbWbQv5fUxKKy7B42pveckCDzbFTxuw6RqhckyGRP87R7L1yH49BV1Krd3mfU7z6p6to6wkBUqmRCxdeiT"}ratevalidator3000
tn_node_epoch_record_signersmaxvalidator107.171010
tn_node_epoch_record_signersmaxobserver17.288
tn_node_epoch_record_missing_signers_total{signer="47oQDzXUa9Kv8fzSat8zoggtm4MsQLWEiS8FG8u1HPff"}ratevalidator10000
tn_node_epoch_record_missing_signers_total{signer="47oQDzXUa9Kv8fzSat8zoggtm4MsQLWEiS8FG8u1HPff"}rateobserver1000
tn_node_epoch_record_missing_signers_total{signer="4W4ApMvFcM9ipf73ecsU25MbSrzHu5m9kqJE5A17jT5e"}ratevalidator10000
tn_node_epoch_record_missing_signers_total{signer="4W4ApMvFcM9ipf73ecsU25MbSrzHu5m9kqJE5A17jT5e"}rateobserver1000
tn_node_epoch_record_missing_signers_total{signer="58pEnTaq2dYqsp6y1gGHu4wf5S9Ux6sJQepAbKcqR4qz"}ratevalidator100.0003560.01650.022
tn_node_epoch_record_missing_signers_total{signer="58pEnTaq2dYqsp6y1gGHu4wf5S9Ux6sJQepAbKcqR4qz"}rateobserver10.0003770.01370.0182
tn_node_epoch_record_missing_signers_total{signer="5RAEhJabcANXvwckXuhzdDJXWnvTQjwCZ7fESk2FfB1r"}ratevalidator10000
tn_node_epoch_record_missing_signers_total{signer="5RAEhJabcANXvwckXuhzdDJXWnvTQjwCZ7fESk2FfB1r"}rateobserver1000
tn_node_epoch_record_missing_signers_total{signer="5d3ZtLyzWxL75opPW68CCqBm1sMEY1KdwV26FRuGfDUw"}ratevalidator10000
tn_node_epoch_record_missing_signers_total{signer="5d3ZtLyzWxL75opPW68CCqBm1sMEY1KdwV26FRuGfDUw"}rateobserver1000
tn_node_epoch_record_missing_signers_total{signer="5txV47P6jAHi1Uvm6sfy7osyiYRzGo3uJoX1SfaVq3n6"}ratevalidator100.0008180.01670.0222
tn_node_epoch_record_missing_signers_total{signer="5txV47P6jAHi1Uvm6sfy7osyiYRzGo3uJoX1SfaVq3n6"}rateobserver10.000920.01670.0222
tn_node_epoch_record_missing_signers_total{signer="A7xHY55JTpwRz2z2wPD4V6axQajsVkGJA1oG9AUjo5W1"}ratevalidator100.0008180.01670.0222
tn_node_epoch_record_missing_signers_total{signer="A7xHY55JTpwRz2z2wPD4V6axQajsVkGJA1oG9AUjo5W1"}rateobserver10.000920.01670.0222
tn_node_epoch_record_missing_signers_total{signer="Ab1dyJbmy75Q7dYxNn9RtQW2maYpA8FYz6i8MaX8RKfq"}ratevalidator1000.01330.0177
tn_node_epoch_record_missing_signers_total{signer="Ab1dyJbmy75Q7dYxNn9RtQW2maYpA8FYz6i8MaX8RKfq"}rateobserver1000
tn_node_epoch_record_missing_signers_total{signer="CaobP5Gbf4YAuKj3qEbhtJu4FQ6MfmMuGJKPUocjqwZZ"}ratevalidator1000.01540.0206
tn_node_epoch_record_missing_signers_total{signer="CaobP5Gbf4YAuKj3qEbhtJu4FQ6MfmMuGJKPUocjqwZZ"}rateobserver10.0003770.01370.0182
tn_node_epoch_record_missing_signers_total{signer="CioqH7gHDTeNpghq5o2w8AZyRjNpheFWsRQqEQCMy6J3"}ratevalidator1000.01580.021
tn_node_epoch_record_missing_signers_total{signer="CioqH7gHDTeNpghq5o2w8AZyRjNpheFWsRQqEQCMy6J3"}rateobserver1000
tn_network_outbound_request_duration_seconds{outcome="failure",class="batch"}p95validator918.2 s29.0 s29.0 s
tn_network_outbound_request_duration_seconds{outcome="failure",class="epoch_record"}p95validator414.5 s29.0 s29.0 s
tn_network_outbound_request_duration_seconds{outcome="failure",class="epoch_record"}p95observer114.5 s29.0 s29.0 s
tn_network_outbound_request_duration_seconds{outcome="failure",class="vote"}p95validator19,506 ms29.0 s29.0 s
tn_network_outbound_request_duration_seconds{outcome="response",class="batch"}p95validator4448 ms847 ms847 ms
tn_network_outbound_request_duration_seconds{outcome="response",class="epoch_record"}p95validator3437 ms479 ms481 ms
tn_network_outbound_request_duration_seconds{outcome="response",class="vote"}p95validator61,993 ms2,381 ms2,384 ms
tn_network_closed_admission_denied_total{network="bench",direction="in"}ratevalidator10000
tn_network_closed_admission_denied_total{network="bench",direction="in"}rateobserver1000
tn_network_closed_admission_denied_total{network="bench",direction="out"}ratevalidator10000
tn_network_closed_admission_denied_total{network="bench",direction="out"}rateobserver1000
tn_network_source_admission_denied_total{network="bench",direction="in",reason="address_full"}ratevalidator10000
tn_network_source_admission_denied_total{network="bench",direction="in",reason="address_full"}rateobserver1000
tn_network_source_admission_denied_total{network="bench",direction="in",reason="duplicate_connection"}ratevalidator10000
tn_network_source_admission_denied_total{network="bench",direction="in",reason="duplicate_connection"}rateobserver1000
tn_network_source_admission_denied_total{network="bench",direction="in",reason="invalid_limits"}ratevalidator10000
tn_network_source_admission_denied_total{network="bench",direction="in",reason="invalid_limits"}rateobserver1000
tn_network_source_admission_denied_total{network="bench",direction="in",reason="peer_full"}ratevalidator10000
tn_network_source_admission_denied_total{network="bench",direction="in",reason="peer_full"}rateobserver1000
tn_network_source_admission_denied_total{network="bench",direction="in",reason="poisoned"}ratevalidator10000
tn_network_source_admission_denied_total{network="bench",direction="in",reason="poisoned"}rateobserver1000
tn_network_source_admission_denied_total{network="bench",direction="in",reason="prefix_full"}ratevalidator10000
tn_network_source_admission_denied_total{network="bench",direction="in",reason="prefix_full"}rateobserver1000
tn_network_source_admission_denied_total{network="bench",direction="in",reason="process_full"}ratevalidator10000
tn_network_source_admission_denied_total{network="bench",direction="in",reason="process_full"}rateobserver1000
tn_network_source_admission_denied_total{network="bench",direction="in",reason="sources_full"}ratevalidator10000
tn_network_source_admission_denied_total{network="bench",direction="in",reason="sources_full"}rateobserver1000
tn_network_source_admission_denied_total{network="bench",direction="in",reason="unsupported_address"}ratevalidator10000
tn_network_source_admission_denied_total{network="bench",direction="in",reason="unsupported_address"}rateobserver1000
tn_network_source_admission_denied_total{network="bench",direction="out",reason="address_full"}ratevalidator10000
tn_network_source_admission_denied_total{network="bench",direction="out",reason="address_full"}rateobserver1000
tn_network_source_admission_denied_total{network="bench",direction="out",reason="duplicate_connection"}ratevalidator10000
tn_network_source_admission_denied_total{network="bench",direction="out",reason="duplicate_connection"}rateobserver1000
tn_network_source_admission_denied_total{network="bench",direction="out",reason="invalid_limits"}ratevalidator10000
tn_network_source_admission_denied_total{network="bench",direction="out",reason="invalid_limits"}rateobserver1000
tn_network_source_admission_denied_total{network="bench",direction="out",reason="peer_full"}ratevalidator10000
tn_network_source_admission_denied_total{network="bench",direction="out",reason="peer_full"}rateobserver1000
tn_network_source_admission_denied_total{network="bench",direction="out",reason="poisoned"}ratevalidator10000
tn_network_source_admission_denied_total{network="bench",direction="out",reason="poisoned"}rateobserver1000
tn_network_source_admission_denied_total{network="bench",direction="out",reason="prefix_full"}ratevalidator10000
tn_network_source_admission_denied_total{network="bench",direction="out",reason="prefix_full"}rateobserver1000
tn_network_source_admission_denied_total{network="bench",direction="out",reason="process_full"}ratevalidator10000
tn_network_source_admission_denied_total{network="bench",direction="out",reason="process_full"}rateobserver1000
tn_network_source_admission_denied_total{network="bench",direction="out",reason="sources_full"}ratevalidator10000
tn_network_source_admission_denied_total{network="bench",direction="out",reason="sources_full"}rateobserver1000
tn_network_source_admission_denied_total{network="bench",direction="out",reason="unsupported_address"}ratevalidator10000
tn_network_source_admission_denied_total{network="bench",direction="out",reason="unsupported_address"}rateobserver1000
tn_network_admission_modemaxvalidator10222
tn_network_admission_modemaxobserver1222
tn_network_admission_fallbackmaxvalidator10011
tn_network_admission_fallbackmaxobserver1000
tn_network_outbound_request_failures_total{kind="connection"}ratevalidator1002.823.99
tn_network_outbound_request_failures_total{kind="connection"}rateobserver1000
tn_network_outbound_request_failures_total{kind="dial"}ratevalidator101295.5138
tn_network_outbound_request_failures_total{kind="dial"}rateobserver1000
tn_network_outbound_request_failures_total{kind="io"}ratevalidator101.9515.616.8
tn_network_outbound_request_failures_total{kind="io"}rateobserver10.4881.091.38
tn_network_outbound_request_failures_total{kind="timeout"}ratevalidator100.4185.25.96
tn_network_outbound_request_failures_total{kind="timeout"}rateobserver10.002830.02020.0222
tn_network_outbound_request_failures_total{kind="unsupported"}ratevalidator10000
tn_network_outbound_request_failures_total{kind="unsupported"}rateobserver1000
tn_network_inbound_requests_shed_total{class="batch",reason="admission"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="batch",reason="admission"}rateobserver1000
tn_network_inbound_requests_shed_total{class="batch",reason="queue_budget"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="batch",reason="queue_budget"}rateobserver1000
tn_network_inbound_requests_shed_total{class="batch",reason="queue_full"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="batch",reason="queue_full"}rateobserver1000
tn_network_inbound_requests_shed_total{class="batch",reason="serve_budget"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="batch",reason="serve_budget"}rateobserver1000
tn_network_inbound_requests_shed_total{class="batch",reason="unsubscribed"}ratevalidator100.01456.782.6
tn_network_inbound_requests_shed_total{class="batch",reason="unsubscribed"}rateobserver1000
tn_network_inbound_requests_shed_total{class="certificate_sync",reason="admission"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="certificate_sync",reason="admission"}rateobserver1000
tn_network_inbound_requests_shed_total{class="certificate_sync",reason="queue_budget"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="certificate_sync",reason="queue_budget"}rateobserver1000
tn_network_inbound_requests_shed_total{class="certificate_sync",reason="queue_full"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="certificate_sync",reason="queue_full"}rateobserver1000
tn_network_inbound_requests_shed_total{class="certificate_sync",reason="serve_budget"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="certificate_sync",reason="serve_budget"}rateobserver1000
tn_network_inbound_requests_shed_total{class="certificate_sync",reason="unsubscribed"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="certificate_sync",reason="unsubscribed"}rateobserver1000
tn_network_inbound_requests_shed_total{class="epoch_record",reason="admission"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="epoch_record",reason="admission"}rateobserver1000
tn_network_inbound_requests_shed_total{class="epoch_record",reason="queue_budget"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="epoch_record",reason="queue_budget"}rateobserver1000
tn_network_inbound_requests_shed_total{class="epoch_record",reason="queue_full"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="epoch_record",reason="queue_full"}rateobserver1000
tn_network_inbound_requests_shed_total{class="epoch_record",reason="serve_budget"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="epoch_record",reason="serve_budget"}rateobserver1000
tn_network_inbound_requests_shed_total{class="epoch_record",reason="unsubscribed"}ratevalidator100.3951.081.4
tn_network_inbound_requests_shed_total{class="epoch_record",reason="unsubscribed"}rateobserver10.1830.4990.62
tn_network_inbound_requests_shed_total{class="gossip",reason="admission"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="gossip",reason="admission"}rateobserver1000
tn_network_inbound_requests_shed_total{class="gossip",reason="queue_budget"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="gossip",reason="queue_budget"}rateobserver1000
tn_network_inbound_requests_shed_total{class="gossip",reason="queue_full"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="gossip",reason="queue_full"}rateobserver1000
tn_network_inbound_requests_shed_total{class="gossip",reason="serve_budget"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="gossip",reason="serve_budget"}rateobserver1000
tn_network_inbound_requests_shed_total{class="gossip",reason="unsubscribed"}ratevalidator100.0130.1330.176
tn_network_inbound_requests_shed_total{class="gossip",reason="unsubscribed"}rateobserver10.01290.02930.0364
tn_network_inbound_requests_shed_total{class="other",reason="admission"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="other",reason="admission"}rateobserver1000
tn_network_inbound_requests_shed_total{class="other",reason="queue_budget"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="other",reason="queue_budget"}rateobserver1000
tn_network_inbound_requests_shed_total{class="other",reason="queue_full"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="other",reason="queue_full"}rateobserver1000
tn_network_inbound_requests_shed_total{class="other",reason="serve_budget"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="other",reason="serve_budget"}rateobserver1000
tn_network_inbound_requests_shed_total{class="other",reason="unsubscribed"}ratevalidator100.01990.2240.288
tn_network_inbound_requests_shed_total{class="other",reason="unsubscribed"}rateobserver1000
tn_network_inbound_requests_shed_total{class="vote",reason="admission"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="vote",reason="admission"}rateobserver1000
tn_network_inbound_requests_shed_total{class="vote",reason="queue_budget"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="vote",reason="queue_budget"}rateobserver1000
tn_network_inbound_requests_shed_total{class="vote",reason="queue_full"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="vote",reason="queue_full"}rateobserver1000
tn_network_inbound_requests_shed_total{class="vote",reason="serve_budget"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="vote",reason="serve_budget"}rateobserver1000
tn_network_inbound_requests_shed_total{class="vote",reason="unsubscribed"}ratevalidator100.25921.831.7
tn_network_inbound_requests_shed_total{class="vote",reason="unsubscribed"}rateobserver1000
tn_network_inbound_request_service_seconds{class="batch"}p95validator560 ms209 ms209 ms
tn_network_inbound_request_service_seconds{class="epoch_record"}p95validator55 ms5 ms5 ms
tn_network_inbound_request_service_seconds{class="epoch_record"}p95observer15 ms5 ms5 ms
tn_network_inbound_request_service_seconds{class="vote"}p95validator7616 ms2,425 ms2,425 ms
tn_network_established_connectionsmaxvalidator1019.32020
tn_network_established_connectionsmaxobserver1192020
tn_worker_sync_streams_shed_total{reason="budget_exhausted"}ratevalidator1000
tn_worker_sync_streams_shed_total{reason="denied"}ratevalidator100.06542.182.91
tn_worker_sync_streams_shed_total{reason="denied"}rateobserver10.01790.650.867
tn_worker_batch_fetch_duration_secondsp95validator105 ms1,618 ms2,180 ms
tn_executor_batch_fetch_failures_totalratevalidator10000
tn_executor_batch_fetch_failures_totalrateobserver1000
tn_primary_votes_received_totalratevalidator1015.117.317.3
tn_primary_votes_received_totalrateobserver1000
tn_primary_vote_request_failures_totalratevalidator100.01641.962.64
tn_primary_vote_request_failures_totalrateobserver1000
tn_primary_certificates_formed_totalratevalidator102.522.882.89
tn_primary_certificates_formed_totalrateobserver1000
tn_primary_certificate_form_duration_secondsp95validator10442 ms838 ms942 ms
tn_engine_outputs_executed_totalratevalidator101.261.491.51
tn_engine_outputs_executed_totalrateobserver11.261.481.53
tn_engine_persist_failures_totalratevalidator10000
tn_engine_persist_failures_totalrateobserver1000
tn_node_consensus_sync_distancemaxvalidator108.39506507
tn_node_consensus_sync_distancemaxobserver12.2310.212
reth_db_table_size{table="AccountChangeSets"}maxvalidator10457 MiB728 MiB729 MiB
reth_db_table_size{table="AccountChangeSets"}maxobserver1458 MiB727 MiB729 MiB
reth_db_table_size{table="AccountsHistory"}maxvalidator1051 MiB80 MiB80 MiB
reth_db_table_size{table="AccountsHistory"}maxobserver151 MiB80 MiB80 MiB
reth_db_table_size{table="HashedAccounts"}maxvalidator1033 MiB58 MiB58 MiB
reth_db_table_size{table="HashedAccounts"}maxobserver133 MiB58 MiB58 MiB
reth_db_table_size{table="HashedStorages"}maxvalidator1041 MiB44 MiB44 MiB
reth_db_table_size{table="HashedStorages"}maxobserver141 MiB44 MiB44 MiB
reth_db_table_size{table="PlainAccountState"}maxvalidator1028 MiB48 MiB48 MiB
reth_db_table_size{table="PlainAccountState"}maxobserver128 MiB48 MiB48 MiB
reth_db_table_size{table="PlainStorageState"}maxvalidator1041 MiB44 MiB44 MiB
reth_db_table_size{table="PlainStorageState"}maxobserver141 MiB44 MiB44 MiB
reth_db_table_size{table="StorageChangeSets"}maxvalidator10537 MiB857 MiB859 MiB
reth_db_table_size{table="StorageChangeSets"}maxobserver1539 MiB857 MiB859 MiB
reth_db_table_size{table="StoragesHistory"}maxvalidator10103 MiB127 MiB127 MiB
reth_db_table_size{table="StoragesHistory"}maxobserver1103 MiB127 MiB127 MiB
reth_db_table_size{table="TransactionHashNumbers"}maxvalidator10998 MiB1.58 GiB1.58 GiB
reth_db_table_size{table="TransactionHashNumbers"}maxobserver11000 MiB1.58 GiB1.58 GiB
reth_db_table_size{table="TransactionSenders"}maxvalidator10514 MiB832 MiB834 MiB
reth_db_table_size{table="TransactionSenders"}maxobserver1516 MiB832 MiB834 MiB
reth_db_freelistmaxvalidator1027.8K51.9K60K
reth_db_freelistmaxobserver127.6K48.2K57.1K
reth_database_transaction_commit_write_duration_secondsp95validator95 ms5 ms5 ms
reth_database_transaction_commit_write_duration_secondsp95observer15 ms5 ms5 ms
reth_database_transaction_commit_sync_duration_secondsp95validator10248 ms465 ms470 ms
reth_database_transaction_commit_sync_duration_secondsp95observer1250 ms370 ms425 ms
reth_database_transaction_commit_whole_duration_secondsp95validator10248 ms465 ms470 ms
reth_database_transaction_commit_whole_duration_secondsp95observer1250 ms370 ms425 ms
tn_engine_recover_secondsp95validator10220 ms944 ms944 ms
tn_engine_recover_secondsp95observer1194 ms686 ms889 ms
tn_engine_evm_secondsp95validator10265 ms562 ms662 ms
tn_engine_evm_secondsp95observer1258 ms514 ms638 ms
tn_engine_root_secondsp95validator10496 ms1,017 ms1,450 ms
tn_engine_root_secondsp95observer1479 ms823 ms888 ms
tn_engine_finish_secondsp95validator10245 ms492 ms492 ms
tn_engine_finish_secondsp95observer1162 ms272 ms319 ms
tn_engine_forward_wait_secondsp95validator102,386 ms24.2 s26.1 s
tn_engine_forward_wait_secondsp95observer199 ms2,294 ms2,369 ms
tn_engine_channel_depthmaxvalidator101.1312.214
tn_engine_channel_depthmaxobserver10.01850.52
tn_engine_executed_transactions_totalratevalidator107.44K14K14K
tn_engine_executed_transactions_totalrateobserver17.38K13.7K13.7K
tn_executor_execution_wait_secondsp95validator10190 ms30.0 s30.0 s
tn_primary_commit_run_ahead_blocked_totalratevalidator10000
tn_primary_commit_run_ahead_blocked_totalrateobserver1000
tn_primary_consensus_exec_lag_roundsmaxvalidator1017.1170172
tn_primary_consensus_exec_lag_roundsmaxobserver13.4118.522
tn_primary_commit_lag_roundsmaxvalidator102.233.256
tn_primary_commit_lag_roundsmaxobserver1000
tn_primary_rejoin_grace_holds_totalratevalidator1000.01340.0178
tn_primary_rejoin_grace_holds_totalrateobserver1000
tn_primary_votes_refused_inactive_totalratevalidator10031.131.4
tn_primary_votes_refused_inactive_totalrateobserver1000
tn_primary_votes_refused_execution_behind_totalratevalidator100.008190.7280.911
tn_primary_votes_refused_execution_behind_totalrateobserver1000
tn_batch_builder_pool_age_secondsp95validator106,447 ms30.0 s30.0 s
tn_batch_builder_builds_total{reason="bytes_full"}ratevalidator10000
tn_batch_builder_builds_total{reason="bytes_full"}rateobserver1000
tn_batch_builder_builds_total{reason="drained"}ratevalidator101.72.963.07
tn_batch_builder_builds_total{reason="drained"}rateobserver1000
tn_batch_builder_builds_total{reason="gas_full"}ratevalidator101.575.85.84
tn_batch_builder_builds_total{reason="gas_full"}rateobserver1000
tn_worker_batch_report_secondsp95validator100 ms2 ms3 ms
tn_primary_digest_wait_secondsp95validator101,378 ms30.0 s30.0 s
tn_primary_headers_proposed_total{reason="max_delay"}ratevalidator100.008590.1220.156
tn_primary_headers_proposed_total{reason="min_delay"}ratevalidator102.532.882.89
tn_primary_headers_proposed_total{reason="parents_and_threshold"}ratevalidator100.01382.552.87
tn_primary_digests_requeued_total{cause="canceled"}ratevalidator100.007092526
tn_primary_digests_requeued_total{cause="committed"}ratevalidator100.01145.376.4
tn_executor_consensus_pack_persist_secondsp95validator107 ms10 ms12 ms
tn_batch_builder_seal_duration_secondsp95validator10259 ms881 ms881 ms
tn_worker_quorum_wait_secondsp95validator10258 ms842 ms842 ms
tn_primary_proposer_pending_digestsmaxvalidator100.87734.840
tn_primary_proposer_pending_digestsmaxobserver1000
reth_transaction_pool_pending_pool_transactionsmaxvalidator104876.27K9.81K
reth_transaction_pool_pending_pool_transactionsmaxobserver1000
reth_transaction_pool_queued_pool_transactionsmaxvalidator103795.69K5.97K
reth_transaction_pool_queued_pool_transactionsmaxobserver1000
reth_transaction_pool_all_transactions_by_hashmaxvalidator102.52K9.12K9.88K
reth_transaction_pool_all_transactions_by_hashmaxobserver1000
reth_storage_providers_database_save_blocks_totalp95validator10176 ms343 ms365 ms
reth_storage_providers_database_save_blocks_totalp95observer1146 ms256 ms267 ms
reth_storage_providers_database_save_blocks_mdbxp95validator9175 ms343 ms365 ms
reth_storage_providers_database_save_blocks_mdbxp95observer1146 ms256 ms267 ms
reth_storage_providers_database_save_blocks_sfp95validator1033 ms83 ms91 ms
reth_storage_providers_database_save_blocks_sfp95observer134 ms52 ms60 ms
reth_storage_providers_database_save_blocks_commit_mdbxp95validator10162 ms400 ms402 ms
reth_storage_providers_database_save_blocks_commit_mdbxp95observer1169 ms335 ms379 ms
reth_storage_providers_database_save_blocks_commit_sfp95validator911 ms16 ms17 ms
reth_storage_providers_database_save_blocks_commit_sfp95observer19 ms11 ms12 ms
reth_storage_providers_database_save_blocks_write_statep95validator1027 ms49 ms55 ms
reth_storage_providers_database_save_blocks_write_statep95observer124 ms43 ms46 ms
reth_storage_providers_database_save_blocks_write_hashed_statep95validator1017 ms28 ms32 ms
reth_storage_providers_database_save_blocks_write_hashed_statep95observer116 ms24 ms25 ms
reth_storage_providers_database_save_blocks_write_trie_updatesp95validator1023 ms36 ms39 ms
reth_storage_providers_database_save_blocks_write_trie_updatesp95observer121 ms33 ms34 ms
reth_database_transaction_commit_preparation_duration_secondsp95validator105 ms5 ms5 ms
reth_database_transaction_commit_preparation_duration_secondsp95observer15 ms5 ms5 ms
reth_database_transaction_commit_gc_wallclock_duration_secondsp95validator105 ms5 ms5 ms
reth_database_transaction_commit_gc_wallclock_duration_secondsp95observer15 ms5 ms5 ms
reth_database_transaction_commit_gc_cputime_duration_secondsp95validator105 ms5 ms5 ms
reth_database_transaction_commit_gc_cputime_duration_secondsp95observer15 ms5 ms5 ms
reth_database_transaction_commit_audit_duration_secondsp95validator95 ms5 ms5 ms
reth_database_transaction_commit_audit_duration_secondsp95observer15 ms5 ms5 ms
reth_database_transaction_commit_ending_duration_secondsp95validator105 ms5 ms5 ms
reth_database_transaction_commit_ending_duration_secondsp95observer15 ms5 ms5 ms
tn_node_switch{name="digest_dedupe"}maxvalidator10000
tn_node_switch{name="digest_dedupe"}maxobserver1000
tn_node_switch{name="epoch_close_task_drain"}maxvalidator10000
tn_node_switch{name="epoch_close_task_drain"}maxobserver1000
tn_node_switch{name="epoch_record_state_check"}maxvalidator10111
tn_node_switch{name="epoch_record_state_check"}maxobserver1111
tn_node_switch{name="mdbx_random_advice"}maxvalidator10111
tn_node_switch{name="mdbx_random_advice"}maxobserver1111
tn_node_switch{name="parent_grace"}maxvalidator10000
tn_node_switch{name="parent_grace"}maxobserver1000
tn_node_switch{name="pipelined_persist"}maxvalidator10000
tn_node_switch{name="pipelined_persist"}maxobserver1000
tn_node_switch{name="quorum_exec_block"}maxvalidator10111
tn_node_switch{name="quorum_exec_block"}maxobserver1111
tn_node_switch{name="quorum_exec_block_starved_rounds"}maxvalidator10000
tn_node_switch{name="quorum_exec_block_starved_rounds"}maxobserver1000
tn_node_switch{name="recovery_threads"}maxvalidator10000
tn_node_switch{name="recovery_threads"}maxobserver1000
tn_node_switch{name="rpc_admission_backlog_outputs"}maxvalidator10000
tn_node_switch{name="rpc_admission_backlog_outputs"}maxobserver1000
tn_node_switch{name="sealed_tx_keep"}maxvalidator10000
tn_node_switch{name="sealed_tx_keep"}maxobserver1000
tn_node_switch{name="vote_queue_fix"}maxvalidator10111
tn_node_switch{name="vote_queue_fix"}maxobserver1111
tn_primary_epoch_entry_rounds_behindmaxvalidator1033535
tn_primary_epoch_entry_rounds_behindmaxobserver1000
tn_reth_mdbx_random_advice_bytesmaxvalidator108192.00 GiB8192.00 GiB8192.00 GiB
tn_reth_mdbx_random_advice_bytesmaxobserver18192.00 GiB8192.00 GiB8192.00 GiB
tn_reth_mdbx_used_bytesmaxvalidator102.87 GiB4.46 GiB4.46 GiB
tn_reth_mdbx_used_bytesmaxobserver12.88 GiB4.46 GiB4.46 GiB
tn_reth_mdbx_page_ops_total{op="cow"}maxvalidator1043.2M74.5M74.8M
tn_reth_mdbx_page_ops_total{op="cow"}maxobserver135M66.8M67.4M
tn_reth_mdbx_page_ops_total{op="msync"}maxvalidator106.37K9.29K9.35K
tn_reth_mdbx_page_ops_total{op="msync"}maxobserver15.32K8.57K8.63K
tn_reth_mdbx_page_ops_total{op="newly"}maxvalidator101.08M2.49M2.5M
tn_reth_mdbx_page_ops_total{op="newly"}maxobserver1821K2.06M2.09M
tn_reth_mdbx_page_ops_total{op="prefault"}maxvalidator10754K1.17M1.17M
tn_reth_mdbx_page_ops_total{op="prefault"}maxobserver1622K1.07M1.08M
tn_reth_mdbx_page_ops_total{op="spill"}maxvalidator10000
tn_reth_mdbx_page_ops_total{op="spill"}maxobserver1000
tn_reth_mdbx_page_ops_total{op="unspill"}maxvalidator10000
tn_reth_mdbx_page_ops_total{op="unspill"}maxobserver1000
tn_reth_mdbx_page_ops_total{op="wops"}maxvalidator10111
tn_reth_mdbx_page_ops_total{op="wops"}maxobserver1111
tn_reth_sealed_txs_markedmaxvalidator10000
tn_reth_sealed_txs_markedmaxobserver1000
tn_reth_sealed_txs_cleared_executed_totalmaxvalidator10000
tn_reth_sealed_txs_cleared_executed_totalmaxobserver1000
tn_reth_sealed_txs_cleared_repooled_totalmaxvalidator10000
tn_reth_sealed_txs_cleared_repooled_totalmaxobserver1000
tn_reth_sealed_txs_dropped_evicted_totalmaxvalidator10000
tn_reth_sealed_txs_dropped_evicted_totalmaxobserver1000
tn_reth_sealed_txs_cleared_lag_totalmaxvalidator10000
tn_reth_sealed_txs_cleared_lag_totalmaxobserver1000
tn_reth_sealed_txs_late_marks_refused_totalmaxvalidator10000
tn_reth_sealed_txs_late_marks_refused_totalmaxobserver1000
tn_batch_builder_sealed_txs_skipped_total{worker="0"}maxvalidator10000
tn_batch_builder_sealed_txs_skipped_total{worker="0"}maxobserver1000
reth_transaction_pool_pending_pool_size_bytesmaxvalidator10173.4 KiB2.2 MiB3.4 MiB
reth_transaction_pool_pending_pool_size_bytesmaxobserver10 B0 B0 B
reth_transaction_pool_queued_pool_size_bytesmaxvalidator10136.3 KiB2.0 MiB2.1 MiB
reth_transaction_pool_queued_pool_size_bytesmaxobserver10 B0 B0 B
reth_transaction_pool_pending_transactions_evictedmaxvalidator10000
reth_transaction_pool_pending_transactions_evictedmaxobserver1000
reth_transaction_pool_queued_transactions_evictedmaxvalidator10000
reth_transaction_pool_queued_transactions_evictedmaxobserver1000
reth_transaction_pool_basefee_transactions_evictedmaxvalidator10000
reth_transaction_pool_basefee_transactions_evictedmaxobserver1000
reth_transaction_pool_removed_transactionsmaxvalidator101.78M3.89M3.9M
reth_transaction_pool_removed_transactionsmaxobserver1000
reth_transaction_pool_invalid_transactionsmaxvalidator10000
reth_transaction_pool_invalid_transactionsmaxobserver1000
tn_worker_duplicate_digest_suppressed_total{worker="0"}maxvalidator10000
tn_worker_duplicate_digest_suppressed_total{worker="0"}maxobserver1000
tn_primary_duplicate_digest_suppressed_totalmaxvalidator10000
tn_primary_duplicate_digest_suppressed_totalmaxobserver1000
tn_engine_recovery_fallback_totalmaxvalidator10000
tn_engine_recovery_fallback_totalmaxobserver1000
tn_node_epoch_close_backlog_outputsmaxvalidator106.471616
tn_node_epoch_close_backlog_outputsmaxobserver11.7522
tn_node_epoch_close_detect_lag_secondsmaxvalidator101,213 ms408.8 s408.8 s
tn_node_epoch_close_detect_lag_secondsmaxobserver11,571 ms2,951 ms2,951 ms
tn_node_epoch_close_drain_secondsmaxvalidator104,476 ms13.0 s13.0 s
tn_node_epoch_close_drain_secondsmaxobserver1946 ms1,351 ms1,351 ms
tn_node_epoch_record_vote_skipped_total{reason="certified_before_close"}maxvalidator31.5633
tn_node_epoch_record_vote_skipped_total{reason="not_in_committee"}maxobserver14.0655
tn_primary_parent_grace_total{outcome="complete"}maxvalidator10000
tn_primary_parent_grace_total{outcome="expired"}maxvalidator10000
tn_primary_parent_grace_total{outcome="preempted"}maxvalidator10000
tn_primary_late_parents_total{origin="older_round"}maxvalidator105.03K8.22K8.22K
tn_primary_late_parents_total{origin="own_round_behind"}maxvalidator1027.5K32.8K32.9K
tn_node_execution_backlog_outputsmaxvalidator109.0422.524
tn_node_execution_backlog_outputsmaxobserver12.239.7512
tn_engine_unpersisted_outputsmaxvalidator10000
tn_engine_unpersisted_outputsmaxobserver1000
tn_engine_built_heightmaxvalidator10000
tn_engine_built_heightmaxobserver1000
tn_engine_close_barrier_holds_totalmaxvalidator10000
tn_engine_close_barrier_holds_totalmaxobserver1000

Commit latency p95

tn_primary_commit_latency_seconds

Round lag

tn_primary_round − tn_primary_committed_round

Engine queued outputs

tn_engine_queued_outputs, max by node

Certification by node

The share of each validator's own headers that formed a certificate, per window (tools/cert-rate.py on the node logs). bench-validator-02, bench-validator-06 and bench-validator-10 certified fewer than 90 % of their own headers in a window where they proposed at least 50. Own leader % is the validator's share of the committed leaders and not committed % the share of its certificates no committed leader reached, both over the run window; the fair share is 100 / 10 validators = 10.0 %. Greyed: fewer than 50 headers proposed, not judged.

ValidatorRegionblast 0 preambleblast 0 warmupblast 0 mixedblast 0 cooldownblast 1 preambleblast 1 warmupblast 1 mixedblast 1 cooldownblast 2 preambleblast 2 warmupblast 2 mixedblast 2 cooldownrunOwn leader %Not committed %
bench-validator-01us-west2 (Los Angeles)100.0100.0100.0100.097.9100.099.9100.0100.0100.099.9100.0100.013.30.1
bench-validator-02 below 90 %europe-west2 (London)101.9100.099.9100.0102.2100.099.8100.0100.099.678.693.088.72.845.8
bench-validator-03australia-southeast1 (Sydney)101.999.6100.1100.0100.0100.099.9100.099.4100.099.9100.3100.013.30.0
bench-validator-04asia-northeast1 (Tokyo)101.999.6100.1101.497.999.599.8100.899.4100.0100.0100.0100.013.70.1
bench-validator-05us-east4 (Ashburn)101.999.6100.1100.097.899.099.7100.0100.0100.099.9–99.97.517.5
bench-validator-06 below 90 %europe-west4 (Netherlands)98.199.599.9100.0100.0100.099.699.299.499.644.592.370.70.361.0
bench-validator-07asia-southeast1 (Singapore)100.0100.0100.1100.097.9100.099.9100.0100.0100.2100.0100.3100.013.70.0
bench-validator-08asia-east2 (Hong Kong)101.999.6100.1101.497.9100.0100.0100.099.4100.299.9100.3100.013.70.0
bench-validator-09asia-northeast3 (Seoul)100.0100.0100.198.6100.0100.099.9100.8100.0100.295.8100.397.813.40.1
bench-validator-10 below 90 %asia-south1 (Mumbai)98.199.199.8101.4100.099.599.8100.0100.0100.068.2100.083.39.87.5

On-node sampler

NodeClassPeak RssAnonPeak RssFilePeak cachePSI memPSI ioPSI cpudb growthstatic_files growthRoot usedOOM Δ
bench-observer-11observer674 MiB4.33 GiB– 0.0 %2.5 %1.0 %4.00 GiB5.24 GiB19.0 %0
bench-validator-01validator2.98 GiB3.51 GiB– 0.0 %3.4 %6.2 %4.00 GiB5.24 GiB21.0 %0
bench-validator-02validator3.00 GiB3.60 GiB– 0.0 %4.1 %6.9 %4.00 GiB5.24 GiB21.0 %0
bench-validator-03validator2.20 GiB3.81 GiB– 0.0 %2.7 %9.3 %4.00 GiB5.24 GiB21.0 %0
bench-validator-04validator2.29 GiB3.71 GiB– 0.0 %3.2 %8.3 %4.00 GiB5.24 GiB20.0 %0
bench-validator-05validator3.47 GiB4.21 GiB– 0.0 %3.0 %7.0 %4.00 GiB5.24 GiB20.0 %0
bench-validator-06validator2.43 GiB3.88 GiB– 0.0 %3.1 %7.7 %4.00 GiB5.24 GiB21.0 %0
bench-validator-07validator2.22 GiB3.50 GiB– 0.0 %3.3 %8.0 %4.00 GiB5.24 GiB21.0 %0
bench-validator-08validator2.46 GiB3.68 GiB– 0.0 %3.1 %8.0 %4.00 GiB5.24 GiB21.0 %0
bench-validator-09validator2.33 GiB3.49 GiB– 0.0 %3.1 %7.3 %4.00 GiB5.24 GiB20.0 %0
bench-validator-10validator2.93 GiB3.88 GiB– 0.0 %3.2 %6.5 %4.00 GiB5.24 GiB21.0 %0

RssAnon vs RssFile

/proc/<pid>/status of the telcoin process on the node with the largest RSS

consensus-db/cache size

Not collected: consensus-db/cache size.

Pressure stall (some, avg10)

/proc/pressure on the node with the highest memory PSI

Headroom

Per role and per node, from sizing.json (p95 over the landed window; PSI and RssAnon max over the whole sampler run). The lines follow the hardware doc's tier rules: CPU p95 over 70 % of vCPUs; memory used p95 over 75 % of RAM, or telcoin RSS p95 over 85 % of RAM; disk write p95 over 60 % of the PD MiB/s limit, or write IOPS p95 over 60 % of the IOPS limit; IO pressure over 20 % or memory pressure over 5 % (PSI some avg10, max); network p95 over 50 % of 1 Gbps; root disk over 80 %. A line crossed on any node puts the role at the limit; a nonzero reading within 15 points of its line is tight; otherwise the role has headroom. OOM kills, disk throttling and degraded disk performance are flagged too.

SignalClassWorst nodeValueShare of capacityLineNodes over / tightStatus
CPU p95validatorbench-validator-0943.6 %43.6 %> 70 %0 / 0 headroom
Memory used p95validatorbench-validator-0527.3 %27.3 %> 75 %0 / 0 headroom
telcoin RSS p95validatorbench-validator-107.04 GB41.0 %> 85 %0 / 0 headroom
PSI memory maxvalidatorbench-validator-010.0 %0.0 %> 5 %0 / 0 headroom
RssAnon peakvalidatorbench-validator-053.73 GB21.7 %info– info
Disk write p95validatorbench-validator-09136 MB/s16.2 %> 60 %0 / 0 headroom
Write IOPS p95validatorbench-validator-0927,05854.1 %> 60 %0 / 10 tight
PSI io maxvalidatorbench-validator-024.1 %4.1 %> 20 %0 / 0 headroom
Root disk maxvalidatorbench-validator-0320.1 %20.1 %> 80 %0 / 0 headroom
Network p95validatorbench-validator-07153 Mbps15.3 %> 50 %0 / 0 headroom
Consensus cache peakvalidator–––info– not collected
OOM killsvalidator––anyany– none
Disk throttlingvalidator––metric type unavailable in projectany– not collected
Disk performance degradedvalidator––any sampleany– none
CPU p95observerbench-observer-1123.9 %23.9 %> 70 %0 / 0 headroom
Memory used p95observerbench-observer-1110.4 %10.4 %> 75 %0 / 0 headroom
telcoin RSS p95observerbench-observer-115.11 GB29.7 %> 85 %0 / 0 headroom
PSI memory maxobserverbench-observer-110.0 %0.0 %> 5 %0 / 0 headroom
RssAnon peakobserverbench-observer-110.71 GB4.1 %info– info
Disk write p95observerbench-observer-11125 MB/s14.9 %> 60 %0 / 0 headroom
Write IOPS p95observerbench-observer-1127,04754.1 %> 60 %0 / 1 tight
PSI io maxobserverbench-observer-112.5 %2.5 %> 20 %0 / 0 headroom
Root disk maxobserverbench-observer-1118.6 %18.6 %> 80 %0 / 0 headroom
Network p95observerbench-observer-1111 Mbps1.1 %> 50 %0 / 0 headroom
Consensus cache peakobserver–––info– not collected
OOM killsobserver––anyany– none
Disk throttlingobserver––metric type unavailable in projectany– not collected
Disk performance degradedobserver––any sampleany– none

Per node

NodeClassCPU p95Memory used p95telcoin RSS p95PSI memory maxRssAnon peakDisk write p95Write IOPS p95PSI io maxRoot disk maxNetwork p95Consensus cache peak
Line> 70 %> 75 %> 85 %> 5 %info> 60 %> 60 %> 20 %> 80 %> 50 %info
bench-validator-01validator40.1 %24.5 %6.84 GB 40 %0.0 %3.20 GB 19 %131 MB/s 16 %26,119 52 %3.4 %19.9 %133 Mbps 13 %–
bench-validator-02validator40.0 %24.5 %6.99 GB 41 %0.0 %3.22 GB 19 %135 MB/s 16 %26,774 54 %4.1 %20.0 %136 Mbps 14 %–
bench-validator-03validator43.4 %19.7 %6.38 GB 37 %0.0 %2.36 GB 14 %135 MB/s 16 %26,633 53 %2.7 %20.1 %144 Mbps 14 %–
bench-validator-04validator41.2 %20.3 %6.33 GB 37 %0.0 %2.46 GB 14 %134 MB/s 16 %26,312 53 %3.2 %19.8 %136 Mbps 14 %–
bench-validator-05validator38.2 %27.3 %6.79 GB 40 %0.0 %3.73 GB 22 %129 MB/s 15 %26,143 52 %3.8 %19.2 %126 Mbps 13 %–
bench-validator-06validator39.3 %21.1 %6.71 GB 39 %0.0 %2.61 GB 15 %136 MB/s 16 %26,729 53 %3.1 %20.0 %133 Mbps 13 %–
bench-validator-07validator42.8 %19.8 %6.06 GB 35 %0.0 %2.38 GB 14 %134 MB/s 16 %26,356 53 %3.3 %20.1 %153 Mbps 15 %–
bench-validator-08validator43.0 %21.2 %6.42 GB 37 %0.0 %2.64 GB 15 %134 MB/s 16 %26,340 53 %3.1 %20.0 %143 Mbps 14 %–
bench-validator-09validator43.6 %20.5 %6.12 GB 36 %0.0 %2.51 GB 15 %136 MB/s 16 %27,058 54 %3.1 %19.8 %141 Mbps 14 %–
bench-validator-10validator40.2 %24.1 %7.04 GB 41 %0.0 %3.15 GB 18 %136 MB/s 16 %26,750 53 %3.2 %20.0 %135 Mbps 14 %–
bench-observer-11observer23.9 %10.4 %5.11 GB 30 %0.0 %0.71 GB 4 %125 MB/s 15 %27,047 54 %2.5 %18.6 %11 Mbps 1 %–

Small figures are the share of capacity: RAM for RSS, the PD limit for disk write (VM cap applied), 1 Gbps for network. Hover a cell for its JSON path.

Verdict

Fleet A vs Fleet B

A = Fleet c3-hdb · c3-highcpu-8 · r20261113-1630
B = Fleet c3-hdb · c3-highcpu-8 · r20261113-1351

Verdict

tight c3-highcpu-8 validators at 8,539 sustained TPS: tight on disk. Disk tight: write IOPS p95 27,058, 54.1 % of the 50,000 IOPS limit (bench-validator-09), tight against the 60 % line. CPU has headroom: CPU p95 43.6 % of 8 vCPUs (bench-validator-09). Memory has headroom: memory used p95 27.3 % of RAM (bench-validator-05); telcoin RSS p95 7.04 GB, 41.0 % of 16 GiB RAM (bench-validator-10); memory pressure (PSI some avg10) max 0.0 % (bench-validator-01). Network has headroom: network p95 153 Mbps, 15.3 % of 1 Gbps (bench-validator-07).

tight c3-highcpu-8 observers at 8,539 sustained TPS: tight on disk. Disk tight: write IOPS p95 27,047, 54.1 % of the 50,000 IOPS limit (bench-observer-11), tight against the 60 % line. CPU has headroom: CPU p95 23.9 % of 8 vCPUs (bench-observer-11). Memory has headroom: memory used p95 10.4 % of RAM (bench-observer-11); telcoin RSS p95 5.11 GB, 29.7 % of 16 GiB RAM (bench-observer-11); memory pressure (PSI some avg10) max 0.0 % (bench-observer-11). Network has headroom: network p95 11 Mbps, 1.1 % of 1 Gbps (bench-observer-11).

at the limit c3-highcpu-8 validators at 8,065 sustained TPS: at the limit on disk. Disk at the limit: write IOPS p95 20,050–20,146, up to 100.7 % of the 20,000 IOPS limit, over the 60 % line on all 10 validators (bench-validator-01, 02, 03, 04, 05, 06, 07, 08, 09, 10); IO pressure (PSI some avg10) max 14.9 % (bench-validator-08), tight against the 20 % line. CPU has headroom: CPU p95 36.0 % of 8 vCPUs (bench-validator-03). Memory has headroom: memory used p95 22.7 % of RAM (bench-validator-06); telcoin RSS p95 6.87 GB, 40.0 % of 16 GiB RAM (bench-validator-06); memory pressure (PSI some avg10) max 0.0 % (bench-validator-01). Network has headroom: network p95 130 Mbps, 13.0 % of 1 Gbps (bench-validator-07). Next step: a faster volume (pd-ssd adds 30 IOPS and 0.48 MiB/s per GB up to the per-VM cap; past the cap, a larger VM).

at the limit c3-highcpu-8 observers at 8,065 sustained TPS: at the limit on disk. Disk at the limit: write IOPS p95 20,071, 100.4 % of the 20,000 IOPS limit on bench-observer-11, over the 60 % line; IO pressure (PSI some avg10) max 13.7 % (bench-observer-11), tight against the 20 % line. CPU has headroom: CPU p95 20.4 % of 8 vCPUs (bench-observer-11). Memory has headroom: memory used p95 12.4 % of RAM (bench-observer-11); telcoin RSS p95 4.54 GB, 26.4 % of 16 GiB RAM (bench-observer-11); memory pressure (PSI some avg10) max 0.0 % (bench-observer-11). Network has headroom: network p95 12 Mbps, 1.2 % of 1 Gbps (bench-observer-11). Next step: a faster volume (pd-ssd adds 30 IOPS and 0.48 MiB/s per GB up to the per-VM cap; past the cap, a larger VM).

Validators: Fleet A (c3-highcpu-8) is tight on disk, with headroom on CPU, memory and network; Fleet B (c3-highcpu-8) is at the limit on disk, with headroom on CPU, memory and network. Disk is at the limit on Fleet B (c3-highcpu-8) only. Fleet A (c3-highcpu-8) stays under the disk lines that Fleet B (c3-highcpu-8) crosses.

Observers: Fleet A (c3-highcpu-8) is tight on disk, with headroom on CPU, memory and network; Fleet B (c3-highcpu-8) is at the limit on disk, with headroom on CPU, memory and network. Disk is at the limit on Fleet B (c3-highcpu-8) only. Fleet A (c3-highcpu-8) stays under the disk lines that Fleet B (c3-highcpu-8) crosses.

KPIFleet AFleet BB ÷ A
Sustained mixed TPS (own)8,539.0 tx/s8,065.4 tx/s0.94×
Sustained mixed TPS (chain)8,539.0 tx/s8,065.4 tx/s0.94×
Landed / accepted99.9 %100.0 %1.00×
Reverted (own)00–
Latency p50858 ms679 ms0.79×
Latency p953,528 ms14,662 ms4.16×
Gas / s (own)365M gas/s269M gas/s0.74×
Capacity at mix16,530.3 tx/s14,103.0 tx/s0.85×

Headroom, worst node per class

SignalClassFleet AFleet BLine
CPU p95validator43.6 % headroom36.0 % headroom> 70 %
Memory used p95validator27.3 % headroom22.7 % headroom> 75 %
telcoin RSS p95validator7.04 GB 41.0 % headroom6.87 GB 40.0 % headroom> 85 %
PSI memory maxvalidator0.0 % headroom0.0 % headroom> 5 %
Disk write p95validator136 MB/s 16.2 % headroom110 MB/s 13.1 % headroom> 60 %
Write IOPS p95validator27,058 54.1 % tight20,146 100.7 % at the limit> 60 %
PSI io maxvalidator4.1 % headroom14.9 % tight> 20 %
Root disk maxvalidator20.1 % headroom17.5 % headroom> 80 %
Network p95validator153 Mbps 15.3 % headroom130 Mbps 13.0 % headroom> 50 %
CPU p95observer23.9 % headroom20.4 % headroom> 70 %
Memory used p95observer10.4 % headroom12.4 % headroom> 75 %
telcoin RSS p95observer5.11 GB 29.7 % headroom4.54 GB 26.4 % headroom> 85 %
PSI memory maxobserver0.0 % headroom0.0 % headroom> 5 %
Disk write p95observer125 MB/s 14.9 % headroom97 MB/s 11.5 % headroom> 60 %
Write IOPS p95observer27,047 54.1 % tight20,071 100.4 % at the limit> 60 %
PSI io maxobserver2.5 % headroom13.7 % tight> 20 %
Root disk maxobserver18.6 % headroom15.9 % headroom> 80 %
Network p95observer11 Mbps 1.1 % headroom12 Mbps 1.2 % headroom> 50 %

Own TPS, trailing 60-s mean

aligned on each fleet's mixed-phase start

CPU, hottest validator

aligned on each fleet's mixed-phase start

Memory used, hottest validator

aligned on each fleet's mixed-phase start

telcoin RSS, hottest validator

aligned on each fleet's mixed-phase start

Disk MiB/s, hottest validator

aligned on each fleet's mixed-phase start

Network sent, hottest validator

aligned on each fleet's mixed-phase start

Reference chains

Not collected: reference chains.