This is the full report of the run “The adaptive vote fix on Hyperdisk Balanced at 50,000 IOPS”. Back to the run page.
r20261114-0000 · Fleet c3-hdb · c3-highcpu-8

Telcoin Network benchmark: mixed workload

Throughput is measured from blocks the chain produced (own senders vs every sender), resource use from Cloud Monitoring, Prometheus and the on-node sampler. Summaries cover the mixed phase. Hover any number for the JSON field it came from.

10 blaster(s), 6K senders, chain 32286, schema tn-blast-combined/v2 · mixed phase 2026-11-14 02:13–02:53 UTC · 1 observers, 10 validators

Preflight: WTEL from genesis (0x0000…37e1), v3 fee tier 3000, v4 available, Multicall3 present · Cooldown ended: all_landed ×2, idle_30s ×8

Headline numbers

Sustained mixed TPS (own)
8,337.1 tx/s
best 60-s mean of our own landed tx in the mixed phase
Sustained mixed TPS (chain)
8,337.1 tx/s
all tx in blocks, same window
Landed / accepted
99.9 %
19.2M of 19.2M accepted
Reverted (own)
0
0.00 % of landed
Latency p50
990 ms
submit → block timestamp; seen p50 6,875 ms
Latency p95
3,879 ms
seen p95 23,151 ms
Gas / s (own)
363M gas/s
chain 363M gas/s
Capacity at mix
16,723.6 tx/s
gas-limit basis 10,733.0 tx/s; block gas limit 30M

Per transaction type

Landed counts only our own senders; latency is submit → block timestamp.

TypeSubmittedAcceptedLandedRevertedUnlandedSkipsGas limitGas used avgp50p95
native_transfer5.52M5.39M5.38M05.67K024.1K21K984 ms3,867 ms
batched_self_send3.06M2.96M2.95M04.89K024.1K21K1,028 ms3,912 ms
uniswap_v2_swap2.75M2.65M2.65M04.12K0180K106K985 ms3,870 ms
stablecoin_transfer2.28M2.21M2.2M03.35K075K43.9K984 ms3,865 ms
wtel_wrap1.83M1.76M1.76M02.71K050K27.9K984 ms3,863 ms
wtel_unwrap1.22M1.18M1.18M01.84K045K35.2K983 ms3,866 ms
stablecoin_approve912K880K879K01.31K060K31.8K984 ms3,862 ms
uniswap_v2_add_liquidity608K587K587K08510250K128K983 ms3,866 ms
uniswap_v4_swap457K442K441K07190250K124K985 ms3,862 ms
contract_deploy454K439K438K06860140K74.2K984 ms3,861 ms
uniswap_v2_remove_liquidity305K295K294K04320220K129K986 ms3,876 ms
uniswap_v3_swap304K294K293K04590220K120K985 ms3,866 ms
faucet_drip126K126K126K014634K150K107K994 ms7,610 ms

Submission by RPC node

NodeSubmittedAcceptedAccepted %Top errorsSend p50Send p95
bench-validator-01
bench-validator-01:8545
2.79M2.78M99.6 %known 6.3K6 ms20 ms
bench-validator-02
bench-validator-02:8545
1.01M954K94.8 %other 36.4K, known 7.53K5 ms28 ms
bench-validator-03
bench-validator-03:8545
2.77M2.76M99.5 %known 6.96K6 ms21 ms
bench-validator-04
bench-validator-04:8545
2.11M2.11M99.9 %known 6216 ms22 ms
bench-validator-05
bench-validator-05:8545
2.37M548K23.1 %other 1.82M, known 8361 ms11 ms
bench-validator-06
bench-validator-06:8545
1.05M994K95.0 %known 26K6 ms29 ms
bench-validator-07
bench-validator-07:8545
3.02M3M99.3 %known 12K7 ms24 ms
bench-validator-08
bench-validator-08:8545
2.6M2.57M98.5 %known 21.1K6 ms19 ms
bench-validator-09
bench-validator-09:8545
2.06M2.01M97.6 %known 26.9K4 ms16 ms
bench-validator-10
bench-validator-10:8545
1.6M1.55M97.1 %known 21.5K6 ms22 ms

Throughput over time

Shaded: phases outside the mixed window. Hairlines: epoch closes. Dashed orange: chaos events.

Landed transactions per second

own senders vs chain total, per block-timestamp second

Gas per second

own vs chain gas used, per block-timestamp second

CPU and memory per node

CPU utilization

compute.googleapis.com/instance/cpu/utilization, 60-s mean, % of all vCPUs; summaries over the mixed phase

Table view
NodeClassMeanp95Peak 60-s meanMax
bench-observer-11observer22.5 %24.2 %27.0 %27.0 %
bench-validator-01validator39.6 %41.9 %51.5 %51.5 %
bench-validator-02validator35.7 %38.4 %48.7 %48.7 %
bench-validator-03validator39.8 %43.3 %50.4 %50.4 %
bench-validator-04validator38.3 %39.8 %51.5 %51.5 %
bench-validator-05validator31.9 %39.4 %49.4 %49.4 %
bench-validator-06validator37.4 %39.4 %48.8 %48.8 %
bench-validator-07validator40.0 %42.7 %50.2 %50.2 %
bench-validator-08validator39.3 %42.7 %49.7 %49.7 %
bench-validator-09validator38.5 %42.5 %49.6 %49.6 %
bench-validator-10validator38.0 %40.6 %50.6 %50.6 %

CPU: mean vs peak 60-s mean

One row per node, over the mixed phase.

telcoin resident memory

agent processes/rss_usage summed over telcoin pids; summaries over the mixed phase

Table view
NodeClassMeanp95Peak 60-s meanMax
bench-observer-11observer3.42 GiB4.86 GiB5.14 GiB5.14 GiB
bench-validator-01validator5.19 GiB6.14 GiB6.17 GiB6.17 GiB
bench-validator-02validator5.44 GiB6.39 GiB6.47 GiB6.47 GiB
bench-validator-03validator5.13 GiB6.13 GiB6.19 GiB6.19 GiB
bench-validator-04validator5.09 GiB6.11 GiB6.19 GiB6.19 GiB
bench-validator-05validator4.82 GiB7.12 GiB7.27 GiB7.27 GiB
bench-validator-06validator5.33 GiB6.49 GiB6.54 GiB6.54 GiB
bench-validator-07validator5.07 GiB6.03 GiB6.16 GiB6.16 GiB
bench-validator-08validator5.38 GiB6.40 GiB6.46 GiB6.46 GiB
bench-validator-09validator5.26 GiB6.24 GiB6.29 GiB6.29 GiB
bench-validator-10validator5.16 GiB6.07 GiB6.15 GiB6.15 GiB

telcoin RSS: mean vs peak 60-s mean

One row per node, over the mixed phase.

Memory used

agent memory/percent_used, state=used; summaries over the mixed phase

Table view
NodeClassMeanp95Peak 60-s meanMax
bench-observer-11observer10.0 %11.1 %12.2 %12.2 %
bench-validator-01validator20.1 %21.0 %22.0 %22.0 %
bench-validator-02validator21.6 %22.6 %22.6 %22.6 %
bench-validator-03validator20.2 %21.2 %21.8 %21.8 %
bench-validator-04validator19.7 %20.6 %21.1 %21.1 %
bench-validator-05validator16.0 %19.6 %19.7 %19.7 %
bench-validator-06validator22.0 %25.1 %25.5 %25.5 %
bench-validator-07validator19.6 %20.5 %21.1 %21.1 %
bench-validator-08validator21.7 %22.9 %23.4 %23.4 %
bench-validator-09validator20.9 %22.2 %23.0 %23.0 %
bench-validator-10validator20.5 %21.5 %23.0 %23.0 %

Memory used: mean vs peak 60-s mean

One row per node, over the mixed phase.

Disk

Disk throttling (validator)
not collected –
metric type unavailable in project
Disk performance degraded (validator)
none –
any sample

Disk read + write throughput

instance/disk/{read,write}_bytes_count rate, all devices; summaries over the mixed phase

Table view
NodeClassMeanp95Peak 60-s meanMaxLimit
bench-observer-11observer107.7 MiB/s116.2 MiB/s119.4 MiB/s119.4 MiB/s800.0 MiB/s
bench-validator-01validator118.3 MiB/s125.9 MiB/s128.3 MiB/s128.3 MiB/s800.0 MiB/s
bench-validator-02validator114.0 MiB/s121.8 MiB/s123.3 MiB/s123.3 MiB/s800.0 MiB/s
bench-validator-03validator118.5 MiB/s125.8 MiB/s127.0 MiB/s127.0 MiB/s800.0 MiB/s
bench-validator-04validator117.5 MiB/s124.4 MiB/s126.7 MiB/s126.7 MiB/s800.0 MiB/s
bench-validator-05validator98.1 MiB/s122.7 MiB/s126.1 MiB/s126.1 MiB/s800.0 MiB/s
bench-validator-06validator118.5 MiB/s127.0 MiB/s128.8 MiB/s128.8 MiB/s800.0 MiB/s
bench-validator-07validator118.7 MiB/s126.6 MiB/s127.4 MiB/s127.4 MiB/s800.0 MiB/s
bench-validator-08validator118.8 MiB/s127.3 MiB/s129.0 MiB/s129.0 MiB/s800.0 MiB/s
bench-validator-09validator118.7 MiB/s125.8 MiB/s130.7 MiB/s130.7 MiB/s800.0 MiB/s
bench-validator-10validator118.6 MiB/s126.5 MiB/s127.8 MiB/s127.8 MiB/s800.0 MiB/s

Disk read + write IOPS

instance/disk/{read,write}_ops_count rate, all devices; summaries over the mixed phase

Table view
NodeClassMeanp95Peak 60-s meanMaxLimit
bench-observer-11observer23.8K IOPS26.1K IOPS27K IOPS27K IOPS50K IOPS
bench-validator-01validator23.9K IOPS26.2K IOPS26.8K IOPS26.8K IOPS50K IOPS
bench-validator-02validator23.1K IOPS25.6K IOPS26K IOPS26K IOPS50K IOPS
bench-validator-03validator23.9K IOPS26.1K IOPS26.4K IOPS26.4K IOPS50K IOPS
bench-validator-04validator23.8K IOPS26K IOPS26.1K IOPS26.1K IOPS50K IOPS
bench-validator-05validator19.7K IOPS25K IOPS25.4K IOPS25.4K IOPS50K IOPS
bench-validator-06validator23.9K IOPS26.3K IOPS26.9K IOPS26.9K IOPS50K IOPS
bench-validator-07validator23.9K IOPS26.1K IOPS26.5K IOPS26.5K IOPS50K IOPS
bench-validator-08validator23.9K IOPS26.4K IOPS26.7K IOPS26.7K IOPS50K IOPS
bench-validator-09validator23.9K IOPS26K IOPS27.1K IOPS27.1K IOPS50K IOPS
bench-validator-10validator23.9K IOPS26K IOPS26.5K IOPS26.5K IOPS50K IOPS
Disk limits used
NodeMachinevCPUDiskIOPS limitMiB/s limitBasis
bench-observer-11c3-highcpu-88hyperdisk-balanced 100 GB50K IOPS800.0 MiB/shyperdisk-balanced 100 GB: provisioned 50,000 IOPS, provisioned 800 MiB/s; c3 8 vCPU cap 50,000 IOPS / 800 MiB/s; reads and writes share this budget
bench-validator-01c3-highcpu-88hyperdisk-balanced 100 GB50K IOPS800.0 MiB/shyperdisk-balanced 100 GB: provisioned 50,000 IOPS, provisioned 800 MiB/s; c3 8 vCPU cap 50,000 IOPS / 800 MiB/s; reads and writes share this budget
bench-validator-02c3-highcpu-88hyperdisk-balanced 100 GB50K IOPS800.0 MiB/shyperdisk-balanced 100 GB: provisioned 50,000 IOPS, provisioned 800 MiB/s; c3 8 vCPU cap 50,000 IOPS / 800 MiB/s; reads and writes share this budget
bench-validator-03c3-highcpu-88hyperdisk-balanced 100 GB50K IOPS800.0 MiB/shyperdisk-balanced 100 GB: provisioned 50,000 IOPS, provisioned 800 MiB/s; c3 8 vCPU cap 50,000 IOPS / 800 MiB/s; reads and writes share this budget
bench-validator-04c3-highcpu-88hyperdisk-balanced 100 GB50K IOPS800.0 MiB/shyperdisk-balanced 100 GB: provisioned 50,000 IOPS, provisioned 800 MiB/s; c3 8 vCPU cap 50,000 IOPS / 800 MiB/s; reads and writes share this budget
bench-validator-05c3-highcpu-88hyperdisk-balanced 100 GB50K IOPS800.0 MiB/shyperdisk-balanced 100 GB: provisioned 50,000 IOPS, provisioned 800 MiB/s; c3 8 vCPU cap 50,000 IOPS / 800 MiB/s; reads and writes share this budget
bench-validator-06c3-highcpu-88hyperdisk-balanced 100 GB50K IOPS800.0 MiB/shyperdisk-balanced 100 GB: provisioned 50,000 IOPS, provisioned 800 MiB/s; c3 8 vCPU cap 50,000 IOPS / 800 MiB/s; reads and writes share this budget
bench-validator-07c3-highcpu-88hyperdisk-balanced 100 GB50K IOPS800.0 MiB/shyperdisk-balanced 100 GB: provisioned 50,000 IOPS, provisioned 800 MiB/s; c3 8 vCPU cap 50,000 IOPS / 800 MiB/s; reads and writes share this budget
bench-validator-08c3-highcpu-88hyperdisk-balanced 100 GB50K IOPS800.0 MiB/shyperdisk-balanced 100 GB: provisioned 50,000 IOPS, provisioned 800 MiB/s; c3 8 vCPU cap 50,000 IOPS / 800 MiB/s; reads and writes share this budget
bench-validator-09c3-highcpu-88hyperdisk-balanced 100 GB50K IOPS800.0 MiB/shyperdisk-balanced 100 GB: provisioned 50,000 IOPS, provisioned 800 MiB/s; c3 8 vCPU cap 50,000 IOPS / 800 MiB/s; reads and writes share this budget
bench-validator-10c3-highcpu-88hyperdisk-balanced 100 GB50K IOPS800.0 MiB/shyperdisk-balanced 100 GB: provisioned 50,000 IOPS, provisioned 800 MiB/s; c3 8 vCPU cap 50,000 IOPS / 800 MiB/s; reads and writes share this budget

Network

Network sent

instance/network/sent_bytes_count rate; summaries over the mixed phase

Table view
NodeClassMeanp95Peak 60-s meanMax
bench-observer-11observer4.7 Mbps20.8 Mbps39.7 Mbps39.7 Mbps
bench-validator-01validator132.0 Mbps138.3 Mbps142.7 Mbps142.7 Mbps
bench-validator-02validator120.2 Mbps129.2 Mbps134.2 Mbps134.2 Mbps
bench-validator-03validator133.9 Mbps143.6 Mbps175.5 Mbps175.5 Mbps
bench-validator-04validator128.9 Mbps133.6 Mbps140.0 Mbps140.0 Mbps
bench-validator-05validator104.2 Mbps126.0 Mbps130.9 Mbps130.9 Mbps
bench-validator-06validator125.4 Mbps131.6 Mbps134.4 Mbps134.4 Mbps
bench-validator-07validator136.8 Mbps149.2 Mbps184.0 Mbps184.0 Mbps
bench-validator-08validator132.2 Mbps141.8 Mbps151.8 Mbps151.8 Mbps
bench-validator-09validator129.2 Mbps137.8 Mbps142.3 Mbps142.3 Mbps
bench-validator-10validator127.6 Mbps138.5 Mbps142.1 Mbps142.1 Mbps

Network received

instance/network/received_bytes_count rate; summaries over the mixed phase

Table view
NodeClassMeanp95Peak 60-s meanMax
bench-observer-11observer8.4 Mbps16.0 Mbps29.0 Mbps29.0 Mbps
bench-validator-01validator13.8 Mbps17.5 Mbps20.8 Mbps20.8 Mbps
bench-validator-02validator12.2 Mbps17.6 Mbps67.1 Mbps67.1 Mbps
bench-validator-03validator14.0 Mbps17.7 Mbps19.4 Mbps19.4 Mbps
bench-validator-04validator12.6 Mbps13.7 Mbps20.3 Mbps20.3 Mbps
bench-validator-05validator21.1 Mbps54.3 Mbps85.7 Mbps85.7 Mbps
bench-validator-06validator10.8 Mbps12.6 Mbps17.7 Mbps17.7 Mbps
bench-validator-07validator14.4 Mbps16.6 Mbps20.1 Mbps20.1 Mbps
bench-validator-08validator13.7 Mbps17.7 Mbps19.5 Mbps19.5 Mbps
bench-validator-09validator12.6 Mbps16.3 Mbps18.8 Mbps18.8 Mbps
bench-validator-10validator11.7 Mbps15.3 Mbps20.4 Mbps20.4 Mbps

Consensus health

MetricQueryClassNodesMedian of node meansWorst peak 60-s meanMax
tn_primary_commit_latency_secondsp95validator81,164 ms4,056 ms5,250 ms
tn_engine_execution_duration_secondsp95validator92,011 ms4,190 ms4,417 ms
tn_engine_execution_duration_secondsp95observer11,771 ms3,253 ms3,667 ms
tn_worker_batch_size_bytesp95validator8234.1 KiB246.4 KiB246.4 KiB
tn_worker_batch_transactionsp95validator94961.48K1.48K
tn_executor_output_transactionsp95validator813.1K24.2K24.2K
tn_primary_consensus_output_bytesmaxvalidator10554.5 KiB1.4 MiB2.6 MiB
tn_primary_consensus_output_bytesmaxobserver1546.0 KiB1.1 MiB1.4 MiB
tn_engine_queued_outputsmaxvalidator105.4488
tn_engine_queued_outputsmaxobserver11.334.56
tn_primary_roundmaxvalidator101.46K2.95K3K
tn_primary_roundmaxobserver11.47K2.94K3K
tn_primary_committed_roundmaxvalidator101.46K2.95K3K
tn_primary_committed_roundmaxobserver11.47K2.94K3K
tn_epoch_currentmaxvalidator106.688
tn_epoch_currentmaxobserver16.688
tn_batch_builder_pending_pool_transactionsmaxvalidator105318.11K8.15K
tn_batch_builder_pending_pool_transactionsmaxobserver1000
reth_process_resident_memory_bytesmaxvalidator105.23 GiB7.27 GiB7.32 GiB
reth_process_resident_memory_bytesmaxobserver13.53 GiB5.14 GiB5.15 GiB
tn_primary_subdags_committed_totalratevalidator101.171.421.42
tn_primary_subdags_committed_totalrateobserver1000
tn_primary_round_laglagvalidator102.233.55
tn_primary_round_laglagobserver1000
tn_batch_builder_seal_failures_totalratevalidator80.0004090.01670.0222
tn_node_modemaxvalidator10011
tn_node_modemaxobserver1000
tn_primary_vote_requests_totalratevalidator102032.933.1
tn_primary_vote_requests_totalrateobserver1000
tn_primary_votes_created_totalratevalidator1019.925.425.6
tn_primary_votes_created_totalrateobserver1000
tn_primary_votes_recast_totalratevalidator10000
tn_primary_votes_recast_totalrateobserver1000
tn_primary_vote_requests_too_new_totalratevalidator10000
tn_primary_vote_requests_too_new_totalrateobserver1000
tn_primary_vote_timeouts_total{stage="exec_wait"}ratevalidator10000
tn_primary_vote_timeouts_total{stage="exec_wait"}rateobserver1000
tn_primary_vote_timeouts_total{stage="lead_sleep"}ratevalidator10000
tn_primary_vote_timeouts_total{stage="lead_sleep"}rateobserver1000
tn_primary_vote_timeouts_total{stage="parents"}ratevalidator1000.08560.114
tn_primary_vote_timeouts_total{stage="parents"}rateobserver1000
tn_primary_vote_timeouts_total{stage="persist"}ratevalidator10000
tn_primary_vote_timeouts_total{stage="persist"}rateobserver1000
tn_primary_vote_timeouts_total{stage="prelock_wait"}ratevalidator10000
tn_primary_vote_timeouts_total{stage="prelock_wait"}rateobserver1000
tn_primary_vote_timeouts_total{stage="sync_batches"}ratevalidator10000
tn_primary_vote_timeouts_total{stage="sync_batches"}rateobserver1000
tn_primary_vote_timeouts_total{stage="validate"}ratevalidator10000
tn_primary_vote_timeouts_total{stage="validate"}rateobserver1000
tn_primary_vote_timeouts_total{stage="vote_store"}ratevalidator10000
tn_primary_vote_timeouts_total{stage="vote_store"}rateobserver1000
tn_primary_vote_stage_seconds{stage="exec_wait"}p95validator100 ms0 ms0 ms
tn_primary_vote_stage_seconds{stage="lead_sleep"}p95validator100 ms0 ms0 ms
tn_primary_vote_stage_seconds{stage="lock"}p95validator90 ms0 ms0 ms
tn_primary_vote_stage_seconds{stage="parents"}p95validator101 ms2 ms2 ms
tn_primary_vote_stage_seconds{stage="persist"}p95validator103 ms4 ms4 ms
tn_primary_vote_stage_seconds{stage="prelock_wait"}p95validator10368 ms2,425 ms2,425 ms
tn_primary_vote_stage_seconds{stage="sync_batches"}p95validator100 ms0 ms0 ms
tn_primary_vote_stage_seconds{stage="validate"}p95validator103 ms5 ms5 ms
tn_primary_vote_stage_seconds{stage="vote_store"}p95validator101 ms1 ms1 ms
tn_primary_leader_rounds_total{leader="1Vge17NWtME42tFv13ygpke1RnwtfRfQ3bB8f9LhvJY",swapped="false",found="false"}ratevalidator100.0007650.01670.0222
tn_primary_leader_rounds_total{leader="1Vge17NWtME42tFv13ygpke1RnwtfRfQ3bB8f9LhvJY",swapped="false",found="true"}ratevalidator100.1180.1560.156
tn_primary_leader_rounds_total{leader="1Vge17NWtME42tFv13ygpke1RnwtfRfQ3bB8f9LhvJY",swapped="true",found="false"}ratevalidator100.0003560.01650.0221
tn_primary_leader_rounds_total{leader="1Vge17NWtME42tFv13ygpke1RnwtfRfQ3bB8f9LhvJY",swapped="true",found="true"}ratevalidator100.03690.08330.111
tn_primary_leader_rounds_total{leader="45NSoQM2SZZt2dus3jPGygK9UREEpqDjwPxJC3Krnp12",swapped="false",found="false"}ratevalidator100.001170.02220.0222
tn_primary_leader_rounds_total{leader="45NSoQM2SZZt2dus3jPGygK9UREEpqDjwPxJC3Krnp12",swapped="false",found="true"}ratevalidator100.1170.1560.156
tn_primary_leader_rounds_total{leader="45NSoQM2SZZt2dus3jPGygK9UREEpqDjwPxJC3Krnp12",swapped="true",found="false"}ratevalidator100.001060.02210.0222
tn_primary_leader_rounds_total{leader="45NSoQM2SZZt2dus3jPGygK9UREEpqDjwPxJC3Krnp12",swapped="true",found="true"}ratevalidator100.03490.1170.156
tn_primary_leader_rounds_total{leader="4T2tfJZmemUhL2MbHGS6w4V83kjeXFZrNXhQqRHr7zVc",swapped="false",found="false"}ratevalidator100.00280.07780.0889
tn_primary_leader_rounds_total{leader="4T2tfJZmemUhL2MbHGS6w4V83kjeXFZrNXhQqRHr7zVc",swapped="false",found="true"}ratevalidator100.07810.1440.156
tn_primary_leader_rounds_total{leader="4T2tfJZmemUhL2MbHGS6w4V83kjeXFZrNXhQqRHr7zVc",swapped="true",found="false"}ratevalidator10000
tn_primary_leader_rounds_total{leader="4T2tfJZmemUhL2MbHGS6w4V83kjeXFZrNXhQqRHr7zVc",swapped="true",found="true"}ratevalidator100.002770.04440.0667
tn_primary_leader_rounds_total{leader="4kX5oKXKBi2yPeW88mrJbQMACeKoA4AGbJ3D5vyRVvTT",swapped="false",found="false"}ratevalidator100.002810.03890.0444
tn_primary_leader_rounds_total{leader="4kX5oKXKBi2yPeW88mrJbQMACeKoA4AGbJ3D5vyRVvTT",swapped="false",found="true"}ratevalidator100.1160.1560.156
tn_primary_leader_rounds_total{leader="4kX5oKXKBi2yPeW88mrJbQMACeKoA4AGbJ3D5vyRVvTT",swapped="true",found="false"}ratevalidator100.000350.01650.0221
tn_primary_leader_rounds_total{leader="4kX5oKXKBi2yPeW88mrJbQMACeKoA4AGbJ3D5vyRVvTT",swapped="true",found="true"}ratevalidator100.02740.09440.133
tn_primary_leader_rounds_total{leader="96CMnptik7z4MPDnp4oax9bHUUvZGCSaYzsMcg33CEcr",swapped="false",found="false"}ratevalidator10000
tn_primary_leader_rounds_total{leader="96CMnptik7z4MPDnp4oax9bHUUvZGCSaYzsMcg33CEcr",swapped="false",found="true"}ratevalidator100.07690.1560.156
tn_primary_leader_rounds_total{leader="96CMnptik7z4MPDnp4oax9bHUUvZGCSaYzsMcg33CEcr",swapped="true",found="false"}ratevalidator100.0003560.01650.0221
tn_primary_leader_rounds_total{leader="96CMnptik7z4MPDnp4oax9bHUUvZGCSaYzsMcg33CEcr",swapped="true",found="true"}ratevalidator100.009820.07780.0889
tn_primary_leader_rounds_total{leader="BhMgY6bDVD8P8eChCuUFK66GUAEFZQDGtg8AcVk3us76",swapped="false",found="false"}ratevalidator100.007130.1140.133
tn_primary_leader_rounds_total{leader="BhMgY6bDVD8P8eChCuUFK66GUAEFZQDGtg8AcVk3us76",swapped="false",found="true"}ratevalidator100.05410.1560.156
tn_primary_leader_rounds_total{leader="BhMgY6bDVD8P8eChCuUFK66GUAEFZQDGtg8AcVk3us76",swapped="true",found="false"}ratevalidator100.0007120.03310.0441
tn_primary_leader_rounds_total{leader="BhMgY6bDVD8P8eChCuUFK66GUAEFZQDGtg8AcVk3us76",swapped="true",found="true"}ratevalidator100.01570.10.133
tn_primary_leader_rounds_total{leader="CoyGfRch9FC9Kp8HYjyyY8NWQM3pw94E13Jc9qBM9tRr",swapped="false",found="false"}ratevalidator10000
tn_primary_leader_rounds_total{leader="CoyGfRch9FC9Kp8HYjyyY8NWQM3pw94E13Jc9qBM9tRr",swapped="false",found="true"}ratevalidator100.1190.1560.156
tn_primary_leader_rounds_total{leader="CoyGfRch9FC9Kp8HYjyyY8NWQM3pw94E13Jc9qBM9tRr",swapped="true",found="false"}ratevalidator100.001180.01670.0222
tn_primary_leader_rounds_total{leader="CoyGfRch9FC9Kp8HYjyyY8NWQM3pw94E13Jc9qBM9tRr",swapped="true",found="true"}ratevalidator100.03570.08330.111
tn_primary_leader_rounds_total{leader="EqPrjKthssuTFThDFhA1UFVvQfH2YCryuvJLa6hqEDLE",swapped="false",found="false"}ratevalidator10000
tn_primary_leader_rounds_total{leader="EqPrjKthssuTFThDFhA1UFVvQfH2YCryuvJLa6hqEDLE",swapped="false",found="true"}ratevalidator100.1190.1560.156
tn_primary_leader_rounds_total{leader="EqPrjKthssuTFThDFhA1UFVvQfH2YCryuvJLa6hqEDLE",swapped="true",found="false"}ratevalidator10000
tn_primary_leader_rounds_total{leader="EqPrjKthssuTFThDFhA1UFVvQfH2YCryuvJLa6hqEDLE",swapped="true",found="true"}ratevalidator100.04390.1280.156
tn_primary_leader_rounds_total{leader="FgN9BDskVaZSgk5fEeXFVzKT1GeZko5tJbPjx5ayVFAp",swapped="false",found="false"}ratevalidator10000
tn_primary_leader_rounds_total{leader="FgN9BDskVaZSgk5fEeXFVzKT1GeZko5tJbPjx5ayVFAp",swapped="false",found="true"}ratevalidator100.009820.1280.133
tn_primary_leader_rounds_total{leader="FgN9BDskVaZSgk5fEeXFVzKT1GeZko5tJbPjx5ayVFAp",swapped="true",found="false"}ratevalidator10000
tn_primary_leader_rounds_total{leader="FgN9BDskVaZSgk5fEeXFVzKT1GeZko5tJbPjx5ayVFAp",swapped="true",found="true"}ratevalidator10000
tn_primary_leader_rounds_total{leader="J5rtTGKhcBys9w1R4FgwK1QL4Ci7R7GUGRRRBUy5oLgT",swapped="false",found="false"}ratevalidator10000
tn_primary_leader_rounds_total{leader="J5rtTGKhcBys9w1R4FgwK1QL4Ci7R7GUGRRRBUy5oLgT",swapped="false",found="true"}ratevalidator100.1180.150.156
tn_primary_leader_rounds_total{leader="J5rtTGKhcBys9w1R4FgwK1QL4Ci7R7GUGRRRBUy5oLgT",swapped="true",found="false"}ratevalidator10000
tn_primary_leader_rounds_total{leader="J5rtTGKhcBys9w1R4FgwK1QL4Ci7R7GUGRRRBUy5oLgT",swapped="true",found="true"}ratevalidator100.03340.08330.111
tn_primary_leader_commits_total{leader="1Vge17NWtME42tFv13ygpke1RnwtfRfQ3bB8f9LhvJY"}ratevalidator100.1550.2220.244
tn_primary_leader_commits_total{leader="45NSoQM2SZZt2dus3jPGygK9UREEpqDjwPxJC3Krnp12"}ratevalidator100.1520.2220.267
tn_primary_leader_commits_total{leader="4T2tfJZmemUhL2MbHGS6w4V83kjeXFZrNXhQqRHr7zVc"}ratevalidator100.0810.150.178
tn_primary_leader_commits_total{leader="4kX5oKXKBi2yPeW88mrJbQMACeKoA4AGbJ3D5vyRVvTT"}ratevalidator100.1420.2390.289
tn_primary_leader_commits_total{leader="96CMnptik7z4MPDnp4oax9bHUUvZGCSaYzsMcg33CEcr"}ratevalidator100.08670.2060.222
tn_primary_leader_commits_total{leader="BhMgY6bDVD8P8eChCuUFK66GUAEFZQDGtg8AcVk3us76"}ratevalidator100.06980.2330.267
tn_primary_leader_commits_total{leader="CoyGfRch9FC9Kp8HYjyyY8NWQM3pw94E13Jc9qBM9tRr"}ratevalidator100.1540.2150.222
tn_primary_leader_commits_total{leader="EqPrjKthssuTFThDFhA1UFVvQfH2YCryuvJLa6hqEDLE"}ratevalidator100.1630.2610.289
tn_primary_leader_commits_total{leader="FgN9BDskVaZSgk5fEeXFVzKT1GeZko5tJbPjx5ayVFAp"}ratevalidator100.009820.1280.133
tn_primary_leader_commits_total{leader="J5rtTGKhcBys9w1R4FgwK1QL4Ci7R7GUGRRRBUy5oLgT"}ratevalidator100.1510.2060.222
tn_engine_queue_wait_secondsp95validator96,943 ms28.0 s28.9 s
tn_engine_queue_wait_secondsp95observer12,246 ms5,887 ms7,062 ms
tn_engine_persist_duration_secondsp95validator8502 ms1,025 ms1,225 ms
tn_engine_persist_duration_secondsp95observer1483 ms886 ms900 ms
tn_engine_busy_milliseconds_totalratevalidator109671.02K1.07K
tn_engine_busy_milliseconds_totalrateobserver18989981.02K
tn_reth_persist_save_blocks_secondsp95validator8270 ms941 ms982 ms
tn_reth_persist_save_blocks_secondsp95observer1247 ms443 ms450 ms
tn_reth_persist_commit_secondsp95validator7256 ms524 ms608 ms
tn_reth_persist_commit_secondsp95observer1254 ms443 ms450 ms
tn_reth_persisted_accounts_totalratevalidator103.01K7.59K7.59K
tn_reth_persisted_accounts_totalrateobserver13.02K7.15K7.15K
tn_reth_persisted_storage_slots_totalratevalidator103.99K4.63K4.79K
tn_reth_persisted_storage_slots_totalrateobserver13.99K4.6K4.62K
tn_node_mode_transitions_total{to="cvv_inactive",reason="epoch_behind"}ratevalidator1000
tn_executor_rejoin_blocked_total{reason="attempts_exhausted"}ratevalidator2000
tn_executor_rejoin_blocked_total{reason="epoch_closed"}ratevalidator20.1821.11.2
tn_executor_rejoin_blocked_total{reason="epoch_mismatch"}ratevalidator2000
tn_executor_rejoin_blocked_total{reason="no_known_header"}ratevalidator2000
tn_executor_rejoin_blocked_total{reason="round_gap"}ratevalidator20.8651.261.31
tn_executor_rejoin_blocked_total{reason="stale_head"}ratevalidator20.01190.1170.156
tn_worker_sync_serves_denied_total{worker="0",tier="committee"}ratevalidator10000
tn_worker_sync_serves_denied_total{worker="0",tier="committee"}rateobserver1000
tn_worker_sync_serves_denied_total{worker="0",tier="public"}ratevalidator10000
tn_worker_sync_serves_denied_total{worker="0",tier="public"}rateobserver1000
tn_worker_batch_requests_denied_total{worker="0",peer="22sZN3dfzGaFxmXDGRmsRAfGVdDaQL5QTF3LucJjK7TCoCjs5KWDh65fySh3as2EY3prEnEvhAUr9jWMq1do9K8Yqb6JuiVdJeKkgaamMCVtKEjEsv2SKkhsznyjCC7B9K6f"}ratevalidator2000
tn_worker_batch_requests_denied_total{worker="0",peer="246yKXSoqMieNxok98Gc4YK1jvcyXB2aZGnCUZVzdd4u7xnreRoG33zjTx2j39eEoWqxrNfrArkRZgdjzrv3uiVWKkFst3EPDDFK1tpD6d8GELQYfoksz5dWTrfWw8XSc23C"}ratevalidator3000
tn_worker_batch_requests_denied_total{worker="0",peer="248YoQPU4YeF32nHAFy4Bpqi6D82pGtcsLeBiTVWkY9d7wZP1jJAigfSYTcz9uLCxYX3P9WCXx4kcmjc9LR8wsPKweKxLJmGkSwvBd1L2tmnUHc7GAR2Lg9DKg7y7qhmkhyU"}ratevalidator400.04440.0444
tn_worker_batch_requests_denied_total{worker="0",peer="24FLp4x96H8a5uLAxPoR7dXZwYdjVgXtAeuZ33AC78exiwYZiw1nGQRFp8FSgVKNHKJ91BwPt3xMTpHuPibwkeSrUjzgVRuBrzeiwg9NxKEs9mqkkfnfQbTo3xRUcGhbaNDy"}ratevalidator300.02220.0222
tn_worker_batch_requests_denied_total{worker="0",peer="mmP1kjNxDQkDKpQtLWQuHL359GeEpJ5GJ92pEzuvpgDYZDyijtJuvdLfKFuBco19CsAtTQ91ySVPASDQpf17rMHdKoiqWfSzPKYnzcS171G1ECrD3fPWSPZUMe8qpsGN4FF"}ratevalidator2000
tn_worker_batch_requests_denied_total{worker="0",peer="o87qNkBiNtFRU21RPZct6PNWZKyErCNs5SE8jKWc2U1p17MceZWVy8TCNoTQzWr1mw46WAJYbQu8jxPG2Mr9cSpR6THfTnjFFh7P4cdQjhQ2ppSCnDwWeyR88maKC7VuW5s"}ratevalidator20.006490.1430.19
tn_worker_batch_requests_denied_total{worker="0",peer="ofFnex8dHPBxBsLF1cMwxkeeGaczu6bZbit1C1rrGifGULALhjyT3NmkXu7yKm7x68XG7PtknmESg64VUDx6tUYqn1xvE15TDY66FSStJ51SLkC16trU3YKUgSgFwfo7usQ"}ratevalidator2000
tn_worker_batch_requests_denied_total{worker="0",peer="oxeGgwHXVvJ7zysJxhdnaTBDBmSwdFF6rTUhzsu2LMYMX6rAjTwdbq3MknqA5dicHehf8fiMSUqirLJrpdouru82oQGjJgwZvi2QNX3QW1DM9xrp5TRNcqRzd1Ljnz57scS"}ratevalidator2000
tn_worker_batch_requests_denied_total{worker="0",peer="ufF6X1ErVVMivUGoNP7Uh8JSxAmghNxnwDd3Z5ovw8yrh89qCbrmuvrNcJPufFQeLbiuCeh5L913BZERvjBeyj4EzfYzaAHYnzx7ZH7W6detPnFTZfSPAYojJCZoNak9PVM"}ratevalidator20.0009170.03340.0334
tn_worker_batch_requests_denied_total{worker="0",peer="xAM3H8UEq3dtRNVRF5cvLSXojnMskoSnjAyMHeuRVc6JdgTaLNNgF3hbgMsJn284GKXQjF5rccMRnJaGHRYCYd1ULPuGnuv7m2Lt1Se3heDfhTb39oeLiTDLdiTqwkhw9qe"}ratevalidator2000
tn_worker_batch_requests_failed_total{worker="0",peer="22sZN3dfzGaFxmXDGRmsRAfGVdDaQL5QTF3LucJjK7TCoCjs5KWDh65fySh3as2EY3prEnEvhAUr9jWMq1do9K8Yqb6JuiVdJeKkgaamMCVtKEjEsv2SKkhsznyjCC7B9K6f"}ratevalidator2000
tn_worker_batch_requests_failed_total{worker="0",peer="246yKXSoqMieNxok98Gc4YK1jvcyXB2aZGnCUZVzdd4u7xnreRoG33zjTx2j39eEoWqxrNfrArkRZgdjzrv3uiVWKkFst3EPDDFK1tpD6d8GELQYfoksz5dWTrfWw8XSc23C"}ratevalidator3000
tn_worker_batch_requests_failed_total{worker="0",peer="248YoQPU4YeF32nHAFy4Bpqi6D82pGtcsLeBiTVWkY9d7wZP1jJAigfSYTcz9uLCxYX3P9WCXx4kcmjc9LR8wsPKweKxLJmGkSwvBd1L2tmnUHc7GAR2Lg9DKg7y7qhmkhyU"}ratevalidator4000
tn_worker_batch_requests_failed_total{worker="0",peer="24FLp4x96H8a5uLAxPoR7dXZwYdjVgXtAeuZ33AC78exiwYZiw1nGQRFp8FSgVKNHKJ91BwPt3xMTpHuPibwkeSrUjzgVRuBrzeiwg9NxKEs9mqkkfnfQbTo3xRUcGhbaNDy"}ratevalidator3000
tn_worker_batch_requests_failed_total{worker="0",peer="mmP1kjNxDQkDKpQtLWQuHL359GeEpJ5GJ92pEzuvpgDYZDyijtJuvdLfKFuBco19CsAtTQ91ySVPASDQpf17rMHdKoiqWfSzPKYnzcS171G1ECrD3fPWSPZUMe8qpsGN4FF"}ratevalidator2000
tn_worker_batch_requests_failed_total{worker="0",peer="o87qNkBiNtFRU21RPZct6PNWZKyErCNs5SE8jKWc2U1p17MceZWVy8TCNoTQzWr1mw46WAJYbQu8jxPG2Mr9cSpR6THfTnjFFh7P4cdQjhQ2ppSCnDwWeyR88maKC7VuW5s"}ratevalidator2000
tn_worker_batch_requests_failed_total{worker="0",peer="ofFnex8dHPBxBsLF1cMwxkeeGaczu6bZbit1C1rrGifGULALhjyT3NmkXu7yKm7x68XG7PtknmESg64VUDx6tUYqn1xvE15TDY66FSStJ51SLkC16trU3YKUgSgFwfo7usQ"}ratevalidator20.004990.08860.132
tn_worker_batch_requests_failed_total{worker="0",peer="oxeGgwHXVvJ7zysJxhdnaTBDBmSwdFF6rTUhzsu2LMYMX6rAjTwdbq3MknqA5dicHehf8fiMSUqirLJrpdouru82oQGjJgwZvi2QNX3QW1DM9xrp5TRNcqRzd1Ljnz57scS"}ratevalidator2000
tn_worker_batch_requests_failed_total{worker="0",peer="ufF6X1ErVVMivUGoNP7Uh8JSxAmghNxnwDd3Z5ovw8yrh89qCbrmuvrNcJPufFQeLbiuCeh5L913BZERvjBeyj4EzfYzaAHYnzx7ZH7W6detPnFTZfSPAYojJCZoNak9PVM"}ratevalidator2000
tn_worker_batch_requests_failed_total{worker="0",peer="xAM3H8UEq3dtRNVRF5cvLSXojnMskoSnjAyMHeuRVc6JdgTaLNNgF3hbgMsJn284GKXQjF5rccMRnJaGHRYCYd1ULPuGnuv7m2Lt1Se3heDfhTb39oeLiTDLdiTqwkhw9qe"}ratevalidator2000
tn_node_epoch_record_signersmaxvalidator108.361010
tn_node_epoch_record_signersmaxobserver18.871010
tn_node_epoch_record_missing_signers_total{signer="1Vge17NWtME42tFv13ygpke1RnwtfRfQ3bB8f9LhvJY"}ratevalidator10000
tn_node_epoch_record_missing_signers_total{signer="1Vge17NWtME42tFv13ygpke1RnwtfRfQ3bB8f9LhvJY"}rateobserver1000
tn_node_epoch_record_missing_signers_total{signer="45NSoQM2SZZt2dus3jPGygK9UREEpqDjwPxJC3Krnp12"}ratevalidator10000
tn_node_epoch_record_missing_signers_total{signer="45NSoQM2SZZt2dus3jPGygK9UREEpqDjwPxJC3Krnp12"}rateobserver1000
tn_node_epoch_record_missing_signers_total{signer="4T2tfJZmemUhL2MbHGS6w4V83kjeXFZrNXhQqRHr7zVc"}ratevalidator100.0007410.01670.0222
tn_node_epoch_record_missing_signers_total{signer="4T2tfJZmemUhL2MbHGS6w4V83kjeXFZrNXhQqRHr7zVc"}rateobserver10.000520.01560.0208
tn_node_epoch_record_missing_signers_total{signer="4kX5oKXKBi2yPeW88mrJbQMACeKoA4AGbJ3D5vyRVvTT"}ratevalidator10000
tn_node_epoch_record_missing_signers_total{signer="4kX5oKXKBi2yPeW88mrJbQMACeKoA4AGbJ3D5vyRVvTT"}rateobserver1000
tn_node_epoch_record_missing_signers_total{signer="96CMnptik7z4MPDnp4oax9bHUUvZGCSaYzsMcg33CEcr"}ratevalidator10000
tn_node_epoch_record_missing_signers_total{signer="96CMnptik7z4MPDnp4oax9bHUUvZGCSaYzsMcg33CEcr"}rateobserver1000
tn_node_epoch_record_missing_signers_total{signer="BhMgY6bDVD8P8eChCuUFK66GUAEFZQDGtg8AcVk3us76"}ratevalidator100.0007410.01670.0222
tn_node_epoch_record_missing_signers_total{signer="BhMgY6bDVD8P8eChCuUFK66GUAEFZQDGtg8AcVk3us76"}rateobserver10.000520.01560.0208
tn_node_epoch_record_missing_signers_total{signer="CoyGfRch9FC9Kp8HYjyyY8NWQM3pw94E13Jc9qBM9tRr"}ratevalidator1000.0130.0173
tn_node_epoch_record_missing_signers_total{signer="CoyGfRch9FC9Kp8HYjyyY8NWQM3pw94E13Jc9qBM9tRr"}rateobserver1000
tn_node_epoch_record_missing_signers_total{signer="EqPrjKthssuTFThDFhA1UFVvQfH2YCryuvJLa6hqEDLE"}ratevalidator100.0007330.01670.0222
tn_node_epoch_record_missing_signers_total{signer="EqPrjKthssuTFThDFhA1UFVvQfH2YCryuvJLa6hqEDLE"}rateobserver10.000520.01560.0208
tn_node_epoch_record_missing_signers_total{signer="FgN9BDskVaZSgk5fEeXFVzKT1GeZko5tJbPjx5ayVFAp"}ratevalidator10000
tn_node_epoch_record_missing_signers_total{signer="FgN9BDskVaZSgk5fEeXFVzKT1GeZko5tJbPjx5ayVFAp"}rateobserver1000
tn_node_epoch_record_missing_signers_total{signer="J5rtTGKhcBys9w1R4FgwK1QL4Ci7R7GUGRRRBUy5oLgT"}ratevalidator10000
tn_node_epoch_record_missing_signers_total{signer="J5rtTGKhcBys9w1R4FgwK1QL4Ci7R7GUGRRRBUy5oLgT"}rateobserver1000
tn_network_outbound_request_duration_seconds{outcome="failure",class="batch"}p95validator215.2 s29.0 s29.0 s
tn_network_outbound_request_duration_seconds{outcome="failure",class="epoch_record"}p95validator529.0 s29.0 s29.0 s
tn_network_outbound_request_duration_seconds{outcome="failure",class="epoch_record"}p95observer129.0 s29.0 s29.0 s
tn_network_outbound_request_duration_seconds{outcome="failure",class="vote"}p95validator511.1 s29.0 s29.0 s
tn_network_outbound_request_duration_seconds{outcome="response",class="batch"}p95validator1483 ms654 ms743 ms
tn_network_outbound_request_duration_seconds{outcome="response",class="epoch_record"}p95validator5436 ms450 ms452 ms
tn_network_outbound_request_duration_seconds{outcome="response",class="epoch_record"}p95observer1240 ms241 ms241 ms
tn_network_outbound_request_duration_seconds{outcome="response",class="vote"}p95validator31,065 ms2,238 ms2,252 ms
tn_network_closed_admission_denied_total{network="bench",direction="in"}ratevalidator10000
tn_network_closed_admission_denied_total{network="bench",direction="in"}rateobserver1000
tn_network_closed_admission_denied_total{network="bench",direction="out"}ratevalidator10000
tn_network_closed_admission_denied_total{network="bench",direction="out"}rateobserver1000
tn_network_source_admission_denied_total{network="bench",direction="in",reason="address_full"}ratevalidator10000
tn_network_source_admission_denied_total{network="bench",direction="in",reason="address_full"}rateobserver1000
tn_network_source_admission_denied_total{network="bench",direction="in",reason="duplicate_connection"}ratevalidator10000
tn_network_source_admission_denied_total{network="bench",direction="in",reason="duplicate_connection"}rateobserver1000
tn_network_source_admission_denied_total{network="bench",direction="in",reason="invalid_limits"}ratevalidator10000
tn_network_source_admission_denied_total{network="bench",direction="in",reason="invalid_limits"}rateobserver1000
tn_network_source_admission_denied_total{network="bench",direction="in",reason="peer_full"}ratevalidator10000
tn_network_source_admission_denied_total{network="bench",direction="in",reason="peer_full"}rateobserver1000
tn_network_source_admission_denied_total{network="bench",direction="in",reason="poisoned"}ratevalidator10000
tn_network_source_admission_denied_total{network="bench",direction="in",reason="poisoned"}rateobserver1000
tn_network_source_admission_denied_total{network="bench",direction="in",reason="prefix_full"}ratevalidator10000
tn_network_source_admission_denied_total{network="bench",direction="in",reason="prefix_full"}rateobserver1000
tn_network_source_admission_denied_total{network="bench",direction="in",reason="process_full"}ratevalidator10000
tn_network_source_admission_denied_total{network="bench",direction="in",reason="process_full"}rateobserver1000
tn_network_source_admission_denied_total{network="bench",direction="in",reason="sources_full"}ratevalidator10000
tn_network_source_admission_denied_total{network="bench",direction="in",reason="sources_full"}rateobserver1000
tn_network_source_admission_denied_total{network="bench",direction="in",reason="unsupported_address"}ratevalidator10000
tn_network_source_admission_denied_total{network="bench",direction="in",reason="unsupported_address"}rateobserver1000
tn_network_source_admission_denied_total{network="bench",direction="out",reason="address_full"}ratevalidator10000
tn_network_source_admission_denied_total{network="bench",direction="out",reason="address_full"}rateobserver1000
tn_network_source_admission_denied_total{network="bench",direction="out",reason="duplicate_connection"}ratevalidator10000
tn_network_source_admission_denied_total{network="bench",direction="out",reason="duplicate_connection"}rateobserver1000
tn_network_source_admission_denied_total{network="bench",direction="out",reason="invalid_limits"}ratevalidator10000
tn_network_source_admission_denied_total{network="bench",direction="out",reason="invalid_limits"}rateobserver1000
tn_network_source_admission_denied_total{network="bench",direction="out",reason="peer_full"}ratevalidator10000
tn_network_source_admission_denied_total{network="bench",direction="out",reason="peer_full"}rateobserver1000
tn_network_source_admission_denied_total{network="bench",direction="out",reason="poisoned"}ratevalidator10000
tn_network_source_admission_denied_total{network="bench",direction="out",reason="poisoned"}rateobserver1000
tn_network_source_admission_denied_total{network="bench",direction="out",reason="prefix_full"}ratevalidator10000
tn_network_source_admission_denied_total{network="bench",direction="out",reason="prefix_full"}rateobserver1000
tn_network_source_admission_denied_total{network="bench",direction="out",reason="process_full"}ratevalidator10000
tn_network_source_admission_denied_total{network="bench",direction="out",reason="process_full"}rateobserver1000
tn_network_source_admission_denied_total{network="bench",direction="out",reason="sources_full"}ratevalidator10000
tn_network_source_admission_denied_total{network="bench",direction="out",reason="sources_full"}rateobserver1000
tn_network_source_admission_denied_total{network="bench",direction="out",reason="unsupported_address"}ratevalidator10000
tn_network_source_admission_denied_total{network="bench",direction="out",reason="unsupported_address"}rateobserver1000
tn_network_admission_modemaxvalidator10222
tn_network_admission_modemaxobserver1222
tn_network_admission_fallbackmaxvalidator10011
tn_network_admission_fallbackmaxobserver1000
tn_network_outbound_request_failures_total{kind="connection"}ratevalidator1001.572.03
tn_network_outbound_request_failures_total{kind="connection"}rateobserver1000
tn_network_outbound_request_failures_total{kind="dial"}ratevalidator1024102118
tn_network_outbound_request_failures_total{kind="dial"}rateobserver1000
tn_network_outbound_request_failures_total{kind="io"}ratevalidator103.211414.5
tn_network_outbound_request_failures_total{kind="io"}rateobserver10.3610.9891.54
tn_network_outbound_request_failures_total{kind="timeout"}ratevalidator100.9684.34.7
tn_network_outbound_request_failures_total{kind="timeout"}rateobserver10.003510.02710.0416
tn_network_outbound_request_failures_total{kind="unsupported"}ratevalidator10000
tn_network_outbound_request_failures_total{kind="unsupported"}rateobserver1000
tn_network_inbound_requests_shed_total{class="batch",reason="admission"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="batch",reason="admission"}rateobserver1000
tn_network_inbound_requests_shed_total{class="batch",reason="queue_budget"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="batch",reason="queue_budget"}rateobserver1000
tn_network_inbound_requests_shed_total{class="batch",reason="queue_full"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="batch",reason="queue_full"}rateobserver1000
tn_network_inbound_requests_shed_total{class="batch",reason="serve_budget"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="batch",reason="serve_budget"}rateobserver1000
tn_network_inbound_requests_shed_total{class="batch",reason="unsubscribed"}ratevalidator100.014355.484.2
tn_network_inbound_requests_shed_total{class="batch",reason="unsubscribed"}rateobserver1000
tn_network_inbound_requests_shed_total{class="certificate_sync",reason="admission"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="certificate_sync",reason="admission"}rateobserver1000
tn_network_inbound_requests_shed_total{class="certificate_sync",reason="queue_budget"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="certificate_sync",reason="queue_budget"}rateobserver1000
tn_network_inbound_requests_shed_total{class="certificate_sync",reason="queue_full"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="certificate_sync",reason="queue_full"}rateobserver1000
tn_network_inbound_requests_shed_total{class="certificate_sync",reason="serve_budget"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="certificate_sync",reason="serve_budget"}rateobserver1000
tn_network_inbound_requests_shed_total{class="certificate_sync",reason="unsubscribed"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="certificate_sync",reason="unsubscribed"}rateobserver1000
tn_network_inbound_requests_shed_total{class="epoch_record",reason="admission"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="epoch_record",reason="admission"}rateobserver1000
tn_network_inbound_requests_shed_total{class="epoch_record",reason="queue_budget"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="epoch_record",reason="queue_budget"}rateobserver1000
tn_network_inbound_requests_shed_total{class="epoch_record",reason="queue_full"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="epoch_record",reason="queue_full"}rateobserver1000
tn_network_inbound_requests_shed_total{class="epoch_record",reason="serve_budget"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="epoch_record",reason="serve_budget"}rateobserver1000
tn_network_inbound_requests_shed_total{class="epoch_record",reason="unsubscribed"}ratevalidator100.4011.621.62
tn_network_inbound_requests_shed_total{class="epoch_record",reason="unsubscribed"}rateobserver10.160.5430.708
tn_network_inbound_requests_shed_total{class="gossip",reason="admission"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="gossip",reason="admission"}rateobserver1000
tn_network_inbound_requests_shed_total{class="gossip",reason="queue_budget"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="gossip",reason="queue_budget"}rateobserver1000
tn_network_inbound_requests_shed_total{class="gossip",reason="queue_full"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="gossip",reason="queue_full"}rateobserver1000
tn_network_inbound_requests_shed_total{class="gossip",reason="serve_budget"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="gossip",reason="serve_budget"}rateobserver1000
tn_network_inbound_requests_shed_total{class="gossip",reason="unsubscribed"}ratevalidator100.02020.1060.123
tn_network_inbound_requests_shed_total{class="gossip",reason="unsubscribed"}rateobserver10.04920.1670.167
tn_network_inbound_requests_shed_total{class="other",reason="admission"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="other",reason="admission"}rateobserver1000
tn_network_inbound_requests_shed_total{class="other",reason="queue_budget"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="other",reason="queue_budget"}rateobserver1000
tn_network_inbound_requests_shed_total{class="other",reason="queue_full"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="other",reason="queue_full"}rateobserver1000
tn_network_inbound_requests_shed_total{class="other",reason="serve_budget"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="other",reason="serve_budget"}rateobserver1000
tn_network_inbound_requests_shed_total{class="other",reason="unsubscribed"}ratevalidator100.03740.2040.282
tn_network_inbound_requests_shed_total{class="other",reason="unsubscribed"}rateobserver1000
tn_network_inbound_requests_shed_total{class="vote",reason="admission"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="vote",reason="admission"}rateobserver1000
tn_network_inbound_requests_shed_total{class="vote",reason="queue_budget"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="vote",reason="queue_budget"}rateobserver1000
tn_network_inbound_requests_shed_total{class="vote",reason="queue_full"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="vote",reason="queue_full"}rateobserver1000
tn_network_inbound_requests_shed_total{class="vote",reason="serve_budget"}ratevalidator10000
tn_network_inbound_requests_shed_total{class="vote",reason="serve_budget"}rateobserver1000
tn_network_inbound_requests_shed_total{class="vote",reason="unsubscribed"}ratevalidator100.2281928.9
tn_network_inbound_requests_shed_total{class="vote",reason="unsubscribed"}rateobserver1000
tn_network_inbound_request_service_seconds{class="batch"}p95validator462 ms241 ms241 ms
tn_network_inbound_request_service_seconds{class="epoch_record"}p95validator65 ms63 ms85 ms
tn_network_inbound_request_service_seconds{class="vote"}p95validator534 ms2,425 ms2,425 ms
tn_network_established_connectionsmaxvalidator1019.52020
tn_network_established_connectionsmaxobserver1192020
tn_worker_sync_streams_shed_total{reason="denied"}ratevalidator100.002860.1170.167
tn_worker_batch_fetch_duration_secondsp95validator85 ms508 ms683 ms
tn_executor_batch_fetch_failures_totalratevalidator10000
tn_executor_batch_fetch_failures_totalrateobserver1000
tn_primary_votes_received_totalratevalidator1014.116.917.1
tn_primary_votes_received_totalrateobserver1000
tn_primary_vote_request_failures_totalratevalidator100.01353.645.16
tn_primary_vote_request_failures_totalrateobserver1000
tn_primary_certificates_formed_totalratevalidator102.252.822.84
tn_primary_certificates_formed_totalrateobserver1000
tn_primary_certificate_form_duration_secondsp95validator8488 ms957 ms1,131 ms
tn_engine_outputs_executed_totalratevalidator101.171.471.56
tn_engine_outputs_executed_totalrateobserver11.171.431.44
tn_engine_persist_failures_totalratevalidator10000
tn_engine_persist_failures_totalrateobserver1000
tn_node_consensus_sync_distancemaxvalidator106.91459460
tn_node_consensus_sync_distancemaxobserver12.157
reth_db_table_size{table="AccountChangeSets"}maxvalidator10454 MiB723 MiB725 MiB
reth_db_table_size{table="AccountChangeSets"}maxobserver1454 MiB723 MiB725 MiB
reth_db_table_size{table="AccountsHistory"}maxvalidator1051 MiB83 MiB83 MiB
reth_db_table_size{table="AccountsHistory"}maxobserver151 MiB83 MiB83 MiB
reth_db_table_size{table="HashedAccounts"}maxvalidator1033 MiB57 MiB57 MiB
reth_db_table_size{table="HashedAccounts"}maxobserver133 MiB57 MiB57 MiB
reth_db_table_size{table="HashedStorages"}maxvalidator1040 MiB43 MiB43 MiB
reth_db_table_size{table="HashedStorages"}maxobserver140 MiB43 MiB43 MiB
reth_db_table_size{table="PlainAccountState"}maxvalidator1027 MiB48 MiB48 MiB
reth_db_table_size{table="PlainAccountState"}maxobserver127 MiB48 MiB48 MiB
reth_db_table_size{table="PlainStorageState"}maxvalidator1040 MiB43 MiB43 MiB
reth_db_table_size{table="PlainStorageState"}maxobserver140 MiB43 MiB43 MiB
reth_db_table_size{table="StorageChangeSets"}maxvalidator10542 MiB869 MiB871 MiB
reth_db_table_size{table="StorageChangeSets"}maxobserver1542 MiB869 MiB871 MiB
reth_db_table_size{table="StoragesHistory"}maxvalidator10101 MiB124 MiB124 MiB
reth_db_table_size{table="StoragesHistory"}maxobserver1101 MiB124 MiB124 MiB
reth_db_table_size{table="TransactionHashNumbers"}maxvalidator10987 MiB1.56 GiB1.57 GiB
reth_db_table_size{table="TransactionHashNumbers"}maxobserver1987 MiB1.56 GiB1.57 GiB
reth_db_table_size{table="TransactionSenders"}maxvalidator10508 MiB824 MiB826 MiB
reth_db_table_size{table="TransactionSenders"}maxobserver1509 MiB823 MiB826 MiB
reth_db_freelistmaxvalidator1034.6K59.2K64.7K
reth_db_freelistmaxobserver134.3K56.7K61.5K
reth_database_transaction_commit_write_duration_secondsp95validator85 ms5 ms5 ms
reth_database_transaction_commit_write_duration_secondsp95observer15 ms5 ms5 ms
reth_database_transaction_commit_sync_duration_secondsp95validator8258 ms460 ms489 ms
reth_database_transaction_commit_sync_duration_secondsp95observer1251 ms435 ms443 ms
reth_database_transaction_commit_whole_duration_secondsp95validator8255 ms460 ms489 ms
reth_database_transaction_commit_whole_duration_secondsp95observer1252 ms435 ms443 ms
tn_engine_recover_secondsp95validator9234 ms964 ms964 ms
tn_engine_recover_secondsp95observer1233 ms666 ms829 ms
tn_engine_evm_secondsp95validator10343 ms844 ms883 ms
tn_engine_evm_secondsp95observer1307 ms721 ms800 ms
tn_engine_root_secondsp95validator8573 ms1,816 ms1,975 ms
tn_engine_root_secondsp95observer1558 ms1,458 ms1,700 ms
tn_engine_finish_secondsp95validator9260 ms766 ms860 ms
tn_engine_finish_secondsp95observer1206 ms406 ms433 ms
tn_engine_forward_wait_secondsp95validator91,616 ms9,547 ms9,618 ms
tn_engine_forward_wait_secondsp95observer10 ms0 ms0 ms
tn_engine_channel_depthmaxvalidator100.5346.758
tn_engine_channel_depthmaxobserver1000
tn_engine_executed_transactions_totalratevalidator107.41K14.1K14.1K
tn_engine_executed_transactions_totalrateobserver17.42K13.5K13.5K
tn_executor_execution_wait_secondsp95validator9189 ms30.0 s30.0 s
tn_primary_commit_run_ahead_blocked_totalratevalidator10000
tn_primary_commit_run_ahead_blocked_totalrateobserver1000
tn_primary_consensus_exec_lag_roundsmaxvalidator1014.399102
tn_primary_consensus_exec_lag_roundsmaxobserver13.319.514
tn_primary_commit_lag_roundsmaxvalidator102.233.55
tn_primary_commit_lag_roundsmaxobserver1000
tn_primary_rejoin_grace_holds_totalratevalidator10000
tn_primary_rejoin_grace_holds_totalrateobserver1000
tn_primary_votes_refused_inactive_totalratevalidator10032.933.1
tn_primary_votes_refused_inactive_totalrateobserver1000
tn_primary_votes_refused_execution_behind_totalratevalidator100.01981.011.38
tn_primary_votes_refused_execution_behind_totalrateobserver1000
tn_batch_builder_pool_age_secondsp95validator85,863 ms30.0 s30.0 s
tn_batch_builder_builds_total{reason="bytes_full"}ratevalidator10000
tn_batch_builder_builds_total{reason="bytes_full"}rateobserver1000
tn_batch_builder_builds_total{reason="drained"}ratevalidator101.63.163.24
tn_batch_builder_builds_total{reason="drained"}rateobserver1000
tn_batch_builder_builds_total{reason="gas_full"}ratevalidator101.515.445.62
tn_batch_builder_builds_total{reason="gas_full"}rateobserver1000
tn_worker_batch_report_secondsp95validator90 ms3 ms4 ms
tn_primary_digest_wait_secondsp95validator81,373 ms30.0 s30.0 s
tn_primary_headers_proposed_total{reason="max_delay"}ratevalidator100.02190.1670.2
tn_primary_headers_proposed_total{reason="min_delay"}ratevalidator102.322.832.84
tn_primary_headers_proposed_total{reason="parents_and_threshold"}ratevalidator100.02980.4560.556
tn_primary_digests_requeued_total{cause="canceled"}ratevalidator100.08153.194.11
tn_primary_digests_requeued_total{cause="committed"}ratevalidator100.00351.411.56
tn_executor_consensus_pack_persist_secondsp95validator87 ms10 ms12 ms
tn_batch_builder_seal_duration_secondsp95validator7260 ms719 ms800 ms
tn_worker_quorum_wait_secondsp95validator9255 ms719 ms800 ms
tn_primary_proposer_pending_digestsmaxvalidator100.9085.2510
tn_primary_proposer_pending_digestsmaxobserver1000
reth_transaction_pool_pending_pool_transactionsmaxvalidator105318.11K8.15K
reth_transaction_pool_pending_pool_transactionsmaxobserver1000
reth_transaction_pool_queued_pool_transactionsmaxvalidator101.88K5.65K6.09K
reth_transaction_pool_queued_pool_transactionsmaxobserver1000
reth_transaction_pool_all_transactions_by_hashmaxvalidator103.76K8.51K9.53K
reth_transaction_pool_all_transactions_by_hashmaxobserver1000
reth_storage_providers_database_save_blocks_totalp95validator7195 ms747 ms790 ms
reth_storage_providers_database_save_blocks_totalp95observer1172 ms357 ms376 ms
reth_storage_providers_database_save_blocks_mdbxp95validator7195 ms747 ms790 ms
reth_storage_providers_database_save_blocks_mdbxp95observer1172 ms357 ms376 ms
reth_storage_providers_database_save_blocks_sfp95validator742 ms101 ms112 ms
reth_storage_providers_database_save_blocks_sfp95observer142 ms106 ms107 ms
reth_storage_providers_database_save_blocks_commit_mdbxp95validator8183 ms439 ms469 ms
reth_storage_providers_database_save_blocks_commit_mdbxp95observer1175 ms308 ms376 ms
reth_storage_providers_database_save_blocks_commit_sfp95validator810 ms13 ms15 ms
reth_storage_providers_database_save_blocks_commit_sfp95observer18 ms11 ms12 ms
reth_storage_providers_database_save_blocks_write_statep95validator831 ms62 ms65 ms
reth_storage_providers_database_save_blocks_write_statep95observer130 ms57 ms64 ms
reth_storage_providers_database_save_blocks_write_hashed_statep95validator720 ms33 ms35 ms
reth_storage_providers_database_save_blocks_write_hashed_statep95observer119 ms28 ms29 ms
reth_storage_providers_database_save_blocks_write_trie_updatesp95validator826 ms41 ms44 ms
reth_storage_providers_database_save_blocks_write_trie_updatesp95observer125 ms38 ms40 ms
reth_database_transaction_commit_preparation_duration_secondsp95validator85 ms5 ms5 ms
reth_database_transaction_commit_preparation_duration_secondsp95observer15 ms5 ms5 ms
reth_database_transaction_commit_gc_wallclock_duration_secondsp95validator85 ms5 ms5 ms
reth_database_transaction_commit_gc_wallclock_duration_secondsp95observer15 ms5 ms5 ms
reth_database_transaction_commit_gc_cputime_duration_secondsp95validator75 ms5 ms5 ms
reth_database_transaction_commit_gc_cputime_duration_secondsp95observer15 ms5 ms5 ms
reth_database_transaction_commit_audit_duration_secondsp95validator85 ms5 ms5 ms
reth_database_transaction_commit_audit_duration_secondsp95observer15 ms5 ms5 ms
reth_database_transaction_commit_ending_duration_secondsp95validator85 ms5 ms5 ms
reth_database_transaction_commit_ending_duration_secondsp95observer15 ms5 ms5 ms
tn_node_switch{name="digest_dedupe"}maxvalidator10000
tn_node_switch{name="digest_dedupe"}maxobserver1000
tn_node_switch{name="epoch_close_task_drain"}maxvalidator10000
tn_node_switch{name="epoch_close_task_drain"}maxobserver1000
tn_node_switch{name="epoch_record_state_check"}maxvalidator10111
tn_node_switch{name="epoch_record_state_check"}maxobserver1111
tn_node_switch{name="mdbx_random_advice"}maxvalidator10111
tn_node_switch{name="mdbx_random_advice"}maxobserver1111
tn_node_switch{name="parent_grace"}maxvalidator10000
tn_node_switch{name="parent_grace"}maxobserver1000
tn_node_switch{name="pipelined_persist"}maxvalidator10000
tn_node_switch{name="pipelined_persist"}maxobserver1000
tn_node_switch{name="quorum_exec_block"}maxvalidator10111
tn_node_switch{name="quorum_exec_block"}maxobserver1111
tn_node_switch{name="quorum_exec_block_starved_rounds"}maxvalidator10444
tn_node_switch{name="quorum_exec_block_starved_rounds"}maxobserver1444
tn_node_switch{name="recovery_threads"}maxvalidator10000
tn_node_switch{name="recovery_threads"}maxobserver1000
tn_node_switch{name="rpc_admission_backlog_outputs"}maxvalidator10000
tn_node_switch{name="rpc_admission_backlog_outputs"}maxobserver1000
tn_node_switch{name="sealed_tx_keep"}maxvalidator10000
tn_node_switch{name="sealed_tx_keep"}maxobserver1000
tn_node_switch{name="vote_queue_fix"}maxvalidator10111
tn_node_switch{name="vote_queue_fix"}maxobserver1111
tn_primary_epoch_entry_rounds_behindmaxvalidator10318.222
tn_primary_epoch_entry_rounds_behindmaxobserver1000
tn_reth_mdbx_random_advice_bytesmaxvalidator108192.00 GiB8192.00 GiB8192.00 GiB
tn_reth_mdbx_random_advice_bytesmaxobserver18192.00 GiB8192.00 GiB8192.00 GiB
tn_reth_mdbx_used_bytesmaxvalidator102.88 GiB4.54 GiB4.54 GiB
tn_reth_mdbx_used_bytesmaxobserver12.88 GiB4.54 GiB4.54 GiB
tn_reth_mdbx_page_ops_total{op="cow"}maxvalidator1042M72M72.2M
tn_reth_mdbx_page_ops_total{op="cow"}maxobserver124.9M52.2M52.7M
tn_reth_mdbx_page_ops_total{op="msync"}maxvalidator106.18K8.84K8.86K
tn_reth_mdbx_page_ops_total{op="msync"}maxobserver13.84K7.15K7.2K
tn_reth_mdbx_page_ops_total{op="newly"}maxvalidator10991K2.3M2.31M
tn_reth_mdbx_page_ops_total{op="newly"}maxobserver1603K1.16M1.19M
tn_reth_mdbx_page_ops_total{op="prefault"}maxvalidator10755K1.2M1.2M
tn_reth_mdbx_page_ops_total{op="prefault"}maxobserver1460K900K907K
tn_reth_mdbx_page_ops_total{op="spill"}maxvalidator10000
tn_reth_mdbx_page_ops_total{op="spill"}maxobserver1000
tn_reth_mdbx_page_ops_total{op="unspill"}maxvalidator10000
tn_reth_mdbx_page_ops_total{op="unspill"}maxobserver1000
tn_reth_mdbx_page_ops_total{op="wops"}maxvalidator10111
tn_reth_mdbx_page_ops_total{op="wops"}maxobserver1111
tn_reth_sealed_txs_markedmaxvalidator10000
tn_reth_sealed_txs_markedmaxobserver1000
tn_reth_sealed_txs_cleared_executed_totalmaxvalidator10000
tn_reth_sealed_txs_cleared_executed_totalmaxobserver1000
tn_reth_sealed_txs_cleared_repooled_totalmaxvalidator10000
tn_reth_sealed_txs_cleared_repooled_totalmaxobserver1000
tn_reth_sealed_txs_dropped_evicted_totalmaxvalidator10000
tn_reth_sealed_txs_dropped_evicted_totalmaxobserver1000
tn_reth_sealed_txs_cleared_lag_totalmaxvalidator10000
tn_reth_sealed_txs_cleared_lag_totalmaxobserver1000
tn_reth_sealed_txs_late_marks_refused_totalmaxvalidator10000
tn_reth_sealed_txs_late_marks_refused_totalmaxobserver1000
tn_batch_builder_sealed_txs_skipped_total{worker="0"}maxvalidator10000
tn_batch_builder_sealed_txs_skipped_total{worker="0"}maxobserver1000
reth_transaction_pool_pending_pool_size_bytesmaxvalidator10189.2 KiB2.8 MiB2.9 MiB
reth_transaction_pool_pending_pool_size_bytesmaxobserver10 B0 B0 B
reth_transaction_pool_queued_pool_size_bytesmaxvalidator10657.8 KiB2.0 MiB2.1 MiB
reth_transaction_pool_queued_pool_size_bytesmaxobserver10 B0 B0 B
reth_transaction_pool_pending_transactions_evictedmaxvalidator10000
reth_transaction_pool_pending_transactions_evictedmaxobserver1000
reth_transaction_pool_queued_transactions_evictedmaxvalidator10000
reth_transaction_pool_queued_transactions_evictedmaxobserver1000
reth_transaction_pool_basefee_transactions_evictedmaxvalidator10000
reth_transaction_pool_basefee_transactions_evictedmaxobserver1000
reth_transaction_pool_removed_transactionsmaxvalidator101.67M3.44M3.44M
reth_transaction_pool_removed_transactionsmaxobserver1000
reth_transaction_pool_invalid_transactionsmaxvalidator10000
reth_transaction_pool_invalid_transactionsmaxobserver1000
tn_worker_duplicate_digest_suppressed_total{worker="0"}maxvalidator10000
tn_worker_duplicate_digest_suppressed_total{worker="0"}maxobserver1000
tn_primary_duplicate_digest_suppressed_totalmaxvalidator10000
tn_primary_duplicate_digest_suppressed_totalmaxobserver1000
tn_engine_recovery_fallback_totalmaxvalidator10000
tn_engine_recovery_fallback_totalmaxobserver1000
tn_node_epoch_close_backlog_outputsmaxvalidator105.351212
tn_node_epoch_close_backlog_outputsmaxobserver11.733
tn_node_epoch_close_detect_lag_secondsmaxvalidator10692 ms372.6 s372.6 s
tn_node_epoch_close_detect_lag_secondsmaxobserver11,179 ms1,330 ms1,330 ms
tn_node_epoch_close_drain_secondsmaxvalidator103,900 ms11.5 s11.5 s
tn_node_epoch_close_drain_secondsmaxobserver11,204 ms2,740 ms2,740 ms
tn_node_epoch_record_vote_skipped_total{reason="certified_before_close"}maxvalidator2111
tn_node_epoch_record_vote_skipped_total{reason="not_in_committee"}maxobserver15.9277
tn_primary_parent_grace_total{outcome="complete"}maxvalidator10000
tn_primary_parent_grace_total{outcome="expired"}maxvalidator10000
tn_primary_parent_grace_total{outcome="preempted"}maxvalidator10000
tn_primary_late_parents_total{origin="older_round"}maxvalidator108.29K11.8K11.8K
tn_primary_late_parents_total{origin="own_round_behind"}maxvalidator1046.6K56.2K56.2K
tn_node_execution_backlog_outputsmaxvalidator107.5417.819
tn_node_execution_backlog_outputsmaxobserver12.25.57
tn_engine_unpersisted_outputsmaxvalidator10000
tn_engine_unpersisted_outputsmaxobserver1000
tn_engine_built_heightmaxvalidator10000
tn_engine_built_heightmaxobserver1000
tn_engine_close_barrier_holds_totalmaxvalidator10000
tn_engine_close_barrier_holds_totalmaxobserver1000

Commit latency p95

tn_primary_commit_latency_seconds

Round lag

tn_primary_round − tn_primary_committed_round

Engine queued outputs

tn_engine_queued_outputs, max by node

Certification by node

The share of each validator's own headers that formed a certificate, per window (tools/cert-rate.py on the node logs). bench-validator-06 and bench-validator-10 certified fewer than 90 % of their own headers in a window where they proposed at least 50. Own leader % is the validator's share of the committed leaders and not committed % the share of its certificates no committed leader reached, both over the run window; the fair share is 100 / 10 validators = 10.0 %. Greyed: fewer than 50 headers proposed, not judged.

ValidatorRegionblast 0 preambleblast 0 warmupblast 0 mixedblast 0 cooldownblast 1 preambleblast 1 warmupblast 1 mixedblast 1 cooldownblast 2 preambleblast 2 warmupblast 2 mixedblast 2 cooldownrunOwn leader %Not committed %
bench-validator-01us-west2 (Los Angeles)101.5100.4100.0100.099.6100.0100.0100.499.299.899.2100.099.612.70.1
bench-validator-02europe-west2 (London)100.0100.0100.0100.099.698.499.3100.096.897.799.6–99.73.734.7
bench-validator-03australia-southeast1 (Sydney)101.5100.4100.0101.1100.499.799.5100.4100.097.999.1100.599.413.00.1
bench-validator-04asia-northeast1 (Tokyo)100.0100.4100.0101.1100.0100.3100.0100.099.299.899.9100.5100.013.60.1
bench-validator-05us-east4 (Ashburn)101.599.6100.0100.099.699.7100.0100.4100.099.8100.0–100.011.30.1
bench-validator-06 below 90 %europe-west4 (Netherlands)100.0100.499.5100.099.698.299.199.5100.097.784.199.091.70.642.8
bench-validator-07asia-southeast1 (Singapore)100.0100.499.9100.099.699.299.6100.495.297.998.3100.099.113.30.1
bench-validator-08asia-east2 (Hong Kong)101.5100.0100.0101.199.699.099.1100.497.697.094.897.197.213.10.1
bench-validator-09asia-northeast3 (Seoul)100.0100.0100.0100.099.699.599.5100.497.696.890.6100.095.112.40.3
bench-validator-10 below 90 %asia-south1 (Mumbai)100.099.6100.0100.099.297.798.9100.099.297.285.7100.092.59.610.5

On-node sampler

NodeClassPeak RssAnonPeak RssFilePeak cachePSI memPSI ioPSI cpudb growthstatic_files growthRoot usedOOM Δ
bench-observer-11observer939 MiB4.23 GiB– 0.0 %2.2 %2.4 %4.00 GiB5.20 GiB19.0 %0
bench-validator-01validator2.59 GiB3.84 GiB– 0.0 %2.6 %10.0 %4.00 GiB5.20 GiB21.0 %0
bench-validator-02validator2.68 GiB3.83 GiB– 0.0 %4.0 %4.8 %4.00 GiB5.20 GiB22.0 %0
bench-validator-03validator2.56 GiB3.78 GiB– 0.0 %2.9 %7.4 %4.00 GiB5.20 GiB21.0 %0
bench-validator-04validator2.46 GiB3.86 GiB– 0.0 %3.0 %8.9 %4.00 GiB5.20 GiB21.0 %0
bench-validator-05validator2.14 GiB5.42 GiB– 0.0 %4.9 %6.5 %4.00 GiB5.14 GiB22.0 %0
bench-validator-06validator3.19 GiB3.56 GiB– 0.0 %3.4 %5.6 %4.00 GiB5.20 GiB21.0 %0
bench-validator-07validator2.45 GiB3.88 GiB– 0.0 %2.9 %8.2 %4.00 GiB5.20 GiB21.0 %0
bench-validator-08validator2.84 GiB3.78 GiB– 0.0 %2.9 %8.0 %4.00 GiB5.20 GiB21.0 %0
bench-validator-09validator2.74 GiB3.77 GiB– 0.0 %2.8 %6.9 %4.00 GiB5.20 GiB21.0 %0
bench-validator-10validator2.77 GiB3.80 GiB– 0.0 %3.5 %5.8 %4.00 GiB5.20 GiB21.0 %0

RssAnon vs RssFile

/proc/<pid>/status of the telcoin process on the node with the largest RSS

consensus-db/cache size

Not collected: consensus-db/cache size.

Pressure stall (some, avg10)

/proc/pressure on the node with the highest memory PSI

Headroom

Per role and per node, from sizing.json (p95 over the landed window; PSI and RssAnon max over the whole sampler run). The lines follow the hardware doc's tier rules: CPU p95 over 70 % of vCPUs; memory used p95 over 75 % of RAM, or telcoin RSS p95 over 85 % of RAM; disk write p95 over 60 % of the PD MiB/s limit, or write IOPS p95 over 60 % of the IOPS limit; IO pressure over 20 % or memory pressure over 5 % (PSI some avg10, max); network p95 over 50 % of 1 Gbps; root disk over 80 %. A line crossed on any node puts the role at the limit; a nonzero reading within 15 points of its line is tight; otherwise the role has headroom. OOM kills, disk throttling and degraded disk performance are flagged too.

SignalClassWorst nodeValueShare of capacityLineNodes over / tightStatus
CPU p95validatorbench-validator-0343.3 %43.3 %> 70 %0 / 0 headroom
Memory used p95validatorbench-validator-0625.1 %25.1 %> 75 %0 / 0 headroom
telcoin RSS p95validatorbench-validator-057.56 GB44.0 %> 85 %0 / 0 headroom
PSI memory maxvalidatorbench-validator-010.0 %0.0 %> 5 %0 / 0 headroom
RssAnon peakvalidatorbench-validator-063.43 GB19.9 %info– info
Disk write p95validatorbench-validator-08133 MB/s15.9 %> 60 %0 / 0 headroom
Write IOPS p95validatorbench-validator-0826,41452.8 %> 60 %0 / 10 tight
PSI io maxvalidatorbench-validator-054.9 %4.9 %> 20 %0 / 0 headroom
Root disk maxvalidatorbench-validator-0521.3 %21.3 %> 80 %0 / 0 headroom
Network p95validatorbench-validator-07149 Mbps14.9 %> 50 %0 / 0 headroom
Consensus cache peakvalidator–––info– not collected
OOM killsvalidator––anyany– none
Disk throttlingvalidator––metric type unavailable in projectany– not collected
Disk performance degradedvalidator––any sampleany– none
CPU p95observerbench-observer-1124.2 %24.2 %> 70 %0 / 0 headroom
Memory used p95observerbench-observer-1111.1 %11.1 %> 75 %0 / 0 headroom
telcoin RSS p95observerbench-observer-115.22 GB30.4 %> 85 %0 / 0 headroom
PSI memory maxobserverbench-observer-110.0 %0.0 %> 5 %0 / 0 headroom
RssAnon peakobserverbench-observer-110.98 GB5.7 %info– info
Disk write p95observerbench-observer-11122 MB/s14.5 %> 60 %0 / 0 headroom
Write IOPS p95observerbench-observer-1126,13952.3 %> 60 %0 / 1 tight
PSI io maxobserverbench-observer-112.2 %2.2 %> 20 %0 / 0 headroom
Root disk maxobserverbench-observer-1118.2 %18.2 %> 80 %0 / 0 headroom
Network p95observerbench-observer-1121 Mbps2.1 %> 50 %0 / 0 headroom
Consensus cache peakobserver–––info– not collected
OOM killsobserver––anyany– none
Disk throttlingobserver––metric type unavailable in projectany– not collected
Disk performance degradedobserver––any sampleany– none

Per node

NodeClassCPU p95Memory used p95telcoin RSS p95PSI memory maxRssAnon peakDisk write p95Write IOPS p95PSI io maxRoot disk maxNetwork p95Consensus cache peak
Line> 70 %> 75 %> 85 %> 5 %info> 60 %> 60 %> 20 %> 80 %> 50 %info
bench-validator-01validator41.9 %21.0 %6.60 GB 38 %0.0 %2.78 GB 16 %132 MB/s 16 %26,160 52 %2.6 %20.7 %138 Mbps 14 %–
bench-validator-02validator38.4 %22.6 %6.87 GB 40 %0.0 %3.04 GB 18 %128 MB/s 15 %25,602 51 %4.0 %20.9 %129 Mbps 13 %–
bench-validator-03validator43.3 %21.2 %6.58 GB 38 %0.0 %2.75 GB 16 %132 MB/s 16 %26,116 52 %2.9 %20.6 %144 Mbps 14 %–
bench-validator-04validator39.8 %20.6 %6.56 GB 38 %0.0 %2.64 GB 15 %130 MB/s 16 %25,953 52 %3.0 %20.5 %134 Mbps 13 %–
bench-validator-05validator39.4 %19.6 %7.56 GB 44 %0.0 %2.30 GB 13 %129 MB/s 15 %24,984 50 %4.9 %21.3 %126 Mbps 13 %–
bench-validator-06validator39.4 %25.1 %6.97 GB 41 %0.0 %3.43 GB 20 %133 MB/s 16 %26,277 53 %3.4 %20.4 %132 Mbps 13 %–
bench-validator-07validator42.7 %20.5 %6.47 GB 38 %0.0 %2.63 GB 15 %133 MB/s 16 %26,113 52 %3.0 %20.4 %149 Mbps 15 %–
bench-validator-08validator42.7 %22.9 %6.87 GB 40 %0.0 %3.04 GB 18 %133 MB/s 16 %26,414 53 %2.9 %20.5 %142 Mbps 14 %–
bench-validator-09validator42.5 %22.2 %6.70 GB 39 %0.0 %2.95 GB 17 %132 MB/s 16 %25,990 52 %2.8 %20.5 %138 Mbps 14 %–
bench-validator-10validator40.6 %21.5 %6.52 GB 38 %0.0 %2.97 GB 17 %133 MB/s 16 %25,953 52 %3.5 %20.5 %139 Mbps 14 %–
bench-observer-11observer24.2 %11.1 %5.22 GB 30 %0.0 %0.98 GB 6 %122 MB/s 15 %26,139 52 %2.2 %18.2 %21 Mbps 2 %–

Small figures are the share of capacity: RAM for RSS, the PD limit for disk write (VM cap applied), 1 Gbps for network. Hover a cell for its JSON path.

Verdict

Fleet A vs Fleet B

A = Fleet c3-hdb · c3-highcpu-8 · r20261114-0000
B = Fleet c3-hdb · c3-highcpu-8 · r20261113-1630

Verdict

tight c3-highcpu-8 validators at 8,337 sustained TPS: tight on disk. Disk tight: write IOPS p95 26,414, 52.8 % of the 50,000 IOPS limit (bench-validator-08), tight against the 60 % line. CPU has headroom: CPU p95 43.3 % of 8 vCPUs (bench-validator-03). Memory has headroom: memory used p95 25.1 % of RAM (bench-validator-06); telcoin RSS p95 7.56 GB, 44.0 % of 16 GiB RAM (bench-validator-05); memory pressure (PSI some avg10) max 0.0 % (bench-validator-01). Network has headroom: network p95 149 Mbps, 14.9 % of 1 Gbps (bench-validator-07).

tight c3-highcpu-8 observers at 8,337 sustained TPS: tight on disk. Disk tight: write IOPS p95 26,139, 52.3 % of the 50,000 IOPS limit (bench-observer-11), tight against the 60 % line. CPU has headroom: CPU p95 24.2 % of 8 vCPUs (bench-observer-11). Memory has headroom: memory used p95 11.1 % of RAM (bench-observer-11); telcoin RSS p95 5.22 GB, 30.4 % of 16 GiB RAM (bench-observer-11); memory pressure (PSI some avg10) max 0.0 % (bench-observer-11). Network has headroom: network p95 21 Mbps, 2.1 % of 1 Gbps (bench-observer-11).

tight c3-highcpu-8 validators at 8,539 sustained TPS: tight on disk. Disk tight: write IOPS p95 27,058, 54.1 % of the 50,000 IOPS limit (bench-validator-09), tight against the 60 % line. CPU has headroom: CPU p95 43.6 % of 8 vCPUs (bench-validator-09). Memory has headroom: memory used p95 27.3 % of RAM (bench-validator-05); telcoin RSS p95 7.04 GB, 41.0 % of 16 GiB RAM (bench-validator-10); memory pressure (PSI some avg10) max 0.0 % (bench-validator-01). Network has headroom: network p95 153 Mbps, 15.3 % of 1 Gbps (bench-validator-07).

tight c3-highcpu-8 observers at 8,539 sustained TPS: tight on disk. Disk tight: write IOPS p95 27,047, 54.1 % of the 50,000 IOPS limit (bench-observer-11), tight against the 60 % line. CPU has headroom: CPU p95 23.9 % of 8 vCPUs (bench-observer-11). Memory has headroom: memory used p95 10.4 % of RAM (bench-observer-11); telcoin RSS p95 5.11 GB, 29.7 % of 16 GiB RAM (bench-observer-11); memory pressure (PSI some avg10) max 0.0 % (bench-observer-11). Network has headroom: network p95 11 Mbps, 1.1 % of 1 Gbps (bench-observer-11).

Validators: Fleet A (c3-highcpu-8) is tight on disk, with headroom on CPU, memory and network; Fleet B (c3-highcpu-8) is tight on disk, with headroom on CPU, memory and network.

Observers: Fleet A (c3-highcpu-8) is tight on disk, with headroom on CPU, memory and network; Fleet B (c3-highcpu-8) is tight on disk, with headroom on CPU, memory and network.

KPIFleet AFleet BB ÷ A
Sustained mixed TPS (own)8,337.1 tx/s8,539.0 tx/s1.02×
Sustained mixed TPS (chain)8,337.1 tx/s8,539.0 tx/s1.02×
Landed / accepted99.9 %99.9 %1.00×
Reverted (own)00–
Latency p50990 ms858 ms0.87×
Latency p953,879 ms3,528 ms0.91×
Gas / s (own)363M gas/s365M gas/s1.01×
Capacity at mix16,723.6 tx/s16,530.3 tx/s0.99×

Headroom, worst node per class

SignalClassFleet AFleet BLine
CPU p95validator43.3 % headroom43.6 % headroom> 70 %
Memory used p95validator25.1 % headroom27.3 % headroom> 75 %
telcoin RSS p95validator7.56 GB 44.0 % headroom7.04 GB 41.0 % headroom> 85 %
PSI memory maxvalidator0.0 % headroom0.0 % headroom> 5 %
Disk write p95validator133 MB/s 15.9 % headroom136 MB/s 16.2 % headroom> 60 %
Write IOPS p95validator26,414 52.8 % tight27,058 54.1 % tight> 60 %
PSI io maxvalidator4.9 % headroom4.1 % headroom> 20 %
Root disk maxvalidator21.3 % headroom20.1 % headroom> 80 %
Network p95validator149 Mbps 14.9 % headroom153 Mbps 15.3 % headroom> 50 %
CPU p95observer24.2 % headroom23.9 % headroom> 70 %
Memory used p95observer11.1 % headroom10.4 % headroom> 75 %
telcoin RSS p95observer5.22 GB 30.4 % headroom5.11 GB 29.7 % headroom> 85 %
PSI memory maxobserver0.0 % headroom0.0 % headroom> 5 %
Disk write p95observer122 MB/s 14.5 % headroom125 MB/s 14.9 % headroom> 60 %
Write IOPS p95observer26,139 52.3 % tight27,047 54.1 % tight> 60 %
PSI io maxobserver2.2 % headroom2.5 % headroom> 20 %
Root disk maxobserver18.2 % headroom18.6 % headroom> 80 %
Network p95observer21 Mbps 2.1 % headroom11 Mbps 1.1 % headroom> 50 %

Own TPS, trailing 60-s mean

aligned on each fleet's mixed-phase start

CPU, hottest validator

aligned on each fleet's mixed-phase start

Memory used, hottest validator

aligned on each fleet's mixed-phase start

telcoin RSS, hottest validator

aligned on each fleet's mixed-phase start

Disk MiB/s, hottest validator

aligned on each fleet's mixed-phase start

Network sent, hottest validator

aligned on each fleet's mixed-phase start

Reference chains

Not collected: reference chains.