Throughput is measured from blocks the chain produced (own senders vs every sender), resource use from Cloud Monitoring, Prometheus and the on-node sampler. Summaries cover the mixed phase. Hover any number for the JSON field it came from.
Landed counts only our own senders; latency is submit → block timestamp.
| Type | Submitted | Accepted | Landed | Reverted | Unlanded | Skips | Gas limit | Gas used avg | p50 | p95 |
|---|---|---|---|---|---|---|---|---|---|---|
| native_transfer | 4M | 3.57M | 3.57M | 0 | 1.61K | 0 | 24.1K | 21K | 3,291 ms | 12,779 ms |
| batched_self_send | 2.19M | 1.84M | 1.84M | 0 | 1.21K | 0 | 24.1K | 21K | 3,375 ms | 13,061 ms |
| uniswap_v2_swap | 1.98M | 1.66M | 1.66M | 0 | 1.22K | 0 | 180K | 106K | 3,299 ms | 12,748 ms |
| stablecoin_transfer | 1.64M | 1.38M | 1.37M | 0 | 978 | 0 | 75K | 43.9K | 3,294 ms | 12,748 ms |
| wtel_wrap | 1.32M | 1.11M | 1.11M | 0 | 814 | 0 | 50K | 27.9K | 3,294 ms | 12,757 ms |
| wtel_unwrap | 874K | 735K | 735K | 0 | 527 | 0 | 45K | 35.2K | 3,298 ms | 12,779 ms |
| stablecoin_approve | 656K | 552K | 552K | 0 | 397 | 0 | 60K | 31.8K | 3,299 ms | 12,755 ms |
| uniswap_v2_add_liquidity | 435K | 365K | 365K | 0 | 268 | 0 | 250K | 128K | 3,295 ms | 12,698 ms |
| contract_deploy | 326K | 274K | 274K | 0 | 202 | 0 | 140K | 74.2K | 3,297 ms | 12,765 ms |
| uniswap_v4_swap | 325K | 273K | 273K | 0 | 194 | 0 | 250K | 124K | 3,294 ms | 12,763 ms |
| uniswap_v3_swap | 221K | 186K | 185K | 0 | 126 | 0 | 220K | 120K | 3,303 ms | 12,750 ms |
| uniswap_v2_remove_liquidity | 220K | 185K | 185K | 0 | 127 | 0 | 220K | 129K | 3,303 ms | 12,744 ms |
| faucet_drip | 126K | 119K | 119K | 0 | 1 | 423K | 150K | 107K | 2,780 ms | 15,269 ms |
| Node | Submitted | Accepted | Accepted % | Top errors | Send p50 | Send p95 |
|---|---|---|---|---|---|---|
| bench-validator-01 bench-validator-01:8545 | 1.7M | 1.69M | 99.5 % | known 3.56K | 6 ms | 19 ms |
| bench-validator-02 bench-validator-02:8545 | 1.74M | 1.74M | 99.6 % | known 3.17K | 7 ms | 24 ms |
| bench-validator-03 bench-validator-03:8545 | 1.53M | 1.51M | 98.6 % | known 12.5K | 8 ms | 24 ms |
| bench-validator-04 bench-validator-04:8545 | 1.57M | 1.57M | 99.9 % | known 194 | 9 ms | 30 ms |
| bench-validator-05 bench-validator-05:8545 | 1.7M | 1.25M | 73.7 % | other 447K, known 390 | 6 ms | 24 ms |
| bench-validator-06 bench-validator-06:8545 | 1.31M | 1.28M | 97.8 % | known 13.2K | 7 ms | 25 ms |
| bench-validator-07 bench-validator-07:8545 | 3.02M | 532K | 17.6 % | other 2.49M, known 63 | 1 ms | 11 ms |
| bench-validator-08 bench-validator-08:8545 | 3.41M | 163K | 4.8 % | other 3.24M, known 1.31K | 1 ms | 3 ms |
| bench-validator-09 bench-validator-09:8545 | 1.36M | 1.34M | 98.2 % | known 12.4K | 9 ms | 30 ms |
| bench-validator-10 bench-validator-10:8545 | 1.28M | 1.25M | 97.9 % | known 13.1K | 8 ms | 24 ms |
Shaded: phases outside the mixed window. Hairlines: epoch closes. Dashed orange: chaos events.
own senders vs chain total, per block-timestamp second
own vs chain gas used, per block-timestamp second
compute.googleapis.com/instance/cpu/utilization, 60-s mean, % of all vCPUs; summaries over the mixed phase
| Node | Class | Mean | p95 | Peak 60-s mean | Max |
|---|---|---|---|---|---|
| bench-observer-11 | observer | 25.2 % | 27.1 % | 27.5 % | 27.5 % |
| bench-validator-01 | validator | 34.9 % | 36.2 % | 36.7 % | 36.7 % |
| bench-validator-02 | validator | 34.4 % | 35.9 % | 37.4 % | 37.4 % |
| bench-validator-03 | validator | 34.6 % | 36.5 % | 37.2 % | 37.2 % |
| bench-validator-04 | validator | 34.8 % | 36.3 % | 36.7 % | 36.7 % |
| bench-validator-05 | validator | 28.7 % | 35.9 % | 38.0 % | 38.0 % |
| bench-validator-06 | validator | 34.5 % | 36.3 % | 36.9 % | 36.9 % |
| bench-validator-07 | validator | 34.1 % | 36.0 % | 36.8 % | 36.8 % |
| bench-validator-08 | validator | 31.9 % | 34.1 % | 34.8 % | 34.8 % |
| bench-validator-09 | validator | 34.5 % | 36.5 % | 37.0 % | 37.0 % |
| bench-validator-10 | validator | 34.4 % | 36.4 % | 37.3 % | 37.3 % |
One row per node, over the mixed phase.
agent processes/rss_usage summed over telcoin pids; summaries over the mixed phase
| Node | Class | Mean | p95 | Peak 60-s mean | Max |
|---|---|---|---|---|---|
| bench-observer-11 | observer | 2.50 GiB | 3.22 GiB | 3.40 GiB | 3.40 GiB |
| bench-validator-01 | validator | 6.39 GiB | 8.05 GiB | 8.25 GiB | 8.25 GiB |
| bench-validator-02 | validator | 5.87 GiB | 7.40 GiB | 7.50 GiB | 7.50 GiB |
| bench-validator-03 | validator | 5.97 GiB | 8.37 GiB | 8.72 GiB | 8.72 GiB |
| bench-validator-04 | validator | 5.76 GiB | 7.25 GiB | 7.55 GiB | 7.55 GiB |
| bench-validator-05 | validator | 4.63 GiB | 6.37 GiB | 6.53 GiB | 6.53 GiB |
| bench-validator-06 | validator | 6.74 GiB | 8.85 GiB | 8.96 GiB | 8.96 GiB |
| bench-validator-07 | validator | 6.14 GiB | 7.77 GiB | 8.06 GiB | 8.06 GiB |
| bench-validator-08 | validator | 6.70 GiB | 8.49 GiB | 8.79 GiB | 8.79 GiB |
| bench-validator-09 | validator | 6.86 GiB | 8.70 GiB | 8.91 GiB | 8.91 GiB |
| bench-validator-10 | validator | 6.77 GiB | 8.80 GiB | 9.03 GiB | 9.03 GiB |
One row per node, over the mixed phase.
agent memory/percent_used, state=used; summaries over the mixed phase
| Node | Class | Mean | p95 | Peak 60-s mean | Max |
|---|---|---|---|---|---|
| bench-observer-11 | observer | 12.4 % | 13.5 % | 13.7 % | 13.7 % |
| bench-validator-01 | validator | 35.0 % | 39.1 % | 39.3 % | 39.3 % |
| bench-validator-02 | validator | 31.8 % | 34.8 % | 35.4 % | 35.4 % |
| bench-validator-03 | validator | 32.5 % | 40.7 % | 41.6 % | 41.6 % |
| bench-validator-04 | validator | 31.2 % | 34.1 % | 34.3 % | 34.3 % |
| bench-validator-05 | validator | 23.6 % | 30.2 % | 32.6 % | 32.6 % |
| bench-validator-06 | validator | 37.5 % | 43.9 % | 44.0 % | 44.0 % |
| bench-validator-07 | validator | 33.6 % | 37.5 % | 37.9 % | 37.9 % |
| bench-validator-08 | validator | 36.9 % | 41.8 % | 41.8 % | 41.8 % |
| bench-validator-09 | validator | 37.9 % | 43.0 % | 44.4 % | 44.4 % |
| bench-validator-10 | validator | 37.3 % | 43.4 % | 43.6 % | 43.6 % |
One row per node, over the mixed phase.
instance/disk/{read,write}_bytes_count rate, all devices; summaries over the mixed phase
| Node | Class | Mean | p95 | Peak 60-s mean | Max | Limit |
|---|---|---|---|---|---|---|
| bench-observer-11 | observer | 225.1 MiB/s | 309.8 MiB/s | 328.6 MiB/s | 328.6 MiB/s | 800.0 MiB/s |
| bench-validator-01 | validator | 179.8 MiB/s | 231.8 MiB/s | 244.3 MiB/s | 244.3 MiB/s | 800.0 MiB/s |
| bench-validator-02 | validator | 167.4 MiB/s | 217.0 MiB/s | 242.4 MiB/s | 242.4 MiB/s | 800.0 MiB/s |
| bench-validator-03 | validator | 166.8 MiB/s | 222.7 MiB/s | 246.7 MiB/s | 246.7 MiB/s | 800.0 MiB/s |
| bench-validator-04 | validator | 170.1 MiB/s | 224.6 MiB/s | 243.5 MiB/s | 243.5 MiB/s | 800.0 MiB/s |
| bench-validator-05 | validator | 134.7 MiB/s | 219.0 MiB/s | 241.5 MiB/s | 241.5 MiB/s | 800.0 MiB/s |
| bench-validator-06 | validator | 165.8 MiB/s | 223.5 MiB/s | 243.7 MiB/s | 243.7 MiB/s | 800.0 MiB/s |
| bench-validator-07 | validator | 187.8 MiB/s | 250.3 MiB/s | 273.1 MiB/s | 273.1 MiB/s | 800.0 MiB/s |
| bench-validator-08 | validator | 253.7 MiB/s | 310.1 MiB/s | 320.2 MiB/s | 320.2 MiB/s | 800.0 MiB/s |
| bench-validator-09 | validator | 164.6 MiB/s | 208.6 MiB/s | 220.7 MiB/s | 220.7 MiB/s | 800.0 MiB/s |
| bench-validator-10 | validator | 166.7 MiB/s | 217.4 MiB/s | 237.0 MiB/s | 237.0 MiB/s | 800.0 MiB/s |
instance/disk/{read,write}_ops_count rate, all devices; summaries over the mixed phase
| Node | Class | Mean | p95 | Peak 60-s mean | Max | Limit |
|---|---|---|---|---|---|---|
| bench-observer-11 | observer | 6.23K IOPS | 9.19K IOPS | 9.54K IOPS | 9.54K IOPS | 20K IOPS |
| bench-validator-01 | validator | 7.35K IOPS | 9.32K IOPS | 10.3K IOPS | 10.3K IOPS | 20K IOPS |
| bench-validator-02 | validator | 7.65K IOPS | 9.67K IOPS | 9.86K IOPS | 9.86K IOPS | 20K IOPS |
| bench-validator-03 | validator | 7.68K IOPS | 9.67K IOPS | 9.83K IOPS | 9.83K IOPS | 20K IOPS |
| bench-validator-04 | validator | 7.57K IOPS | 9.58K IOPS | 10K IOPS | 10K IOPS | 20K IOPS |
| bench-validator-05 | validator | 6.13K IOPS | 9.4K IOPS | 9.96K IOPS | 9.96K IOPS | 20K IOPS |
| bench-validator-06 | validator | 7.7K IOPS | 9.76K IOPS | 9.97K IOPS | 9.97K IOPS | 20K IOPS |
| bench-validator-07 | validator | 6.98K IOPS | 9.51K IOPS | 9.58K IOPS | 9.58K IOPS | 20K IOPS |
| bench-validator-08 | validator | 5.09K IOPS | 8.01K IOPS | 8.84K IOPS | 8.84K IOPS | 20K IOPS |
| bench-validator-09 | validator | 7.67K IOPS | 9.43K IOPS | 9.73K IOPS | 9.73K IOPS | 20K IOPS |
| bench-validator-10 | validator | 7.67K IOPS | 9.92K IOPS | 10.1K IOPS | 10.1K IOPS | 20K IOPS |
| Node | Machine | vCPU | Disk | IOPS limit | MiB/s limit | Basis |
|---|---|---|---|---|---|---|
| bench-observer-11 | c3-highcpu-8 | 8 | hyperdisk-balanced 100 GB | 20K IOPS | 800.0 MiB/s | hyperdisk-balanced 100 GB: provisioned 20,000 IOPS, provisioned 800 MiB/s; c3 8 vCPU cap 50,000 IOPS / 800 MiB/s; reads and writes share this budget |
| bench-validator-01 | c3-highcpu-8 | 8 | hyperdisk-balanced 100 GB | 20K IOPS | 800.0 MiB/s | hyperdisk-balanced 100 GB: provisioned 20,000 IOPS, provisioned 800 MiB/s; c3 8 vCPU cap 50,000 IOPS / 800 MiB/s; reads and writes share this budget |
| bench-validator-02 | c3-highcpu-8 | 8 | hyperdisk-balanced 100 GB | 20K IOPS | 800.0 MiB/s | hyperdisk-balanced 100 GB: provisioned 20,000 IOPS, provisioned 800 MiB/s; c3 8 vCPU cap 50,000 IOPS / 800 MiB/s; reads and writes share this budget |
| bench-validator-03 | c3-highcpu-8 | 8 | hyperdisk-balanced 100 GB | 20K IOPS | 800.0 MiB/s | hyperdisk-balanced 100 GB: provisioned 20,000 IOPS, provisioned 800 MiB/s; c3 8 vCPU cap 50,000 IOPS / 800 MiB/s; reads and writes share this budget |
| bench-validator-04 | c3-highcpu-8 | 8 | hyperdisk-balanced 100 GB | 20K IOPS | 800.0 MiB/s | hyperdisk-balanced 100 GB: provisioned 20,000 IOPS, provisioned 800 MiB/s; c3 8 vCPU cap 50,000 IOPS / 800 MiB/s; reads and writes share this budget |
| bench-validator-05 | c3-highcpu-8 | 8 | hyperdisk-balanced 100 GB | 20K IOPS | 800.0 MiB/s | hyperdisk-balanced 100 GB: provisioned 20,000 IOPS, provisioned 800 MiB/s; c3 8 vCPU cap 50,000 IOPS / 800 MiB/s; reads and writes share this budget |
| bench-validator-06 | c3-highcpu-8 | 8 | hyperdisk-balanced 100 GB | 20K IOPS | 800.0 MiB/s | hyperdisk-balanced 100 GB: provisioned 20,000 IOPS, provisioned 800 MiB/s; c3 8 vCPU cap 50,000 IOPS / 800 MiB/s; reads and writes share this budget |
| bench-validator-07 | c3-highcpu-8 | 8 | hyperdisk-balanced 100 GB | 20K IOPS | 800.0 MiB/s | hyperdisk-balanced 100 GB: provisioned 20,000 IOPS, provisioned 800 MiB/s; c3 8 vCPU cap 50,000 IOPS / 800 MiB/s; reads and writes share this budget |
| bench-validator-08 | c3-highcpu-8 | 8 | hyperdisk-balanced 100 GB | 20K IOPS | 800.0 MiB/s | hyperdisk-balanced 100 GB: provisioned 20,000 IOPS, provisioned 800 MiB/s; c3 8 vCPU cap 50,000 IOPS / 800 MiB/s; reads and writes share this budget |
| bench-validator-09 | c3-highcpu-8 | 8 | hyperdisk-balanced 100 GB | 20K IOPS | 800.0 MiB/s | hyperdisk-balanced 100 GB: provisioned 20,000 IOPS, provisioned 800 MiB/s; c3 8 vCPU cap 50,000 IOPS / 800 MiB/s; reads and writes share this budget |
| bench-validator-10 | c3-highcpu-8 | 8 | hyperdisk-balanced 100 GB | 20K IOPS | 800.0 MiB/s | hyperdisk-balanced 100 GB: provisioned 20,000 IOPS, provisioned 800 MiB/s; c3 8 vCPU cap 50,000 IOPS / 800 MiB/s; reads and writes share this budget |
instance/network/sent_bytes_count rate; summaries over the mixed phase
| Node | Class | Mean | p95 | Peak 60-s mean | Max |
|---|---|---|---|---|---|
| bench-observer-11 | observer | 0.9 Mbps | 1.2 Mbps | 1.3 Mbps | 1.3 Mbps |
| bench-validator-01 | validator | 81.8 Mbps | 85.8 Mbps | 89.0 Mbps | 89.0 Mbps |
| bench-validator-02 | validator | 81.8 Mbps | 86.9 Mbps | 88.5 Mbps | 88.5 Mbps |
| bench-validator-03 | validator | 83.5 Mbps | 99.1 Mbps | 126.2 Mbps | 126.2 Mbps |
| bench-validator-04 | validator | 81.3 Mbps | 87.5 Mbps | 96.6 Mbps | 96.6 Mbps |
| bench-validator-05 | validator | 66.0 Mbps | 85.3 Mbps | 88.9 Mbps | 88.9 Mbps |
| bench-validator-06 | validator | 80.0 Mbps | 86.4 Mbps | 88.2 Mbps | 88.2 Mbps |
| bench-validator-07 | validator | 77.5 Mbps | 82.7 Mbps | 85.7 Mbps | 85.7 Mbps |
| bench-validator-08 | validator | 73.0 Mbps | 79.2 Mbps | 83.3 Mbps | 83.3 Mbps |
| bench-validator-09 | validator | 85.5 Mbps | 103.5 Mbps | 119.2 Mbps | 119.2 Mbps |
| bench-validator-10 | validator | 86.6 Mbps | 99.1 Mbps | 155.0 Mbps | 155.0 Mbps |
instance/network/received_bytes_count rate; summaries over the mixed phase
| Node | Class | Mean | p95 | Peak 60-s mean | Max |
|---|---|---|---|---|---|
| bench-observer-11 | observer | 5.9 Mbps | 13.3 Mbps | 40.0 Mbps | 40.0 Mbps |
| bench-validator-01 | validator | 8.1 Mbps | 9.0 Mbps | 9.4 Mbps | 9.4 Mbps |
| bench-validator-02 | validator | 8.1 Mbps | 8.7 Mbps | 9.7 Mbps | 9.7 Mbps |
| bench-validator-03 | validator | 7.8 Mbps | 8.9 Mbps | 9.0 Mbps | 9.0 Mbps |
| bench-validator-04 | validator | 7.8 Mbps | 8.9 Mbps | 9.7 Mbps | 9.7 Mbps |
| bench-validator-05 | validator | 10.8 Mbps | 19.3 Mbps | 62.8 Mbps | 62.8 Mbps |
| bench-validator-06 | validator | 7.4 Mbps | 8.6 Mbps | 8.8 Mbps | 8.8 Mbps |
| bench-validator-07 | validator | 16.2 Mbps | 33.2 Mbps | 55.0 Mbps | 55.0 Mbps |
| bench-validator-08 | validator | 17.4 Mbps | 30.9 Mbps | 54.9 Mbps | 54.9 Mbps |
| bench-validator-09 | validator | 7.5 Mbps | 8.8 Mbps | 9.0 Mbps | 9.0 Mbps |
| bench-validator-10 | validator | 7.5 Mbps | 8.7 Mbps | 8.7 Mbps | 8.7 Mbps |
| Metric | Query | Class | Nodes | Median of node means | Worst peak 60-s mean | Max |
|---|---|---|---|---|---|---|
| tn_primary_commit_latency_seconds | p95 | validator | 9 | 4,343 ms | 30.0 s | 30.0 s |
| tn_engine_execution_duration_seconds | p95 | validator | 10 | 5,781 ms | 23.9 s | 26.5 s |
| tn_engine_execution_duration_seconds | p95 | observer | 1 | 5,611 ms | 9,372 ms | 9,550 ms |
| tn_worker_batch_size_bytes | p95 | validator | 9 | 241.8 KiB | 246.4 KiB | 246.4 KiB |
| tn_worker_batch_transactions | p95 | validator | 9 | 484 | 1.48K | 1.48K |
| tn_executor_output_transactions | p95 | validator | 10 | 29.1K | 46.2K | 47.2K |
| tn_primary_consensus_output_bytes | max | validator | 10 | 1016.0 KiB | 2.1 MiB | 3.0 MiB |
| tn_primary_consensus_output_bytes | max | observer | 1 | 1013.8 KiB | 2.0 MiB | 2.6 MiB |
| tn_engine_queued_outputs | max | validator | 10 | 3.31 | 8 | 8 |
| tn_engine_queued_outputs | max | observer | 1 | 4.6 | 8 | 8 |
| tn_primary_round | max | validator | 10 | 715 | 2.94K | 2.96K |
| tn_primary_round | max | observer | 1 | 715 | 2.94K | 2.95K |
| tn_primary_committed_round | max | validator | 10 | 713 | 2.94K | 2.95K |
| tn_primary_committed_round | max | observer | 1 | 715 | 2.94K | 2.95K |
| tn_epoch_current | max | validator | 10 | 3.35 | 4 | 4 |
| tn_epoch_current | max | observer | 1 | 3.34 | 4 | 4 |
| tn_batch_builder_pending_pool_transactions | max | validator | 10 | 515 | 9.62K | 9.62K |
| tn_batch_builder_pending_pool_transactions | max | observer | 1 | 0 | 0 | 0 |
| reth_process_resident_memory_bytes | max | validator | 10 | 6.39 GiB | 9.04 GiB | 9.11 GiB |
| reth_process_resident_memory_bytes | max | observer | 1 | 2.50 GiB | 3.39 GiB | 3.40 GiB |
| tn_primary_subdags_committed_total | rate | validator | 10 | 0.432 | 0.9 | 1.09 |
| tn_primary_subdags_committed_total | rate | observer | 1 | 0 | 0 | 0 |
| tn_primary_round_lag | lag | validator | 10 | 2.05 | 29.8 | 38 |
| tn_primary_round_lag | lag | observer | 1 | 0 | 0 | 0 |
| tn_batch_builder_seal_failures_total | rate | validator | 4 | 0 | 0.0167 | 0.0222 |
| tn_node_mode | max | validator | 10 | 0 | 1 | 1 |
| tn_node_mode | max | observer | 1 | 0 | 0 | 0 |
| tn_primary_vote_requests_total | rate | validator | 10 | 6.75 | 19.3 | 21.4 |
| tn_primary_vote_requests_total | rate | observer | 1 | 0 | 0 | 0 |
| tn_primary_votes_created_total | rate | validator | 10 | 6.69 | 16.7 | 19.3 |
| tn_primary_votes_created_total | rate | observer | 1 | 0 | 0 | 0 |
| tn_primary_votes_recast_total | rate | validator | 10 | 0 | 0 | 0 |
| tn_primary_votes_recast_total | rate | observer | 1 | 0 | 0 | 0 |
| tn_primary_vote_requests_too_new_total | rate | validator | 10 | 0 | 11.3 | 13.4 |
| tn_primary_vote_requests_too_new_total | rate | observer | 1 | 0 | 0 | 0 |
| tn_primary_vote_timeouts_total{stage="exec_wait"} | rate | validator | 10 | 0 | 0.539 | 0.822 |
| tn_primary_vote_timeouts_total{stage="exec_wait"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_primary_vote_timeouts_total{stage="lead_sleep"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_primary_vote_timeouts_total{stage="lead_sleep"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_primary_vote_timeouts_total{stage="parents"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_primary_vote_timeouts_total{stage="parents"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_primary_vote_timeouts_total{stage="persist"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_primary_vote_timeouts_total{stage="persist"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_primary_vote_timeouts_total{stage="sync_batches"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_primary_vote_timeouts_total{stage="sync_batches"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_primary_vote_timeouts_total{stage="validate"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_primary_vote_timeouts_total{stage="validate"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_primary_vote_timeouts_total{stage="vote_store"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_primary_vote_timeouts_total{stage="vote_store"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_primary_vote_stage_seconds{stage="exec_wait"} | p95 | validator | 8 | 1,764 ms | 9,458 ms | 9,583 ms |
| tn_primary_vote_stage_seconds{stage="lead_sleep"} | p95 | validator | 8 | 0 ms | 0 ms | 0 ms |
| tn_primary_vote_stage_seconds{stage="lock"} | p95 | validator | 8 | 298 ms | 9,750 ms | 9,750 ms |
| tn_primary_vote_stage_seconds{stage="parents"} | p95 | validator | 8 | 1 ms | 3 ms | 4 ms |
| tn_primary_vote_stage_seconds{stage="persist"} | p95 | validator | 8 | 45 ms | 175 ms | 186 ms |
| tn_primary_vote_stage_seconds{stage="sync_batches"} | p95 | validator | 8 | 0 ms | 1 ms | 2 ms |
| tn_primary_vote_stage_seconds{stage="validate"} | p95 | validator | 8 | 3 ms | 5 ms | 5 ms |
| tn_primary_vote_stage_seconds{stage="vote_store"} | p95 | validator | 8 | 1 ms | 12 ms | 15 ms |
| tn_primary_leader_rounds_total{leader="2UYxCFESCqw61C7pEFpzxAzcdu1aeTFTRC6pcgZivJG2",swapped="false",found="false"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_primary_leader_rounds_total{leader="2UYxCFESCqw61C7pEFpzxAzcdu1aeTFTRC6pcgZivJG2",swapped="false",found="true"} | rate | validator | 10 | 0.0285 | 0.0778 | 0.111 |
| tn_primary_leader_rounds_total{leader="2UYxCFESCqw61C7pEFpzxAzcdu1aeTFTRC6pcgZivJG2",swapped="true",found="false"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_primary_leader_rounds_total{leader="2UYxCFESCqw61C7pEFpzxAzcdu1aeTFTRC6pcgZivJG2",swapped="true",found="true"} | rate | validator | 10 | 0.00799 | 0.0667 | 0.0889 |
| tn_primary_leader_rounds_total{leader="3GBP8A9iAAinoV4Cme8WzPoXfYRkErZ8GHx78FgaZazB",swapped="false",found="false"} | rate | validator | 10 | 0.00471 | 0.0556 | 0.0667 |
| tn_primary_leader_rounds_total{leader="3GBP8A9iAAinoV4Cme8WzPoXfYRkErZ8GHx78FgaZazB",swapped="false",found="true"} | rate | validator | 10 | 0.0383 | 0.0889 | 0.111 |
| tn_primary_leader_rounds_total{leader="3GBP8A9iAAinoV4Cme8WzPoXfYRkErZ8GHx78FgaZazB",swapped="true",found="false"} | rate | validator | 10 | 0.0025 | 0.0667 | 0.0889 |
| tn_primary_leader_rounds_total{leader="3GBP8A9iAAinoV4Cme8WzPoXfYRkErZ8GHx78FgaZazB",swapped="true",found="true"} | rate | validator | 10 | 0.0118 | 0.0611 | 0.0889 |
| tn_primary_leader_rounds_total{leader="3VrEiVQncxBCKqm5AN8KTtvrSShmeCT6bBQswYYtAutD",swapped="false",found="false"} | rate | validator | 10 | 0.0024 | 0.0556 | 0.0667 |
| tn_primary_leader_rounds_total{leader="3VrEiVQncxBCKqm5AN8KTtvrSShmeCT6bBQswYYtAutD",swapped="false",found="true"} | rate | validator | 10 | 0.0283 | 0.0944 | 0.111 |
| tn_primary_leader_rounds_total{leader="3VrEiVQncxBCKqm5AN8KTtvrSShmeCT6bBQswYYtAutD",swapped="true",found="false"} | rate | validator | 10 | 0.00115 | 0.044 | 0.066 |
| tn_primary_leader_rounds_total{leader="3VrEiVQncxBCKqm5AN8KTtvrSShmeCT6bBQswYYtAutD",swapped="true",found="true"} | rate | validator | 10 | 0.01 | 0.0611 | 0.0889 |
| tn_primary_leader_rounds_total{leader="7CgjP1y53obfnVnshMfKtLaaDuMWD7Eys21AQjYjjNsn",swapped="false",found="false"} | rate | validator | 10 | 0.00208 | 0.0333 | 0.0444 |
| tn_primary_leader_rounds_total{leader="7CgjP1y53obfnVnshMfKtLaaDuMWD7Eys21AQjYjjNsn",swapped="false",found="true"} | rate | validator | 10 | 0.0411 | 0.1 | 0.111 |
| tn_primary_leader_rounds_total{leader="7CgjP1y53obfnVnshMfKtLaaDuMWD7Eys21AQjYjjNsn",swapped="true",found="false"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_primary_leader_rounds_total{leader="7CgjP1y53obfnVnshMfKtLaaDuMWD7Eys21AQjYjjNsn",swapped="true",found="true"} | rate | validator | 10 | 0.0131 | 0.0667 | 0.0889 |
| tn_primary_leader_rounds_total{leader="AhKq47odXwQXCiQJqJRtvrpufzSh2twSUUijT6Jw4bWq",swapped="false",found="false"} | rate | validator | 10 | 0.00417 | 0.0667 | 0.0889 |
| tn_primary_leader_rounds_total{leader="AhKq47odXwQXCiQJqJRtvrpufzSh2twSUUijT6Jw4bWq",swapped="false",found="true"} | rate | validator | 10 | 0.00833 | 0.0944 | 0.111 |
| tn_primary_leader_rounds_total{leader="AhKq47odXwQXCiQJqJRtvrpufzSh2twSUUijT6Jw4bWq",swapped="true",found="false"} | rate | validator | 10 | 0.00163 | 0.0444 | 0.0667 |
| tn_primary_leader_rounds_total{leader="AhKq47odXwQXCiQJqJRtvrpufzSh2twSUUijT6Jw4bWq",swapped="true",found="true"} | rate | validator | 10 | 0.00333 | 0.0556 | 0.0667 |
| tn_primary_leader_rounds_total{leader="CsBXvWKVeeTEDzk4kqHeRd7gjHs9VSTU68Z5Zsrrw2AJ",swapped="false",found="false"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_primary_leader_rounds_total{leader="CsBXvWKVeeTEDzk4kqHeRd7gjHs9VSTU68Z5Zsrrw2AJ",swapped="false",found="true"} | rate | validator | 10 | 0.0457 | 0.1 | 0.111 |
| tn_primary_leader_rounds_total{leader="CsBXvWKVeeTEDzk4kqHeRd7gjHs9VSTU68Z5Zsrrw2AJ",swapped="true",found="false"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_primary_leader_rounds_total{leader="CsBXvWKVeeTEDzk4kqHeRd7gjHs9VSTU68Z5Zsrrw2AJ",swapped="true",found="true"} | rate | validator | 10 | 0.0151 | 0.0444 | 0.0667 |
| tn_primary_leader_rounds_total{leader="G38nSkpHUZfYUQEeXNzD5SuoKjyapZU5WEgwe5tVddye",swapped="false",found="false"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_primary_leader_rounds_total{leader="G38nSkpHUZfYUQEeXNzD5SuoKjyapZU5WEgwe5tVddye",swapped="false",found="true"} | rate | validator | 10 | 0.0453 | 0.0944 | 0.111 |
| tn_primary_leader_rounds_total{leader="G38nSkpHUZfYUQEeXNzD5SuoKjyapZU5WEgwe5tVddye",swapped="true",found="false"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_primary_leader_rounds_total{leader="G38nSkpHUZfYUQEeXNzD5SuoKjyapZU5WEgwe5tVddye",swapped="true",found="true"} | rate | validator | 10 | 0.0171 | 0.0667 | 0.0667 |
| tn_primary_leader_rounds_total{leader="GYZ77es5DWi4ChjbRqcGQBFdM63pQHLLWBX4jhhNznBR",swapped="false",found="false"} | rate | validator | 10 | 0.00283 | 0.0609 | 0.0667 |
| tn_primary_leader_rounds_total{leader="GYZ77es5DWi4ChjbRqcGQBFdM63pQHLLWBX4jhhNznBR",swapped="false",found="true"} | rate | validator | 10 | 0.0213 | 0.0944 | 0.111 |
| tn_primary_leader_rounds_total{leader="GYZ77es5DWi4ChjbRqcGQBFdM63pQHLLWBX4jhhNznBR",swapped="true",found="false"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_primary_leader_rounds_total{leader="GYZ77es5DWi4ChjbRqcGQBFdM63pQHLLWBX4jhhNznBR",swapped="true",found="true"} | rate | validator | 10 | 0.0112 | 0.0833 | 0.111 |
| tn_primary_leader_rounds_total{leader="HDKF9cCXBezTpvjiz14LQiEZCfn7MY2SY3pBoNuRauMn",swapped="false",found="false"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_primary_leader_rounds_total{leader="HDKF9cCXBezTpvjiz14LQiEZCfn7MY2SY3pBoNuRauMn",swapped="false",found="true"} | rate | validator | 10 | 0.0428 | 0.0833 | 0.111 |
| tn_primary_leader_rounds_total{leader="HDKF9cCXBezTpvjiz14LQiEZCfn7MY2SY3pBoNuRauMn",swapped="true",found="false"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_primary_leader_rounds_total{leader="HDKF9cCXBezTpvjiz14LQiEZCfn7MY2SY3pBoNuRauMn",swapped="true",found="true"} | rate | validator | 10 | 0.00983 | 0.05 | 0.0667 |
| tn_primary_leader_rounds_total{leader="vp782ooaY6XgWBo6vH5D844aXbcVpYdaxwJTHi7CEWs",swapped="false",found="false"} | rate | validator | 10 | 0.000773 | 0.033 | 0.044 |
| tn_primary_leader_rounds_total{leader="vp782ooaY6XgWBo6vH5D844aXbcVpYdaxwJTHi7CEWs",swapped="false",found="true"} | rate | validator | 10 | 0.0253 | 0.0833 | 0.111 |
| tn_primary_leader_rounds_total{leader="vp782ooaY6XgWBo6vH5D844aXbcVpYdaxwJTHi7CEWs",swapped="true",found="false"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_primary_leader_rounds_total{leader="vp782ooaY6XgWBo6vH5D844aXbcVpYdaxwJTHi7CEWs",swapped="true",found="true"} | rate | validator | 10 | 0.00806 | 0.0611 | 0.0889 |
| tn_primary_leader_commits_total{leader="2UYxCFESCqw61C7pEFpzxAzcdu1aeTFTRC6pcgZivJG2"} | rate | validator | 10 | 0.0367 | 0.111 | 0.156 |
| tn_primary_leader_commits_total{leader="3GBP8A9iAAinoV4Cme8WzPoXfYRkErZ8GHx78FgaZazB"} | rate | validator | 10 | 0.0501 | 0.111 | 0.133 |
| tn_primary_leader_commits_total{leader="3VrEiVQncxBCKqm5AN8KTtvrSShmeCT6bBQswYYtAutD"} | rate | validator | 10 | 0.0383 | 0.133 | 0.178 |
| tn_primary_leader_commits_total{leader="7CgjP1y53obfnVnshMfKtLaaDuMWD7Eys21AQjYjjNsn"} | rate | validator | 10 | 0.0535 | 0.133 | 0.2 |
| tn_primary_leader_commits_total{leader="AhKq47odXwQXCiQJqJRtvrpufzSh2twSUUijT6Jw4bWq"} | rate | validator | 10 | 0.0117 | 0.139 | 0.156 |
| tn_primary_leader_commits_total{leader="CsBXvWKVeeTEDzk4kqHeRd7gjHs9VSTU68Z5Zsrrw2AJ"} | rate | validator | 10 | 0.0608 | 0.133 | 0.156 |
| tn_primary_leader_commits_total{leader="G38nSkpHUZfYUQEeXNzD5SuoKjyapZU5WEgwe5tVddye"} | rate | validator | 10 | 0.0624 | 0.144 | 0.156 |
| tn_primary_leader_commits_total{leader="GYZ77es5DWi4ChjbRqcGQBFdM63pQHLLWBX4jhhNznBR"} | rate | validator | 10 | 0.0325 | 0.139 | 0.178 |
| tn_primary_leader_commits_total{leader="HDKF9cCXBezTpvjiz14LQiEZCfn7MY2SY3pBoNuRauMn"} | rate | validator | 10 | 0.0526 | 0.122 | 0.156 |
| tn_primary_leader_commits_total{leader="vp782ooaY6XgWBo6vH5D844aXbcVpYdaxwJTHi7CEWs"} | rate | validator | 10 | 0.0333 | 0.144 | 0.178 |
| tn_engine_queue_wait_seconds | p95 | validator | 10 | 15.8 s | 29.7 s | 30.0 s |
| tn_engine_queue_wait_seconds | p95 | observer | 1 | 26.0 s | 29.0 s | 29.0 s |
| tn_engine_persist_duration_seconds | p95 | validator | 8 | 1,148 ms | 8,469 ms | 9,125 ms |
| tn_engine_persist_duration_seconds | p95 | observer | 1 | 1,699 ms | 4,365 ms | 4,583 ms |
| tn_engine_busy_milliseconds_total | rate | validator | 10 | 965 | 1.12K | 1.21K |
| tn_engine_busy_milliseconds_total | rate | observer | 1 | 981 | 1.05K | 1.1K |
| tn_reth_persist_save_blocks_seconds | p95 | validator | 9 | 413 ms | 8,469 ms | 9,125 ms |
| tn_reth_persist_save_blocks_seconds | p95 | observer | 1 | 615 ms | 4,083 ms | 4,583 ms |
| tn_reth_persist_commit_seconds | p95 | validator | 9 | 780 ms | 1,988 ms | 2,250 ms |
| tn_reth_persist_commit_seconds | p95 | observer | 1 | 934 ms | 1,065 ms | 1,300 ms |
| tn_reth_persisted_accounts_total | rate | validator | 10 | 2K | 2.36K | 2.53K |
| tn_reth_persisted_accounts_total | rate | observer | 1 | 1.98K | 2.34K | 2.43K |
| tn_reth_persisted_storage_slots_total | rate | validator | 10 | 2.76K | 3.28K | 3.39K |
| tn_reth_persisted_storage_slots_total | rate | observer | 1 | 2.73K | 3.18K | 3.27K |
| tn_node_mode_transitions_total{to="cvv_inactive",reason="epoch_behind"} | rate | validator | 1 | 0 | 0 | 0 |
| tn_node_mode_transitions_total{to="cvv_inactive",reason="outside_gc_window"} | rate | validator | 2 | 0 | 0 | 0 |
| tn_executor_rejoin_blocked_total{reason="epoch_closed"} | rate | validator | 3 | 0.0242 | 0.383 | 0.4 |
| tn_executor_rejoin_blocked_total{reason="epoch_mismatch"} | rate | validator | 3 | 0 | 0 | 0 |
| tn_executor_rejoin_blocked_total{reason="no_known_header"} | rate | validator | 3 | 0 | 0 | 0 |
| tn_executor_rejoin_blocked_total{reason="round_gap"} | rate | validator | 3 | 0.224 | 0.428 | 1.11 |
| tn_executor_rejoin_blocked_total{reason="stale_head"} | rate | validator | 3 | 0.112 | 0.294 | 0.356 |
| tn_worker_sync_serves_denied_total{worker="0",tier="committee"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_worker_sync_serves_denied_total{worker="0",tier="committee"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_worker_sync_serves_denied_total{worker="0",tier="public"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_worker_sync_serves_denied_total{worker="0",tier="public"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_worker_batch_requests_denied_total{worker="0",peer="2144FHRwrcvmEkUTAChHeoVdYn6vt1mQLCRLaSfVuieBgpKo9aMinwYFiqG8M3A61AEcbHZciXpvFFVYf9WipKxLTBUeZuDohtvK7FAtbmQBuc26DnJrRCY4tkqAhpZJkTGB"} | rate | validator | 2 | 0.00644 | 0.366 | 0.366 |
| tn_worker_batch_requests_denied_total{worker="0",peer="21hNc4w4BkZF6F6jQvFr1E36NpvJ2veVAzh1Zyy7C4kNgBfFCu423Ncw8t18vgbLt4PdtKeW8fZUJJA7emBzDbPnE8dDyY59bw64hftwyb8bNnXFDRXx1RUYabsqUNbBShYT"} | rate | validator | 3 | 0 | 0.507 | 0.507 |
| tn_worker_batch_requests_denied_total{worker="0",peer="nE5uq6wTBpXvRMnqjwm51T7ysmhfHshFJpXwuTpqcUBqqLyvgPVKF2i1vWik1vk1avXzV9Tu9bcrGJkSd2Yn2YhYMsZapcduTXK3Da2s9mrHJmqUA2ZyXXfuMd2nH3y1mty"} | rate | validator | 2 | 0.00639 | 0.338 | 0.338 |
| tn_worker_batch_requests_denied_total{worker="0",peer="nZtVqRH4VuyHtF3ASV74DMPQ93mDi1vcntRBsVoBwKWaz1tkbSLyGtkZD6wJq21M4znSfATuS9uVkgsb1TkxepdgKm9b6Rvug4aW2NpW1iKFNcHv7dBcmoz3ik4nT63PbzU"} | rate | validator | 1 | 0.00149 | 0.0844 | 0.0844 |
| tn_worker_batch_requests_denied_total{worker="0",peer="omnUG8kmgM38zGPxu8w39WpDqPtgjinaJV6pUPRtV81Phho5JxSaczk6PNrFC24bcWgL6Sucw2hBNrTFMMg94DAFfsPpYLgabACTpoHfq2kEocssSxyxjBmjDkrNCBtacp2"} | rate | validator | 2 | 0.0114 | 0.647 | 0.647 |
| tn_worker_batch_requests_denied_total{worker="0",peer="sHv24SnwtmMr3yzEHBqH5tFoNWDkzVdGLLMr8oaHDpcGSdiZV5SrK2jD5Fcp8J5zZVDkiZ1PGGaB7DYLJKCgtE1tvhexUZuR4Shz6gE2KTWb4xrGdPPySgX4YQXFzG4rMd8"} | rate | validator | 2 | 0 | 0 | 0 |
| tn_worker_batch_requests_denied_total{worker="0",peer="sUDaca6agCTkShRNgnS31xLNdovgGqqCSn1hxgvSd6aD29FJhnhubtkQikashEAecU5xJEJfuWERN9CMhzD17HJYbHpbeK9n4oMLeDwjg5uexrvpVexNCQ1j3TvRu9Ccigk"} | rate | validator | 1 | 0.00099 | 0.0563 | 0.0563 |
| tn_worker_batch_requests_denied_total{worker="0",peer="xDya61TkRgwqXhL621p5f5sqN4SxRG9RmTwLMdk2vAzqdXsqCtp5FTkUrPbw9zZTSm8s5b9TaDHvduCgFJgsSAVkavhmaj6gQmp6SVvogR2k9g9UaJuR7MWZBnfe7x5HgUP"} | rate | validator | 2 | 0.0114 | 0.647 | 0.647 |
| tn_worker_batch_requests_denied_total{worker="0",peer="xQAX3CxFYar64P2zTGvebVPT5JqMQe9pTyAnvLJmCHm7gxKWQ4SVcYho956GYuwVRUN5LzAhoV2VZodvzT2ePSFzS7iQMmb4xmAjvfwCDZa7V8TvcrqJaCxtGaQzBcZhyzD"} | rate | validator | 2 | 0.0139 | 0.788 | 0.788 |
| tn_worker_batch_requests_denied_total{worker="0",peer="zXUFxNGREuf8atZat5FZBk1NG5beET3h5KEfUFEnkXvnnMBJXPXaPvAYzj9Ny5Hur4ASkkDhwFgheryjVsqyU7oab9f5D5oZrAPSUVkbDFXW3GEoBzKbKGAr2TQBq84oriY"} | rate | validator | 3 | 0 | 0.281 | 0.281 |
| tn_worker_batch_requests_denied_total{worker="0",peer="zZTwQkGwmXD9pUpeNa5ehNm8GmdG85N3s8nsv6tZ4jpt5rHMZU881cXpvJUrvZYERZtkLb6QVnxKht9XhxjFeMV8ahr8GtG2b5nz5Ee75iVSqx6249xRKifoWcwWrvpP6TV"} | rate | validator | 2 | 0.00198 | 0.113 | 0.113 |
| tn_worker_batch_requests_failed_total{worker="0",peer="2144FHRwrcvmEkUTAChHeoVdYn6vt1mQLCRLaSfVuieBgpKo9aMinwYFiqG8M3A61AEcbHZciXpvFFVYf9WipKxLTBUeZuDohtvK7FAtbmQBuc26DnJrRCY4tkqAhpZJkTGB"} | rate | validator | 2 | 0.00297 | 0.169 | 0.169 |
| tn_worker_batch_requests_failed_total{worker="0",peer="21hNc4w4BkZF6F6jQvFr1E36NpvJ2veVAzh1Zyy7C4kNgBfFCu423Ncw8t18vgbLt4PdtKeW8fZUJJA7emBzDbPnE8dDyY59bw64hftwyb8bNnXFDRXx1RUYabsqUNbBShYT"} | rate | validator | 3 | 0 | 0.0282 | 0.0282 |
| tn_worker_batch_requests_failed_total{worker="0",peer="nE5uq6wTBpXvRMnqjwm51T7ysmhfHshFJpXwuTpqcUBqqLyvgPVKF2i1vWik1vk1avXzV9Tu9bcrGJkSd2Yn2YhYMsZapcduTXK3Da2s9mrHJmqUA2ZyXXfuMd2nH3y1mty"} | rate | validator | 2 | 0.00347 | 0.197 | 0.197 |
| tn_worker_batch_requests_failed_total{worker="0",peer="nZtVqRH4VuyHtF3ASV74DMPQ93mDi1vcntRBsVoBwKWaz1tkbSLyGtkZD6wJq21M4znSfATuS9uVkgsb1TkxepdgKm9b6Rvug4aW2NpW1iKFNcHv7dBcmoz3ik4nT63PbzU"} | rate | validator | 1 | 0 | 0 | 0 |
| tn_worker_batch_requests_failed_total{worker="0",peer="omnUG8kmgM38zGPxu8w39WpDqPtgjinaJV6pUPRtV81Phho5JxSaczk6PNrFC24bcWgL6Sucw2hBNrTFMMg94DAFfsPpYLgabACTpoHfq2kEocssSxyxjBmjDkrNCBtacp2"} | rate | validator | 2 | 0.000495 | 0.0282 | 0.0282 |
| tn_worker_batch_requests_failed_total{worker="0",peer="sHv24SnwtmMr3yzEHBqH5tFoNWDkzVdGLLMr8oaHDpcGSdiZV5SrK2jD5Fcp8J5zZVDkiZ1PGGaB7DYLJKCgtE1tvhexUZuR4Shz6gE2KTWb4xrGdPPySgX4YQXFzG4rMd8"} | rate | validator | 2 | 0 | 0 | 0 |
| tn_worker_batch_requests_failed_total{worker="0",peer="sUDaca6agCTkShRNgnS31xLNdovgGqqCSn1hxgvSd6aD29FJhnhubtkQikashEAecU5xJEJfuWERN9CMhzD17HJYbHpbeK9n4oMLeDwjg5uexrvpVexNCQ1j3TvRu9Ccigk"} | rate | validator | 1 | 0.000495 | 0.0282 | 0.0282 |
| tn_worker_batch_requests_failed_total{worker="0",peer="xDya61TkRgwqXhL621p5f5sqN4SxRG9RmTwLMdk2vAzqdXsqCtp5FTkUrPbw9zZTSm8s5b9TaDHvduCgFJgsSAVkavhmaj6gQmp6SVvogR2k9g9UaJuR7MWZBnfe7x5HgUP"} | rate | validator | 2 | 0.00545 | 0.31 | 0.31 |
| tn_worker_batch_requests_failed_total{worker="0",peer="xQAX3CxFYar64P2zTGvebVPT5JqMQe9pTyAnvLJmCHm7gxKWQ4SVcYho956GYuwVRUN5LzAhoV2VZodvzT2ePSFzS7iQMmb4xmAjvfwCDZa7V8TvcrqJaCxtGaQzBcZhyzD"} | rate | validator | 2 | 0.00693 | 0.394 | 0.394 |
| tn_worker_batch_requests_failed_total{worker="0",peer="zXUFxNGREuf8atZat5FZBk1NG5beET3h5KEfUFEnkXvnnMBJXPXaPvAYzj9Ny5Hur4ASkkDhwFgheryjVsqyU7oab9f5D5oZrAPSUVkbDFXW3GEoBzKbKGAr2TQBq84oriY"} | rate | validator | 3 | 0 | 0.0282 | 0.0282 |
| tn_worker_batch_requests_failed_total{worker="0",peer="zZTwQkGwmXD9pUpeNa5ehNm8GmdG85N3s8nsv6tZ4jpt5rHMZU881cXpvJUrvZYERZtkLb6QVnxKht9XhxjFeMV8ahr8GtG2b5nz5Ee75iVSqx6249xRKifoWcwWrvpP6TV"} | rate | validator | 2 | 0.00099 | 0.0563 | 0.0563 |
| tn_node_epoch_record_signers | max | validator | 10 | 7.74 | 10 | 10 |
| tn_node_epoch_record_signers | max | observer | 1 | 8.19 | 10 | 10 |
| tn_node_epoch_record_missing_signers_total{signer="2UYxCFESCqw61C7pEFpzxAzcdu1aeTFTRC6pcgZivJG2"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_node_epoch_record_missing_signers_total{signer="2UYxCFESCqw61C7pEFpzxAzcdu1aeTFTRC6pcgZivJG2"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_node_epoch_record_missing_signers_total{signer="3GBP8A9iAAinoV4Cme8WzPoXfYRkErZ8GHx78FgaZazB"} | rate | validator | 10 | 0 | 0.0152 | 0.0203 |
| tn_node_epoch_record_missing_signers_total{signer="3GBP8A9iAAinoV4Cme8WzPoXfYRkErZ8GHx78FgaZazB"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_node_epoch_record_missing_signers_total{signer="3VrEiVQncxBCKqm5AN8KTtvrSShmeCT6bBQswYYtAutD"} | rate | validator | 10 | 0.000365 | 0.0165 | 0.022 |
| tn_node_epoch_record_missing_signers_total{signer="3VrEiVQncxBCKqm5AN8KTtvrSShmeCT6bBQswYYtAutD"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_node_epoch_record_missing_signers_total{signer="7CgjP1y53obfnVnshMfKtLaaDuMWD7Eys21AQjYjjNsn"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_node_epoch_record_missing_signers_total{signer="7CgjP1y53obfnVnshMfKtLaaDuMWD7Eys21AQjYjjNsn"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_node_epoch_record_missing_signers_total{signer="AhKq47odXwQXCiQJqJRtvrpufzSh2twSUUijT6Jw4bWq"} | rate | validator | 10 | 0.000365 | 0.0165 | 0.022 |
| tn_node_epoch_record_missing_signers_total{signer="AhKq47odXwQXCiQJqJRtvrpufzSh2twSUUijT6Jw4bWq"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_node_epoch_record_missing_signers_total{signer="CsBXvWKVeeTEDzk4kqHeRd7gjHs9VSTU68Z5Zsrrw2AJ"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_node_epoch_record_missing_signers_total{signer="CsBXvWKVeeTEDzk4kqHeRd7gjHs9VSTU68Z5Zsrrw2AJ"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_node_epoch_record_missing_signers_total{signer="G38nSkpHUZfYUQEeXNzD5SuoKjyapZU5WEgwe5tVddye"} | rate | validator | 10 | 0.000365 | 0.0158 | 0.021 |
| tn_node_epoch_record_missing_signers_total{signer="G38nSkpHUZfYUQEeXNzD5SuoKjyapZU5WEgwe5tVddye"} | rate | observer | 1 | 0.000424 | 0.0137 | 0.0182 |
| tn_node_epoch_record_missing_signers_total{signer="GYZ77es5DWi4ChjbRqcGQBFdM63pQHLLWBX4jhhNznBR"} | rate | validator | 10 | 0.000779 | 0.0167 | 0.0222 |
| tn_node_epoch_record_missing_signers_total{signer="GYZ77es5DWi4ChjbRqcGQBFdM63pQHLLWBX4jhhNznBR"} | rate | observer | 1 | 0.000424 | 0.0137 | 0.0182 |
| tn_node_epoch_record_missing_signers_total{signer="HDKF9cCXBezTpvjiz14LQiEZCfn7MY2SY3pBoNuRauMn"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_node_epoch_record_missing_signers_total{signer="HDKF9cCXBezTpvjiz14LQiEZCfn7MY2SY3pBoNuRauMn"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_node_epoch_record_missing_signers_total{signer="vp782ooaY6XgWBo6vH5D844aXbcVpYdaxwJTHi7CEWs"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_node_epoch_record_missing_signers_total{signer="vp782ooaY6XgWBo6vH5D844aXbcVpYdaxwJTHi7CEWs"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_network_outbound_request_duration_seconds{outcome="failure",class="batch"} | p95 | validator | 8 | 15.9 s | 29.0 s | 29.0 s |
| tn_network_outbound_request_duration_seconds{outcome="failure",class="epoch_record"} | p95 | validator | 5 | 14.7 s | 29.0 s | 29.0 s |
| tn_network_outbound_request_duration_seconds{outcome="failure",class="vote"} | p95 | validator | 6 | 28.2 s | 29.0 s | 29.0 s |
| tn_network_outbound_request_duration_seconds{outcome="response",class="batch"} | p95 | validator | 6 | 444 ms | 477 ms | 477 ms |
| tn_network_outbound_request_duration_seconds{outcome="response",class="epoch_record"} | p95 | validator | 6 | 459 ms | 494 ms | 499 ms |
| tn_network_outbound_request_duration_seconds{outcome="response",class="vote"} | p95 | validator | 5 | 3,910 ms | 8,543 ms | 8,718 ms |
| tn_network_closed_admission_denied_total{network="bench",direction="in"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_network_closed_admission_denied_total{network="bench",direction="in"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_network_closed_admission_denied_total{network="bench",direction="out"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_network_closed_admission_denied_total{network="bench",direction="out"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_network_source_admission_denied_total{network="bench",direction="in",reason="address_full"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_network_source_admission_denied_total{network="bench",direction="in",reason="address_full"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_network_source_admission_denied_total{network="bench",direction="in",reason="duplicate_connection"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_network_source_admission_denied_total{network="bench",direction="in",reason="duplicate_connection"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_network_source_admission_denied_total{network="bench",direction="in",reason="invalid_limits"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_network_source_admission_denied_total{network="bench",direction="in",reason="invalid_limits"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_network_source_admission_denied_total{network="bench",direction="in",reason="peer_full"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_network_source_admission_denied_total{network="bench",direction="in",reason="peer_full"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_network_source_admission_denied_total{network="bench",direction="in",reason="poisoned"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_network_source_admission_denied_total{network="bench",direction="in",reason="poisoned"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_network_source_admission_denied_total{network="bench",direction="in",reason="prefix_full"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_network_source_admission_denied_total{network="bench",direction="in",reason="prefix_full"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_network_source_admission_denied_total{network="bench",direction="in",reason="process_full"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_network_source_admission_denied_total{network="bench",direction="in",reason="process_full"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_network_source_admission_denied_total{network="bench",direction="in",reason="sources_full"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_network_source_admission_denied_total{network="bench",direction="in",reason="sources_full"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_network_source_admission_denied_total{network="bench",direction="in",reason="unsupported_address"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_network_source_admission_denied_total{network="bench",direction="in",reason="unsupported_address"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_network_source_admission_denied_total{network="bench",direction="out",reason="address_full"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_network_source_admission_denied_total{network="bench",direction="out",reason="address_full"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_network_source_admission_denied_total{network="bench",direction="out",reason="duplicate_connection"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_network_source_admission_denied_total{network="bench",direction="out",reason="duplicate_connection"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_network_source_admission_denied_total{network="bench",direction="out",reason="invalid_limits"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_network_source_admission_denied_total{network="bench",direction="out",reason="invalid_limits"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_network_source_admission_denied_total{network="bench",direction="out",reason="peer_full"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_network_source_admission_denied_total{network="bench",direction="out",reason="peer_full"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_network_source_admission_denied_total{network="bench",direction="out",reason="poisoned"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_network_source_admission_denied_total{network="bench",direction="out",reason="poisoned"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_network_source_admission_denied_total{network="bench",direction="out",reason="prefix_full"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_network_source_admission_denied_total{network="bench",direction="out",reason="prefix_full"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_network_source_admission_denied_total{network="bench",direction="out",reason="process_full"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_network_source_admission_denied_total{network="bench",direction="out",reason="process_full"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_network_source_admission_denied_total{network="bench",direction="out",reason="sources_full"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_network_source_admission_denied_total{network="bench",direction="out",reason="sources_full"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_network_source_admission_denied_total{network="bench",direction="out",reason="unsupported_address"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_network_source_admission_denied_total{network="bench",direction="out",reason="unsupported_address"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_network_admission_mode | max | validator | 10 | 2 | 2 | 2 |
| tn_network_admission_mode | max | observer | 1 | 2 | 2 | 2 |
| tn_network_admission_fallback | max | validator | 10 | 0 | 0 | 0 |
| tn_network_admission_fallback | max | observer | 1 | 0 | 0 | 0 |
| tn_network_outbound_request_failures_total{kind="connection"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_network_outbound_request_failures_total{kind="connection"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_network_outbound_request_failures_total{kind="dial"} | rate | validator | 10 | 9.96 | 67.5 | 83.9 |
| tn_network_outbound_request_failures_total{kind="dial"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_network_outbound_request_failures_total{kind="io"} | rate | validator | 10 | 1.84 | 7.81 | 9.91 |
| tn_network_outbound_request_failures_total{kind="io"} | rate | observer | 1 | 0.312 | 0.763 | 0.948 |
| tn_network_outbound_request_failures_total{kind="timeout"} | rate | validator | 10 | 17.6 | 65.9 | 86.1 |
| tn_network_outbound_request_failures_total{kind="timeout"} | rate | observer | 1 | 0.00332 | 0.0209 | 0.0222 |
| tn_network_outbound_request_failures_total{kind="unsupported"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_network_outbound_request_failures_total{kind="unsupported"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_network_inbound_requests_shed_total{class="batch",reason="admission"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_network_inbound_requests_shed_total{class="batch",reason="admission"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_network_inbound_requests_shed_total{class="batch",reason="queue_budget"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_network_inbound_requests_shed_total{class="batch",reason="queue_budget"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_network_inbound_requests_shed_total{class="batch",reason="queue_full"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_network_inbound_requests_shed_total{class="batch",reason="queue_full"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_network_inbound_requests_shed_total{class="batch",reason="serve_budget"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_network_inbound_requests_shed_total{class="batch",reason="serve_budget"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_network_inbound_requests_shed_total{class="batch",reason="unsubscribed"} | rate | validator | 10 | 0.117 | 33.8 | 41.6 |
| tn_network_inbound_requests_shed_total{class="batch",reason="unsubscribed"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_network_inbound_requests_shed_total{class="certificate_sync",reason="admission"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_network_inbound_requests_shed_total{class="certificate_sync",reason="admission"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_network_inbound_requests_shed_total{class="certificate_sync",reason="queue_budget"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_network_inbound_requests_shed_total{class="certificate_sync",reason="queue_budget"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_network_inbound_requests_shed_total{class="certificate_sync",reason="queue_full"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_network_inbound_requests_shed_total{class="certificate_sync",reason="queue_full"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_network_inbound_requests_shed_total{class="certificate_sync",reason="serve_budget"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_network_inbound_requests_shed_total{class="certificate_sync",reason="serve_budget"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_network_inbound_requests_shed_total{class="certificate_sync",reason="unsubscribed"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_network_inbound_requests_shed_total{class="certificate_sync",reason="unsubscribed"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_network_inbound_requests_shed_total{class="epoch_record",reason="admission"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_network_inbound_requests_shed_total{class="epoch_record",reason="admission"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_network_inbound_requests_shed_total{class="epoch_record",reason="queue_budget"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_network_inbound_requests_shed_total{class="epoch_record",reason="queue_budget"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_network_inbound_requests_shed_total{class="epoch_record",reason="queue_full"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_network_inbound_requests_shed_total{class="epoch_record",reason="queue_full"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_network_inbound_requests_shed_total{class="epoch_record",reason="serve_budget"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_network_inbound_requests_shed_total{class="epoch_record",reason="serve_budget"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_network_inbound_requests_shed_total{class="epoch_record",reason="unsubscribed"} | rate | validator | 10 | 0.197 | 1.1 | 1.42 |
| tn_network_inbound_requests_shed_total{class="epoch_record",reason="unsubscribed"} | rate | observer | 1 | 0.00433 | 0.0404 | 0.0444 |
| tn_network_inbound_requests_shed_total{class="gossip",reason="admission"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_network_inbound_requests_shed_total{class="gossip",reason="admission"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_network_inbound_requests_shed_total{class="gossip",reason="queue_budget"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_network_inbound_requests_shed_total{class="gossip",reason="queue_budget"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_network_inbound_requests_shed_total{class="gossip",reason="queue_full"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_network_inbound_requests_shed_total{class="gossip",reason="queue_full"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_network_inbound_requests_shed_total{class="gossip",reason="serve_budget"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_network_inbound_requests_shed_total{class="gossip",reason="serve_budget"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_network_inbound_requests_shed_total{class="gossip",reason="unsubscribed"} | rate | validator | 10 | 0.0318 | 1.17 | 1.63 |
| tn_network_inbound_requests_shed_total{class="gossip",reason="unsubscribed"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_network_inbound_requests_shed_total{class="other",reason="admission"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_network_inbound_requests_shed_total{class="other",reason="admission"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_network_inbound_requests_shed_total{class="other",reason="queue_budget"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_network_inbound_requests_shed_total{class="other",reason="queue_budget"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_network_inbound_requests_shed_total{class="other",reason="queue_full"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_network_inbound_requests_shed_total{class="other",reason="queue_full"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_network_inbound_requests_shed_total{class="other",reason="serve_budget"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_network_inbound_requests_shed_total{class="other",reason="serve_budget"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_network_inbound_requests_shed_total{class="other",reason="unsubscribed"} | rate | validator | 10 | 0.0282 | 0.112 | 0.135 |
| tn_network_inbound_requests_shed_total{class="other",reason="unsubscribed"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_network_inbound_requests_shed_total{class="vote",reason="admission"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_network_inbound_requests_shed_total{class="vote",reason="admission"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_network_inbound_requests_shed_total{class="vote",reason="queue_budget"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_network_inbound_requests_shed_total{class="vote",reason="queue_budget"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_network_inbound_requests_shed_total{class="vote",reason="queue_full"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_network_inbound_requests_shed_total{class="vote",reason="queue_full"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_network_inbound_requests_shed_total{class="vote",reason="serve_budget"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_network_inbound_requests_shed_total{class="vote",reason="serve_budget"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_network_inbound_requests_shed_total{class="vote",reason="unsubscribed"} | rate | validator | 10 | 0.498 | 4.58 | 6.67 |
| tn_network_inbound_requests_shed_total{class="vote",reason="unsubscribed"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_network_inbound_request_service_seconds{class="batch"} | p95 | validator | 3 | 17 ms | 21 ms | 21 ms |
| tn_network_inbound_request_service_seconds{class="epoch_record"} | p95 | validator | 4 | 7 ms | 155 ms | 228 ms |
| tn_network_inbound_request_service_seconds{class="epoch_record"} | p95 | observer | 1 | 6 ms | 36 ms | 80 ms |
| tn_network_inbound_request_service_seconds{class="vote"} | p95 | validator | 7 | 1,233 ms | 9,747 ms | 9,750 ms |
| tn_network_established_connections | max | validator | 10 | 19.6 | 21 | 21 |
| tn_network_established_connections | max | observer | 1 | 19.8 | 21 | 21 |
| tn_worker_sync_streams_shed_total{reason="budget_exhausted"} | rate | validator | 2 | 0 | 0 | 0 |
| tn_worker_sync_streams_shed_total{reason="denied"} | rate | validator | 10 | 0.00692 | 3.02 | 3.02 |
| tn_worker_sync_streams_shed_total{reason="denied"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_worker_batch_fetch_duration_seconds | p95 | validator | 9 | 9 ms | 1,437 ms | 2,312 ms |
| tn_executor_batch_fetch_failures_total | rate | validator | 10 | 0 | 0 | 0 |
| tn_executor_batch_fetch_failures_total | rate | observer | 1 | 0 | 0 | 0 |
| tn_primary_votes_received_total | rate | validator | 10 | 5.34 | 11.3 | 13.1 |
| tn_primary_votes_received_total | rate | observer | 1 | 0 | 0 | 0 |
| tn_primary_vote_request_failures_total | rate | validator | 10 | 0.795 | 3.61 | 4.73 |
| tn_primary_vote_request_failures_total | rate | observer | 1 | 0 | 0 | 0 |
| tn_primary_certificates_formed_total | rate | validator | 10 | 0.864 | 1.88 | 2.18 |
| tn_primary_certificates_formed_total | rate | observer | 1 | 0 | 0 | 0 |
| tn_primary_certificate_form_duration_seconds | p95 | validator | 10 | 1,001 ms | 4,930 ms | 9,250 ms |
| tn_engine_outputs_executed_total | rate | validator | 10 | 0.433 | 0.894 | 1.2 |
| tn_engine_outputs_executed_total | rate | observer | 1 | 0.427 | 0.767 | 1.36 |
| tn_engine_persist_failures_total | rate | validator | 10 | 0 | 0 | 0 |
| tn_engine_persist_failures_total | rate | observer | 1 | 0 | 0 | 0 |
| tn_node_consensus_sync_distance | max | validator | 10 | 3.88 | 173 | 190 |
| tn_node_consensus_sync_distance | max | observer | 1 | 11.3 | 21 | 28 |
| reth_db_table_size{table="AccountChangeSets"} | max | validator | 10 | 220 MiB | 396 MiB | 396 MiB |
| reth_db_table_size{table="AccountChangeSets"} | max | observer | 1 | 217 MiB | 395 MiB | 396 MiB |
| reth_db_table_size{table="AccountsHistory"} | max | validator | 10 | 25 MiB | 52 MiB | 52 MiB |
| reth_db_table_size{table="AccountsHistory"} | max | observer | 1 | 25 MiB | 52 MiB | 52 MiB |
| reth_db_table_size{table="HashedAccounts"} | max | validator | 10 | 16 MiB | 31 MiB | 31 MiB |
| reth_db_table_size{table="HashedAccounts"} | max | observer | 1 | 16 MiB | 31 MiB | 31 MiB |
| reth_db_table_size{table="HashedStorages"} | max | validator | 10 | 22 MiB | 24 MiB | 24 MiB |
| reth_db_table_size{table="HashedStorages"} | max | observer | 1 | 22 MiB | 24 MiB | 24 MiB |
| reth_db_table_size{table="PlainAccountState"} | max | validator | 10 | 14 MiB | 26 MiB | 26 MiB |
| reth_db_table_size{table="PlainAccountState"} | max | observer | 1 | 13 MiB | 26 MiB | 26 MiB |
| reth_db_table_size{table="PlainStorageState"} | max | validator | 10 | 21 MiB | 24 MiB | 24 MiB |
| reth_db_table_size{table="PlainStorageState"} | max | observer | 1 | 21 MiB | 24 MiB | 24 MiB |
| reth_db_table_size{table="StorageChangeSets"} | max | validator | 10 | 268 MiB | 515 MiB | 515 MiB |
| reth_db_table_size{table="StorageChangeSets"} | max | observer | 1 | 265 MiB | 513 MiB | 515 MiB |
| reth_db_table_size{table="StoragesHistory"} | max | validator | 10 | 53 MiB | 69 MiB | 69 MiB |
| reth_db_table_size{table="StoragesHistory"} | max | observer | 1 | 53 MiB | 69 MiB | 69 MiB |
| reth_db_table_size{table="TransactionHashNumbers"} | max | validator | 10 | 498 MiB | 876 MiB | 876 MiB |
| reth_db_table_size{table="TransactionHashNumbers"} | max | observer | 1 | 492 MiB | 874 MiB | 876 MiB |
| reth_db_table_size{table="TransactionSenders"} | max | validator | 10 | 256 MiB | 451 MiB | 451 MiB |
| reth_db_table_size{table="TransactionSenders"} | max | observer | 1 | 254 MiB | 450 MiB | 451 MiB |
| reth_db_freelist | max | validator | 10 | 39K | 54.8K | 58.5K |
| reth_db_freelist | max | observer | 1 | 39.1K | 52.9K | 57.6K |
| reth_database_transaction_commit_write_duration_seconds | p95 | validator | 10 | 5 ms | 5 ms | 5 ms |
| reth_database_transaction_commit_write_duration_seconds | p95 | observer | 1 | 5 ms | 5 ms | 5 ms |
| reth_database_transaction_commit_sync_duration_seconds | p95 | validator | 9 | 711 ms | 1,825 ms | 2,162 ms |
| reth_database_transaction_commit_sync_duration_seconds | p95 | observer | 1 | 930 ms | 1,023 ms | 1,150 ms |
| reth_database_transaction_commit_whole_duration_seconds | p95 | validator | 9 | 711 ms | 1,825 ms | 2,162 ms |
| reth_database_transaction_commit_whole_duration_seconds | p95 | observer | 1 | 930 ms | 1,023 ms | 1,150 ms |
tn_primary_commit_latency_seconds
tn_primary_round − tn_primary_committed_round
tn_engine_queued_outputs, max by node
The share of each validator's own headers that formed a certificate, per window (tools/cert-rate.py on the node logs). bench-validator-08, bench-validator-09 and bench-validator-10 certified fewer than 90 % of their own headers in a window where they proposed at least 50. Own leader % is the validator's share of the committed leaders and not committed % the share of its certificates no committed leader reached, both over the run window; the fair share is 100 / 10 validators = 10.0 %. Greyed: fewer than 50 headers proposed, not judged.
| Validator | Region | phase preamble | phase warmup | phase mixed | phase cooldown | run | Own leader % | Not committed % |
|---|---|---|---|---|---|---|---|---|
| bench-validator-01 | us-west2 (Los Angeles) | 95.8 | 97.1 | 99.7 | 100.0 | 99.3 | 14.1 | 0.2 |
| bench-validator-02 | europe-west2 (London) | 100.0 | 100.0 | 99.6 | 99.7 | 99.7 | 11.5 | 1.1 |
| bench-validator-03 | australia-southeast1 (Sydney) | 94.0 | 92.4 | 95.3 | 99.7 | 95.6 | 12.9 | 2.5 |
| bench-validator-04 | asia-northeast1 (Tokyo) | 100.0 | 100.6 | 99.9 | 99.7 | 99.9 | 14.4 | 0.2 |
| bench-validator-05 | us-east4 (Ashburn) | 100.0 | 100.6 | 99.9 | – | 100.0 | 13.9 | 0.3 |
| bench-validator-06 | europe-west4 (Netherlands) | 100.0 | 100.0 | 93.9 | 99.7 | 95.4 | 8.0 | 2.0 |
| bench-validator-07 | asia-southeast1 (Singapore) | 100.0 | 100.6 | 99.9 | – | 100.0 | 16.3 | 3.1 |
| bench-validator-08 below 90 % | asia-east2 (Hong Kong) | 58.6 | 80.2 | 96.7 | 70.0 | 85.3 | 8.4 | 6.8 |
| bench-validator-09 below 90 % | asia-northeast3 (Seoul) | 97.7 | 80.2 | 84.7 | 99.7 | 87.2 | 12.2 | 1.2 |
| bench-validator-10 below 90 % | asia-south1 (Mumbai) | 84.7 | 91.3 | 94.7 | 100.0 | 94.4 | 8.6 | 5.9 |
| Node | Class | Peak RssAnon | Peak RssFile | Peak cache | PSI mem | PSI io | PSI cpu | db growth | static_files growth | Root used | OOM Δ |
|---|---|---|---|---|---|---|---|---|---|---|---|
| bench-observer-11 | observer | 1.14 GiB | 2.45 GiB | – | 0.0 % | 12.6 % | 1.7 % | 0 B | 2.83 GiB | 64.0 % | 0 |
| bench-validator-01 | validator | 5.37 GiB | 2.95 GiB | – | 0.0 % | 11.0 % | 4.4 % | 0 B | 2.83 GiB | 65.0 % | 0 |
| bench-validator-02 | validator | 4.72 GiB | 2.87 GiB | – | 0.0 % | 10.5 % | 4.3 % | 0 B | 2.83 GiB | 65.0 % | 0 |
| bench-validator-03 | validator | 5.78 GiB | 2.95 GiB | – | 0.0 % | 11.6 % | 4.1 % | 0 B | 2.83 GiB | 65.0 % | 0 |
| bench-validator-04 | validator | 4.60 GiB | 2.97 GiB | – | 0.0 % | 11.9 % | 4.3 % | 0 B | 2.83 GiB | 65.0 % | 0 |
| bench-validator-05 | validator | 4.40 GiB | 2.31 GiB | – | 0.0 % | 11.5 % | 4.4 % | 0 B | 2.69 GiB | 57.0 % | 0 |
| bench-validator-06 | validator | 6.12 GiB | 2.92 GiB | – | 0.0 % | 11.0 % | 4.3 % | 0 B | 2.83 GiB | 65.0 % | 0 |
| bench-validator-07 | validator | 5.19 GiB | 2.95 GiB | – | 0.0 % | 12.6 % | 4.0 % | 0 B | 2.83 GiB | 65.0 % | 0 |
| bench-validator-08 | validator | 5.79 GiB | 3.14 GiB | – | 0.1 % | 17.4 % | 4.5 % | 0 B | 2.83 GiB | 63.0 % | 0 |
| bench-validator-09 | validator | 6.29 GiB | 2.96 GiB | – | 0.0 % | 10.5 % | 4.6 % | 0 B | 2.83 GiB | 65.0 % | 0 |
| bench-validator-10 | validator | 6.08 GiB | 2.91 GiB | – | 0.0 % | 10.2 % | 4.5 % | 0 B | 2.83 GiB | 65.0 % | 0 |
/proc/<pid>/status of the telcoin process on the node with the largest RSS
Not collected: consensus-db/cache size.
/proc/pressure on the node with the highest memory PSI
Per role and per node, from sizing.json (p95 over the landed window; PSI and RssAnon max over the whole sampler run). The lines follow the hardware doc's tier rules: CPU p95 over 70 % of vCPUs; memory used p95 over 75 % of RAM, or telcoin RSS p95 over 85 % of RAM; disk write p95 over 60 % of the PD MiB/s limit, or write IOPS p95 over 60 % of the IOPS limit; IO pressure over 20 % or memory pressure over 5 % (PSI some avg10, max); network p95 over 50 % of 1 Gbps; root disk over 80 %. A line crossed on any node puts the role at the limit; a nonzero reading within 15 points of its line is tight; otherwise the role has headroom. OOM kills, disk throttling and degraded disk performance are flagged too.
| Signal | Class | Worst node | Value | Share of capacity | Line | Nodes over / tight | Status |
|---|---|---|---|---|---|---|---|
| CPU p95 | validator | bench-validator-09 | 36.7 % | 36.7 % | > 70 % | 0 / 0 | headroom |
| Memory used p95 | validator | bench-validator-06 | 43.5 % | 43.5 % | > 75 % | 0 / 0 | headroom |
| telcoin RSS p95 | validator | bench-validator-10 | 9.28 GB | 54.0 % | > 85 % | 0 / 0 | headroom |
| PSI memory max | validator | bench-validator-08 | 0.1 % | 0.1 % | > 5 % | 0 / 1 | tight |
| RssAnon peak | validator | bench-validator-09 | 6.75 GB | 39.3 % | info | – | info |
| Disk write p95 | validator | bench-validator-08 | 325 MB/s | 38.7 % | > 60 % | 0 / 0 | headroom |
| Write IOPS p95 | validator | bench-validator-10 | 9,882 | 49.4 % | > 60 % | 0 / 8 | tight |
| PSI io max | validator | bench-validator-08 | 17.4 % | 17.4 % | > 20 % | 0 / 10 | tight |
| Root disk max | validator | bench-validator-01 | 63.9 % | 63.9 % | > 80 % | 0 / 0 | headroom |
| Network p95 | validator | bench-validator-09 | 103 Mbps | 10.3 % | > 50 % | 0 / 0 | headroom |
| Consensus cache peak | validator | – | – | – | info | – | not collected |
| OOM kills | validator | – | – | any | any | – | none |
| Disk throttling | validator | – | – | metric type unavailable in project | any | – | not collected |
| Disk performance degraded | validator | – | – | any sample | any | – | none |
| CPU p95 | observer | bench-observer-11 | 27.4 % | 27.4 % | > 70 % | 0 / 0 | headroom |
| Memory used p95 | observer | bench-observer-11 | 13.5 % | 13.5 % | > 75 % | 0 / 0 | headroom |
| telcoin RSS p95 | observer | bench-observer-11 | 3.45 GB | 20.1 % | > 85 % | 0 / 0 | headroom |
| PSI memory max | observer | bench-observer-11 | 0.0 % | 0.0 % | > 5 % | 0 / 0 | headroom |
| RssAnon peak | observer | bench-observer-11 | 1.22 GB | 7.1 % | info | – | info |
| Disk write p95 | observer | bench-observer-11 | 325 MB/s | 38.7 % | > 60 % | 0 / 0 | headroom |
| Write IOPS p95 | observer | bench-observer-11 | 8,830 | 44.1 % | > 60 % | 0 / 0 | headroom |
| PSI io max | observer | bench-observer-11 | 12.6 % | 12.6 % | > 20 % | 0 / 1 | tight |
| Root disk max | observer | bench-observer-11 | 62.1 % | 62.1 % | > 80 % | 0 / 0 | headroom |
| Network p95 | observer | bench-observer-11 | 13 Mbps | 1.3 % | > 50 % | 0 / 0 | headroom |
| Consensus cache peak | observer | – | – | – | info | – | not collected |
| OOM kills | observer | – | – | any | any | – | none |
| Disk throttling | observer | – | – | metric type unavailable in project | any | – | not collected |
| Disk performance degraded | observer | – | – | any sample | any | – | none |
| Node | Class | CPU p95 | Memory used p95 | telcoin RSS p95 | PSI memory max | RssAnon peak | Disk write p95 | Write IOPS p95 | PSI io max | Root disk max | Network p95 | Consensus cache peak |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Line | > 70 % | > 75 % | > 85 % | > 5 % | info | > 60 % | > 60 % | > 20 % | > 80 % | > 50 % | info | |
| bench-validator-01 | validator | 36.4 % | 38.9 % | 8.62 GB 50 % | 0.0 % | 5.76 GB 34 % | 243 MB/s 29 % | 9,282 46 % | 11.0 % | 63.9 % | 86 Mbps 9 % | – |
| bench-validator-02 | validator | 35.9 % | 34.7 % | 7.88 GB 46 % | 0.0 % | 5.07 GB 30 % | 227 MB/s 27 % | 9,621 48 % | 10.5 % | 63.6 % | 87 Mbps 9 % | – |
| bench-validator-03 | validator | 36.6 % | 40.3 % | 8.76 GB 51 % | 0.0 % | 6.20 GB 36 % | 233 MB/s 28 % | 9,629 48 % | 11.6 % | 62.8 % | 99 Mbps 10 % | – |
| bench-validator-04 | validator | 36.4 % | 34.1 % | 7.69 GB 45 % | 0.0 % | 4.94 GB 29 % | 235 MB/s 28 % | 9,552 48 % | 11.9 % | 62.7 % | 90 Mbps 9 % | – |
| bench-validator-05 | validator | 36.3 % | 29.9 % | 6.04 GB 35 % | 0.0 % | 5.12 GB 30 % | 230 MB/s 27 % | 8,916 45 % | 11.5 % | 55.5 % | 86 Mbps 9 % | – |
| bench-validator-06 | validator | 36.4 % | 43.5 % | 9.27 GB 54 % | 0.0 % | 6.58 GB 38 % | 234 MB/s 28 % | 9,697 48 % | 11.0 % | 62.8 % | 88 Mbps 9 % | – |
| bench-validator-07 | validator | 36.1 % | 37.5 % | 8.24 GB 48 % | 0.0 % | 5.57 GB 32 % | 262 MB/s 31 % | 9,262 46 % | 12.6 % | 61.5 % | 83 Mbps 8 % | – |
| bench-validator-08 | validator | 34.2 % | 41.6 % | 9.07 GB 53 % | 0.1 % | 6.21 GB 36 % | 325 MB/s 39 % | 7,956 40 % | 17.4 % | 59.4 % | 80 Mbps 8 % | – |
| bench-validator-09 | validator | 36.7 % | 43.0 % | 9.18 GB 53 % | 0.0 % | 6.75 GB 39 % | 220 MB/s 26 % | 9,346 47 % | 10.5 % | 62.9 % | 103 Mbps 10 % | – |
| bench-validator-10 | validator | 36.5 % | 43.4 % | 9.28 GB 54 % | 0.0 % | 6.53 GB 38 % | 228 MB/s 27 % | 9,882 49 % | 10.2 % | 63.0 % | 99 Mbps 10 % | – |
| bench-observer-11 | observer | 27.4 % | 13.5 % | 3.45 GB 20 % | 0.0 % | 1.22 GB 7 % | 325 MB/s 39 % | 8,830 44 % | 12.6 % | 62.1 % | 13 Mbps 1 % | – |
Small figures are the share of capacity: RAM for RSS, the PD limit for disk write (VM cap applied), 1 Gbps for network. Hover a cell for its JSON path.
A = Fleet c3-hdb · c3-highcpu-8 · r20261105-0803
B = Fleet c3-hdb · c3-highcpu-8 · r20261103-1922
tight c3-highcpu-8 validators at 5,525 sustained TPS: tight on memory and disk. Memory tight: memory pressure (PSI some avg10) max 0.1 % (bench-validator-08), tight against the 5 % line. Disk tight: write IOPS p95 9,882, 49.4 % of the 20,000 IOPS limit (bench-validator-10), tight against the 60 % line; IO pressure (PSI some avg10) max 17.4 % (bench-validator-08), tight against the 20 % line. CPU has headroom: CPU p95 36.7 % of 8 vCPUs (bench-validator-09). Network has headroom: network p95 103 Mbps, 10.3 % of 1 Gbps (bench-validator-09).
tight c3-highcpu-8 observers at 5,525 sustained TPS: tight on disk. Disk tight: IO pressure (PSI some avg10) max 12.6 % (bench-observer-11), tight against the 20 % line. CPU has headroom: CPU p95 27.4 % of 8 vCPUs (bench-observer-11). Memory has headroom: memory used p95 13.5 % of RAM (bench-observer-11); telcoin RSS p95 3.45 GB, 20.1 % of 16 GiB RAM (bench-observer-11); memory pressure (PSI some avg10) max 0.0 % (bench-observer-11). Network has headroom: network p95 13 Mbps, 1.3 % of 1 Gbps (bench-observer-11).
at the limit c3-highcpu-8 validators at 7,599 sustained TPS: at the limit on disk. Disk at the limit: write p95 602–613 MB/s, up to 73.1 % of the 800 MiB/s limit, over the 60 % line on 9 of 10 validators (bench-validator-01, 02, 03, 04, 06, 07, 08, 09, 10); IO pressure (PSI some avg10) max 25.8–33.2 %, over the 20 % line on all 10 validators (bench-validator-01, 02, 03, 04, 05, 06, 07, 08, 09, 10); write IOPS p95 10,292, 51.5 % of the 20,000 IOPS limit (bench-validator-05), tight against the 60 % line. CPU has headroom: CPU p95 27.4 % of 8 vCPUs (bench-validator-09). Memory has headroom: memory used p95 34.8 % of RAM (bench-validator-06); telcoin RSS p95 7.83 GB, 45.6 % of 16 GiB RAM (bench-validator-06); memory pressure (PSI some avg10) max 0.0 % (bench-validator-01). Network has headroom: network p95 113 Mbps, 11.3 % of 1 Gbps (bench-validator-07). Next step: a faster volume (pd-ssd adds 30 IOPS and 0.48 MiB/s per GB up to the per-VM cap; past the cap, a larger VM).
at the limit c3-highcpu-8 observers at 7,599 sustained TPS: at the limit on disk. Disk at the limit: write p95 526 MB/s, 62.8 % of the 800 MiB/s limit on bench-observer-11, over the 60 % line; IO pressure (PSI some avg10) max 32.0 % on bench-observer-11, over the 20 % line. CPU has headroom: CPU p95 14.3 % of 8 vCPUs (bench-observer-11). Memory has headroom: memory used p95 11.0 % of RAM (bench-observer-11); telcoin RSS p95 3.45 GB, 20.1 % of 16 GiB RAM (bench-observer-11); memory pressure (PSI some avg10) max 0.0 % (bench-observer-11). Network has headroom: network p95 5 Mbps, 0.5 % of 1 Gbps (bench-observer-11). Next step: a faster volume (pd-ssd adds 30 IOPS and 0.48 MiB/s per GB up to the per-VM cap; past the cap, a larger VM).
Validators: Fleet A (c3-highcpu-8) is tight on memory and disk, with headroom on CPU and network; Fleet B (c3-highcpu-8) is at the limit on disk, with headroom on CPU, memory and network. Disk is at the limit on Fleet B (c3-highcpu-8) only. Fleet A (c3-highcpu-8) stays under the disk lines that Fleet B (c3-highcpu-8) crosses.
Observers: Fleet A (c3-highcpu-8) is tight on disk, with headroom on CPU, memory and network; Fleet B (c3-highcpu-8) is at the limit on disk, with headroom on CPU, memory and network. Disk is at the limit on Fleet B (c3-highcpu-8) only. Fleet A (c3-highcpu-8) stays under the disk lines that Fleet B (c3-highcpu-8) crosses.
| KPI | Fleet A | Fleet B | B ÷ A |
|---|---|---|---|
| Sustained mixed TPS (own) | 5,524.8 tx/s | 7,599.3 tx/s | 1.38× |
| Sustained mixed TPS (chain) | 5,524.8 tx/s | 7,599.3 tx/s | 1.38× |
| Landed / accepted | 99.9 % | 99.6 % | 1.00× |
| Reverted (own) | 0 | 0 | – |
| Latency p50 | 3,303 ms | 3,494 ms | 1.06× |
| Latency p95 | 12,837 ms | 14,173 ms | 1.10× |
| Gas / s (own) | 231M gas/s | 198M gas/s | 0.86× |
| Capacity at mix | 9,431.4 tx/s | 8,037.3 tx/s | 0.85× |
| Signal | Class | Fleet A | Fleet B | Line |
|---|---|---|---|---|
| CPU p95 | validator | 36.7 % headroom | 27.4 % headroom | > 70 % |
| Memory used p95 | validator | 43.5 % headroom | 34.8 % headroom | > 75 % |
| telcoin RSS p95 | validator | 9.28 GB 54.0 % headroom | 7.83 GB 45.6 % headroom | > 85 % |
| PSI memory max | validator | 0.1 % tight | 0.0 % headroom | > 5 % |
| Disk write p95 | validator | 325 MB/s 38.7 % headroom | 613 MB/s 73.1 % at the limit | > 60 % |
| Write IOPS p95 | validator | 9,882 49.4 % tight | 10,292 51.5 % tight | > 60 % |
| PSI io max | validator | 17.4 % tight | 33.2 % at the limit | > 20 % |
| Root disk max | validator | 63.9 % headroom | 13.3 % headroom | > 80 % |
| Network p95 | validator | 103 Mbps 10.3 % headroom | 113 Mbps 11.3 % headroom | > 50 % |
| CPU p95 | observer | 27.4 % headroom | 14.3 % headroom | > 70 % |
| Memory used p95 | observer | 13.5 % headroom | 11.0 % headroom | > 75 % |
| telcoin RSS p95 | observer | 3.45 GB 20.1 % headroom | 3.45 GB 20.1 % headroom | > 85 % |
| PSI memory max | observer | 0.0 % headroom | 0.0 % headroom | > 5 % |
| Disk write p95 | observer | 325 MB/s 38.7 % headroom | 526 MB/s 62.8 % at the limit | > 60 % |
| Write IOPS p95 | observer | 8,830 44.1 % headroom | 5,493 27.5 % headroom | > 60 % |
| PSI io max | observer | 12.6 % tight | 32.0 % at the limit | > 20 % |
| Root disk max | observer | 62.1 % headroom | 12.1 % headroom | > 80 % |
| Network p95 | observer | 13 Mbps 1.3 % headroom | 5 Mbps 0.5 % headroom | > 50 % |
aligned on each fleet's mixed-phase start
aligned on each fleet's mixed-phase start
aligned on each fleet's mixed-phase start
aligned on each fleet's mixed-phase start
aligned on each fleet's mixed-phase start
aligned on each fleet's mixed-phase start
Not collected: reference chains.