Throughput is measured from blocks the chain produced (own senders vs every sender), resource use from Cloud Monitoring, Prometheus and the on-node sampler. Summaries cover the mixed phase. Hover any number for the JSON field it came from.
Landed counts only our own senders; latency is submit → block timestamp.
| Type | Submitted | Accepted | Landed | Reverted | Unlanded | Skips | Gas limit | Gas used avg | p50 | p95 |
|---|---|---|---|---|---|---|---|---|---|---|
| native_transfer | 2.78M | 2.66M | 2.65M | 0 | 7.18K | 0 | 24.1K | 21K | 2,801 ms | 14,359 ms |
| batched_self_send | 1.49M | 1.4M | 1.39M | 0 | 6.17K | 0 | 24.1K | 21K | 2,895 ms | 14,770 ms |
| uniswap_v2_swap | 1.34M | 1.25M | 1.25M | 0 | 5.28K | 0 | 180K | 106K | 2,807 ms | 14,349 ms |
| stablecoin_transfer | 1.12M | 1.05M | 1.04M | 0 | 4.22K | 0 | 75K | 43.9K | 2,804 ms | 14,377 ms |
| wtel_wrap | 888K | 831K | 827K | 0 | 3.41K | 0 | 50K | 27.9K | 2,808 ms | 14,409 ms |
| wtel_unwrap | 594K | 555K | 553K | 0 | 2.25K | 0 | 45K | 35.2K | 2,805 ms | 14,386 ms |
| stablecoin_approve | 448K | 419K | 418K | 0 | 1.72K | 0 | 60K | 31.8K | 2,798 ms | 14,321 ms |
| uniswap_v2_add_liquidity | 297K | 278K | 277K | 0 | 1.14K | 0 | 250K | 128K | 2,805 ms | 14,271 ms |
| uniswap_v4_swap | 225K | 210K | 209K | 0 | 873 | 0 | 250K | 124K | 2,793 ms | 14,322 ms |
| contract_deploy | 224K | 209K | 208K | 0 | 857 | 0 | 140K | 74.2K | 2,804 ms | 14,324 ms |
| uniswap_v3_swap | 149K | 140K | 139K | 0 | 585 | 0 | 220K | 120K | 2,819 ms | 14,339 ms |
| uniswap_v2_remove_liquidity | 149K | 139K | 139K | 0 | 587 | 0 | 220K | 129K | 2,796 ms | 14,283 ms |
| faucet_drip | 125K | 122K | 122K | 0 | 97 | 244K | 150K | 107K | 2,153 ms | 7,925 ms |
| Node | Submitted | Accepted | Accepted % | Top errors | Send p50 | Send p95 |
|---|---|---|---|---|---|---|
| bench-validator-01 bench-validator-01:8545 | 695K | 690K | 99.3 % | known 1.58K | 6 ms | 24 ms |
| bench-validator-02 bench-validator-02:8545 | 1.17M | 1.14M | 97.4 % | known 15.9K | 7 ms | 29 ms |
| bench-validator-03 bench-validator-03:8545 | 992K | 989K | 99.7 % | known 1.33K | 9 ms | 32 ms |
| bench-validator-04 bench-validator-04:8545 | 940K | 903K | 96.1 % | known 10.6K | 12 ms | 37 ms |
| bench-validator-05 bench-validator-05:8545 | 1.94M | 226K | 11.7 % | other 1.7M, known 5.55K | 1 ms | 10 ms |
| bench-validator-06 bench-validator-06:8545 | 1.03M | 994K | 96.8 % | known 15.6K | 8 ms | 29 ms |
| bench-validator-07 bench-validator-07:8545 | 1.5M | 1.48M | 98.9 % | known 8.02K | 7 ms | 23 ms |
| bench-validator-08 bench-validator-08:8545 | 575K | 530K | 92.1 % | known 10.9K | 9 ms | 36 ms |
| bench-validator-09 bench-validator-09:8545 | 1.52M | 1.5M | 98.8 % | known 8.55K | 7 ms | 24 ms |
| bench-validator-10 bench-validator-10:8545 | 899K | 865K | 96.2 % | known 14.8K | 8 ms | 29 ms |
Shaded: phases outside the mixed window. Hairlines: epoch closes. Dashed orange: chaos events.
own senders vs chain total, per block-timestamp second
own vs chain gas used, per block-timestamp second
compute.googleapis.com/instance/cpu/utilization, 60-s mean, % of all vCPUs; summaries over the mixed phase
| Node | Class | Mean | p95 | Peak 60-s mean | Max |
|---|---|---|---|---|---|
| bench-observer-11 | observer | 10.7 % | 12.0 % | 13.4 % | 13.4 % |
| bench-validator-01 | validator | 17.5 % | 20.5 % | 20.7 % | 20.7 % |
| bench-validator-02 | validator | 18.2 % | 19.8 % | 20.5 % | 20.5 % |
| bench-validator-03 | validator | 17.7 % | 19.8 % | 22.2 % | 22.2 % |
| bench-validator-04 | validator | 18.0 % | 20.6 % | 20.9 % | 20.9 % |
| bench-validator-05 | validator | 16.0 % | 20.3 % | 21.8 % | 21.8 % |
| bench-validator-06 | validator | 18.1 % | 20.6 % | 20.9 % | 20.9 % |
| bench-validator-07 | validator | 18.5 % | 20.2 % | 20.5 % | 20.5 % |
| bench-validator-08 | validator | 17.4 % | 20.4 % | 22.2 % | 22.2 % |
| bench-validator-09 | validator | 18.4 % | 20.5 % | 20.8 % | 20.8 % |
| bench-validator-10 | validator | 18.1 % | 19.5 % | 19.9 % | 19.9 % |
One row per node, over the mixed phase.
agent processes/rss_usage summed over telcoin pids; summaries over the mixed phase
| Node | Class | Mean | p95 | Peak 60-s mean | Max |
|---|---|---|---|---|---|
| bench-observer-11 | observer | 2.73 GiB | 3.70 GiB | 3.90 GiB | 3.90 GiB |
| bench-validator-01 | validator | 5.59 GiB | 7.32 GiB | 7.52 GiB | 7.52 GiB |
| bench-validator-02 | validator | 6.22 GiB | 7.65 GiB | 8.11 GiB | 8.11 GiB |
| bench-validator-03 | validator | 5.66 GiB | 7.10 GiB | 7.25 GiB | 7.25 GiB |
| bench-validator-04 | validator | 5.57 GiB | 6.85 GiB | 7.10 GiB | 7.10 GiB |
| bench-validator-05 | validator | 4.56 GiB | 5.60 GiB | 5.98 GiB | 5.98 GiB |
| bench-validator-06 | validator | 5.84 GiB | 7.36 GiB | 7.51 GiB | 7.51 GiB |
| bench-validator-07 | validator | 5.66 GiB | 6.97 GiB | 7.16 GiB | 7.16 GiB |
| bench-validator-08 | validator | 6.06 GiB | 7.30 GiB | 7.41 GiB | 7.41 GiB |
| bench-validator-09 | validator | 5.72 GiB | 7.16 GiB | 7.33 GiB | 7.33 GiB |
| bench-validator-10 | validator | 5.97 GiB | 7.53 GiB | 7.88 GiB | 7.88 GiB |
One row per node, over the mixed phase.
agent memory/percent_used, state=used; summaries over the mixed phase
| Node | Class | Mean | p95 | Peak 60-s mean | Max |
|---|---|---|---|---|---|
| bench-observer-11 | observer | 10.4 % | 11.7 % | 12.5 % | 12.5 % |
| bench-validator-01 | validator | 27.1 % | 32.3 % | 33.1 % | 33.1 % |
| bench-validator-02 | validator | 30.7 % | 34.6 % | 36.4 % | 36.4 % |
| bench-validator-03 | validator | 27.2 % | 30.7 % | 32.0 % | 32.0 % |
| bench-validator-04 | validator | 26.7 % | 29.6 % | 30.6 % | 30.6 % |
| bench-validator-05 | validator | 18.8 % | 27.0 % | 27.1 % | 27.1 % |
| bench-validator-06 | validator | 28.3 % | 32.8 % | 33.4 % | 33.4 % |
| bench-validator-07 | validator | 27.2 % | 30.4 % | 30.6 % | 30.6 % |
| bench-validator-08 | validator | 29.5 % | 31.9 % | 32.0 % | 32.0 % |
| bench-validator-09 | validator | 27.6 % | 31.1 % | 31.9 % | 31.9 % |
| bench-validator-10 | validator | 29.2 % | 34.1 % | 35.4 % | 35.4 % |
One row per node, over the mixed phase.
instance/disk/{read,write}_bytes_count rate, all devices; summaries over the mixed phase
| Node | Class | Mean | p95 | Peak 60-s mean | Max | Limit |
|---|---|---|---|---|---|---|
| bench-observer-11 | observer | 480.9 MiB/s | 559.1 MiB/s | 567.1 MiB/s | 567.1 MiB/s | 800.0 MiB/s |
| bench-validator-01 | validator | 499.5 MiB/s | 548.8 MiB/s | 552.1 MiB/s | 552.1 MiB/s | 800.0 MiB/s |
| bench-validator-02 | validator | 509.5 MiB/s | 565.1 MiB/s | 571.9 MiB/s | 571.9 MiB/s | 800.0 MiB/s |
| bench-validator-03 | validator | 495.0 MiB/s | 551.6 MiB/s | 554.4 MiB/s | 554.4 MiB/s | 800.0 MiB/s |
| bench-validator-04 | validator | 523.0 MiB/s | 563.7 MiB/s | 567.7 MiB/s | 567.7 MiB/s | 800.0 MiB/s |
| bench-validator-05 | validator | 401.7 MiB/s | 516.5 MiB/s | 544.4 MiB/s | 544.4 MiB/s | 800.0 MiB/s |
| bench-validator-06 | validator | 521.9 MiB/s | 562.8 MiB/s | 567.1 MiB/s | 567.1 MiB/s | 800.0 MiB/s |
| bench-validator-07 | validator | 496.0 MiB/s | 560.5 MiB/s | 563.2 MiB/s | 563.2 MiB/s | 800.0 MiB/s |
| bench-validator-08 | validator | 469.9 MiB/s | 558.1 MiB/s | 569.2 MiB/s | 569.2 MiB/s | 800.0 MiB/s |
| bench-validator-09 | validator | 497.1 MiB/s | 565.4 MiB/s | 567.5 MiB/s | 567.5 MiB/s | 800.0 MiB/s |
| bench-validator-10 | validator | 520.2 MiB/s | 567.6 MiB/s | 572.8 MiB/s | 572.8 MiB/s | 800.0 MiB/s |
instance/disk/{read,write}_ops_count rate, all devices; summaries over the mixed phase
| Node | Class | Mean | p95 | Peak 60-s mean | Max | Limit |
|---|---|---|---|---|---|---|
| bench-observer-11 | observer | 3.14K IOPS | 5.21K IOPS | 5.73K IOPS | 5.73K IOPS | 20K IOPS |
| bench-validator-01 | validator | 2.98K IOPS | 5.06K IOPS | 5.51K IOPS | 5.51K IOPS | 20K IOPS |
| bench-validator-02 | validator | 2.83K IOPS | 4.18K IOPS | 4.57K IOPS | 4.57K IOPS | 20K IOPS |
| bench-validator-03 | validator | 3.08K IOPS | 5.32K IOPS | 5.38K IOPS | 5.38K IOPS | 20K IOPS |
| bench-validator-04 | validator | 2.48K IOPS | 2.68K IOPS | 2.7K IOPS | 2.7K IOPS | 20K IOPS |
| bench-validator-05 | validator | 4.04K IOPS | 8.91K IOPS | 9.46K IOPS | 9.46K IOPS | 20K IOPS |
| bench-validator-06 | validator | 2.55K IOPS | 2.87K IOPS | 2.89K IOPS | 2.89K IOPS | 20K IOPS |
| bench-validator-07 | validator | 3.08K IOPS | 4.89K IOPS | 5.11K IOPS | 5.11K IOPS | 20K IOPS |
| bench-validator-08 | validator | 3.61K IOPS | 5.9K IOPS | 6.5K IOPS | 6.5K IOPS | 20K IOPS |
| bench-validator-09 | validator | 3.05K IOPS | 4.84K IOPS | 5.5K IOPS | 5.5K IOPS | 20K IOPS |
| bench-validator-10 | validator | 2.56K IOPS | 2.83K IOPS | 2.94K IOPS | 2.94K IOPS | 20K IOPS |
| Node | Machine | vCPU | Disk | IOPS limit | MiB/s limit | Basis |
|---|---|---|---|---|---|---|
| bench-observer-11 | c3-highcpu-8 | 8 | hyperdisk-balanced 100 GB | 20K IOPS | 800.0 MiB/s | hyperdisk-balanced 100 GB: provisioned 20,000 IOPS, provisioned 800 MiB/s; c3 8 vCPU cap 50,000 IOPS / 800 MiB/s; reads and writes share this budget |
| bench-validator-01 | c3-highcpu-8 | 8 | hyperdisk-balanced 100 GB | 20K IOPS | 800.0 MiB/s | hyperdisk-balanced 100 GB: provisioned 20,000 IOPS, provisioned 800 MiB/s; c3 8 vCPU cap 50,000 IOPS / 800 MiB/s; reads and writes share this budget |
| bench-validator-02 | c3-highcpu-8 | 8 | hyperdisk-balanced 100 GB | 20K IOPS | 800.0 MiB/s | hyperdisk-balanced 100 GB: provisioned 20,000 IOPS, provisioned 800 MiB/s; c3 8 vCPU cap 50,000 IOPS / 800 MiB/s; reads and writes share this budget |
| bench-validator-03 | c3-highcpu-8 | 8 | hyperdisk-balanced 100 GB | 20K IOPS | 800.0 MiB/s | hyperdisk-balanced 100 GB: provisioned 20,000 IOPS, provisioned 800 MiB/s; c3 8 vCPU cap 50,000 IOPS / 800 MiB/s; reads and writes share this budget |
| bench-validator-04 | c3-highcpu-8 | 8 | hyperdisk-balanced 100 GB | 20K IOPS | 800.0 MiB/s | hyperdisk-balanced 100 GB: provisioned 20,000 IOPS, provisioned 800 MiB/s; c3 8 vCPU cap 50,000 IOPS / 800 MiB/s; reads and writes share this budget |
| bench-validator-05 | c3-highcpu-8 | 8 | hyperdisk-balanced 100 GB | 20K IOPS | 800.0 MiB/s | hyperdisk-balanced 100 GB: provisioned 20,000 IOPS, provisioned 800 MiB/s; c3 8 vCPU cap 50,000 IOPS / 800 MiB/s; reads and writes share this budget |
| bench-validator-06 | c3-highcpu-8 | 8 | hyperdisk-balanced 100 GB | 20K IOPS | 800.0 MiB/s | hyperdisk-balanced 100 GB: provisioned 20,000 IOPS, provisioned 800 MiB/s; c3 8 vCPU cap 50,000 IOPS / 800 MiB/s; reads and writes share this budget |
| bench-validator-07 | c3-highcpu-8 | 8 | hyperdisk-balanced 100 GB | 20K IOPS | 800.0 MiB/s | hyperdisk-balanced 100 GB: provisioned 20,000 IOPS, provisioned 800 MiB/s; c3 8 vCPU cap 50,000 IOPS / 800 MiB/s; reads and writes share this budget |
| bench-validator-08 | c3-highcpu-8 | 8 | hyperdisk-balanced 100 GB | 20K IOPS | 800.0 MiB/s | hyperdisk-balanced 100 GB: provisioned 20,000 IOPS, provisioned 800 MiB/s; c3 8 vCPU cap 50,000 IOPS / 800 MiB/s; reads and writes share this budget |
| bench-validator-09 | c3-highcpu-8 | 8 | hyperdisk-balanced 100 GB | 20K IOPS | 800.0 MiB/s | hyperdisk-balanced 100 GB: provisioned 20,000 IOPS, provisioned 800 MiB/s; c3 8 vCPU cap 50,000 IOPS / 800 MiB/s; reads and writes share this budget |
| bench-validator-10 | c3-highcpu-8 | 8 | hyperdisk-balanced 100 GB | 20K IOPS | 800.0 MiB/s | hyperdisk-balanced 100 GB: provisioned 20,000 IOPS, provisioned 800 MiB/s; c3 8 vCPU cap 50,000 IOPS / 800 MiB/s; reads and writes share this budget |
instance/network/sent_bytes_count rate; summaries over the mixed phase
| Node | Class | Mean | p95 | Peak 60-s mean | Max |
|---|---|---|---|---|---|
| bench-observer-11 | observer | 1.0 Mbps | 1.2 Mbps | 1.4 Mbps | 1.4 Mbps |
| bench-validator-01 | validator | 59.0 Mbps | 70.9 Mbps | 82.5 Mbps | 82.5 Mbps |
| bench-validator-02 | validator | 61.1 Mbps | 70.9 Mbps | 71.9 Mbps | 71.9 Mbps |
| bench-validator-03 | validator | 60.0 Mbps | 70.7 Mbps | 79.1 Mbps | 79.1 Mbps |
| bench-validator-04 | validator | 61.2 Mbps | 73.6 Mbps | 76.4 Mbps | 76.4 Mbps |
| bench-validator-05 | validator | 55.0 Mbps | 77.8 Mbps | 84.6 Mbps | 84.6 Mbps |
| bench-validator-06 | validator | 60.9 Mbps | 69.8 Mbps | 70.2 Mbps | 70.2 Mbps |
| bench-validator-07 | validator | 62.9 Mbps | 71.6 Mbps | 73.6 Mbps | 73.6 Mbps |
| bench-validator-08 | validator | 62.9 Mbps | 81.6 Mbps | 99.6 Mbps | 99.6 Mbps |
| bench-validator-09 | validator | 62.3 Mbps | 71.3 Mbps | 73.6 Mbps | 73.6 Mbps |
| bench-validator-10 | validator | 60.2 Mbps | 67.5 Mbps | 72.2 Mbps | 72.2 Mbps |
instance/network/received_bytes_count rate; summaries over the mixed phase
| Node | Class | Mean | p95 | Peak 60-s mean | Max |
|---|---|---|---|---|---|
| bench-observer-11 | observer | 3.3 Mbps | 3.8 Mbps | 4.2 Mbps | 4.2 Mbps |
| bench-validator-01 | validator | 5.4 Mbps | 6.5 Mbps | 7.3 Mbps | 7.3 Mbps |
| bench-validator-02 | validator | 6.1 Mbps | 7.4 Mbps | 8.1 Mbps | 8.1 Mbps |
| bench-validator-03 | validator | 5.8 Mbps | 7.9 Mbps | 8.0 Mbps | 8.0 Mbps |
| bench-validator-04 | validator | 5.8 Mbps | 8.2 Mbps | 8.4 Mbps | 8.4 Mbps |
| bench-validator-05 | validator | 12.2 Mbps | 31.7 Mbps | 43.9 Mbps | 43.9 Mbps |
| bench-validator-06 | validator | 6.0 Mbps | 7.1 Mbps | 7.7 Mbps | 7.7 Mbps |
| bench-validator-07 | validator | 6.7 Mbps | 7.8 Mbps | 7.9 Mbps | 7.9 Mbps |
| bench-validator-08 | validator | 5.3 Mbps | 7.2 Mbps | 7.5 Mbps | 7.5 Mbps |
| bench-validator-09 | validator | 6.7 Mbps | 8.5 Mbps | 8.9 Mbps | 8.9 Mbps |
| bench-validator-10 | validator | 5.8 Mbps | 7.4 Mbps | 8.2 Mbps | 8.2 Mbps |
| Metric | Query | Class | Nodes | Median of node means | Worst peak 60-s mean | Max |
|---|---|---|---|---|---|---|
| tn_primary_commit_latency_seconds | p95 | validator | 10 | 3,378 ms | 26.9 s | 30.0 s |
| tn_engine_execution_duration_seconds | p95 | validator | 10 | 4,255 ms | 12.2 s | 17.0 s |
| tn_engine_execution_duration_seconds | p95 | observer | 1 | 3,984 ms | 6,219 ms | 7,000 ms |
| tn_worker_batch_size_bytes | p95 | validator | 10 | 237.8 KiB | 245.8 KiB | 246.2 KiB |
| tn_worker_batch_transactions | p95 | validator | 10 | 487 | 1.48K | 1.48K |
| tn_executor_output_transactions | p95 | validator | 10 | 20.4K | 51.2K | 75K |
| tn_primary_consensus_output_bytes | max | validator | 10 | 677.2 KiB | 2.4 MiB | 4.4 MiB |
| tn_primary_consensus_output_bytes | max | observer | 1 | 691.5 KiB | 2.0 MiB | 2.3 MiB |
| tn_engine_queued_outputs | max | validator | 10 | 4.37 | 8 | 8 |
| tn_engine_queued_outputs | max | observer | 1 | 2.13 | 7.75 | 8 |
| tn_primary_round | max | validator | 10 | 803 | 1.78K | 1.78K |
| tn_primary_round | max | observer | 1 | 786 | 1.75K | 1.78K |
| tn_primary_committed_round | max | validator | 10 | 801 | 1.78K | 1.78K |
| tn_primary_committed_round | max | observer | 1 | 786 | 1.75K | 1.78K |
| tn_epoch_current | max | validator | 10 | 7.77 | 9 | 9 |
| tn_epoch_current | max | observer | 1 | 7.77 | 9 | 9 |
| tn_batch_builder_pending_pool_transactions | max | validator | 10 | 387 | 9.74K | 9.74K |
| tn_batch_builder_pending_pool_transactions | max | observer | 1 | 0 | 0 | 0 |
| reth_process_resident_memory_bytes | max | validator | 10 | 5.71 GiB | 8.08 GiB | 8.12 GiB |
| reth_process_resident_memory_bytes | max | observer | 1 | 2.74 GiB | 3.89 GiB | 3.91 GiB |
| tn_primary_subdags_committed_total | rate | validator | 10 | 0.472 | 0.767 | 0.844 |
| tn_primary_subdags_committed_total | rate | observer | 1 | 0 | 0 | 0 |
| tn_primary_round_lag | lag | validator | 10 | 2.13 | 4.75 | 7 |
| tn_primary_round_lag | lag | observer | 1 | 0 | 0 | 0 |
| tn_batch_builder_seal_failures_total | rate | validator | 5 | 0 | 0 | 0 |
| tn_node_mode | max | validator | 10 | 0 | 1 | 1 |
| tn_node_mode | max | observer | 1 | 0 | 0 | 0 |
| tn_primary_vote_requests_total | rate | validator | 10 | 8.7 | 27 | 27.8 |
| tn_primary_vote_requests_total | rate | observer | 1 | 0 | 0 | 0 |
| tn_primary_votes_created_total | rate | validator | 10 | 7.24 | 14 | 14 |
| tn_primary_votes_created_total | rate | observer | 1 | 0 | 0 | 0 |
| tn_primary_votes_recast_total | rate | validator | 10 | 0 | 0 | 0 |
| tn_primary_votes_recast_total | rate | observer | 1 | 0 | 0 | 0 |
| tn_primary_vote_requests_too_new_total | rate | validator | 10 | 0 | 0 | 0 |
| tn_primary_vote_requests_too_new_total | rate | observer | 1 | 0 | 0 | 0 |
| tn_primary_vote_timeouts_total{stage="exec_wait"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_primary_vote_timeouts_total{stage="exec_wait"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_primary_vote_timeouts_total{stage="lead_sleep"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_primary_vote_timeouts_total{stage="lead_sleep"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_primary_vote_timeouts_total{stage="parents"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_primary_vote_timeouts_total{stage="parents"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_primary_vote_timeouts_total{stage="persist"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_primary_vote_timeouts_total{stage="persist"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_primary_vote_timeouts_total{stage="sync_batches"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_primary_vote_timeouts_total{stage="sync_batches"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_primary_vote_timeouts_total{stage="validate"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_primary_vote_timeouts_total{stage="validate"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_primary_vote_timeouts_total{stage="vote_store"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_primary_vote_timeouts_total{stage="vote_store"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_primary_vote_stage_seconds{stage="exec_wait"} | p95 | validator | 10 | 1,503 ms | 2,425 ms | 2,425 ms |
| tn_primary_vote_stage_seconds{stage="lead_sleep"} | p95 | validator | 10 | 0 ms | 0 ms | 0 ms |
| tn_primary_vote_stage_seconds{stage="lock"} | p95 | validator | 10 | 1,562 ms | 9,709 ms | 9,739 ms |
| tn_primary_vote_stage_seconds{stage="parents"} | p95 | validator | 10 | 1 ms | 19 ms | 27 ms |
| tn_primary_vote_stage_seconds{stage="persist"} | p95 | validator | 10 | 190 ms | 232 ms | 246 ms |
| tn_primary_vote_stage_seconds{stage="sync_batches"} | p95 | validator | 10 | 0 ms | 0 ms | 0 ms |
| tn_primary_vote_stage_seconds{stage="validate"} | p95 | validator | 10 | 2 ms | 4 ms | 4 ms |
| tn_primary_vote_stage_seconds{stage="vote_store"} | p95 | validator | 10 | 12 ms | 55 ms | 80 ms |
| tn_primary_leader_rounds_total{leader="2xQKratGEXurhrZgwZV1kaxJTtUDbskRJvyJTYRg4gAw",swapped="false",found="false"} | rate | validator | 10 | 0.00264 | 0.0333 | 0.0444 |
| tn_primary_leader_rounds_total{leader="2xQKratGEXurhrZgwZV1kaxJTtUDbskRJvyJTYRg4gAw",swapped="false",found="true"} | rate | validator | 10 | 0.0182 | 0.0833 | 0.111 |
| tn_primary_leader_rounds_total{leader="2xQKratGEXurhrZgwZV1kaxJTtUDbskRJvyJTYRg4gAw",swapped="true",found="false"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_primary_leader_rounds_total{leader="2xQKratGEXurhrZgwZV1kaxJTtUDbskRJvyJTYRg4gAw",swapped="true",found="true"} | rate | validator | 10 | 0.00347 | 0.0333 | 0.0444 |
| tn_primary_leader_rounds_total{leader="35UGYEik14WmgYJYWzf8e97o1jZtv6FDaLVz8SoCJiTU",swapped="false",found="false"} | rate | validator | 10 | 0.00417 | 0.05 | 0.0667 |
| tn_primary_leader_rounds_total{leader="35UGYEik14WmgYJYWzf8e97o1jZtv6FDaLVz8SoCJiTU",swapped="false",found="true"} | rate | validator | 10 | 0.0457 | 0.0889 | 0.111 |
| tn_primary_leader_rounds_total{leader="35UGYEik14WmgYJYWzf8e97o1jZtv6FDaLVz8SoCJiTU",swapped="true",found="false"} | rate | validator | 10 | 0.000833 | 0.0167 | 0.0222 |
| tn_primary_leader_rounds_total{leader="35UGYEik14WmgYJYWzf8e97o1jZtv6FDaLVz8SoCJiTU",swapped="true",found="true"} | rate | validator | 10 | 0.0112 | 0.0815 | 0.0889 |
| tn_primary_leader_rounds_total{leader="3cTfbmwcFSdAFQE5G4Chtdjc5H2zBtutnJDes8KuYXov",swapped="false",found="false"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_primary_leader_rounds_total{leader="3cTfbmwcFSdAFQE5G4Chtdjc5H2zBtutnJDes8KuYXov",swapped="false",found="true"} | rate | validator | 10 | 0.0499 | 0.0889 | 0.111 |
| tn_primary_leader_rounds_total{leader="3cTfbmwcFSdAFQE5G4Chtdjc5H2zBtutnJDes8KuYXov",swapped="true",found="false"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_primary_leader_rounds_total{leader="3cTfbmwcFSdAFQE5G4Chtdjc5H2zBtutnJDes8KuYXov",swapped="true",found="true"} | rate | validator | 10 | 0.0107 | 0.0667 | 0.0667 |
| tn_primary_leader_rounds_total{leader="4zoSra4cxG6XAr6MS7duBukTBdvV7SswEswuFhKdTRX3",swapped="false",found="false"} | rate | validator | 10 | 0.00203 | 0.0444 | 0.0444 |
| tn_primary_leader_rounds_total{leader="4zoSra4cxG6XAr6MS7duBukTBdvV7SswEswuFhKdTRX3",swapped="false",found="true"} | rate | validator | 10 | 0.0332 | 0.0889 | 0.111 |
| tn_primary_leader_rounds_total{leader="4zoSra4cxG6XAr6MS7duBukTBdvV7SswEswuFhKdTRX3",swapped="true",found="false"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_primary_leader_rounds_total{leader="4zoSra4cxG6XAr6MS7duBukTBdvV7SswEswuFhKdTRX3",swapped="true",found="true"} | rate | validator | 10 | 0.00542 | 0.0389 | 0.0444 |
| tn_primary_leader_rounds_total{leader="9qbgQeg8qruCGKrjgbE3RZCeCzppUYPwzC5ZGs5zbrWN",swapped="false",found="false"} | rate | validator | 10 | 0.00116 | 0.0331 | 0.0441 |
| tn_primary_leader_rounds_total{leader="9qbgQeg8qruCGKrjgbE3RZCeCzppUYPwzC5ZGs5zbrWN",swapped="false",found="true"} | rate | validator | 10 | 0.0486 | 0.0889 | 0.0889 |
| tn_primary_leader_rounds_total{leader="9qbgQeg8qruCGKrjgbE3RZCeCzppUYPwzC5ZGs5zbrWN",swapped="true",found="false"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_primary_leader_rounds_total{leader="9qbgQeg8qruCGKrjgbE3RZCeCzppUYPwzC5ZGs5zbrWN",swapped="true",found="true"} | rate | validator | 10 | 0.0108 | 0.05 | 0.0667 |
| tn_primary_leader_rounds_total{leader="Br3WMnwqL5zXNtK4VwriZZyoRze3GRR9jzoYt4kRKaMu",swapped="false",found="false"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_primary_leader_rounds_total{leader="Br3WMnwqL5zXNtK4VwriZZyoRze3GRR9jzoYt4kRKaMu",swapped="false",found="true"} | rate | validator | 10 | 0.0497 | 0.0833 | 0.111 |
| tn_primary_leader_rounds_total{leader="Br3WMnwqL5zXNtK4VwriZZyoRze3GRR9jzoYt4kRKaMu",swapped="true",found="false"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_primary_leader_rounds_total{leader="Br3WMnwqL5zXNtK4VwriZZyoRze3GRR9jzoYt4kRKaMu",swapped="true",found="true"} | rate | validator | 10 | 0.0112 | 0.0611 | 0.0889 |
| tn_primary_leader_rounds_total{leader="CraLT3xtoP81xHLUVnUPzhnTcrbssD1DXga1dFjSTbxy",swapped="false",found="false"} | rate | validator | 10 | 0.00863 | 0.0722 | 0.0889 |
| tn_primary_leader_rounds_total{leader="CraLT3xtoP81xHLUVnUPzhnTcrbssD1DXga1dFjSTbxy",swapped="false",found="true"} | rate | validator | 10 | 0.0203 | 0.0889 | 0.0889 |
| tn_primary_leader_rounds_total{leader="CraLT3xtoP81xHLUVnUPzhnTcrbssD1DXga1dFjSTbxy",swapped="true",found="false"} | rate | validator | 10 | 0.000744 | 0.0331 | 0.0441 |
| tn_primary_leader_rounds_total{leader="CraLT3xtoP81xHLUVnUPzhnTcrbssD1DXga1dFjSTbxy",swapped="true",found="true"} | rate | validator | 10 | 0.00958 | 0.0667 | 0.0667 |
| tn_primary_leader_rounds_total{leader="Cy8GPpAWuRTXmb3MRTxMLnehaF3Q9VL6WedTrNQkLT4J",swapped="false",found="false"} | rate | validator | 10 | 0.00417 | 0.05 | 0.0667 |
| tn_primary_leader_rounds_total{leader="Cy8GPpAWuRTXmb3MRTxMLnehaF3Q9VL6WedTrNQkLT4J",swapped="false",found="true"} | rate | validator | 10 | 0.0164 | 0.0722 | 0.0889 |
| tn_primary_leader_rounds_total{leader="Cy8GPpAWuRTXmb3MRTxMLnehaF3Q9VL6WedTrNQkLT4J",swapped="true",found="false"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_primary_leader_rounds_total{leader="Cy8GPpAWuRTXmb3MRTxMLnehaF3Q9VL6WedTrNQkLT4J",swapped="true",found="true"} | rate | validator | 10 | 0.0025 | 0.0333 | 0.0444 |
| tn_primary_leader_rounds_total{leader="FKvYJLZuwBRW37NmRrSh9VH7BBv95yLfGSb5Ckn1xhZp",swapped="false",found="false"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_primary_leader_rounds_total{leader="FKvYJLZuwBRW37NmRrSh9VH7BBv95yLfGSb5Ckn1xhZp",swapped="false",found="true"} | rate | validator | 10 | 0.0497 | 0.0833 | 0.0889 |
| tn_primary_leader_rounds_total{leader="FKvYJLZuwBRW37NmRrSh9VH7BBv95yLfGSb5Ckn1xhZp",swapped="true",found="false"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_primary_leader_rounds_total{leader="FKvYJLZuwBRW37NmRrSh9VH7BBv95yLfGSb5Ckn1xhZp",swapped="true",found="true"} | rate | validator | 10 | 0.0141 | 0.0556 | 0.0667 |
| tn_primary_leader_rounds_total{leader="GBSG38zYEybFE4ELTMRQFQWkSLZLn5goracd4TXx2tX2",swapped="false",found="false"} | rate | validator | 10 | 0.000363 | 0.0165 | 0.0221 |
| tn_primary_leader_rounds_total{leader="GBSG38zYEybFE4ELTMRQFQWkSLZLn5goracd4TXx2tX2",swapped="false",found="true"} | rate | validator | 10 | 0.0488 | 0.0889 | 0.0889 |
| tn_primary_leader_rounds_total{leader="GBSG38zYEybFE4ELTMRQFQWkSLZLn5goracd4TXx2tX2",swapped="true",found="false"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_primary_leader_rounds_total{leader="GBSG38zYEybFE4ELTMRQFQWkSLZLn5goracd4TXx2tX2",swapped="true",found="true"} | rate | validator | 10 | 0.0133 | 0.0611 | 0.0889 |
| tn_primary_leader_commits_total{leader="2xQKratGEXurhrZgwZV1kaxJTtUDbskRJvyJTYRg4gAw"} | rate | validator | 10 | 0.0217 | 0.0833 | 0.111 |
| tn_primary_leader_commits_total{leader="35UGYEik14WmgYJYWzf8e97o1jZtv6FDaLVz8SoCJiTU"} | rate | validator | 10 | 0.0569 | 0.167 | 0.178 |
| tn_primary_leader_commits_total{leader="3cTfbmwcFSdAFQE5G4Chtdjc5H2zBtutnJDes8KuYXov"} | rate | validator | 10 | 0.0601 | 0.156 | 0.156 |
| tn_primary_leader_commits_total{leader="4zoSra4cxG6XAr6MS7duBukTBdvV7SswEswuFhKdTRX3"} | rate | validator | 10 | 0.0386 | 0.1 | 0.111 |
| tn_primary_leader_commits_total{leader="9qbgQeg8qruCGKrjgbE3RZCeCzppUYPwzC5ZGs5zbrWN"} | rate | validator | 10 | 0.0594 | 0.128 | 0.156 |
| tn_primary_leader_commits_total{leader="Br3WMnwqL5zXNtK4VwriZZyoRze3GRR9jzoYt4kRKaMu"} | rate | validator | 10 | 0.061 | 0.122 | 0.156 |
| tn_primary_leader_commits_total{leader="CraLT3xtoP81xHLUVnUPzhnTcrbssD1DXga1dFjSTbxy"} | rate | validator | 10 | 0.0299 | 0.15 | 0.156 |
| tn_primary_leader_commits_total{leader="Cy8GPpAWuRTXmb3MRTxMLnehaF3Q9VL6WedTrNQkLT4J"} | rate | validator | 10 | 0.0189 | 0.0889 | 0.111 |
| tn_primary_leader_commits_total{leader="FKvYJLZuwBRW37NmRrSh9VH7BBv95yLfGSb5Ckn1xhZp"} | rate | validator | 10 | 0.0639 | 0.128 | 0.156 |
| tn_primary_leader_commits_total{leader="GBSG38zYEybFE4ELTMRQFQWkSLZLn5goracd4TXx2tX2"} | rate | validator | 10 | 0.0621 | 0.139 | 0.178 |
| tn_engine_queue_wait_seconds | p95 | validator | 10 | 18.9 s | 29.0 s | 29.0 s |
| tn_engine_queue_wait_seconds | p95 | observer | 1 | 7,649 ms | 28.9 s | 29.0 s |
| tn_engine_persist_duration_seconds | p95 | validator | 10 | 2,473 ms | 5,652 ms | 6,750 ms |
| tn_engine_persist_duration_seconds | p95 | observer | 1 | 2,417 ms | 2,919 ms | 3,500 ms |
| tn_engine_busy_milliseconds_total | rate | validator | 10 | 979 | 1.04K | 1.2K |
| tn_engine_busy_milliseconds_total | rate | observer | 1 | 920 | 1.02K | 1.05K |
| tn_reth_persist_save_blocks_seconds | p95 | validator | 10 | 398 ms | 2,731 ms | 3,375 ms |
| tn_reth_persist_save_blocks_seconds | p95 | observer | 1 | 307 ms | 622 ms | 700 ms |
| tn_reth_persist_commit_seconds | p95 | validator | 10 | 2,340 ms | 2,425 ms | 2,425 ms |
| tn_reth_persist_commit_seconds | p95 | observer | 1 | 2,334 ms | 2,425 ms | 2,425 ms |
| tn_reth_persisted_accounts_total | rate | validator | 10 | 1.57K | 2.58K | 2.86K |
| tn_reth_persisted_accounts_total | rate | observer | 1 | 1.56K | 2.15K | 2.35K |
| tn_reth_persisted_storage_slots_total | rate | validator | 10 | 2.17K | 3.41K | 3.78K |
| tn_reth_persisted_storage_slots_total | rate | observer | 1 | 2.15K | 2.85K | 3.1K |
| tn_node_mode_transitions_total{to="cvv_inactive",reason="epoch_behind"} | rate | validator | 1 | 0 | 0 | 0 |
| tn_executor_rejoin_blocked_total{reason="attempts_exhausted"} | rate | validator | 1 | 0 | 0 | 0 |
| tn_executor_rejoin_blocked_total{reason="epoch_closed"} | rate | validator | 1 | 0.11 | 0.528 | 0.578 |
| tn_executor_rejoin_blocked_total{reason="epoch_mismatch"} | rate | validator | 1 | 0 | 0 | 0 |
| tn_executor_rejoin_blocked_total{reason="no_known_header"} | rate | validator | 1 | 0 | 0 | 0 |
| tn_executor_rejoin_blocked_total{reason="round_gap"} | rate | validator | 1 | 0.281 | 0.533 | 0.578 |
| tn_executor_rejoin_blocked_total{reason="stale_head"} | rate | validator | 1 | 0.0941 | 0.344 | 0.422 |
| tn_worker_sync_serves_denied_total{worker="0",tier="committee"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_worker_sync_serves_denied_total{worker="0",tier="committee"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_worker_sync_serves_denied_total{worker="0",tier="public"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_worker_sync_serves_denied_total{worker="0",tier="public"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_worker_batch_requests_denied_total{worker="0",peer="21N2vp3GaZwbCt1bqcBkrg3e5ShjP3Ss4pDF92MtrtYNAVrxq1LNQyFeAty226eBSt7ywnRkAg8XcxztrJcLkSw82uJREWMvzAhnkZzMYLhSAxEggfxpnk41L398JkQheqr4"} | rate | validator | 2 | 0.146 | 3.52 | 4.69 |
| tn_worker_batch_requests_denied_total{worker="0",peer="22Pq8MwWk8jvrnMCaqVCSkTW3JKgC8sJwJ4V9sDAL7nypan2zh2YntVGVYnjM1J67BW6iw87wHL3GGxttJd8kv1evHyzv8X7wyCH6wsGP83Ua7f83EdWFELtZpCLzBpi7NRs"} | rate | validator | 2 | 0.059 | 0.933 | 1.2 |
| tn_worker_batch_requests_denied_total{worker="0",peer="237jbxzcELWRFg5wb9XtWG7Vw9kacjAceJsiJ8P65XGgPgaA8wMRZKp1j7noKyimEdMEgbf5Xt1pZiGi7MSWDmyriPS3ENeZ7bqw7qKkYskJ4qNPz5F7aaCCGdfc9LZd1D9t"} | rate | validator | 3 | 0.0959 | 2.67 | 3.56 |
| tn_worker_batch_requests_denied_total{worker="0",peer="241nv7f9rufDfN5aw3izxGs4awWbc2EeBnX8JeczXZb18KCxvgb7NPjGsdcMjQE8LA7TeFbeK7qW9W4LggWM15UphDvCVENdU7SwKD7qro7bzcQmVCssHULsDKcSMcxHYfqT"} | rate | validator | 3 | 0 | 1.37 | 1.37 |
| tn_worker_batch_requests_denied_total{worker="0",peer="242iCdm1PnngYQ4Y3WTxt9ScYT9BmEJumr77vEaVnDMCj8b5bagbaRBafTLgt1Y4HuBtbC25CURfxhdm5jv3Yoen85Zy5BRHrXcjnJyGxGNmpiqzDPxsAojWpz7EH5g4STRw"} | rate | validator | 2 | 0.104 | 2.55 | 3.4 |
| tn_worker_batch_requests_denied_total{worker="0",peer="mY4mb1gk2adED3HTP3r8aLZhGqcCZmNDfzirXWyomvXB72kkd7SxQKNz857yzy98TS12X7LckY23G8coxSowXyMm93MxfbGviStaxD2g1cRp6Jmh68DbLoN9VVCT8ZX9Zxs"} | rate | validator | 3 | 0.00323 | 2.28 | 3 |
| tn_worker_batch_requests_denied_total{worker="0",peer="n6cEqCe39mRWFoZ3mGZE4nbyGbhun8VSFMCWDvAJugFVR1WPLrgNiTyrBHoqFixkvQzr3ADuRD6X1XVfwjR5S4iQvq89DndaFwk8UGLze2JwuKD9zhazyQTwqrh7j5YKcPc"} | rate | validator | 2 | 0.123 | 3.02 | 4.02 |
| tn_worker_batch_requests_denied_total{worker="0",peer="ojV9SY5hHE4EEsZJzZLwqVpDfVfDBqUPnEsmNX9NBLUarrhPiggeSZonTDsctzH5bYoU9Wiu9RKc9sjja7hGHek8JHyZHwmYgiLD4awMtwJVa5Maig318iRyqXdw8cGsavt"} | rate | validator | 2 | 0.315 | 4.25 | 5.67 |
| tn_worker_batch_requests_denied_total{worker="0",peer="orWZphKCbZg4MyMgHMp5mzQ8gtRwYsSnzNGbgZDU73oRMbtSjag6qYmEdjqGJquoooJRRKPHX2H7WdioR6SUmvmgPomMje5g3VspvcseJF1VoFpMA8VMdWB9ug2WS31HdLM"} | rate | validator | 2 | 0.0939 | 2.3 | 3.07 |
| tn_worker_batch_requests_denied_total{worker="0",peer="s434VVj1mKtfRuhV9ATgLq5tp1xEixtd42JZh7uYbjMUZdVPmq4YY6yLCoU497LfdukihSg3ytdqwgjGk7vxKqviUsWXQyGhyckLXLSvmBF87cz9e3ZcANSL55TK54eHp84"} | rate | validator | 2 | 0.0864 | 2.12 | 2.82 |
| tn_worker_batch_requests_denied_total{worker="0",peer="z38HGKMnkMqbCiuDq3WbgzUxjSB3LvTm25K1d7Qg3DHLdmDuZwwaGGEaxxVc3FZQ5M5xVs8kzCys6TJE3FdUfnqQftxXMB4rivn4mZXfaMe4vgD5kQib9pp8zj6hZQMHbiQ"} | rate | validator | 2 | 0.0762 | 1.87 | 2.49 |
| tn_worker_batch_requests_failed_total{worker="0",peer="21N2vp3GaZwbCt1bqcBkrg3e5ShjP3Ss4pDF92MtrtYNAVrxq1LNQyFeAty226eBSt7ywnRkAg8XcxztrJcLkSw82uJREWMvzAhnkZzMYLhSAxEggfxpnk41L398JkQheqr4"} | rate | validator | 2 | 0.0264 | 0.646 | 0.8 |
| tn_worker_batch_requests_failed_total{worker="0",peer="22Pq8MwWk8jvrnMCaqVCSkTW3JKgC8sJwJ4V9sDAL7nypan2zh2YntVGVYnjM1J67BW6iw87wHL3GGxttJd8kv1evHyzv8X7wyCH6wsGP83Ua7f83EdWFELtZpCLzBpi7NRs"} | rate | validator | 2 | 0.0395 | 0.967 | 1.29 |
| tn_worker_batch_requests_failed_total{worker="0",peer="237jbxzcELWRFg5wb9XtWG7Vw9kacjAceJsiJ8P65XGgPgaA8wMRZKp1j7noKyimEdMEgbf5Xt1pZiGi7MSWDmyriPS3ENeZ7bqw7qKkYskJ4qNPz5F7aaCCGdfc9LZd1D9t"} | rate | validator | 3 | 0.0348 | 0.933 | 1.24 |
| tn_worker_batch_requests_failed_total{worker="0",peer="241nv7f9rufDfN5aw3izxGs4awWbc2EeBnX8JeczXZb18KCxvgb7NPjGsdcMjQE8LA7TeFbeK7qW9W4LggWM15UphDvCVENdU7SwKD7qro7bzcQmVCssHULsDKcSMcxHYfqT"} | rate | validator | 3 | 0 | 0.358 | 0.422 |
| tn_worker_batch_requests_failed_total{worker="0",peer="242iCdm1PnngYQ4Y3WTxt9ScYT9BmEJumr77vEaVnDMCj8b5bagbaRBafTLgt1Y4HuBtbC25CURfxhdm5jv3Yoen85Zy5BRHrXcjnJyGxGNmpiqzDPxsAojWpz7EH5g4STRw"} | rate | validator | 2 | 0.0415 | 1 | 1.33 |
| tn_worker_batch_requests_failed_total{worker="0",peer="mY4mb1gk2adED3HTP3r8aLZhGqcCZmNDfzirXWyomvXB72kkd7SxQKNz857yzy98TS12X7LckY23G8coxSowXyMm93MxfbGviStaxD2g1cRp6Jmh68DbLoN9VVCT8ZX9Zxs"} | rate | validator | 3 | 0 | 0.599 | 0.797 |
| tn_worker_batch_requests_failed_total{worker="0",peer="n6cEqCe39mRWFoZ3mGZE4nbyGbhun8VSFMCWDvAJugFVR1WPLrgNiTyrBHoqFixkvQzr3ADuRD6X1XVfwjR5S4iQvq89DndaFwk8UGLze2JwuKD9zhazyQTwqrh7j5YKcPc"} | rate | validator | 2 | 0.0229 | 0.467 | 0.622 |
| tn_worker_batch_requests_failed_total{worker="0",peer="ojV9SY5hHE4EEsZJzZLwqVpDfVfDBqUPnEsmNX9NBLUarrhPiggeSZonTDsctzH5bYoU9Wiu9RKc9sjja7hGHek8JHyZHwmYgiLD4awMtwJVa5Maig318iRyqXdw8cGsavt"} | rate | validator | 2 | 0.0575 | 0.831 | 1.11 |
| tn_worker_batch_requests_failed_total{worker="0",peer="orWZphKCbZg4MyMgHMp5mzQ8gtRwYsSnzNGbgZDU73oRMbtSjag6qYmEdjqGJquoooJRRKPHX2H7WdioR6SUmvmgPomMje5g3VspvcseJF1VoFpMA8VMdWB9ug2WS31HdLM"} | rate | validator | 2 | 0.047 | 0.729 | 0.933 |
| tn_worker_batch_requests_failed_total{worker="0",peer="s434VVj1mKtfRuhV9ATgLq5tp1xEixtd42JZh7uYbjMUZdVPmq4YY6yLCoU497LfdukihSg3ytdqwgjGk7vxKqviUsWXQyGhyckLXLSvmBF87cz9e3ZcANSL55TK54eHp84"} | rate | validator | 2 | 0.0347 | 0.85 | 1.13 |
| tn_worker_batch_requests_failed_total{worker="0",peer="z38HGKMnkMqbCiuDq3WbgzUxjSB3LvTm25K1d7Qg3DHLdmDuZwwaGGEaxxVc3FZQ5M5xVs8kzCys6TJE3FdUfnqQftxXMB4rivn4mZXfaMe4vgD5kQib9pp8zj6hZQMHbiQ"} | rate | validator | 2 | 0.0942 | 1.34 | 1.77 |
| tn_node_epoch_record_signers | max | validator | 10 | 8.12 | 10 | 10 |
| tn_node_epoch_record_signers | max | observer | 1 | 8.03 | 9 | 9 |
| tn_node_epoch_record_missing_signers_total{signer="2xQKratGEXurhrZgwZV1kaxJTtUDbskRJvyJTYRg4gAw"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_node_epoch_record_missing_signers_total{signer="2xQKratGEXurhrZgwZV1kaxJTtUDbskRJvyJTYRg4gAw"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_node_epoch_record_missing_signers_total{signer="35UGYEik14WmgYJYWzf8e97o1jZtv6FDaLVz8SoCJiTU"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_node_epoch_record_missing_signers_total{signer="35UGYEik14WmgYJYWzf8e97o1jZtv6FDaLVz8SoCJiTU"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_node_epoch_record_missing_signers_total{signer="3cTfbmwcFSdAFQE5G4Chtdjc5H2zBtutnJDes8KuYXov"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_node_epoch_record_missing_signers_total{signer="3cTfbmwcFSdAFQE5G4Chtdjc5H2zBtutnJDes8KuYXov"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_node_epoch_record_missing_signers_total{signer="4zoSra4cxG6XAr6MS7duBukTBdvV7SswEswuFhKdTRX3"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_node_epoch_record_missing_signers_total{signer="4zoSra4cxG6XAr6MS7duBukTBdvV7SswEswuFhKdTRX3"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_node_epoch_record_missing_signers_total{signer="9qbgQeg8qruCGKrjgbE3RZCeCzppUYPwzC5ZGs5zbrWN"} | rate | validator | 10 | 0.00078 | 0.0167 | 0.0222 |
| tn_node_epoch_record_missing_signers_total{signer="9qbgQeg8qruCGKrjgbE3RZCeCzppUYPwzC5ZGs5zbrWN"} | rate | observer | 1 | 0.000575 | 0.0167 | 0.0222 |
| tn_node_epoch_record_missing_signers_total{signer="Br3WMnwqL5zXNtK4VwriZZyoRze3GRR9jzoYt4kRKaMu"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_node_epoch_record_missing_signers_total{signer="Br3WMnwqL5zXNtK4VwriZZyoRze3GRR9jzoYt4kRKaMu"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_node_epoch_record_missing_signers_total{signer="CraLT3xtoP81xHLUVnUPzhnTcrbssD1DXga1dFjSTbxy"} | rate | validator | 10 | 0.00078 | 0.0167 | 0.0222 |
| tn_node_epoch_record_missing_signers_total{signer="CraLT3xtoP81xHLUVnUPzhnTcrbssD1DXga1dFjSTbxy"} | rate | observer | 1 | 0.000575 | 0.0167 | 0.0222 |
| tn_node_epoch_record_missing_signers_total{signer="Cy8GPpAWuRTXmb3MRTxMLnehaF3Q9VL6WedTrNQkLT4J"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_node_epoch_record_missing_signers_total{signer="Cy8GPpAWuRTXmb3MRTxMLnehaF3Q9VL6WedTrNQkLT4J"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_node_epoch_record_missing_signers_total{signer="FKvYJLZuwBRW37NmRrSh9VH7BBv95yLfGSb5Ckn1xhZp"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_node_epoch_record_missing_signers_total{signer="FKvYJLZuwBRW37NmRrSh9VH7BBv95yLfGSb5Ckn1xhZp"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_node_epoch_record_missing_signers_total{signer="GBSG38zYEybFE4ELTMRQFQWkSLZLn5goracd4TXx2tX2"} | rate | validator | 10 | 0.00078 | 0.0167 | 0.0222 |
| tn_node_epoch_record_missing_signers_total{signer="GBSG38zYEybFE4ELTMRQFQWkSLZLn5goracd4TXx2tX2"} | rate | observer | 1 | 0.000538 | 0.0156 | 0.0208 |
| tn_network_outbound_request_duration_seconds{outcome="failure",class="batch"} | p95 | validator | 4 | 21.1 s | 29.0 s | 29.0 s |
| tn_network_outbound_request_duration_seconds{outcome="failure",class="epoch_record"} | p95 | validator | 3 | 29.0 s | 29.0 s | 29.0 s |
| tn_network_outbound_request_duration_seconds{outcome="failure",class="vote"} | p95 | validator | 5 | 25.6 s | 29.0 s | 29.0 s |
| tn_network_outbound_request_duration_seconds{outcome="response",class="batch"} | p95 | validator | 4 | 467 ms | 622 ms | 622 ms |
| tn_network_outbound_request_duration_seconds{outcome="response",class="epoch_record"} | p95 | validator | 2 | 445 ms | 542 ms | 700 ms |
| tn_network_outbound_request_duration_seconds{outcome="response",class="epoch_record"} | p95 | observer | 1 | 243 ms | 248 ms | 250 ms |
| tn_network_outbound_request_duration_seconds{outcome="response",class="vote"} | p95 | validator | 4 | 5,654 ms | 9,464 ms | 9,490 ms |
| tn_network_closed_admission_denied_total{network="bench",direction="in"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_network_closed_admission_denied_total{network="bench",direction="in"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_network_closed_admission_denied_total{network="bench",direction="out"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_network_closed_admission_denied_total{network="bench",direction="out"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_network_source_admission_denied_total{network="bench",direction="in",reason="address_full"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_network_source_admission_denied_total{network="bench",direction="in",reason="address_full"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_network_source_admission_denied_total{network="bench",direction="in",reason="duplicate_connection"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_network_source_admission_denied_total{network="bench",direction="in",reason="duplicate_connection"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_network_source_admission_denied_total{network="bench",direction="in",reason="invalid_limits"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_network_source_admission_denied_total{network="bench",direction="in",reason="invalid_limits"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_network_source_admission_denied_total{network="bench",direction="in",reason="peer_full"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_network_source_admission_denied_total{network="bench",direction="in",reason="peer_full"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_network_source_admission_denied_total{network="bench",direction="in",reason="poisoned"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_network_source_admission_denied_total{network="bench",direction="in",reason="poisoned"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_network_source_admission_denied_total{network="bench",direction="in",reason="prefix_full"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_network_source_admission_denied_total{network="bench",direction="in",reason="prefix_full"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_network_source_admission_denied_total{network="bench",direction="in",reason="process_full"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_network_source_admission_denied_total{network="bench",direction="in",reason="process_full"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_network_source_admission_denied_total{network="bench",direction="in",reason="sources_full"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_network_source_admission_denied_total{network="bench",direction="in",reason="sources_full"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_network_source_admission_denied_total{network="bench",direction="in",reason="unsupported_address"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_network_source_admission_denied_total{network="bench",direction="in",reason="unsupported_address"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_network_source_admission_denied_total{network="bench",direction="out",reason="address_full"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_network_source_admission_denied_total{network="bench",direction="out",reason="address_full"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_network_source_admission_denied_total{network="bench",direction="out",reason="duplicate_connection"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_network_source_admission_denied_total{network="bench",direction="out",reason="duplicate_connection"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_network_source_admission_denied_total{network="bench",direction="out",reason="invalid_limits"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_network_source_admission_denied_total{network="bench",direction="out",reason="invalid_limits"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_network_source_admission_denied_total{network="bench",direction="out",reason="peer_full"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_network_source_admission_denied_total{network="bench",direction="out",reason="peer_full"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_network_source_admission_denied_total{network="bench",direction="out",reason="poisoned"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_network_source_admission_denied_total{network="bench",direction="out",reason="poisoned"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_network_source_admission_denied_total{network="bench",direction="out",reason="prefix_full"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_network_source_admission_denied_total{network="bench",direction="out",reason="prefix_full"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_network_source_admission_denied_total{network="bench",direction="out",reason="process_full"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_network_source_admission_denied_total{network="bench",direction="out",reason="process_full"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_network_source_admission_denied_total{network="bench",direction="out",reason="sources_full"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_network_source_admission_denied_total{network="bench",direction="out",reason="sources_full"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_network_source_admission_denied_total{network="bench",direction="out",reason="unsupported_address"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_network_source_admission_denied_total{network="bench",direction="out",reason="unsupported_address"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_network_admission_mode | max | validator | 10 | 2 | 2 | 2 |
| tn_network_admission_mode | max | observer | 1 | 2 | 2 | 2 |
| tn_network_admission_fallback | max | validator | 10 | 0 | 1 | 1 |
| tn_network_admission_fallback | max | observer | 1 | 0 | 0 | 0 |
| tn_network_outbound_request_failures_total{kind="connection"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_network_outbound_request_failures_total{kind="connection"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_network_outbound_request_failures_total{kind="dial"} | rate | validator | 10 | 11.4 | 54.7 | 65.9 |
| tn_network_outbound_request_failures_total{kind="dial"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_network_outbound_request_failures_total{kind="io"} | rate | validator | 10 | 1.57 | 7.08 | 8.53 |
| tn_network_outbound_request_failures_total{kind="io"} | rate | observer | 1 | 0.222 | 0.777 | 1.02 |
| tn_network_outbound_request_failures_total{kind="timeout"} | rate | validator | 10 | 17.9 | 53.4 | 69.8 |
| tn_network_outbound_request_failures_total{kind="timeout"} | rate | observer | 1 | 0.00396 | 0.0267 | 0.0416 |
| tn_network_outbound_request_failures_total{kind="unsupported"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_network_outbound_request_failures_total{kind="unsupported"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_network_inbound_requests_shed_total{class="batch",reason="admission"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_network_inbound_requests_shed_total{class="batch",reason="admission"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_network_inbound_requests_shed_total{class="batch",reason="queue_budget"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_network_inbound_requests_shed_total{class="batch",reason="queue_budget"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_network_inbound_requests_shed_total{class="batch",reason="queue_full"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_network_inbound_requests_shed_total{class="batch",reason="queue_full"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_network_inbound_requests_shed_total{class="batch",reason="serve_budget"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_network_inbound_requests_shed_total{class="batch",reason="serve_budget"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_network_inbound_requests_shed_total{class="batch",reason="unsubscribed"} | rate | validator | 10 | 0.0153 | 30.1 | 39 |
| tn_network_inbound_requests_shed_total{class="batch",reason="unsubscribed"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_network_inbound_requests_shed_total{class="certificate_sync",reason="admission"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_network_inbound_requests_shed_total{class="certificate_sync",reason="admission"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_network_inbound_requests_shed_total{class="certificate_sync",reason="queue_budget"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_network_inbound_requests_shed_total{class="certificate_sync",reason="queue_budget"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_network_inbound_requests_shed_total{class="certificate_sync",reason="queue_full"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_network_inbound_requests_shed_total{class="certificate_sync",reason="queue_full"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_network_inbound_requests_shed_total{class="certificate_sync",reason="serve_budget"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_network_inbound_requests_shed_total{class="certificate_sync",reason="serve_budget"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_network_inbound_requests_shed_total{class="certificate_sync",reason="unsubscribed"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_network_inbound_requests_shed_total{class="certificate_sync",reason="unsubscribed"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_network_inbound_requests_shed_total{class="epoch_record",reason="admission"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_network_inbound_requests_shed_total{class="epoch_record",reason="admission"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_network_inbound_requests_shed_total{class="epoch_record",reason="queue_budget"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_network_inbound_requests_shed_total{class="epoch_record",reason="queue_budget"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_network_inbound_requests_shed_total{class="epoch_record",reason="queue_full"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_network_inbound_requests_shed_total{class="epoch_record",reason="queue_full"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_network_inbound_requests_shed_total{class="epoch_record",reason="serve_budget"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_network_inbound_requests_shed_total{class="epoch_record",reason="serve_budget"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_network_inbound_requests_shed_total{class="epoch_record",reason="unsubscribed"} | rate | validator | 10 | 0.239 | 0.996 | 1.21 |
| tn_network_inbound_requests_shed_total{class="epoch_record",reason="unsubscribed"} | rate | observer | 1 | 0.0183 | 0.0639 | 0.0833 |
| tn_network_inbound_requests_shed_total{class="gossip",reason="admission"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_network_inbound_requests_shed_total{class="gossip",reason="admission"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_network_inbound_requests_shed_total{class="gossip",reason="queue_budget"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_network_inbound_requests_shed_total{class="gossip",reason="queue_budget"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_network_inbound_requests_shed_total{class="gossip",reason="queue_full"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_network_inbound_requests_shed_total{class="gossip",reason="queue_full"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_network_inbound_requests_shed_total{class="gossip",reason="serve_budget"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_network_inbound_requests_shed_total{class="gossip",reason="serve_budget"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_network_inbound_requests_shed_total{class="gossip",reason="unsubscribed"} | rate | validator | 10 | 0 | 0.189 | 0.245 |
| tn_network_inbound_requests_shed_total{class="gossip",reason="unsubscribed"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_network_inbound_requests_shed_total{class="other",reason="admission"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_network_inbound_requests_shed_total{class="other",reason="admission"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_network_inbound_requests_shed_total{class="other",reason="queue_budget"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_network_inbound_requests_shed_total{class="other",reason="queue_budget"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_network_inbound_requests_shed_total{class="other",reason="queue_full"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_network_inbound_requests_shed_total{class="other",reason="queue_full"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_network_inbound_requests_shed_total{class="other",reason="serve_budget"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_network_inbound_requests_shed_total{class="other",reason="serve_budget"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_network_inbound_requests_shed_total{class="other",reason="unsubscribed"} | rate | validator | 10 | 0.0216 | 0.117 | 0.141 |
| tn_network_inbound_requests_shed_total{class="other",reason="unsubscribed"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_network_inbound_requests_shed_total{class="vote",reason="admission"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_network_inbound_requests_shed_total{class="vote",reason="admission"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_network_inbound_requests_shed_total{class="vote",reason="queue_budget"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_network_inbound_requests_shed_total{class="vote",reason="queue_budget"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_network_inbound_requests_shed_total{class="vote",reason="queue_full"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_network_inbound_requests_shed_total{class="vote",reason="queue_full"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_network_inbound_requests_shed_total{class="vote",reason="serve_budget"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_network_inbound_requests_shed_total{class="vote",reason="serve_budget"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_network_inbound_requests_shed_total{class="vote",reason="unsubscribed"} | rate | validator | 10 | 0.237 | 19.4 | 26.3 |
| tn_network_inbound_requests_shed_total{class="vote",reason="unsubscribed"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_network_inbound_request_service_seconds{class="batch"} | p95 | validator | 4 | 51 ms | 80 ms | 102 ms |
| tn_network_inbound_request_service_seconds{class="epoch_record"} | p95 | validator | 4 | 32 ms | 1,295 ms | 1,750 ms |
| tn_network_inbound_request_service_seconds{class="epoch_record"} | p95 | observer | 1 | 34 ms | 180 ms | 195 ms |
| tn_network_inbound_request_service_seconds{class="vote"} | p95 | validator | 6 | 3,401 ms | 9,710 ms | 9,718 ms |
| tn_network_established_connections | max | validator | 10 | 19.4 | 21 | 21 |
| tn_network_established_connections | max | observer | 1 | 18.8 | 20 | 20 |
| tn_worker_sync_streams_shed_total{reason="budget_exhausted"} | rate | validator | 9 | 0.00068 | 0.0946 | 0.0946 |
| tn_worker_sync_streams_shed_total{reason="budget_exhausted"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_worker_sync_streams_shed_total{reason="denied"} | rate | validator | 10 | 0.284 | 8.59 | 11.4 |
| tn_worker_sync_streams_shed_total{reason="denied"} | rate | observer | 1 | 0.177 | 4.04 | 4.04 |
| tn_worker_batch_fetch_duration_seconds | p95 | validator | 10 | 6 ms | 3,262 ms | 4,583 ms |
| tn_executor_batch_fetch_failures_total | rate | validator | 10 | 0 | 0 | 0 |
| tn_executor_batch_fetch_failures_total | rate | observer | 1 | 0 | 0 | 0 |
| tn_primary_votes_received_total | rate | validator | 10 | 5.82 | 9.47 | 10.4 |
| tn_primary_votes_received_total | rate | observer | 1 | 0 | 0 | 0 |
| tn_primary_vote_request_failures_total | rate | validator | 10 | 0.0427 | 2.38 | 3.03 |
| tn_primary_vote_request_failures_total | rate | observer | 1 | 0 | 0 | 0 |
| tn_primary_certificates_formed_total | rate | validator | 10 | 0.969 | 1.58 | 1.73 |
| tn_primary_certificates_formed_total | rate | observer | 1 | 0 | 0 | 0 |
| tn_primary_certificate_form_duration_seconds | p95 | validator | 10 | 1,203 ms | 9,750 ms | 9,750 ms |
| tn_engine_outputs_executed_total | rate | validator | 10 | 0.474 | 0.767 | 0.822 |
| tn_engine_outputs_executed_total | rate | observer | 1 | 0.472 | 0.772 | 0.778 |
| tn_engine_persist_failures_total | rate | validator | 10 | 0 | 0 | 0 |
| tn_engine_persist_failures_total | rate | observer | 1 | 0 | 0 | 0 |
| tn_node_consensus_sync_distance | max | validator | 10 | 4.86 | 181 | 183 |
| tn_node_consensus_sync_distance | max | observer | 1 | 2.67 | 8.5 | 11 |
| reth_db_table_size{table="AccountChangeSets"} | max | validator | 10 | 280 MiB | 418 MiB | 419 MiB |
| reth_db_table_size{table="AccountChangeSets"} | max | observer | 1 | 281 MiB | 417 MiB | 419 MiB |
| reth_db_table_size{table="AccountsHistory"} | max | validator | 10 | 30 MiB | 48 MiB | 48 MiB |
| reth_db_table_size{table="AccountsHistory"} | max | observer | 1 | 30 MiB | 48 MiB | 48 MiB |
| reth_db_table_size{table="HashedAccounts"} | max | validator | 10 | 19 MiB | 31 MiB | 31 MiB |
| reth_db_table_size{table="HashedAccounts"} | max | observer | 1 | 19 MiB | 31 MiB | 31 MiB |
| reth_db_table_size{table="HashedStorages"} | max | validator | 10 | 42 MiB | 46 MiB | 46 MiB |
| reth_db_table_size{table="HashedStorages"} | max | observer | 1 | 42 MiB | 46 MiB | 46 MiB |
| reth_db_table_size{table="PlainAccountState"} | max | validator | 10 | 16 MiB | 26 MiB | 26 MiB |
| reth_db_table_size{table="PlainAccountState"} | max | observer | 1 | 16 MiB | 26 MiB | 26 MiB |
| reth_db_table_size{table="PlainStorageState"} | max | validator | 10 | 41 MiB | 46 MiB | 46 MiB |
| reth_db_table_size{table="PlainStorageState"} | max | observer | 1 | 42 MiB | 46 MiB | 46 MiB |
| reth_db_table_size{table="StorageChangeSets"} | max | validator | 10 | 341 MiB | 539 MiB | 541 MiB |
| reth_db_table_size{table="StorageChangeSets"} | max | observer | 1 | 341 MiB | 538 MiB | 541 MiB |
| reth_db_table_size{table="StoragesHistory"} | max | validator | 10 | 92 MiB | 109 MiB | 109 MiB |
| reth_db_table_size{table="StoragesHistory"} | max | observer | 1 | 92 MiB | 109 MiB | 109 MiB |
| reth_db_table_size{table="TransactionHashNumbers"} | max | validator | 10 | 646 MiB | 937 MiB | 939 MiB |
| reth_db_table_size{table="TransactionHashNumbers"} | max | observer | 1 | 646 MiB | 936 MiB | 939 MiB |
| reth_db_table_size{table="TransactionSenders"} | max | validator | 10 | 332 MiB | 482 MiB | 483 MiB |
| reth_db_table_size{table="TransactionSenders"} | max | observer | 1 | 333 MiB | 481 MiB | 483 MiB |
| reth_db_freelist | max | validator | 10 | 38.8K | 77.2K | 80.7K |
| reth_db_freelist | max | observer | 1 | 38.9K | 76.3K | 79.9K |
| reth_database_transaction_commit_write_duration_seconds | p95 | validator | 10 | 5 ms | 5 ms | 5 ms |
| reth_database_transaction_commit_write_duration_seconds | p95 | observer | 1 | 5 ms | 5 ms | 5 ms |
| reth_database_transaction_commit_sync_duration_seconds | p95 | validator | 10 | 2,333 ms | 2,425 ms | 2,425 ms |
| reth_database_transaction_commit_sync_duration_seconds | p95 | observer | 1 | 2,333 ms | 2,425 ms | 2,425 ms |
| reth_database_transaction_commit_whole_duration_seconds | p95 | validator | 10 | 2,333 ms | 2,425 ms | 2,425 ms |
| reth_database_transaction_commit_whole_duration_seconds | p95 | observer | 1 | 2,333 ms | 2,425 ms | 2,425 ms |
| tn_engine_recover_seconds | p95 | validator | 10 | 258 ms | 2,324 ms | 2,369 ms |
| tn_engine_recover_seconds | p95 | observer | 1 | 245 ms | 634 ms | 700 ms |
| tn_engine_evm_seconds | p95 | validator | 10 | 510 ms | 1,432 ms | 1,825 ms |
| tn_engine_evm_seconds | p95 | observer | 1 | 479 ms | 1,347 ms | 1,788 ms |
| tn_engine_root_seconds | p95 | validator | 10 | 945 ms | 2,948 ms | 3,750 ms |
| tn_engine_root_seconds | p95 | observer | 1 | 841 ms | 2,188 ms | 2,350 ms |
| tn_engine_finish_seconds | p95 | validator | 10 | 288 ms | 1,025 ms | 1,300 ms |
| tn_engine_finish_seconds | p95 | observer | 1 | 207 ms | 553 ms | 762 ms |
| tn_engine_forward_wait_seconds | p95 | validator | 10 | 278 ms | 9,077 ms | 9,571 ms |
| tn_engine_forward_wait_seconds | p95 | observer | 1 | 113 ms | 3,703 ms | 4,708 ms |
| tn_engine_channel_depth | max | validator | 10 | 0.0125 | 1 | 4 |
| tn_engine_channel_depth | max | observer | 1 | 0.00625 | 0.25 | 1 |
| tn_engine_executed_transactions_total | rate | validator | 10 | 3.49K | 5.45K | 5.96K |
| tn_engine_executed_transactions_total | rate | observer | 1 | 3.47K | 4.41K | 4.91K |
| tn_executor_execution_wait_seconds | p95 | validator | 10 | 5 ms | 30.0 s | 30.0 s |
| tn_primary_commit_run_ahead_blocked_total | rate | validator | 10 | 0 | 0 | 0 |
| tn_primary_commit_run_ahead_blocked_total | rate | observer | 1 | 0 | 0 | 0 |
| tn_primary_consensus_exec_lag_rounds | max | validator | 10 | 10.4 | 58.5 | 62 |
| tn_primary_consensus_exec_lag_rounds | max | observer | 1 | 5.34 | 19 | 24 |
| tn_primary_commit_lag_rounds | max | validator | 10 | 2.13 | 4.75 | 7 |
| tn_primary_commit_lag_rounds | max | observer | 1 | 0 | 0 | 0 |
| tn_primary_rejoin_grace_holds_total | rate | validator | 10 | 0 | 0 | 0 |
| tn_primary_rejoin_grace_holds_total | rate | observer | 1 | 0 | 0 | 0 |
| tn_primary_votes_refused_inactive_total | rate | validator | 10 | 0 | 27 | 27.8 |
| tn_primary_votes_refused_inactive_total | rate | observer | 1 | 0 | 0 | 0 |
| tn_primary_votes_refused_execution_behind_total | rate | validator | 10 | 1.05 | 7.99 | 8 |
| tn_primary_votes_refused_execution_behind_total | rate | observer | 1 | 0 | 0 | 0 |
| tn_batch_builder_pool_age_seconds | p95 | validator | 10 | 13.3 s | 30.0 s | 30.0 s |
| tn_batch_builder_builds_total{reason="bytes_full"} | rate | validator | 10 | 0 | 0 | 0 |
| tn_batch_builder_builds_total{reason="bytes_full"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_batch_builder_builds_total{reason="drained"} | rate | validator | 10 | 0.615 | 1.52 | 1.56 |
| tn_batch_builder_builds_total{reason="drained"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_batch_builder_builds_total{reason="gas_full"} | rate | validator | 10 | 0.806 | 2.87 | 3.31 |
| tn_batch_builder_builds_total{reason="gas_full"} | rate | observer | 1 | 0 | 0 | 0 |
| tn_worker_batch_report_seconds | p95 | validator | 10 | 0 ms | 2 ms | 2 ms |
| tn_primary_digest_wait_seconds | p95 | validator | 10 | 2,563 ms | 24.5 s | 30.0 s |
| tn_primary_headers_proposed_total{reason="max_delay"} | rate | validator | 10 | 0.0324 | 0.144 | 0.178 |
| tn_primary_headers_proposed_total{reason="min_delay"} | rate | validator | 10 | 0.849 | 1.54 | 1.62 |
| tn_primary_headers_proposed_total{reason="parents_and_threshold"} | rate | validator | 10 | 0.113 | 1.24 | 1.29 |
| tn_primary_digests_requeued_total{cause="canceled"} | rate | validator | 10 | 0.0529 | 12.9 | 13.3 |
| tn_primary_digests_requeued_total{cause="committed"} | rate | validator | 10 | 0.0296 | 0.356 | 0.422 |
| tn_executor_consensus_pack_persist_seconds | p95 | validator | 10 | 229 ms | 445 ms | 462 ms |
| tn_batch_builder_seal_duration_seconds | p95 | validator | 10 | 274 ms | 788 ms | 788 ms |
| tn_worker_quorum_wait_seconds | p95 | validator | 10 | 270 ms | 788 ms | 788 ms |
| tn_primary_proposer_pending_digests | max | validator | 10 | 1.39 | 77.8 | 83 |
| tn_primary_proposer_pending_digests | max | observer | 1 | 0 | 0 | 0 |
| reth_transaction_pool_pending_pool_transactions | max | validator | 10 | 386 | 8.94K | 9.74K |
| reth_transaction_pool_pending_pool_transactions | max | observer | 1 | 0 | 0 | 0 |
| reth_transaction_pool_queued_pool_transactions | max | validator | 10 | 1.95K | 9.32K | 9.32K |
| reth_transaction_pool_queued_pool_transactions | max | observer | 1 | 0 | 0 | 0 |
| reth_transaction_pool_all_transactions_by_hash | max | validator | 10 | 4.06K | 10K | 13.9K |
| reth_transaction_pool_all_transactions_by_hash | max | observer | 1 | 0 | 0 | 0 |
| reth_storage_providers_database_save_blocks_total | p95 | validator | 10 | 277 ms | 1,901 ms | 2,473 ms |
| reth_storage_providers_database_save_blocks_total | p95 | observer | 1 | 217 ms | 373 ms | 373 ms |
| reth_storage_providers_database_save_blocks_mdbx | p95 | validator | 10 | 277 ms | 1,901 ms | 2,473 ms |
| reth_storage_providers_database_save_blocks_mdbx | p95 | observer | 1 | 217 ms | 373 ms | 373 ms |
| reth_storage_providers_database_save_blocks_sf | p95 | validator | 10 | 46 ms | 369 ms | 553 ms |
| reth_storage_providers_database_save_blocks_sf | p95 | observer | 1 | 58 ms | 142 ms | 186 ms |
| reth_storage_providers_database_save_blocks_commit_mdbx | p95 | validator | 10 | 1,419 ms | 1,889 ms | 1,907 ms |
| reth_storage_providers_database_save_blocks_commit_mdbx | p95 | observer | 1 | 1,513 ms | 1,771 ms | 1,803 ms |
| reth_storage_providers_database_save_blocks_commit_sf | p95 | validator | 10 | 218 ms | 1,102 ms | 1,121 ms |
| reth_storage_providers_database_save_blocks_commit_sf | p95 | observer | 1 | 109 ms | 271 ms | 277 ms |
| reth_storage_providers_database_save_blocks_write_state | p95 | validator | 10 | 54 ms | 373 ms | 501 ms |
| reth_storage_providers_database_save_blocks_write_state | p95 | observer | 1 | 43 ms | 74 ms | 85 ms |
| reth_storage_providers_database_save_blocks_write_hashed_state | p95 | validator | 10 | 33 ms | 391 ms | 483 ms |
| reth_storage_providers_database_save_blocks_write_hashed_state | p95 | observer | 1 | 26 ms | 37 ms | 45 ms |
| reth_storage_providers_database_save_blocks_write_trie_updates | p95 | validator | 10 | 40 ms | 198 ms | 259 ms |
| reth_storage_providers_database_save_blocks_write_trie_updates | p95 | observer | 1 | 31 ms | 45 ms | 48 ms |
| reth_database_transaction_commit_preparation_duration_seconds | p95 | validator | 10 | 5 ms | 22 ms | 29 ms |
| reth_database_transaction_commit_preparation_duration_seconds | p95 | observer | 1 | 5 ms | 5 ms | 5 ms |
| reth_database_transaction_commit_gc_wallclock_duration_seconds | p95 | validator | 10 | 5 ms | 49 ms | 70 ms |
| reth_database_transaction_commit_gc_wallclock_duration_seconds | p95 | observer | 1 | 5 ms | 5 ms | 5 ms |
| reth_database_transaction_commit_gc_cputime_duration_seconds | p95 | validator | 10 | 5 ms | 5 ms | 5 ms |
| reth_database_transaction_commit_gc_cputime_duration_seconds | p95 | observer | 1 | 5 ms | 5 ms | 5 ms |
| reth_database_transaction_commit_audit_duration_seconds | p95 | validator | 10 | 5 ms | 5 ms | 5 ms |
| reth_database_transaction_commit_audit_duration_seconds | p95 | observer | 1 | 5 ms | 5 ms | 5 ms |
| reth_database_transaction_commit_ending_duration_seconds | p95 | validator | 10 | 5 ms | 5 ms | 5 ms |
| reth_database_transaction_commit_ending_duration_seconds | p95 | observer | 1 | 5 ms | 5 ms | 5 ms |
tn_primary_commit_latency_seconds
tn_primary_round − tn_primary_committed_round
tn_engine_queued_outputs, max by node
The share of each validator's own headers that formed a certificate, per window (tools/cert-rate.py on the node logs). bench-validator-04, bench-validator-08 and bench-validator-10 certified fewer than 90 % of their own headers in a window where they proposed at least 50. Own leader % is the validator's share of the committed leaders and not committed % the share of its certificates no committed leader reached, both over the run window; the fair share is 100 / 10 validators = 10.0 %. Greyed: fewer than 50 headers proposed, not judged.
| Validator | Region | phase preamble | phase warmup | phase mixed | phase cooldown | run | Own leader % | Not committed % |
|---|---|---|---|---|---|---|---|---|
| bench-validator-01 | us-west2 (Los Angeles) | 100.5 | 99.5 | 99.8 | 100.0 | 99.9 | 12.6 | 0.2 |
| bench-validator-02 | europe-west2 (London) | 98.4 | 99.5 | 98.3 | 99.8 | 98.6 | 10.8 | 8.5 |
| bench-validator-03 | australia-southeast1 (Sydney) | 100.0 | 100.0 | 99.7 | 99.8 | 99.8 | 13.2 | 0.7 |
| bench-validator-04 below 90 % | asia-northeast1 (Tokyo) | 99.5 | 53.0 | 49.6 | 99.8 | 60.2 | 6.7 | 2.7 |
| bench-validator-05 | us-east4 (Ashburn) | 100.5 | 99.5 | 100.1 | 100.0 | 100.1 | 12.2 | 8.6 |
| bench-validator-06 | europe-west4 (Netherlands) | 99.5 | 100.0 | 98.9 | 99.8 | 99.1 | 10.4 | 9.4 |
| bench-validator-07 | asia-southeast1 (Singapore) | 100.5 | 99.5 | 97.7 | 99.8 | 98.3 | 13.0 | 2.0 |
| bench-validator-08 below 90 % | asia-east2 (Hong Kong) | 100.0 | 54.0 | 44.4 | 99.8 | 56.4 | 6.1 | 2.6 |
| bench-validator-09 | asia-northeast3 (Seoul) | 100.0 | 100.0 | 90.2 | 99.8 | 92.8 | 12.1 | 2.0 |
| bench-validator-10 below 90 % | asia-south1 (Mumbai) | 97.9 | 97.5 | 67.9 | 99.6 | 76.2 | 8.9 | 7.0 |
| Node | Class | Peak RssAnon | Peak RssFile | Peak cache | PSI mem | PSI io | PSI cpu | db growth | static_files growth | Root used | OOM Δ |
|---|---|---|---|---|---|---|---|---|---|---|---|
| bench-observer-11 | observer | 1011 MiB | 2.93 GiB | – | 0.0 % | 32.8 % | 0.4 % | 0 B | 2.90 GiB | 14.0 % | 0 |
| bench-validator-01 | validator | 4.37 GiB | 3.18 GiB | – | 0.0 % | 30.6 % | 1.7 % | 0 B | 2.90 GiB | 16.0 % | 0 |
| bench-validator-02 | validator | 4.93 GiB | 3.19 GiB | – | 0.0 % | 31.3 % | 2.2 % | 0 B | 2.90 GiB | 16.0 % | 0 |
| bench-validator-03 | validator | 4.22 GiB | 3.21 GiB | – | 0.0 % | 31.4 % | 2.0 % | 0 B | 2.90 GiB | 16.0 % | 0 |
| bench-validator-04 | validator | 3.97 GiB | 3.15 GiB | – | 0.0 % | 32.1 % | 2.7 % | 0 B | 2.90 GiB | 16.0 % | 0 |
| bench-validator-05 | validator | 3.37 GiB | 3.28 GiB | – | 0.0 % | 29.4 % | 1.1 % | 0 B | 2.90 GiB | 16.0 % | 0 |
| bench-validator-06 | validator | 4.44 GiB | 3.15 GiB | – | 0.0 % | 31.6 % | 2.0 % | 0 B | 2.90 GiB | 16.0 % | 0 |
| bench-validator-07 | validator | 4.05 GiB | 3.18 GiB | – | 0.0 % | 30.9 % | 2.0 % | 0 B | 2.90 GiB | 16.0 % | 0 |
| bench-validator-08 | validator | 4.25 GiB | 3.21 GiB | – | 0.0 % | 34.2 % | 1.5 % | 0 B | 2.90 GiB | 16.0 % | 0 |
| bench-validator-09 | validator | 4.19 GiB | 3.20 GiB | – | 0.0 % | 31.6 % | 2.0 % | 0 B | 2.90 GiB | 16.0 % | 0 |
| bench-validator-10 | validator | 4.76 GiB | 3.10 GiB | – | 0.0 % | 32.4 % | 2.0 % | 0 B | 2.90 GiB | 16.0 % | 0 |
/proc/<pid>/status of the telcoin process on the node with the largest RSS
Not collected: consensus-db/cache size.
/proc/pressure on the node with the highest memory PSI
Per role and per node, from sizing.json (p95 over the landed window; PSI and RssAnon max over the whole sampler run). The lines follow the hardware doc's tier rules: CPU p95 over 70 % of vCPUs; memory used p95 over 75 % of RAM, or telcoin RSS p95 over 85 % of RAM; disk write p95 over 60 % of the PD MiB/s limit, or write IOPS p95 over 60 % of the IOPS limit; IO pressure over 20 % or memory pressure over 5 % (PSI some avg10, max); network p95 over 50 % of 1 Gbps; root disk over 80 %. A line crossed on any node puts the role at the limit; a nonzero reading within 15 points of its line is tight; otherwise the role has headroom. OOM kills, disk throttling and degraded disk performance are flagged too.
| Signal | Class | Worst node | Value | Share of capacity | Line | Nodes over / tight | Status |
|---|---|---|---|---|---|---|---|
| CPU p95 | validator | bench-validator-08 | 22.2 % | 22.2 % | > 70 % | 0 / 0 | headroom |
| Memory used p95 | validator | bench-validator-02 | 34.2 % | 34.2 % | > 75 % | 0 / 0 | headroom |
| telcoin RSS p95 | validator | bench-validator-02 | 8.11 GB | 47.2 % | > 85 % | 0 / 0 | headroom |
| PSI memory max | validator | bench-validator-01 | 0.0 % | 0.0 % | > 5 % | 0 / 0 | headroom |
| RssAnon peak | validator | bench-validator-02 | 5.29 GB | 30.8 % | info | – | info |
| Disk write p95 | validator | bench-validator-10 | 595 MB/s | 70.9 % | > 60 % | 10 / 0 | at the limit |
| Write IOPS p95 | validator | bench-validator-05 | 8,909 | 44.5 % | > 60 % | 0 / 0 | headroom |
| PSI io max | validator | bench-validator-08 | 34.2 % | 34.2 % | > 20 % | 10 / 0 | at the limit |
| Root disk max | validator | bench-validator-07 | 15.6 % | 15.6 % | > 80 % | 0 / 0 | headroom |
| Network p95 | validator | bench-validator-08 | 82 Mbps | 8.2 % | > 50 % | 0 / 0 | headroom |
| Consensus cache peak | validator | – | – | – | info | – | not collected |
| OOM kills | validator | – | – | any | any | – | none |
| Disk throttling | validator | – | – | metric type unavailable in project | any | – | not collected |
| Disk performance degraded | validator | – | – | any sample | any | – | none |
| CPU p95 | observer | bench-observer-11 | 13.2 % | 13.2 % | > 70 % | 0 / 0 | headroom |
| Memory used p95 | observer | bench-observer-11 | 11.7 % | 11.7 % | > 75 % | 0 / 0 | headroom |
| telcoin RSS p95 | observer | bench-observer-11 | 3.79 GB | 22.1 % | > 85 % | 0 / 0 | headroom |
| PSI memory max | observer | bench-observer-11 | 0.0 % | 0.0 % | > 5 % | 0 / 0 | headroom |
| RssAnon peak | observer | bench-observer-11 | 1.07 GB | 6.2 % | info | – | info |
| Disk write p95 | observer | bench-observer-11 | 588 MB/s | 70.1 % | > 60 % | 1 / 0 | at the limit |
| Write IOPS p95 | observer | bench-observer-11 | 5,206 | 26.0 % | > 60 % | 0 / 0 | headroom |
| PSI io max | observer | bench-observer-11 | 32.8 % | 32.8 % | > 20 % | 1 / 0 | at the limit |
| Root disk max | observer | bench-observer-11 | 13.2 % | 13.2 % | > 80 % | 0 / 0 | headroom |
| Network p95 | observer | bench-observer-11 | 4 Mbps | 0.4 % | > 50 % | 0 / 0 | headroom |
| Consensus cache peak | observer | – | – | – | info | – | not collected |
| OOM kills | observer | – | – | any | any | – | none |
| Disk throttling | observer | – | – | metric type unavailable in project | any | – | not collected |
| Disk performance degraded | observer | – | – | any sample | any | – | none |
| Node | Class | CPU p95 | Memory used p95 | telcoin RSS p95 | PSI memory max | RssAnon peak | Disk write p95 | Write IOPS p95 | PSI io max | Root disk max | Network p95 | Consensus cache peak |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Line | > 70 % | > 75 % | > 85 % | > 5 % | info | > 60 % | > 60 % | > 20 % | > 80 % | > 50 % | info | |
| bench-validator-01 | validator | 20.7 % | 31.4 % | 7.49 GB 44 % | 0.0 % | 4.69 GB 27 % | 576 MB/s 69 % | 5,057 25 % | 30.6 % | 15.6 % | 71 Mbps 7 % | – |
| bench-validator-02 | validator | 20.5 % | 34.2 % | 8.11 GB 47 % | 0.0 % | 5.29 GB 31 % | 593 MB/s 71 % | 4,137 21 % | 31.3 % | 15.6 % | 71 Mbps 7 % | – |
| bench-validator-03 | validator | 22.2 % | 30.1 % | 7.47 GB 43 % | 0.0 % | 4.53 GB 26 % | 580 MB/s 69 % | 5,323 27 % | 31.4 % | 15.5 % | 73 Mbps 7 % | – |
| bench-validator-04 | validator | 20.9 % | 29.0 % | 7.30 GB 42 % | 0.0 % | 4.26 GB 25 % | 591 MB/s 70 % | 2,683 13 % | 32.1 % | 15.5 % | 74 Mbps 7 % | – |
| bench-validator-05 | validator | 21.2 % | 27.0 % | 5.88 GB 34 % | 0.0 % | 3.62 GB 21 % | 547 MB/s 65 % | 8,909 45 % | 29.4 % | 15.1 % | 78 Mbps 8 % | – |
| bench-validator-06 | validator | 20.9 % | 32.0 % | 7.78 GB 45 % | 0.0 % | 4.77 GB 28 % | 590 MB/s 70 % | 2,866 14 % | 31.6 % | 15.5 % | 70 Mbps 7 % | – |
| bench-validator-07 | validator | 20.5 % | 30.0 % | 7.40 GB 43 % | 0.0 % | 4.35 GB 25 % | 588 MB/s 70 % | 4,885 24 % | 30.9 % | 15.6 % | 72 Mbps 7 % | – |
| bench-validator-08 | validator | 22.2 % | 31.9 % | 7.75 GB 45 % | 0.0 % | 4.56 GB 27 % | 591 MB/s 70 % | 5,897 29 % | 34.2 % | 15.6 % | 82 Mbps 8 % | – |
| bench-validator-09 | validator | 20.8 % | 30.7 % | 7.53 GB 44 % | 0.0 % | 4.50 GB 26 % | 593 MB/s 71 % | 4,838 24 % | 33.4 % | 15.6 % | 73 Mbps 7 % | – |
| bench-validator-10 | validator | 19.9 % | 33.3 % | 7.87 GB 46 % | 0.0 % | 5.11 GB 30 % | 595 MB/s 71 % | 2,829 14 % | 32.4 % | 15.5 % | 68 Mbps 7 % | – |
| bench-observer-11 | observer | 13.2 % | 11.7 % | 3.79 GB 22 % | 0.0 % | 1.07 GB 6 % | 588 MB/s 70 % | 5,206 26 % | 32.8 % | 13.2 % | 4 Mbps 0 % | – |
Small figures are the share of capacity: RAM for RSS, the PD limit for disk write (VM cap applied), 1 Gbps for network. Hover a cell for its JSON path.
A = Fleet c3-hdb · c3-highcpu-8 · r20261107-1504
B = Fleet c3-hdb · c3-highcpu-8 · r20261105-0803
at the limit c3-highcpu-8 validators at 4,939 sustained TPS: at the limit on disk. Disk at the limit: write p95 547–595 MB/s, up to 70.9 % of the 800 MiB/s limit, over the 60 % line on all 10 validators (bench-validator-01, 02, 03, 04, 05, 06, 07, 08, 09, 10); IO pressure (PSI some avg10) max 29.4–34.2 %, over the 20 % line on all 10 validators (bench-validator-01, 02, 03, 04, 05, 06, 07, 08, 09, 10). CPU has headroom: CPU p95 22.2 % of 8 vCPUs (bench-validator-08). Memory has headroom: memory used p95 34.2 % of RAM (bench-validator-02); telcoin RSS p95 8.11 GB, 47.2 % of 16 GiB RAM (bench-validator-02); memory pressure (PSI some avg10) max 0.0 % (bench-validator-01). Network has headroom: network p95 82 Mbps, 8.2 % of 1 Gbps (bench-validator-08). Next step: a faster volume (pd-ssd adds 30 IOPS and 0.48 MiB/s per GB up to the per-VM cap; past the cap, a larger VM).
at the limit c3-highcpu-8 observers at 4,939 sustained TPS: at the limit on disk. Disk at the limit: write p95 588 MB/s, 70.1 % of the 800 MiB/s limit on bench-observer-11, over the 60 % line; IO pressure (PSI some avg10) max 32.8 % on bench-observer-11, over the 20 % line. CPU has headroom: CPU p95 13.2 % of 8 vCPUs (bench-observer-11). Memory has headroom: memory used p95 11.7 % of RAM (bench-observer-11); telcoin RSS p95 3.79 GB, 22.1 % of 16 GiB RAM (bench-observer-11); memory pressure (PSI some avg10) max 0.0 % (bench-observer-11). Network has headroom: network p95 4 Mbps, 0.4 % of 1 Gbps (bench-observer-11). Next step: a faster volume (pd-ssd adds 30 IOPS and 0.48 MiB/s per GB up to the per-VM cap; past the cap, a larger VM).
tight c3-highcpu-8 validators at 5,525 sustained TPS: tight on memory and disk. Memory tight: memory pressure (PSI some avg10) max 0.1 % (bench-validator-08), tight against the 5 % line. Disk tight: write IOPS p95 9,882, 49.4 % of the 20,000 IOPS limit (bench-validator-10), tight against the 60 % line; IO pressure (PSI some avg10) max 17.4 % (bench-validator-08), tight against the 20 % line. CPU has headroom: CPU p95 36.7 % of 8 vCPUs (bench-validator-09). Network has headroom: network p95 103 Mbps, 10.3 % of 1 Gbps (bench-validator-09).
tight c3-highcpu-8 observers at 5,525 sustained TPS: tight on disk. Disk tight: IO pressure (PSI some avg10) max 12.6 % (bench-observer-11), tight against the 20 % line. CPU has headroom: CPU p95 27.4 % of 8 vCPUs (bench-observer-11). Memory has headroom: memory used p95 13.5 % of RAM (bench-observer-11); telcoin RSS p95 3.45 GB, 20.1 % of 16 GiB RAM (bench-observer-11); memory pressure (PSI some avg10) max 0.0 % (bench-observer-11). Network has headroom: network p95 13 Mbps, 1.3 % of 1 Gbps (bench-observer-11).
Validators: Fleet A (c3-highcpu-8) is at the limit on disk, with headroom on CPU, memory and network; Fleet B (c3-highcpu-8) is tight on memory and disk, with headroom on CPU and network. Disk is at the limit on Fleet A (c3-highcpu-8) only. Fleet B (c3-highcpu-8) stays under the disk lines that Fleet A (c3-highcpu-8) crosses.
Observers: Fleet A (c3-highcpu-8) is at the limit on disk, with headroom on CPU, memory and network; Fleet B (c3-highcpu-8) is tight on disk, with headroom on CPU, memory and network. Disk is at the limit on Fleet A (c3-highcpu-8) only. Fleet B (c3-highcpu-8) stays under the disk lines that Fleet A (c3-highcpu-8) crosses.
| KPI | Fleet A | Fleet B | B ÷ A |
|---|---|---|---|
| Sustained mixed TPS (own) | 4,939.2 tx/s | 5,524.8 tx/s | 1.12× |
| Sustained mixed TPS (chain) | 4,939.2 tx/s | 5,524.8 tx/s | 1.12× |
| Landed / accepted | 99.6 % | 99.9 % | 1.00× |
| Reverted (own) | 0 | 0 | – |
| Latency p50 | 2,809 ms | 3,303 ms | 1.18× |
| Latency p95 | 14,404 ms | 12,837 ms | 0.89× |
| Gas / s (own) | 174M gas/s | 231M gas/s | 1.33× |
| Capacity at mix | 7,705.4 tx/s | 9,431.4 tx/s | 1.22× |
| Signal | Class | Fleet A | Fleet B | Line |
|---|---|---|---|---|
| CPU p95 | validator | 22.2 % headroom | 36.7 % headroom | > 70 % |
| Memory used p95 | validator | 34.2 % headroom | 43.5 % headroom | > 75 % |
| telcoin RSS p95 | validator | 8.11 GB 47.2 % headroom | 9.28 GB 54.0 % headroom | > 85 % |
| PSI memory max | validator | 0.0 % headroom | 0.1 % tight | > 5 % |
| Disk write p95 | validator | 595 MB/s 70.9 % at the limit | 325 MB/s 38.7 % headroom | > 60 % |
| Write IOPS p95 | validator | 8,909 44.5 % headroom | 9,882 49.4 % tight | > 60 % |
| PSI io max | validator | 34.2 % at the limit | 17.4 % tight | > 20 % |
| Root disk max | validator | 15.6 % headroom | 63.9 % headroom | > 80 % |
| Network p95 | validator | 82 Mbps 8.2 % headroom | 103 Mbps 10.3 % headroom | > 50 % |
| CPU p95 | observer | 13.2 % headroom | 27.4 % headroom | > 70 % |
| Memory used p95 | observer | 11.7 % headroom | 13.5 % headroom | > 75 % |
| telcoin RSS p95 | observer | 3.79 GB 22.1 % headroom | 3.45 GB 20.1 % headroom | > 85 % |
| PSI memory max | observer | 0.0 % headroom | 0.0 % headroom | > 5 % |
| Disk write p95 | observer | 588 MB/s 70.1 % at the limit | 325 MB/s 38.7 % headroom | > 60 % |
| Write IOPS p95 | observer | 5,206 26.0 % headroom | 8,830 44.1 % headroom | > 60 % |
| PSI io max | observer | 32.8 % at the limit | 12.6 % tight | > 20 % |
| Root disk max | observer | 13.2 % headroom | 62.1 % headroom | > 80 % |
| Network p95 | observer | 4 Mbps 0.4 % headroom | 13 Mbps 1.3 % headroom | > 50 % |
aligned on each fleet's mixed-phase start
aligned on each fleet's mixed-phase start
aligned on each fleet's mixed-phase start
aligned on each fleet's mixed-phase start
aligned on each fleet's mixed-phase start
aligned on each fleet's mixed-phase start
Not collected: reference chains.