feat(metrics): instrument the thread pools that CPU work runs on

[vicsn]

Motivation

The node runs three parallelism frameworks that each size themselves independently and know nothing of one another. On a 64-core host that is 128 tokio worker threads (2 * num_cores), up to 512 tokio blocking threads, and 64 rayon threads (num_cores).

They are not independent in practice. Every CPU-bound operation is staged through spawn_blocking! and then fans out onto the rayon global pool, so up to 512 callers can inject into a pool only as wide as the machine, with no priorities to separate a batch signature that a peer is waiting on from a cold-cache block verification during catch-up sync.

Whether that actually convoys, and what any of the three pools should be sized to, cannot be answered from reading the code. There are no numbers. This PR adds them; it changes no pool sizes and no scheduling.

What this adds

Thread names, which everything else depends on. The rayon global pool was built without thread_name, so its threads were unnamed, and tokio applies one name to both its workers and its blocking threads. Per-thread CPU time therefore could not be attributed to a pool at all. Threads are now rayon-N, tokio-worker-N and tokio-blocking-N.

Two things to know when reading these off a running process: Linux truncates a thread name to 15 bytes, which cuts the index off tokio-blocking-, so group by prefix rather than parsing the number; and blocking threads idle out and respawn, so their indices climb over the life of the process.

snarkos_cpu_blocking_wait_secs, the gap between submitting a spawn_blocking! task and that task starting. This is where queueing shows up before work ever reaches rayon, aggregated over every call site.

snarkos_tokio_*, sampled from tokio's own counters once a second:

Metric	Answers
snarkos_tokio_worker_busy_cores	How many cores' worth of work the async runtime is actually doing — the figure that sizes worker_threads
snarkos_tokio_worker_busy_ratio_max	Whether load is spread across workers or landing on a few
snarkos_tokio_worker_parks	How much of the runtime's time goes to sleeping and waking
snarkos_tokio_global_queue_depth, snarkos_tokio_alive_tasks, snarkos_tokio_workers	Backlog and runtime shape

Tokio reports busy time and park counts as totals since startup, which say nothing on their own about load right now, so these are published as the change over each sampling interval.

snarkos_rayon_threads, the global pool's width. Worth reading rather than assuming, because num_cpus::get() honours the cgroup CFS quota, so a container's pool need not match the host's core count.

Behind RUSTFLAGS="--cfg tokio_unstable", compiled out of a normal build: snarkos_tokio_blocking_threads, snarkos_tokio_blocking_threads_idle, snarkos_tokio_blocking_queue_depth, snarkos_tokio_worker_steals, snarkos_tokio_worker_noops. Blocking pool occupancy against its 512 cap is the number that pairs with the wait histogram above, and it is only available here.

On work stealing

Worth measuring for tokio, not available for rayon.

Tokio exposes worker_steal_count and worker_noop_count, both included above. A high noop count is a worker waking, finding nothing, and parking again — pure overhead, and the direct signal that 2 * num_cores workers is more than the workload needs.

rayon-core 1.13 exposes no counters at all: no steal count, no queue depth, no thread-state API. So rayon is measured by outcome instead — its width from snarkos_rayon_threads, its saturation from the per-thread CPU time that thread naming now makes attributable, and its queueing from the blocking-pool wait that precedes it. Steal counts are a mechanism metric anyway; the decision-relevant outcomes are idle hunting and convoying, and both are visible without them.

snarkVM bump

Cargo.toml moves the snarkVM rev from e0f68fae to 708b2af912, the head of ProvableHQ/snarkVM#3416, which supplies the VM-side half of the picture:

• snarkvm_vm_check_transaction_cache_hit_total / _miss_total and snarkvm_vm_check_transaction_duration_seconds labelled by cache — whether verification work is real proof checking or a hit on the ~1M-entry partially_verified_transactions cache. This distinguishes the block-production path (warm, so not rayon-width-bound) from catch-up sync (cold, so it is).
• snarkvm_vm_check_transaction_in_flight and snarkvm_vm_check_transaction_duplicate_in_flight — concurrent verifications, which is what would convoy rayon, and how much of that is redundant.
• snarkvm_vm_speculate_*, snarkvm_vm_prepare_for_speculate_*, snarkvm_vm_atomic_speculate_* — how long the sequential, storage-bound part of block production holds things up.

That rev also carries unrelated snarkVM staging commits picked up along the way. snarkVM#3416 is itself still a draft, so this cannot merge before it does.

How to use it

One devnet, one knob per run, A/B against a baseline, reading p99 rather than the mean and discarding the warm-up. The questions to settle, in order:

1. Is worker_busy_cores far below the worker count? Then 128 workers is oversubscription, and the noop count says how much it costs.
2. Is blocking_wait_secs non-zero at p99? Then work is queueing before rayon, and the blocking pool's occupancy says whether the 512 cap or rayon's width is the constraint.
3. Do the rayon threads saturate during cold-cache sync but not during steady state? That decides whether rayon should be sized for sync or for block production, and whether the two need separating at all.

Testing

cargo clippy --workspace --all-targets --all-features -- -D warnings passes, as does the tokio_unstable build. cargo test -p snarkos-node-bft --lib passes, 123 tests.

.ci/test_devnet.sh passes every functional check — all nodes reach the minimum height, transactions confirm, the REST suite passes — then fails the validator log-size cap at 2.28 MB against a 2 MB limit. Nothing here adds a log line, and I have not checked whether that cap trips on staging too.