blockwatcher-rpc

blockwatcher-rpc is a resilient pool of named remote endpoints, chain- and transport-agnostic: the consumer supplies each operation (any async request/response call over its own connection handle) and its error classification, and the pool supplies everything around the request: endpoint selection by priority, per-endpoint circuit breakers, preemptive rate limiting, a whole-call deadline, per-attempt timeouts, retry by error class, one concurrency permit held across retries, health probing, and endpoint-labeled metrics, per the crate doc comment (crates/blockwatcher-rpc/src/lib.rs). It is a module crate: blockwatcher-evm’s evm-rpc and evm-mempool sources are consumers, for example, but nothing about its own dependencies or its public surface names EVM, HTTP, or any other chain or transport at all.

This page covers the pool’s own mechanics (selection, the breaker, the limiter, retry and backoff, health probing) and the one seam that makes “chain- and transport-agnostic” true structurally rather than just by convention. Observability § Other metrics on this endpoint already documents every metric this crate exports from an operator’s point of view; this page does not restate that table.

Key takeaways

• blockwatcher-rpc is a resilient pool of named remote endpoints, chain- and transport-agnostic: the consumer supplies the operation and its error classification, and the pool supplies selection, breaking, rate limiting, retry, and health probing.
• Nothing about its own dependencies or public surface names EVM, HTTP, or any other chain or transport; blockwatcher-evm’s sources are consumers, not a special case.
• This page covers the pool’s own mechanics and the seam that makes it chain- and transport-agnostic structurally, not the metrics it exports, which observability already documents.

Responsibilities

• Run one caller-supplied async operation against a pool of named endpoints to completion: pick an endpoint, bound the attempt with a timeout, retry by the failure’s ErrorClass, and return either a served value or a typed exhaustion reason: Pool::execute (crates/blockwatcher-rpc/src/pool.rs).
• Select an endpoint by priority tier (High before Low; within a tier, config order under the default Ordered strategy, or a weighted rotation under RoundRobin), skipping one whose breaker does not currently admit or whose rate limiter has no token available for this call: Pool::select/Pool::select_pinned (pool.rs).
• Trip a per-endpoint circuit breaker, Breaker, after a configurable run of consecutive classified failures, admit one half-open trial after a cooldown, and reset it on any success (breaker.rs).
• Enforce a preemptive per-endpoint rate limit (TokenBucket, whose burst capacity equals its configured rate) before a call ever leaves the pool (limiter.rs).
• Retry a Transient failure or an attempt timeout on a doubling backoff; retry a RateLimited failure on the same schedule without touching the breaker; return a Permanent or RetryNarrower failure immediately, with its retry budget unburned, all inside Pool::execute (pool.rs).
• Sweep every endpoint whose breaker is not strictly open, concurrently, under the pool’s attempt timeout, without ever feeding a probe’s result back into the breaker itself: Pool::probe_health (pool.rs).
• Publish blockwatcher_rpc_* metrics through six *_TOTAL constants for every attempt, failure, breaker transition, rate-limit wait, exhaustion, and probe failure, each labeled endpoint (pool.rs).

Not this crate’s job: knowing what a request means, what wire format it takes, or what chain it targets: the consumer’s own connection type C and async closure carry all of that, and Pool<C> never inspects either, per the crate doc comment (lib.rs); resolving an env:NAME secret reference into a URL: that lives in blockwatcher-types::SecretRef, called once by whichever module crate constructs a pool (see The env:/url_secret indirection below); retrying a sink delivery or deciding when a match becomes a dead letter: that is blockwatcher-core’s deliver_with_retry (crates/blockwatcher-core/src/pipeline/delivery.rs), a wholly separate retry loop for a wholly separate kind of call; caching a response: what is safe to cache and for how long is a property of the consumer’s domain, not of transport resilience, so this crate does not attempt it at all, per the crate doc comment (lib.rs).

Key types and traits

The seam: generic over a connection, not a trait

There is no trait Endpoint or trait Transport anywhere in this crate: Pool<C> is generic over a bare connection type C with no bound on C at all (impl<C> Pool<C>, pool.rs). The only contract a consumer must satisfy sits on the error type Pool::execute’s operation returns: E: Classify + std::error::Error (pool.rs), where Classify is declared in blockwatcher_ports (crates/blockwatcher-ports/src/error.rs) and merely re-exported here. That single method, fn class(&self) -> ErrorClass (one of Transient/Permanent/RateLimited/RetryNarrower, blockwatcher-ports/src/error.rs), is everything the pool needs to decide whether an attempt’s failure feeds the breaker, retries on backoff, or returns immediately.

blockwatcher-evm is the concrete instance: it parameterizes Pool<EvmEndpoint>, where EvmEndpoint { client: reqwest::Client, url: url::Url } (crates/blockwatcher-evm/src/jsonrpc.rs) is its own HTTP connection handle, and implements Classify for EvmRpcError directly on its own error enum (jsonrpc.rs), mapping HTTP status codes and JSON-RPC error bodies onto the four classes. blockwatcher-rpc never sees reqwest, url::Url, or anything JSON-RPC-shaped: it only ever calls op(&endpoint.conn) and reads back a Result<T, E> it did not construct.

flowchart LR
    subgraph consumer["blockwatcher-evm (a consumer)"]
        ep["EvmEndpoint<br/>{ client, url }"]
        err["EvmRpcError<br/>impl Classify"]
        op["closure: |ep| jsonrpc::call(ep, ...)"]
    end
    subgraph rpcpool["blockwatcher-rpc::Pool&lt;C&gt;"]
        sel["select()<br/>priority + breaker + limiter"]
        exec["execute()<br/>timeout, retry, backoff"]
        brk[("Breaker<br/>per endpoint")]
        lim[("TokenBucket<br/>per endpoint")]
    end
    ep -->|"C = EvmEndpoint"| rpcpool
    op -->|"op: Fn(&amp;C) -> Fut&lt;Result&lt;T, E&gt;&gt;"| exec
    exec --> sel
    sel --> brk
    sel --> lim
    exec -->|"E::class()"| err
    exec -->|"metrics::counter!<br/>blockwatcher_rpc_*"| metrics[("process-global<br/>metrics::Recorder")]

Selection: priority, breaker, limiter, strategy

Pool::select (pool.rs) walks Priority::High endpoints first, then Priority::Low. Within a tier, the walk follows the tier’s rotation ring — endpoint indices in configuration order, each repeated its EndpointConfig::weight — and where it starts is the PoolConfig::strategy choice: Ordered (the default) always starts at the ring’s head, so the first admissible endpoint in configuration order wins every call and config order stays meaningful as the operator’s preference order; RoundRobin starts one position later per selection, spreading consecutive calls across the tier’s admissible endpoints in proportion to their weights. Rotation is best-effort, not strict fairness: the cursor advances once per selection whether or not the endpoint it landed on was admissible, so a broken endpoint’s slot redirects to its successor. Weights are inert under Ordered. Within a tier, an excluded endpoint or one whose breaker does not admit is skipped outright; among the rest, an endpoint whose rate limiter has no token available right now is skipped for this call rather than waited on, so one endpoint’s replenish interval never stalls a call another endpoint could serve immediately. Low is consulted only when zero High endpoints are admissible: a High endpoint that is merely rate-limited still counts as admissible, because being over quota is not ill health, and spilling to Low on quota would leak load past the operator’s priority intent (pool.rs, doc comment). ExecuteOptions::pin bypasses this walk entirely and asks for one named endpoint by name; the pool never substitutes another endpoint behind a pin: a pin that cannot serve fails as PinUnavailable, naming why (pool.rs).

Circuit breaker

stateDiagram-v2
    [*] --> Closed
    Closed --> Closed: success<br/>(resets streak)
    Closed --> Open: threshold consecutive<br/>classified failures
    Open --> HalfOpen: cooldown elapses
    HalfOpen --> Closed: trial succeeds
    HalfOpen --> Open: trial fails<br/>(re-opens for a full cooldown)

Breaker (breaker.rs, crate-private) tracks consecutive_failures and opened_at. admits is true when not strictly open; is_open is true only within cooldown of the failure that tripped it. record_failure returns whether this call is what transitioned the breaker from admitting to strictly open (breaker.rs); a metric counting “breaker opened” events must use that return value, not every failure, since the same streak keeps failing past the threshold without re-tripping, and a failed half-open trial is itself a fresh open transition. Only an attempt timeout or a Transient-classified failure ever calls record_failure; RateLimited explicitly does not, since quota exhaustion is not endpoint ill health (pool.rs).

Rate limiter

TokenBucket (limiter.rs, crate-private) is a per-endpoint token bucket whose burst capacity equals its configured rps: a fresh bucket can serve rps calls immediately, then refills continuously (not in fixed windows) at that same rate. next_available reports the exact wait until one token exists, which is what lets the pool, when every candidate in a tier is rate-limited, sleep exactly the shortest wait across candidates rather than a heuristic (pool.rs).

Retry, backoff, and the deadline

execute acquires one Semaphore permit up front and holds it across every retry; the whole-call deadline bounds even the wait for that permit (pool.rs). Each attempt then runs under min(attempt_timeout, time remaining before deadline) (pool.rs). On the result:

• Permanent and RetryNarrower return immediately, retries unburned, since the pool cannot repair either by trying again (pool.rs).
• An attempt timeout or a Transient failure records a breaker failure and retries after a backoff that doubles from backoff_initial up to backoff_max (pool.rs).
• A RateLimited failure retries on the same doubling schedule but never touches the breaker (pool.rs).

This is retry over one RPC call, distinct from blockwatcher-core’s deliver_with_retry for sink deliveries (Delivery guarantees), and the two retry loops share no code and answer different questions: this one asks “should the pool try another attempt against an endpoint,” that one asks “should the engine try another delivery to a sink, or dead-letter the match.”

Health probing

Pool::probe_health (pool.rs) runs a caller-supplied probe concurrently against every endpoint whose breaker is not strictly open: an open endpoint is omitted from the result entirely (“recovering, no data”), not represented as a failure. Results never feed the breakers back: passive health comes from real traffic through execute, and probing is observation layered on top of it, never a second input into it (pool.rs). A sweep bypasses both the concurrency permit and every endpoint’s rate limiter on purpose, so liveness probing is never starved by the same quota it exists to observe (pool.rs).

The env:/url_secret indirection, and where it actually lives

Nothing in this crate parses an env: reference or a url_secret field: EndpointConfig carries only a name, a Priority, and an optional RateLimit (pool.rs); it has no URL field at all. The indirection an operator writes as "url_secret": "env:SEPOLIA_RPC_URL" resolves in two other places, neither of them blockwatcher-rpc:

• The reference format itself (SecretRef::parse/SecretRef::resolve) is defined once, in the vocabulary crate, crates/blockwatcher-types/src/secret.rs. resolve is the only I/O that crate performs, and it never caches the result: env: is read fresh at each call, though for an environment variable that changes nothing an operator can observe, since a process’s environment is fixed at spawn.
• The call site that actually invokes it, for evm-rpc’s endpoint pool, is resolve_endpoint_url in crates/blockwatcher-evm/src/registry.rs (documented in full on blockwatcher-evm), run once, at source-construction time, before a single EvmEndpoint or Pool is built. By the time an EndpointConfig reaches this crate, the secret has already been resolved into whatever connection value C carries; the pool itself never holds, logs, or has any way to name a secret reference at all.

Neighbours

blockwatcher-rpc depends on, in production:

• blockwatcher-ports: the trait boundary; only Classify/ErrorClass are actually used, re-exported at the crate root
• thiserror: the derive behind PoolError/AttemptError/PoolSetupError/ProbeFailure
• tokio (sync/time/macros/rt features): the semaphore, timers, and async runtime every retry and probe loop runs on
• metrics: the facade the pool’s six named constants emit through
• tracing: this crate’s own structured logging

and, in [dev-dependencies] only:

• blockwatcher-testkit: RecordingMetrics, used to assert on emitted counter names and labels instead of a real Prometheus backend
• tokio (rt-multi-thread/test-util features): multi-threaded and paused-time async tests

The following crates depend on it directly (per the dependency table):

• blockwatcher-evm: parameterizes Pool<EvmEndpoint> for both evm-rpc and evm-mempool, and implements Classify on its own EvmRpcError
• blockwatcher-evm-testkit: a direct production dependency (crates/blockwatcher-evm-testkit/Cargo.toml), not only a transitive one through blockwatcher-evm: single_endpoint_pool (blockwatcher-evm-testkit/src/pool.rs) builds a real Pool<EvmEndpoint> directly, so every EVM source test that points a module at a mock node goes through this crate’s own types

Reading the source

1. Start at lib.rs: the crate doc comment states the whole contract in one paragraph, and the pub use lines are the entire public surface, drawn from error.rs, blockwatcher_ports, and pool.rs.
2. error.rs: Attempt/AttemptError/PoolError, each generic over the consumer’s own error type E: std::error::Error.
3. breaker.rs: Breaker, its methods, and the doc comment’s closed/open/half-open state machine; read this before pool.rs, the only module that calls it.
4. limiter.rs: TokenBucket, try_take/next_available; read this before pool.rs’s select/select_pinned, its only callers.
5. pool.rs: Pool<C>, select/select_pinned, execute, probe_health, in that order; the *_TOTAL metric constants sit near the top of the file, each documented with the exact condition that increments it.