Keyboard shortcuts

Press or to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Observability

blockwatcher exposes two independent views of a running instance: the HTTP API’s GET /status (a point-in-time JSON snapshot of every pipeline, covered on that page) and a Prometheus scrape endpoint carrying the same events as cumulative counters and gauges since process start. This page covers the second one: how to turn it on, exactly what it exports, and what to alert on.

flowchart LR
    rpc["RPC endpoints<br/>(external chains)"]
    sources["Sources<br/>evm-rpc · evm-mempool"]
    sinks["Sinks<br/>webhook · script · log"]
    storage["Storage<br/>checkpoints · dead letters · resources"]
    api["REST API"]
    metrics["Metrics"]
    engine["engine<br/>bounded channels · checkpoints"]

    subgraph pipeline["Engine pipeline"]
        direction LR
        decoder["Decoder"]
        matcher["Matcher<br/>predicates"]
        gate["Gate<br/>threshold · max_once"]
        decoder --> matcher
        matcher --> gate
    end

    rpc --> sources
    sources -->|"decode and match"| decoder
    gate --> sinks
    api -->|"manages resources"| storage
    storage <--> engine
    engine -->|"drives"| pipeline
    engine -.->|"reports"| metrics

    classDef module fill:none,stroke:#a9a3e3
    classDef core fill:none,stroke:#8a8d86,stroke-dasharray: 5 5

    class rpc,sources,sinks,decoder,matcher,gate module
    class engine,api,storage,metrics core
classDef dim fill:none,stroke:#999999,color:#999999,opacity:0.35
classDef focus fill:#ffd43b,stroke:#d9480f,stroke-width:3px,color:#1a1a1a
class rpc,sources,sinks,storage,api,engine,decoder,matcher,gate dim
class metrics focus
click sources "../concepts/selectors.html"
click decoder "../concepts/chain-agnosticism.html"
click matcher "../concepts/predicates.html"
click gate "../concepts/gates.html"
click sinks "../concepts/delivery.html"
click storage "../concepts/resources.html"
click api "http-api.html"
click metrics "observability.html"
click engine "../concepts/pipeline.html"

Key takeaways

  • blockwatcher exposes two independent views of a running instance: /status (point-in-time JSON) and /metrics (cumulative Prometheus counters and gauges), forked from the same events but never derived from each other.
  • The metrics listener is disabled by default and, once enabled, runs on its own socket with no authentication and no relation to the API’s bearer token.
  • Every pipeline metric is defined once as a named constant and carries a pipeline label; sink-scoped emissions also carry a sink label except the counters marked “pipeline only”.
  • blockwatcher-rpc’s connection pool and evm-mempool sources publish their own metrics under separate prefixes, outside the per-network pipeline contract.
  • A rising dead-letter count, a growing in_flight_events, and a restart failure are the alerting signals this page calls out explicitly.

Turning it on: [metrics]

Disabled by default, for the same reason [api] is: a listener nobody asked for is a listener nobody remembered to firewall.

[metrics]
enabled = true
listen = "127.0.0.1:9090"

Once enabled, GET /metrics on listen serves Prometheus exposition text (crates/blockwatcher-metrics/src/lib.rs): no path prefix, no authentication, and no relation to the API’s labelled [auth] table. The metrics listener is a separate axum::Router on its own socket, bound independently of the control plane. A deployment that wants the scrape endpoint restricted to its own network should bind it to a private address or firewall the port; blockwatcher itself puts nothing in front of it. Recorder upkeep (flushing accumulated Prometheus internals) runs on a 5-second interval for as long as the process is up, so a scrape always sees fresh state rather than data queued since the last request. See Configuration reference for the full [metrics] section and its BLOCKWATCHER_METRICS__LISTEN override.

The pipeline metric contract

Every metric below is defined once, as a named constant, in crates/blockwatcher-core/src/metrics.rs: the module’s own doc comment calls these names “a contract: an exporter and its dashboards are built against them,” fixed once shipped. Every emission carries a pipeline label naming the network id; an emission scoped to one sink also carries a sink label, except the rows marked “pipeline only” below, which stay pipeline-scoped even though the event they count is sink-adjacent. An emission scoped to one gate also carries monitor and gate (module name).

Full metric reference
MetricTypeLabelsMeaning
blockwatcher_events_decoded_totalcounterpipelineOne raw payload decoded successfully against one active monitor’s selector. Counted per monitor-decode attempt, not per raw event: one payload against three active monitors adds up to three.
blockwatcher_events_undecodable_totalcounterpipelineA payload a monitor’s selector could not decode, counted per monitor-decode attempt for the same reason as above.
blockwatcher_matches_totalcounterpipelineA decoded event whose monitor had no predicate, or whose predicate evaluated true.
blockwatcher_match_errors_totalcounterpipelineA predicate that type-checked at write time nonetheless failed to evaluate at runtime. Should stay at zero; a non-zero value points at a matcher bug, not an operator mistake.
blockwatcher_deliveries_totalcounterpipeline, sinkA sink event delivered successfully to one sink: a lone match, a retraction, or a digest. A Digest counts as one delivery however many matches it bundled: this counter tracks deliveries, not matches, so an aggregating sink’s value here is lower than its match count by design.
blockwatcher_digest_deliveries_totalcounterpipeline, sinkA SinkEvent::Digest delivered successfully to one sink. Every digest delivery also increments blockwatcher_deliveries_total once, the same as any other successful delivery; this counter narrows that total to the digest share, so blockwatcher_digest_deliveries_total / blockwatcher_deliveries_total for one sink is the fraction of its deliveries that were batched rather than single matches.
blockwatcher_delivery_retries_totalcounterpipeline, sinkOne delivery attempt that failed and is about to retry, incremented per retry, not per final outcome.
blockwatcher_dead_letters_totalcounterpipeline, sinkA sink event durably recorded as a dead letter: a delivery (match or retraction) that exhausted its retry budget, or a match a throttle policy suppressed without attempting (its record carries attempts 0). A dead-lettered digest adds one row here per contained match, never one row for the digest as a whole.
blockwatcher_dead_letters_pruned_totalcounterpipeline (pipeline only)The dead_letter_retention cap dropped the oldest dead letter(s) for this pipeline as part of the same write that recorded a new one. Deliberately carries no sink label: retention is a per-pipeline cap, not a per-sink one. Fires from either place a dead letter is recorded: a sink’s own delivery exhaustion, or the one-shot retract path during an invalidate. A non-zero rate means an operator can no longer list or replay everything that failed on that pipeline; omitting the key (the default) never fires.
blockwatcher_deliveries_throttled_totalcounterpipeline, sinkA Match a sink’s throttle policy suppressed and dead-lettered instead of attempting, because that sink had already spent its window’s delivery budget. A failed delivery never counts here: only a success spends budget, so this counts admission refusals, not delivery failures. Counted per suppressed match, not per suppressed delivery: a throttled digest of N matches adds N here, agreeing with the N dead-letter rows it produces, even though the digest itself spent only one unit of the throttle budget.
blockwatcher_dispatch_failed_totalcounterpipeline (pipeline only)A match that never reached a sink worker at all: the worker’s channel was closed, or (should be unreachable) the monitor named a sink with no running worker. Distinct from a dead letter: this never got as far as a delivery attempt.
blockwatcher_events_misrouted_totalcounterpipelineA raw event arrived carrying a different network’s id than the pipeline it reached. Always zero in a correctly wired deployment; non-zero means a source is emitting for someone else’s network.
blockwatcher_checkpoint_write_failures_totalcounterpipelineA checkpoint persist that failed against storage and is being retried on a doubling backoff.
blockwatcher_checkpoint_regressions_refused_totalcounterpipelineA checkpoint the writer refused because its cursor was older than one the same writer instance already persisted. Always zero under a correct source; non-zero means a source emitted a regressing cursor mid-run.
blockwatcher_in_flight_eventsgaugepipelineHow many dispatched matches are still outstanding (neither delivered nor dead-lettered), sampled once a second and on every checkpoint publish or retry. A value that never drains is the same stall GET /status’s in_flight_events field shows.
blockwatcher_pipelines_aborted_totalcounterpipeline (pipeline only)A pipeline forced through a hard abort during drain: the shutdown deadline was missed and a straggling task had to be cancelled outright, or a restart’s own drain hit the same deadline. Deliberately carries no sink label: an abort is a property of the whole pipeline, not any one delivery.
blockwatcher_pipelines_quiesce_timeout_totalcounterpipeline (pipeline only)A Storage::quiesce call, made after an escalated drain (or re-attempted for a debt an earlier one left owed) to wait out whatever an aborted task’s storage calls left running, that did not resolve within its budget. Always zero under a healthy backend. Counted once at stop_pipeline itself no matter which caller reached it: a restart, a gate-envelope change or a gated monitor’s delete, and a reorg invalidation all refuse in the same shape on this timeout (spawning a replacement, dropping a held journal, or pruning and retracting, respectively, could each race the outstanding write), leaving the network abandoned. A whole-engine shutdown facing the same timeout only warns and counts it, since the process is exiting either way.
blockwatcher_dead_letter_write_failures_totalcounterpipeline, sinkA dead-letter record itself failed to write and is being retried forever. The only signal an exporter has that a sink’s dead-letter storage is stuck: the retry never gives up on its own.
blockwatcher_pipeline_source_restarts_totalcounterpipelineThe restart supervisor actually respawned this pipeline after its source exited before its own cancellation fired.
blockwatcher_pipeline_source_restart_failures_totalcounterpipelineA restart attempt the supervisor made and that itself failed, for a reason other than the engine shutting down. The network is left with no running pipeline and nothing further will retry it. Distinct from blockwatcher_source_invalidation_failures_total, which counts a failed invalidate rather than a failed crash-restart.
blockwatcher_source_invalidations_totalcounterpipelineA source returned Invalidated { from }. Counted once per handled invalidate for that network.
blockwatcher_source_invalidation_failures_totalcounterpipelineA handled invalidate that failed to finish retract or rewind. The network is left with no running pipeline and an unrewound checkpoint until an operator acts. Distinct from blockwatcher_pipeline_source_restart_failures_total.
blockwatcher_retracts_totalcounterpipeline, sinkA Retracted event handed to the one-shot invalidate pass. A successful deliver also increments blockwatcher_deliveries_total.
blockwatcher_journal_gap_totalcounterpipelineThe invalidate cursor plus journal_depth did not cover the checkpoint high-water mark, so some delivered match ids were already pruned. Never silent: also logged at error.
blockwatcher_rewinds_refused_totalcounterpipelineA non-zero rate means an invalidation arrived that no checkpoint write could apply backward: its cursor was not behind the stored checkpoint, no checkpoint was stored at all, or the post-drain checkpoint read failed. Stored state was left as it was (an absent checkpoint is not fabricated) and the pipeline restarted; expect replayed duplicates, never gaps. The common cause is benign: the proven fork sits at the tracker’s own newest emitted block, inside the checkpoint’s own range, so refusing the rewind is the correct outcome. Suspect a stalled sink only when refusals on one network pair with retract or dead-letter anomalies there too.
blockwatcher_operator_actions_totalcounteraction, plus pipeline or monitorA control-surface mutation whose persist has succeeded. action is stable snake_case (pause_monitor, resume_monitor, pause_network, resume_network, discard_dead_letter, discard_dead_letters). Identity is pipeline (network id) or monitor as appropriate. One counter, not one name per action.
blockwatcher_gate_hits_totalcounterpipeline, monitor, gateA timestamped hit offered to on_hit.
blockwatcher_gate_emits_totalcounterpipeline, monitor, gateAn Emit applied (one per mint, not per contained match).
blockwatcher_gate_discards_totalcounterpipeline, monitor, gateA Discard applied.
blockwatcher_gate_untimestamped_totalcounterpipeline, monitor, gatePredicate-true, no usable block.timestamp; not inserted.
blockwatcher_gate_hits_dropped_totalcounterpipeline, monitor, gateOldest hold dropped because of the 10_000 cap.
blockwatcher_gate_persist_failed_totalcounterpipeline, monitor, gateStorage put failed; that cursor is stalled.
blockwatcher_gate_decision_invalid_totalcounterpipeline, monitor, gateon_hit returned a decision the engine rejected; this hit discarded. Should stay at zero.
blockwatcher_gate_journal_heldgaugepipeline, monitor, gateHits currently held, set after every evaluation. Sustained values near gate_hits_cap are the signal to look at gate write volume.

That is every metric blockwatcher-core exports through the metrics facade. A test in metrics.rs pins each constant’s string literal against the contract, so a typo in the definition itself cannot silently drift from what a dashboard expects.

The same events, twice

Each pipeline event forks into two independent destinations, neither derived from the other:

flowchart LR
    event["pipeline event<br/>decode, match, deliver..."] --> atomics["PipelineCounters<br/>plain atomics"]
    event --> facade["metrics facade<br/>named constants"]
    atomics --> status["GET /status<br/>point-in-time JSON"]
    facade --> recorder["prometheus recorder<br/>5s flush"]
    recorder --> scrape["GET /metrics<br/>exposition text"]

Every row above has a twin: crates/blockwatcher-core/src/counters.rs’s PipelineCounters holds one plain atomic per status field (decoded, undecodable, matched, match_errors, delivered, dead_lettered, dispatch_failed, checkpoint_write_failed, dead_letter_write_failed, misrouted, checkpoint_regressions_refused, untimestamped, gate_persist_failed, gate_hits_dropped, gated, gate_emitted); the Prometheus contract additionally exposes in_flight_events, pipelines_aborted, the restart counters, the invalidation-failure counter, the invalidate / retract / journal-gap counters, dead_letters_pruned, and deliveries_throttled and digest_deliveries, which have no per-network JSON counterpart), and both are incremented from the same call site. The atomics feed GET /status’s counters object directly: a point-in-time read, no labels, no cardinality cost, nothing to scrape, while the Prometheus constants feed a time series an alerting rule can take a rate() of. Neither is derived from the other; they are two independent destinations for the same underlying event, which is why the numbers agree at any instant but serve different jobs: reach for /status when debugging one network right now, and for /metrics when watching every network’s trend over time.

Other metrics on this endpoint

blockwatcher-rpc’s connection pool (the load-balancing and circuit-breaking layer evm-rpc and evm-mempool both sit on) publishes its own operational metrics onto the same process-wide recorder, under a blockwatcher_rpc_ prefix (crates/blockwatcher-rpc/src/pool.rs). They are not part of the per-network contract above and carry an endpoint label (the RPC endpoint name from a network’s endpoints[] config) rather than pipeline:

MetricLabelsMeaning
blockwatcher_rpc_attempts_totalendpointOne attempt execute made against an endpoint, whether it succeeded or not.
blockwatcher_rpc_attempt_failures_totalendpoint, classAn attempt that did not succeed; class is the failure’s ErrorClass rendering, or "timeout" for the attempt’s own deadline elapsing.
blockwatcher_rpc_breaker_opened_totalendpointA consecutive-failure streak that tripped that endpoint’s breaker from admitting to open.
blockwatcher_rpc_rate_limit_waits_total(none)An execute iteration that slept because every candidate endpoint was rate-limited: pool-wide, not attributable to one endpoint.
blockwatcher_rpc_exhausted_totalreasonAn execute call that ended with no served value; reason is deadline_exceeded, no_endpoint_available, or pin_unavailable.
blockwatcher_rpc_probe_failures_totalendpointA periodic health probe against an endpoint that did not produce a value.

evm-rpc sources publish metrics of their own under a blockwatcher_evm_ prefix (crates/blockwatcher-evm/src/source/rpc/run.rs); the reorg counter carries pipeline and depth, the skip counter carries only pipeline, and the contradiction and range-split counters carry pipeline and endpoint:

MetricLabelsMeaning
blockwatcher_evm_reorgs_totalpipeline, depthOne linkage break this source classified. depth is within_confirmations (retried in place), beyond_confirmations (proven fork, Invalidated { from }), or beyond_window (no tracked ancestor matched; Invalidated { from } just below the oldest tracked height, bounded by the tracker’s own depth rather than a proven fork block).
blockwatcher_evm_bloom_skips_totalpipelineOne leaf window whose eth_getLogs call was skipped because every fetched header’s logsBloom refuted every monitored address, or every monitored topic0, in the filter: either dimension refuted alone already rules out a match, since eth_getLogs requires both to hold. Counted once the window’s fetch returns, whether or not the window later survives reorg verification and is actually emitted.
blockwatcher_evm_bloom_contradictions_totalpipeline, endpointOne log whose own block’s fetched bloom failed to admit that log’s own address and topic0, proof that the named endpoint’s blooms do not describe the logs it returns. The first one disables bloom_screen for the running source instance that observed it; a nonzero rate afterward on the same running instance means every window since has been fetched in full rather than screened. That disabling does not survive a restart of the pipeline’s source, so it is not a durable fix on its own; see the alerting hint below.
blockwatcher_evm_range_splits_totalpipeline, endpointOne leaf a provider forced out of a wider eth_getLogs range it refused to serve in full. Counted once per leaf (so a window split into three leaves increments this twice, for the two beyond the first), against the endpoint that forced the split.

evm-mempool sources publish metrics of their own, under the same blockwatcher_evm_ prefix (crates/blockwatcher-evm/src/source/mempool/run.rs). The skip counter carries endpoint; the reconnect counter does not, since a lost subscription is not attributable to one endpoint:

MetricLabelsMeaning
blockwatcher_evm_mempool_skips_totalpipeline, endpoint, reasonOne subscription hash that produced no emission for a reason outside this source’s control; reason is gone (the lookup answered null because the transaction was mined or evicted first) or lookup_failed (the pool gave up). Neither is an error: a pending stream is best-effort by nature.
blockwatcher_evm_mempool_reconnects_totalpipeline, reasonOne lost WebSocket subscription this source had to reconnect from; reason is dial_failed, stream_closed, transport_error, or idle_timeout.

Alerting hints

Dead-letter count rising. increase(blockwatcher_dead_letters_total[15m]) > 0 sustained, for one sink label, means that sink is failing permanently or exhausting retries, not a one-off blip. Pull the reason off GET /networks/{id}/dead-letters (each entry’s reason string leads with transient: or permanent:) before deciding whether to fix the sink target or just replay.

A sink’s throttle is swallowing everything. increase(blockwatcher_deliveries_throttled_total[15m]) sustained at roughly the same rate as increase(blockwatcher_matches_total[15m]) for one sink label means that sink’s throttle policy is dead-lettering nearly every match rather than occasionally shedding a burst. Widen max_deliveries or window_ms for that sink, or pause the monitors feeding it, rather than leaving it to accumulate dead letters that need replaying later. This match-for-match comparison only holds for a sink with no aggregate policy: blockwatcher_deliveries_throttled_total counts suppressed matches while blockwatcher_deliveries_total counts a digest as one delivery regardless of how many matches it carried, so for an aggregating sink the throttled-versus-delivered ratio is not match-over-match. Compare against blockwatcher_digest_deliveries_total instead to see how many of that sink’s deliveries were digests in the first place.

Checkpoint lag growing. Watch blockwatcher_in_flight_events for a value that climbs and never comes back down, alongside blockwatcher_checkpoint_write_failures_total ticking upward. The first says a delivery is stuck open (check which sink’s queue is backed up), and the second says storage itself is the problem; either one means the network’s checkpoint has stopped advancing, which is exactly the signal GET /status’s lag field surfaces from the other side.

Threshold never fires. Check blockwatcher_gate_untimestamped_total, whether count is higher than traffic, and whether invalidate drained holds (must not). Compare event timestamps, not wall clock. GET /status pipeline counters gated (Retain+Discard) and gate_emitted (one per Emit) are the point-in-time twins.

Checkpoint stuck and persist_failed ticking. Same class as a closed sink: fix storage; do not skip the event. Watch blockwatcher_gate_persist_failed_total.

Tip reorg “lost” a burst. If holds with cursor ≤ from were dropped, that is a bug (drain-all). Replay will not restore them.

Gate log lines carry network, monitor, and gate (module name).

A network’s source has stopped coming back. blockwatcher_pipeline_source_restart_failures_total > 0 for a pipeline label means the restart supervisor tried and failed to bring that network’s source back up, and, per the metric’s own contract, nothing further will retry it on its own; the pipeline is down until an operator intervenes. This is a stronger signal than an occasional blockwatcher_pipeline_source_restarts_total tick, which just means a source exited and was cleanly respawned. A failed invalidate is a different abandonment: watch blockwatcher_source_invalidation_failures_total for that.

A deep invalidate failed to finish. blockwatcher_source_invalidation_failures_total > 0 for a pipeline label means retract or rewind did not complete. The checkpoint is left unrewound and the source is not restarted; this is not a crash-restart failure. Inspect dead letters for type: retracted payloads, then intervene.

A deep invalidate left a journal gap. increase(blockwatcher_journal_gap_total[15m]) > 0 for a pipeline label means that network’s invalidate cursor plus journal_depth did not cover the checkpoint high-water mark, so some already-delivered match ids cannot be retracted. The matching error log names the network, from, the checkpoint, and the configured depth. Raise journal_depth (it must stay well above confirmation depth) and treat consumer-side undo for ids older than the window as an operator problem: the engine will still rewind and restart.

A source keeps invalidating. increase(blockwatcher_source_invalidations_total[15m]) climbing on one pipeline is unusual for a quiet chain; pair it with blockwatcher_retracts_total to see whether sinks are actually receiving undos. An unrecovered retract leaves the checkpoint unrewound: watch blockwatcher_source_invalidation_failures_total, blockwatcher_dead_letters_total, and in_flight_events on that network rather than expecting a restart.

A reorg past the tracked window. increase(blockwatcher_evm_reorgs_total[15m]) with depth="beyond_window" on one pipeline means no tracked ancestor matched the live chain. The source invalidates from just below the oldest tracked height, a rewind bounded by the tracker’s own depth rather than genesis: the engine retracts journaled deliveries above that bound and restarts the source, the same path as any other invalidation, so expect blockwatcher_source_invalidations_total to tick alongside this depth label.

A bloom-skip rate of zero is not itself a fault. blockwatcher_evm_bloom_skips_total staying flat on a network with a selective monitor (one whose filter names addresses or topic0s) and an active chain is informational, not a fault: it means bloom_screen is disabled for that network, or the merged filter is broad enough (no addresses and no topic0s) that there is nothing left for a header’s bloom to refute. Read it alongside the filter a network’s monitors actually produce before treating it as a signal of anything gone wrong.

A bloom-contradiction count that is not zero. increase(blockwatcher_evm_bloom_contradictions_total[15m]) > 0 for a pipeline and endpoint pair means that endpoint served a header bloom that failed to admit a log it returned for the same block, which can only happen when its blooms do not describe its own logs. Two things follow, and neither is automatic:

  • The running source instance that observed the contradiction has already disabled bloom_screen for itself, so no further window on that particular instance is at risk. That protection belongs to the instance, not the endpoint or the process: a restart of that pipeline’s source (the supervisor recovering from an exit, a proven-reorg invalidation, or a monitor change that escalates to a restart) builds a fresh instance with screening enabled again, against the same endpoint that already proved its blooms unreliable. Set bloom_screen = false for that network to make the disabling survive a restart; nothing else does.
  • The contradiction proves the endpoint’s blooms are wrong, but says nothing about when the endpoint started serving wrong ones. Every window this source screened earlier in its run rested on the same untrustworthy blooms; the contradiction is only the first one this source happened to catch because a later window’s own headers still forced a real fetch. Read blockwatcher_evm_bloom_skips_total for the same pipeline to see how many windows were screened before the trip, and re-scan that range through POST /monitors/{id}/test’s fetch mode (which never screens) or a fresh network with start_block covering it: the checkpoint has already advanced past those windows, and nothing rewinds them on its own.

Read the matching warning log for the network, endpoint, the contradicting window’s own block range, the first contradicting block, and the contradiction count, and treat that endpoint’s blooms as untrusted for any other purpose too.

An evm-mempool connection is flapping. increase(blockwatcher_evm_mempool_reconnects_total[15m]) > N for a pipeline label means that network’s WebSocket subscription keeps dropping and reconnecting; GET /status’s source.status only ever shows whichever state (live, degraded, catching_up) is current at scrape time, so a short-lived flap between two scrapes is invisible there even though it loses whatever was pending in the node’s mempool during the gap.