Observability
blockwatcher exposes two independent views of a running instance: the
HTTP API’s GET /status (a point-in-time JSON
snapshot of every pipeline, covered on that page) and a Prometheus
scrape endpoint carrying the same events as cumulative counters and
gauges since process start. This page covers the second one: how to turn
it on, exactly what it exports, and what to alert on.
flowchart LR
rpc["RPC endpoints<br/>(external chains)"]
sources["Sources<br/>evm-rpc · evm-mempool"]
sinks["Sinks<br/>webhook · script · log"]
storage["Storage<br/>checkpoints · dead letters · resources"]
api["REST API"]
metrics["Metrics"]
engine["engine<br/>bounded channels · checkpoints"]
subgraph pipeline["Engine pipeline"]
direction LR
decoder["Decoder"]
matcher["Matcher<br/>predicates"]
gate["Gate<br/>threshold · max_once"]
decoder --> matcher
matcher --> gate
end
rpc --> sources
sources -->|"decode and match"| decoder
gate --> sinks
api -->|"manages resources"| storage
storage <--> engine
engine -->|"drives"| pipeline
engine -.->|"reports"| metrics
classDef module fill:none,stroke:#a9a3e3
classDef core fill:none,stroke:#8a8d86,stroke-dasharray: 5 5
class rpc,sources,sinks,decoder,matcher,gate module
class engine,api,storage,metrics core
classDef dim fill:none,stroke:#999999,color:#999999,opacity:0.35
classDef focus fill:#ffd43b,stroke:#d9480f,stroke-width:3px,color:#1a1a1a
class rpc,sources,sinks,storage,api,engine,decoder,matcher,gate dim
class metrics focus
click sources "../concepts/selectors.html"
click decoder "../concepts/chain-agnosticism.html"
click matcher "../concepts/predicates.html"
click gate "../concepts/gates.html"
click sinks "../concepts/delivery.html"
click storage "../concepts/resources.html"
click api "http-api.html"
click metrics "observability.html"
click engine "../concepts/pipeline.html"
Key takeaways
- blockwatcher exposes two independent views of a running instance:
/status(point-in-time JSON) and/metrics(cumulative Prometheus counters and gauges), forked from the same events but never derived from each other. - The metrics listener is disabled by default and, once enabled, runs on its own socket with no authentication and no relation to the API’s bearer token.
- Every pipeline metric is defined once as a named constant and carries a
pipelinelabel; sink-scoped emissions also carry asinklabel except the counters marked “pipeline only”. blockwatcher-rpc’s connection pool andevm-mempoolsources publish their own metrics under separate prefixes, outside the per-network pipeline contract.- A rising dead-letter count, a growing
in_flight_events, and a restart failure are the alerting signals this page calls out explicitly.
Turning it on: [metrics]
Disabled by default, for the same reason [api] is: a listener nobody
asked for is a listener nobody remembered to firewall.
[metrics]
enabled = true
listen = "127.0.0.1:9090"
Once enabled, GET /metrics on listen serves Prometheus exposition
text (crates/blockwatcher-metrics/src/lib.rs): no path prefix, no
authentication, and no relation to the API’s labelled [auth] table. The metrics
listener is a separate axum::Router on its own socket, bound
independently of the control plane. A deployment that wants the scrape
endpoint restricted to its own network should bind it to a private
address or firewall the port; blockwatcher itself puts nothing in front of it.
Recorder upkeep (flushing accumulated Prometheus internals) runs on a
5-second interval for as long as the process is up, so a scrape always
sees fresh state rather than data queued since the last request. See
Configuration reference for the full
[metrics] section and its BLOCKWATCHER_METRICS__LISTEN override.
The pipeline metric contract
Every metric below is defined once, as a named constant, in
crates/blockwatcher-core/src/metrics.rs: the module’s own doc comment calls
these names “a contract: an exporter and its dashboards are built
against them,” fixed once shipped. Every emission carries a pipeline
label naming the network id; an
emission scoped to one sink also
carries a sink label, except the rows marked “pipeline only”
below, which stay pipeline-scoped even though the event they count is
sink-adjacent. An emission scoped to one gate
also carries monitor and gate (module name).
Full metric reference
| Metric | Type | Labels | Meaning |
|---|---|---|---|
blockwatcher_events_decoded_total | counter | pipeline | One raw payload decoded successfully against one active monitor’s selector. Counted per monitor-decode attempt, not per raw event: one payload against three active monitors adds up to three. |
blockwatcher_events_undecodable_total | counter | pipeline | A payload a monitor’s selector could not decode, counted per monitor-decode attempt for the same reason as above. |
blockwatcher_matches_total | counter | pipeline | A decoded event whose monitor had no predicate, or whose predicate evaluated true. |
blockwatcher_match_errors_total | counter | pipeline | A predicate that type-checked at write time nonetheless failed to evaluate at runtime. Should stay at zero; a non-zero value points at a matcher bug, not an operator mistake. |
blockwatcher_deliveries_total | counter | pipeline, sink | A sink event delivered successfully to one sink: a lone match, a retraction, or a digest. A Digest counts as one delivery however many matches it bundled: this counter tracks deliveries, not matches, so an aggregating sink’s value here is lower than its match count by design. |
blockwatcher_digest_deliveries_total | counter | pipeline, sink | A SinkEvent::Digest delivered successfully to one sink. Every digest delivery also increments blockwatcher_deliveries_total once, the same as any other successful delivery; this counter narrows that total to the digest share, so blockwatcher_digest_deliveries_total / blockwatcher_deliveries_total for one sink is the fraction of its deliveries that were batched rather than single matches. |
blockwatcher_delivery_retries_total | counter | pipeline, sink | One delivery attempt that failed and is about to retry, incremented per retry, not per final outcome. |
blockwatcher_dead_letters_total | counter | pipeline, sink | A sink event durably recorded as a dead letter: a delivery (match or retraction) that exhausted its retry budget, or a match a throttle policy suppressed without attempting (its record carries attempts 0). A dead-lettered digest adds one row here per contained match, never one row for the digest as a whole. |
blockwatcher_dead_letters_pruned_total | counter | pipeline (pipeline only) | The dead_letter_retention cap dropped the oldest dead letter(s) for this pipeline as part of the same write that recorded a new one. Deliberately carries no sink label: retention is a per-pipeline cap, not a per-sink one. Fires from either place a dead letter is recorded: a sink’s own delivery exhaustion, or the one-shot retract path during an invalidate. A non-zero rate means an operator can no longer list or replay everything that failed on that pipeline; omitting the key (the default) never fires. |
blockwatcher_deliveries_throttled_total | counter | pipeline, sink | A Match a sink’s throttle policy suppressed and dead-lettered instead of attempting, because that sink had already spent its window’s delivery budget. A failed delivery never counts here: only a success spends budget, so this counts admission refusals, not delivery failures. Counted per suppressed match, not per suppressed delivery: a throttled digest of N matches adds N here, agreeing with the N dead-letter rows it produces, even though the digest itself spent only one unit of the throttle budget. |
blockwatcher_dispatch_failed_total | counter | pipeline (pipeline only) | A match that never reached a sink worker at all: the worker’s channel was closed, or (should be unreachable) the monitor named a sink with no running worker. Distinct from a dead letter: this never got as far as a delivery attempt. |
blockwatcher_events_misrouted_total | counter | pipeline | A raw event arrived carrying a different network’s id than the pipeline it reached. Always zero in a correctly wired deployment; non-zero means a source is emitting for someone else’s network. |
blockwatcher_checkpoint_write_failures_total | counter | pipeline | A checkpoint persist that failed against storage and is being retried on a doubling backoff. |
blockwatcher_checkpoint_regressions_refused_total | counter | pipeline | A checkpoint the writer refused because its cursor was older than one the same writer instance already persisted. Always zero under a correct source; non-zero means a source emitted a regressing cursor mid-run. |
blockwatcher_in_flight_events | gauge | pipeline | How many dispatched matches are still outstanding (neither delivered nor dead-lettered), sampled once a second and on every checkpoint publish or retry. A value that never drains is the same stall GET /status’s in_flight_events field shows. |
blockwatcher_pipelines_aborted_total | counter | pipeline (pipeline only) | A pipeline forced through a hard abort during drain: the shutdown deadline was missed and a straggling task had to be cancelled outright, or a restart’s own drain hit the same deadline. Deliberately carries no sink label: an abort is a property of the whole pipeline, not any one delivery. |
blockwatcher_pipelines_quiesce_timeout_total | counter | pipeline (pipeline only) | A Storage::quiesce call, made after an escalated drain (or re-attempted for a debt an earlier one left owed) to wait out whatever an aborted task’s storage calls left running, that did not resolve within its budget. Always zero under a healthy backend. Counted once at stop_pipeline itself no matter which caller reached it: a restart, a gate-envelope change or a gated monitor’s delete, and a reorg invalidation all refuse in the same shape on this timeout (spawning a replacement, dropping a held journal, or pruning and retracting, respectively, could each race the outstanding write), leaving the network abandoned. A whole-engine shutdown facing the same timeout only warns and counts it, since the process is exiting either way. |
blockwatcher_dead_letter_write_failures_total | counter | pipeline, sink | A dead-letter record itself failed to write and is being retried forever. The only signal an exporter has that a sink’s dead-letter storage is stuck: the retry never gives up on its own. |
blockwatcher_pipeline_source_restarts_total | counter | pipeline | The restart supervisor actually respawned this pipeline after its source exited before its own cancellation fired. |
blockwatcher_pipeline_source_restart_failures_total | counter | pipeline | A restart attempt the supervisor made and that itself failed, for a reason other than the engine shutting down. The network is left with no running pipeline and nothing further will retry it. Distinct from blockwatcher_source_invalidation_failures_total, which counts a failed invalidate rather than a failed crash-restart. |
blockwatcher_source_invalidations_total | counter | pipeline | A source returned Invalidated { from }. Counted once per handled invalidate for that network. |
blockwatcher_source_invalidation_failures_total | counter | pipeline | A handled invalidate that failed to finish retract or rewind. The network is left with no running pipeline and an unrewound checkpoint until an operator acts. Distinct from blockwatcher_pipeline_source_restart_failures_total. |
blockwatcher_retracts_total | counter | pipeline, sink | A Retracted event handed to the one-shot invalidate pass. A successful deliver also increments blockwatcher_deliveries_total. |
blockwatcher_journal_gap_total | counter | pipeline | The invalidate cursor plus journal_depth did not cover the checkpoint high-water mark, so some delivered match ids were already pruned. Never silent: also logged at error. |
blockwatcher_rewinds_refused_total | counter | pipeline | A non-zero rate means an invalidation arrived that no checkpoint write could apply backward: its cursor was not behind the stored checkpoint, no checkpoint was stored at all, or the post-drain checkpoint read failed. Stored state was left as it was (an absent checkpoint is not fabricated) and the pipeline restarted; expect replayed duplicates, never gaps. The common cause is benign: the proven fork sits at the tracker’s own newest emitted block, inside the checkpoint’s own range, so refusing the rewind is the correct outcome. Suspect a stalled sink only when refusals on one network pair with retract or dead-letter anomalies there too. |
blockwatcher_operator_actions_total | counter | action, plus pipeline or monitor | A control-surface mutation whose persist has succeeded. action is stable snake_case (pause_monitor, resume_monitor, pause_network, resume_network, discard_dead_letter, discard_dead_letters). Identity is pipeline (network id) or monitor as appropriate. One counter, not one name per action. |
blockwatcher_gate_hits_total | counter | pipeline, monitor, gate | A timestamped hit offered to on_hit. |
blockwatcher_gate_emits_total | counter | pipeline, monitor, gate | An Emit applied (one per mint, not per contained match). |
blockwatcher_gate_discards_total | counter | pipeline, monitor, gate | A Discard applied. |
blockwatcher_gate_untimestamped_total | counter | pipeline, monitor, gate | Predicate-true, no usable block.timestamp; not inserted. |
blockwatcher_gate_hits_dropped_total | counter | pipeline, monitor, gate | Oldest hold dropped because of the 10_000 cap. |
blockwatcher_gate_persist_failed_total | counter | pipeline, monitor, gate | Storage put failed; that cursor is stalled. |
blockwatcher_gate_decision_invalid_total | counter | pipeline, monitor, gate | on_hit returned a decision the engine rejected; this hit discarded. Should stay at zero. |
blockwatcher_gate_journal_held | gauge | pipeline, monitor, gate | Hits currently held, set after every evaluation. Sustained values near gate_hits_cap are the signal to look at gate write volume. |
That is every metric blockwatcher-core exports through the metrics facade.
A test in metrics.rs pins each constant’s string literal against the
contract, so a typo in the definition itself cannot silently drift from
what a dashboard expects.
The same events, twice
Each pipeline event forks into two independent destinations, neither derived from the other:
flowchart LR
event["pipeline event<br/>decode, match, deliver..."] --> atomics["PipelineCounters<br/>plain atomics"]
event --> facade["metrics facade<br/>named constants"]
atomics --> status["GET /status<br/>point-in-time JSON"]
facade --> recorder["prometheus recorder<br/>5s flush"]
recorder --> scrape["GET /metrics<br/>exposition text"]
Every row above has a twin: crates/blockwatcher-core/src/counters.rs’s
PipelineCounters holds one plain atomic per status field (decoded,
undecodable, matched, match_errors, delivered, dead_lettered,
dispatch_failed, checkpoint_write_failed, dead_letter_write_failed,
misrouted, checkpoint_regressions_refused, untimestamped,
gate_persist_failed, gate_hits_dropped, gated, gate_emitted); the
Prometheus contract additionally exposes in_flight_events,
pipelines_aborted, the restart counters, the invalidation-failure
counter, the invalidate / retract / journal-gap counters,
dead_letters_pruned, and deliveries_throttled and digest_deliveries,
which have no per-network JSON counterpart), and both are incremented from the same call site. The atomics feed GET /status’s counters object directly:
a point-in-time read, no labels, no cardinality cost, nothing to scrape,
while the Prometheus constants feed a time series an alerting rule can
take a rate() of. Neither is derived from the other; they are two
independent destinations for the same underlying event, which is why the
numbers agree at any instant but serve different jobs: reach for
/status when debugging one network right now, and for /metrics when
watching every network’s trend over time.
Other metrics on this endpoint
blockwatcher-rpc’s connection pool (the load-balancing and circuit-breaking
layer evm-rpc and evm-mempool both sit on) publishes its own
operational metrics onto the same process-wide recorder, under a
blockwatcher_rpc_ prefix (crates/blockwatcher-rpc/src/pool.rs). They are
not part of the per-network contract above and carry an endpoint label
(the RPC endpoint name from a network’s endpoints[] config) rather than
pipeline:
| Metric | Labels | Meaning |
|---|---|---|
blockwatcher_rpc_attempts_total | endpoint | One attempt execute made against an endpoint, whether it succeeded or not. |
blockwatcher_rpc_attempt_failures_total | endpoint, class | An attempt that did not succeed; class is the failure’s ErrorClass rendering, or "timeout" for the attempt’s own deadline elapsing. |
blockwatcher_rpc_breaker_opened_total | endpoint | A consecutive-failure streak that tripped that endpoint’s breaker from admitting to open. |
blockwatcher_rpc_rate_limit_waits_total | (none) | An execute iteration that slept because every candidate endpoint was rate-limited: pool-wide, not attributable to one endpoint. |
blockwatcher_rpc_exhausted_total | reason | An execute call that ended with no served value; reason is deadline_exceeded, no_endpoint_available, or pin_unavailable. |
blockwatcher_rpc_probe_failures_total | endpoint | A periodic health probe against an endpoint that did not produce a value. |
evm-rpc sources publish metrics of their own under a blockwatcher_evm_
prefix (crates/blockwatcher-evm/src/source/rpc/run.rs); the reorg counter
carries pipeline and depth, the skip counter carries only pipeline, and
the contradiction and range-split counters carry pipeline and endpoint:
| Metric | Labels | Meaning |
|---|---|---|
blockwatcher_evm_reorgs_total | pipeline, depth | One linkage break this source classified. depth is within_confirmations (retried in place), beyond_confirmations (proven fork, Invalidated { from }), or beyond_window (no tracked ancestor matched; Invalidated { from } just below the oldest tracked height, bounded by the tracker’s own depth rather than a proven fork block). |
blockwatcher_evm_bloom_skips_total | pipeline | One leaf window whose eth_getLogs call was skipped because every fetched header’s logsBloom refuted every monitored address, or every monitored topic0, in the filter: either dimension refuted alone already rules out a match, since eth_getLogs requires both to hold. Counted once the window’s fetch returns, whether or not the window later survives reorg verification and is actually emitted. |
blockwatcher_evm_bloom_contradictions_total | pipeline, endpoint | One log whose own block’s fetched bloom failed to admit that log’s own address and topic0, proof that the named endpoint’s blooms do not describe the logs it returns. The first one disables bloom_screen for the running source instance that observed it; a nonzero rate afterward on the same running instance means every window since has been fetched in full rather than screened. That disabling does not survive a restart of the pipeline’s source, so it is not a durable fix on its own; see the alerting hint below. |
blockwatcher_evm_range_splits_total | pipeline, endpoint | One leaf a provider forced out of a wider eth_getLogs range it refused to serve in full. Counted once per leaf (so a window split into three leaves increments this twice, for the two beyond the first), against the endpoint that forced the split. |
evm-mempool sources publish metrics of their own, under the same
blockwatcher_evm_ prefix (crates/blockwatcher-evm/src/source/mempool/run.rs).
The skip counter carries endpoint; the reconnect counter does not, since a
lost subscription is not attributable to one endpoint:
| Metric | Labels | Meaning |
|---|---|---|
blockwatcher_evm_mempool_skips_total | pipeline, endpoint, reason | One subscription hash that produced no emission for a reason outside this source’s control; reason is gone (the lookup answered null because the transaction was mined or evicted first) or lookup_failed (the pool gave up). Neither is an error: a pending stream is best-effort by nature. |
blockwatcher_evm_mempool_reconnects_total | pipeline, reason | One lost WebSocket subscription this source had to reconnect from; reason is dial_failed, stream_closed, transport_error, or idle_timeout. |
Alerting hints
Dead-letter count rising. increase(blockwatcher_dead_letters_total[15m]) > 0
sustained, for one sink label, means that sink is failing permanently
or exhausting retries, not a one-off blip. Pull the reason off
GET /networks/{id}/dead-letters (each entry’s reason string leads
with transient: or permanent:) before deciding whether to fix the
sink target or just replay.
A sink’s throttle is swallowing everything.
increase(blockwatcher_deliveries_throttled_total[15m]) sustained at roughly
the same rate as increase(blockwatcher_matches_total[15m]) for one sink
label means that sink’s throttle policy is dead-lettering nearly every match
rather than occasionally shedding a burst. Widen max_deliveries or
window_ms for that sink, or pause the monitors feeding it, rather than
leaving it to accumulate dead letters that need replaying later. This
match-for-match comparison only holds for a sink with no aggregate policy:
blockwatcher_deliveries_throttled_total counts suppressed matches while
blockwatcher_deliveries_total counts a digest as one delivery regardless of
how many matches it carried, so for an aggregating sink the
throttled-versus-delivered ratio is not match-over-match. Compare against
blockwatcher_digest_deliveries_total instead to see how many of that
sink’s deliveries were digests in the first place.
Checkpoint lag growing. Watch blockwatcher_in_flight_events for a value
that climbs and never comes back down, alongside
blockwatcher_checkpoint_write_failures_total ticking upward. The first says a
delivery is stuck open (check which sink’s queue is backed up), and the
second says storage itself is the problem; either one means the
network’s checkpoint has stopped advancing, which is exactly the signal
GET /status’s lag field surfaces from the other side.
Threshold never fires. Check blockwatcher_gate_untimestamped_total,
whether count is higher than traffic, and whether invalidate drained
holds (must not). Compare event timestamps, not wall clock. GET /status
pipeline counters gated (Retain+Discard) and gate_emitted (one per
Emit) are the point-in-time twins.
Checkpoint stuck and persist_failed ticking. Same class as a closed
sink: fix storage; do not skip the event. Watch
blockwatcher_gate_persist_failed_total.
Tip reorg “lost” a burst. If holds with cursor ≤ from were dropped,
that is a bug (drain-all). Replay will not restore them.
Gate log lines carry network, monitor, and gate (module name).
A network’s source has stopped coming back.
blockwatcher_pipeline_source_restart_failures_total > 0 for a pipeline
label means the restart supervisor tried and failed to bring that
network’s source back up, and, per the metric’s own contract, nothing
further will retry it on its own; the pipeline is down until an operator
intervenes. This is a stronger signal than an occasional
blockwatcher_pipeline_source_restarts_total tick, which just means a source
exited and was cleanly respawned. A failed invalidate is a different
abandonment: watch blockwatcher_source_invalidation_failures_total for that.
A deep invalidate failed to finish.
blockwatcher_source_invalidation_failures_total > 0 for a pipeline label
means retract or rewind did not complete. The checkpoint is left
unrewound and the source is not restarted; this is not a crash-restart
failure. Inspect dead letters for type: retracted payloads, then
intervene.
A deep invalidate left a journal gap.
increase(blockwatcher_journal_gap_total[15m]) > 0 for a pipeline label means
that network’s invalidate cursor plus journal_depth did not cover the
checkpoint high-water mark, so some already-delivered match ids cannot be
retracted. The matching error log names the network, from, the
checkpoint, and the configured depth. Raise journal_depth (it must stay
well above confirmation depth) and treat consumer-side undo for ids older
than the window as an operator problem: the engine will still rewind and
restart.
A source keeps invalidating.
increase(blockwatcher_source_invalidations_total[15m]) climbing on one
pipeline is unusual for a quiet chain; pair it with blockwatcher_retracts_total
to see whether sinks are actually receiving undos. An unrecovered retract
leaves the checkpoint unrewound: watch
blockwatcher_source_invalidation_failures_total, blockwatcher_dead_letters_total,
and in_flight_events on that network rather than expecting a restart.
A reorg past the tracked window.
increase(blockwatcher_evm_reorgs_total[15m]) with depth="beyond_window" on
one pipeline means no tracked ancestor matched the live chain. The source
invalidates from just below the oldest tracked height, a rewind bounded by
the tracker’s own depth rather than genesis: the engine retracts journaled
deliveries above that bound and restarts the source, the same path as any
other invalidation, so expect blockwatcher_source_invalidations_total to
tick alongside this depth label.
A bloom-skip rate of zero is not itself a fault.
blockwatcher_evm_bloom_skips_total staying flat on a network with a
selective monitor (one whose filter names addresses or topic0s) and an
active chain is informational, not a fault: it means bloom_screen is
disabled for that network, or the merged filter is broad enough (no
addresses and no topic0s) that there is nothing left for a header’s bloom
to refute. Read it alongside the filter a network’s monitors actually
produce before treating it as a signal of anything gone wrong.
A bloom-contradiction count that is not zero.
increase(blockwatcher_evm_bloom_contradictions_total[15m]) > 0 for a
pipeline and endpoint pair means that endpoint served a header bloom
that failed to admit a log it returned for the same block, which can only
happen when its blooms do not describe its own logs. Two things follow,
and neither is automatic:
- The running source instance that observed the contradiction has already
disabled
bloom_screenfor itself, so no further window on that particular instance is at risk. That protection belongs to the instance, not the endpoint or the process: a restart of that pipeline’s source (the supervisor recovering from an exit, a proven-reorg invalidation, or a monitor change that escalates to a restart) builds a fresh instance with screening enabled again, against the same endpoint that already proved its blooms unreliable. Setbloom_screen = falsefor that network to make the disabling survive a restart; nothing else does. - The contradiction proves the endpoint’s blooms are wrong, but says
nothing about when the endpoint started serving wrong ones. Every window
this source screened earlier in its run rested on the same untrustworthy
blooms; the
contradiction is only the first one this source happened to catch
because a later window’s own headers still forced a real fetch. Read
blockwatcher_evm_bloom_skips_totalfor the samepipelineto see how many windows were screened before the trip, and re-scan that range throughPOST /monitors/{id}/test’sfetchmode (which never screens) or a fresh network withstart_blockcovering it: the checkpoint has already advanced past those windows, and nothing rewinds them on its own.
Read the matching warning log for the network, endpoint, the contradicting window’s own block range, the first contradicting block, and the contradiction count, and treat that endpoint’s blooms as untrusted for any other purpose too.
An evm-mempool connection is flapping.
increase(blockwatcher_evm_mempool_reconnects_total[15m]) > N for a pipeline
label means that network’s WebSocket subscription keeps dropping and
reconnecting; GET /status’s source.status only ever shows whichever
state (live, degraded, catching_up) is current at scrape time, so a
short-lived flap between two scrapes is invisible there even though it
loses whatever was pending in the node’s mempool during the gap.