Troubleshooting
A symptom-first index into failure modes that are either expected behavior worth recognizing, or genuine problems with a specific fix. Every quoted string below is copied from the code that produces it, not paraphrased, so it’s safe to grep the same text against your own logs.
| Symptom | Diagnosis | Fix |
|---|---|---|
Every API call answers 401 with {"error":{"code":"unauthorized","message":"a bearer token is required"}} | No Authorization header, a malformed one, or a token that doesn’t match any row of [auth]. Checked by require_bearer (crates/blockwatcher-api/src/auth/mod.rs). GET /health is the exemption. | Send Authorization: Bearer <token>. Confirm the env:NAME variable that row’s secret names is set in the process’s environment: it’s read at request time, not cached, so a variable exported after boot still works, but an unset one fails every request identically. |
Every mutating API call answers 403 | The token authenticated but its scope is below that MethodRouter’s minimum (read cannot PUT a sink). Distinct from 401. | Use a token whose scope is at least operate or admin as the route table requires. |
A resource write answers 422 invalid_resource naming an unknown field '…' or unknown event/function '…' | A typo in a predicate field path or a selector’s events/functions name. Predicate field lookups get an edit-distance suggestion (unknown field 'args.vlaue' — did you mean 'args.value'?', the suggestion helpers and field_error in crates/blockwatcher-expr/src/typecheck/diagnostics.rs); an unknown selector name gets the spec’s first declared name of that kind instead (kind_suggestion in crates/blockwatcher-evm/src/decoder/selector.rs). Both port errors share one message shape: unknown field '{field}'{ — did you mean '{s}'?} (crates/blockwatcher-ports/src/error.rs:89,112). | Read the suggestion: it’s often exactly right. If not, confirm the real spelling with GET /specs/{id}/schema, which reflects the spec’s compiled schema, not the raw ABI text. |
A monitor never produces a match: decoded/matched stay at 0 in GET /status’s counters | Four independent causes look identical from the outside: the monitor (or its network) is paused; the selector names the wrong kind for its source (an events selector on an evm-mempool network can never match; see Selectors); the predicate is stricter than intended and every occurrence resolves Unknown/False; or, on evm-rpc, bloom_screen has skipped eth_getLogs believing every block in a window empty (see the row below). A paused monitor is excluded from decode interest entirely, in build_monitor_set (crates/blockwatcher-core/src/compile.rs), so it shows as decoded: 0 exactly like a monitor that’s simply seeing nothing yet. | Check source.status on GET /status first: paused there is the control-plane pause view, distinct from a source genuinely having nothing to report. Confirm the selector kind matches what the network’s source module actually produces. Temporarily drop the predicate field (absent matches everything a selector decodes) and see whether matched starts climbing; if it does, the predicate was the filter. |
No matches on a network whose blocks demonstrably contain them, and blockwatcher_evm_bloom_skips_total climbing, with blockwatcher_evm_bloom_contradictions_total still at zero | bloom_screen (default true) trusts every fetched header’s logsBloom to prove absence before skipping eth_getLogs (crates/blockwatcher-evm/src/source/rpc/bloom.rs, scan.rs). A running source instance disables the screen for itself the first time a fetched window’s own log contradicts that block’s own bloom, so a skip that is still costing matches with the contradiction counter flat means either this endpoint’s blooms are wrong in a way that never contradicts a log the same endpoint has actually returned (consistently absent, consistently zeroed, or wrong on exactly the blocks this filter never happens to match), or the instance that would have caught it has since restarted and is screening again from a clean slate; see the row below either way. | Set bloom_screen = false for that network (crates/blockwatcher-evm/src/source/rpc/config.rs) and confirm the matches appear. |
blockwatcher_evm_bloom_contradictions_total is nonzero for a network | An endpoint returned a log whose own address or topic0 was not admitted by that log’s own block’s bloom, proof that endpoint’s blooms do not describe its own logs (crates/blockwatcher-evm/src/source/rpc/bloom.rs’s admits_log, called from scan.rs). The source instance that observed it has already disabled bloom_screen for itself, logged once as evm-rpc: endpoint's logs bloom does not describe its own returned logs; disabling bloom screening against this endpoint for the lifetime of this source instance, which a restart does not preserve; set bloom_screen = false for this network to keep screening off across a restart too. That disabling belongs to the running instance, not the endpoint or the process: a pipeline restart (the supervisor recovering an exited source, a proven-reorg invalidation, or a monitor change that escalates to a restart) rebuilds the source and resumes screening against the same endpoint. The contradiction also only proves the blooms are wrong, not since when; every window this source screened earlier in the same run rested on the same untrustworthy blooms, and the checkpoint has already advanced past them. | Set bloom_screen = false for that network (crates/blockwatcher-evm/src/source/rpc/config.rs) so the setting, not the latch, is what survives a restart. Then read blockwatcher_evm_bloom_skips_total for the same pipeline to see how many windows were screened before the trip, and re-scan that range (POST /monitors/{id}/test’s fetch mode never screens) to recover whatever those windows may have missed; nothing rewinds it automatically. |
The same match (same content, same or, for evm-mempool, a different id) is delivered again after a restart | Expected: this is at-least-once delivery, not a bug. The checkpoint had not yet advanced past that event when the process stopped, so the restart re-decodes and re-dispatches it. | Deduplicate on the match’s id (deterministic given the same network, monitor, and decoded content, via MatchId::derive: crates/blockwatcher-types/src/id.rs); for evm-mempool specifically, deduplicate on the transaction’s own hash field instead, since that source’s id is not stable across a restart. |
dead_lettered climbs in GET /status’s counters, and GET /networks/{id}/dead-letters keeps growing | The sink is down, misconfigured, or permanently rejecting deliveries. Every entry’s reason string leads with its ErrorClass rendering (transient, permanent, rate_limited, or retry_narrower) followed by the underlying message, built as format!("{class}: {message}") (crates/blockwatcher-core/src/pipeline/sink_worker.rs), e.g. permanent: webhook host returned 500 five times. A letter whose payload has "type":"retracted" cannot be replayed. | Read the reason field to tell “sink refused this outright” from “we ran out of retry budget.” Fix the sink (a bad url_secret, an endpoint that’s actually down) or its config, then POST /networks/{id}/dead-letters/{match_id}/replay for a Match payload: success deletes the letter (204); another exhaustion bumps attempts/reason again and answers 502 replay_failed rather than dropping it. Retract letters are refused with "dead letter payload is a retraction; only match events can be replayed". |
Logs show invalidate cursor is older than the retained journal window; some match ids cannot be retracted and blockwatcher_journal_gap_total ticks | A deep invalidate rewound past what journal_depth still retained, so some already-delivered match ids have no Retracted event. Rewind and restart still happen (crates/blockwatcher-core/src/engine/invalidate.rs). | Raise [engine].journal_depth well above the source’s confirmation window (default 1024). Consumers must handle ids older than the window as an operator problem; the gap is never silent. |
A network stops advancing after a source invalidate, in_flight_events stays up, no replacement matches, and blockwatcher_source_invalidation_failures_total ticks | An unrecovered retract: the retract pass failed or dead-lettered without completing, so the checkpoint is left unrewound and the source is not restarted (handle_invalidation, crates/blockwatcher-core/src/engine/invalidate.rs). This is not a crash-restart failure (blockwatcher_pipeline_source_restart_failures_total stays at zero). | Inspect GET /networks/{id}/dead-letters for type: retracted payloads, fix the sink, and intervene; the supervisor will not retry this control path on its own the way it retries a crashed source. |
An evm-mempool network shows source.status: live, but nothing is ever delivered | The subscription itself succeeded: Live/Degraded/CatchingUp/Starting (crates/blockwatcher-ports/src/source.rs) only tracks the WebSocket connection’s own health, via dial_failed, stream_closed, transport_error, or idle_timeout (crates/blockwatcher-evm/src/source/mempool/run.rs), not whether the provider actually forwards newPendingTransactions notifications at any useful rate. Some providers, especially free or shared tiers, throttle or silently limit this feed even while answering every other RPC call normally; there is no distinct error for this, because from blockwatcher’s side nothing has failed. | Confirm the provider’s plan actually supports mempool streaming (check its docs, or watch whether any hashes arrive by temporarily widening the monitor’s functions list). On a genuinely idle testnet, no one may have called the watched function recently at all; that’s not a provider problem, just a quiet mempool. |
blockwatcher check <dir> exits 1 instead of printing ok: N networks, N specs, N sinks, N monitors | check constructs every module the seed references (crates/blockwatcher/src/check.rs), so it fails on the same things a real boot would: an unregistered module name (unknown {family} module '{name}'; available: […], crates/blockwatcher-core/src/error.rs:9-14), an unresolvable reference, an uncompilable predicate or selector, or a secret an env:NAME reference names that isn’t set in check’s own environment. | Read the printed message: it names the offending record or module, never a value that might be a secret. Export every variable a seed’s url_secret references name before running check, exactly as they’d need to be set for a real boot. |
Monitor write 422 gate requires 'block.timestamp' | Selectors’ schemas have no block.timestamp (mempool-only decode, or a spec that never put the header in the tree) | Drop the gate, or select an event/source whose decoder exposes that path. Do not expect the engine to use wall time. |
| Three Transfers, no digest | Hits span more than window_ms in block time; or count not yet reached; or untimestamped counter is up | Inspect timestamps in the decoded event, not wall clock. Check blockwatcher_gate_hits_total vs gate_emits_total. |
| Second burst never fires after the first digest | Expecting a sliding hour that keeps T1; threshold uses session reset | After fire, T1–T3 are gone; T4–T6 are a new session. That is the specified behaviour. |
max_once later hits not in dead letters | Discard is not throttle | Expected. Use throttle if you need replay. |
| After a 2-block reorg, threshold under-fires | Journal was drain-all (bug) | Holds with cursor ≤ from must survive. File a bug; replay cannot reconstruct them. |
Boot refuses immediately with binding the api listener on 127.0.0.1:8080: Address already in use (os error 48) (or the same for metrics) | Something else (often a previous blockwatcher process that didn’t exit cleanly) already holds that port. Binding happens before the engine starts specifically so this surfaces as a boot failure rather than something a running process discovers later (crates/blockwatcher/src/run.rs:34-41,265-269). | Find and stop whatever holds the port (lsof -i :8080 or equivalent), or change [api].listen/[metrics].listen (or their BLOCKWATCHER_API__LISTEN/BLOCKWATCHER_METRICS__LISTEN overrides) to a free address. |
Reading the exit code
A process that already ran and stopped tells you more than any log line
if you check its exit code, from the same table as the CLI’s own
--help text (crates/blockwatcher/src/cli.rs):
| Code | Meaning |
|---|---|
0 | Clean drain, or a check that passed. |
1 | Config, seed, or boot failure, or a check that refused. |
2 | Shutdown aborted at least one pipeline at the drain deadline. |
64 | The command line itself didn’t parse (unrecognized argument '…', a flag missing its value, etc.). |
See Configuration reference § CLI flags for the full command grammar these codes apply to.
When there’s nothing wrong
One thing that looks like a problem and is actually the system working as designed:
RUST_LOGunset still prints boot lines, restart notices, and shutdown reports. It defaults toinfo, not silence, ininit_tracing(crates/blockwatcher/src/lib.rs): every diagnostic goes to stderr, never stdout, so it never mixes into match output on a process running with no[api]configured at all.