blockwatcher

blockwatcher runs as a long-lived service that operators reconfigure entirely through its REST API: adding networks, contract specs, or delivery targets never requires stopping the process. Under the hood, it turns incoming on-chain data into typed values using the contract specifications you register, then runs your own conditions against that data to decide what counts as a match. Because none of this logic is chain-specific, chain support comes from swappable modules rather than the core itself, for example Ethereum and other EVM chains (the EVM module family ships with blockwatcher; other chain families plug in as modules without touching the core). It hands off each match to whatever endpoint, script, or log file you’ve wired up.

How to read this wiki

The whole system in one picture

flowchart LR
    rpc["RPC endpoints<br/>(external chains)"]
    sources["Sources<br/>evm-rpc · evm-mempool"]
    sinks["Sinks<br/>webhook · script · log"]
    storage["Storage<br/>checkpoints · dead letters · resources"]
    api["REST API"]
    metrics["Metrics"]
    engine["engine<br/>bounded channels · checkpoints"]

    subgraph pipeline["Engine pipeline"]
        direction LR
        decoder["Decoder"]
        matcher["Matcher<br/>predicates"]
        gate["Gate<br/>threshold · max_once"]
        decoder --> matcher
        matcher --> gate
    end

    rpc --> sources
    sources -->|"decode and match"| decoder
    gate --> sinks
    api -->|"manages resources"| storage
    storage <--> engine
    engine -->|"drives"| pipeline
    engine -.->|"reports"| metrics

    classDef module fill:none,stroke:#a9a3e3
    classDef core fill:none,stroke:#8a8d86,stroke-dasharray: 5 5

    class rpc,sources,sinks,decoder,matcher,gate module
    class engine,api,storage,metrics core
click sources "concepts/selectors.html"
click decoder "concepts/chain-agnosticism.html"
click matcher "concepts/predicates.html"
click gate "concepts/gates.html"
click sinks "concepts/delivery.html"
click storage "concepts/resources.html"
click api "reference/http-api.html"
click metrics "reference/observability.html"
click engine "concepts/pipeline.html"

Local development

Build and serve the wiki:

cargo install mdbook mdbook-mermaid --locked
mdbook serve docs/wiki --open

Release notes

Consumer-visible changes are tracked in the root CHANGELOG.md.

Understanding blockwatcher

This part answers what blockwatcher is before any of it needs to run: the problem it exists to solve, the shape of the pipeline it builds for you, and the vocabulary the rest of the wiki assumes you already have. Read it first if you are deciding whether blockwatcher fits your problem, or if a later page uses a term you have not seen defined yet.

• What is blockwatcher?: the problem blockwatcher solves and how it differs from a bespoke listener script.
• How it works in one picture: the pipeline every network runs, traced step by step.
• Glossary: every term the wiki uses, defined against the actual code.

What is blockwatcher?

Smart contracts emit events and receive calls constantly: a transfer, an approval, a call to an administrative function, a change of ownership. A team that cares about a handful of these conditions is usually left with two bad options: poll a block explorer’s API on a timer and hope its rate limits and uptime cooperate, or write and operate a bespoke script against an RPC endpoint for every condition, one script per rule, each with its own crash handling and no shared guarantees. Neither approach scales past a few rules, and both leave a team owning infrastructure that has nothing to do with the business rule they actually care about.

blockwatcher is a long-running service built to take that watching-and-deciding work off a team’s hands, for any number of rules across any number of chains it has a module for.

What blockwatcher does

At the center of an blockwatcher deployment sit four kinds of resource, and configuring all four is what makes something happen. A network tells blockwatcher where to look: a chain and the RPC endpoint(s) to pull activity from. A contract spec tells it what a piece of on-chain data means (an ABI, compiled once into a schema the rest of the system reasons about from then on). A monitor ties a network to a spec, names which addresses and which events or functions to pay attention to via its selectors, and optionally narrows that further with a predicate, a condition over the decoded fields. An optional gate can require several such hits, or cap alerts, before anything is delivered. A sink says where a hit should go.

Wire all four together and the loop needs no further attention from you: blockwatcher keeps ingesting from the network and checking every new occurrence against every active monitor, dispatching whatever counts as a match; see How it works for that loop traced step by step. None of this needs a process restart to change: every one of the four resource kinds above is created, edited, and deleted through a REST API while blockwatcher keeps running, and an edit takes effect on the running pipeline within the same request.

What blockwatcher is not

blockwatcher doesn’t keep a queryable history. It has no store of past chain state you can run open-ended historical questions against: it reacts to activity as it arrives (or, for a one-off test run, over a bounded range you specify through the API), and what it retains about anything it has already processed is limited to its own operational bookkeeping: checkpoints, dead letters, counters.

It also isn’t a place to build dashboards or discover trends. There is no aggregation layer, no time-series rollup, no chart: a match either reaches a destination you control or ends up in blockwatcher’s dead-letter store; anything you do with the resulting stream happens downstream, in whatever receives it.

And it doesn’t participate in a chain. blockwatcher holds no key, has no consensus role, and issues no transactions of its own: it only reads, through whatever RPC endpoint a network names, and its own reliability is bounded by that endpoint’s.

The four promises

No silent gaps. A slow sink never causes blockwatcher to drop anything to keep up: the pipeline holds events in its internal queues under backpressure rather than discarding work, so a struggling delivery target slows the whole pipeline down instead of losing data out of it. Recovering from a crash follows the same rule: blockwatcher resumes exactly where its last persisted checkpoint says it left off, which can mean re-sending a match it had already delivered but never means skipping one it hadn’t reached yet. The one place this promise does not hold is the evm-mempool source: a pending transaction has no fixed chain position, only an arrival order, so a restart there can genuinely lose whatever was mid-flight. Full mechanics: Delivery guarantees.

Bad configuration never fires silently. The failure mode this closes is specific: a monitor that looks correctly wired, runs for months, and never once fires because one field name has a typo three levels deep in a predicate: nothing about a monitor like that announces its own mistake. blockwatcher forecloses it by parsing, type-checking, and compiling every predicate and selector against the schema its monitor actually exposes the moment it’s written, rejecting anything it can’t resolve and naming its best guess at the field you meant. Full mechanics: Predicates and the expression language.

No chain knowledge leaks into the core. Every chain family, however it encodes its own data, normalizes into the same canonical value model before anything past the decoder ever touches it (arbitrary-precision integers included, so a 256-bit token amount survives decode to delivery without losing precision). That boundary is enforced, not just assumed: the engine that runs pipelines, tracks checkpoints, and serves the API never imports a chain SDK, an HTTP client, or a storage driver, and a CI gate breaks the build the moment a core crate tries. Full mechanics: Chain-agnosticism.

Swap any stage without touching the rest. How blockwatcher reads a chain, decodes its data, evaluates conditions, gates hits, delivers matches, and persists state are six independent choices, each an interchangeable implementation behind a small port trait, selected in config by name. Only EVM chain modules ship with blockwatcher, but the boundary is what will let a new chain family, or a new delivery target, arrive as its own module without the engine changing to accommodate it. Full mechanics: Modules and trade-offs.

How it works in one picture

Every network you register runs its own pipeline: a chain-specific source feeding a chain-agnostic decode-and-match stage, fanning out to whichever sinks the matching monitors name. A separate engine wires those pipelines up from resources managed over the REST API, and owns the bounded queues and checkpoint bookkeeping that make delivery safe to resume after a restart.

flowchart LR
    subgraph pipeline["one pipeline per network"]
        direction LR
        rpc("RPC source"):::module
        mem("Mempool source"):::module
        dec("Decoder"):::module
        mat("Matcher"):::module
        gate("Gate"):::core
        sw("Sink worker<br/>throttle · aggregate"):::core
        wh("Webhook sink"):::module
        scr("Script sink"):::module
        lg("Log sink"):::module
        rpc --> dec
        mem --> dec
        dec --> mat
        mat --> gate
        gate --> sw
        sw --> wh
        sw --> scr
        sw --> lg
    end

    api("REST API"):::core -->|writes resources| eng
    sto("Storage port<br/>checkpoints · dead letters"):::module <--> eng
    eng("engine<br/>bounded channels · checkpoint tracker"):::core --> pipeline
    eng -. interest hints .-> rpc
    eng -. interest hints .-> mem

    classDef module fill:none,stroke:#a9a3e3
    classDef core fill:none,stroke:#8a8d86,stroke-dasharray: 5 5

Boxes drawn with a solid outline are swappable modules: an operator picks evm-rpc or evm-mempool for the source, webhook, script, or log for each sink, and so on. The engine and the REST API, drawn with a dashed outline, are fixed: every pipeline is built the same way regardless of which modules fill its module slots.

One event’s journey

1. A source pulls activity. The RPC source (module name evm-rpc) polls its configured endpoints for new blocks and the logs and transactions in them; the Mempool source (evm-mempool) instead subscribes to one node’s pending-transaction feed. Either way, what reaches the next stage is a raw, chain-native payload tagged with a cursor, the source’s own position marker.

2. The decoder normalizes it. Decoder takes that raw payload and the compiled schema from the relevant contract spec, and turns it into blockwatcher’s canonical value model, the same shape of data regardless of which chain family produced it. Nothing past this point knows or cares that the source was EVM-specific.

3. The matcher evaluates every monitor. For each of the network’s active monitors whose selectors apply to this occurrence, Matcher runs the monitor’s compiled predicate (if it has one) against the decoded fields. A monitor with no predicate accepts everything its selectors already narrowed down to.

4. A gate may stay quiet. If the monitor names a gate, the engine offers this hit and the persisted journal to that module. Retain or Discard mints nothing and does not stall the checkpoint. Emit mints a match or digest and continues as today.

5. A match is fanned out. Every monitor that accepted the occurrence (Emit / passthrough) produces one match with a deterministic id, and the engine routes it to a Sink worker for every sink that monitor’s actions name. A sink with no throttle or aggregate configured delivers that match immediately; a sink with one of those set instead holds it in a window, dropping or bundling matches per the policy, and eventually delivers a plain match or a digest of everything the window collected. Either way delivery is one attempt per sink, independently retried.

6. The checkpoint waits for the whole event. The engine only advances the network’s checkpoint past this event once every match it produced (across every monitor and every sink) has either been delivered or exhausted its retries and become a dead letter. Quiet gate hits are already done (outstanding 0). An event that produced no matches at all completes immediately; an event with five dispatched deliveries waits for all five.

If the process crashes anywhere in that sequence, it resumes on restart from the last checkpoint that step 6 actually persisted. Anything from an event at or before that checkpoint is done: matches were delivered or dead-lettered before the checkpoint advanced. Anything after it is redone from scratch, meaning a match already delivered right before the crash can be delivered again, but nothing already accepted onto a queue is ever skipped. A duplicate is possible; a gap is not.

Glossary

Every term below is defined by how blockwatcher’s code actually behaves, not by a general blockchain-industry sense of the word. Terms are alphabetical; each links to the others it depends on.

At-least-once delivery

blockwatcher’s delivery guarantee for a source whose cursor is a real chain position: a network’s checkpoint advances past an event only once every sink event that event produced has either reached its sink or been recorded as a dead letter after exhausting retries, never while a delivery is still outstanding. Quiet gate hits are not deliveries. An emitted match after a gate is still at-least-once (replay can duplicate). Both Match and Retracted are at-least-once; a consumer must treat a redelivered duplicate as the same occurrence (idempotent on match_id). Because the checkpoint is what a restart resumes from, the only way delivery can go wrong across a crash is repeating work already sent, never dropping work that was never sent. evm-mempool is the one shipped source this does not cover: its cursor is a per-run arrival counter rather than a chain position, so a crash there can lose whatever was in flight, and consumers should deduplicate its matches on the transaction hash rather than the match id.

Backpressure

What happens instead of dropping data when one pipeline stage runs behind another. A pipeline’s stages (source, decode-and-match, per-sink delivery) are connected by fixed-size queues; a stage that produces faster than the next one consumes simply waits for room in that queue before it can hand off more work. A source therefore never sees “buffer full, discard”: it sees its own send call take longer, which is the entire backpressure mechanism.

Canonical value model

The one representation every decoder normalizes chain-native data into, and the only representation a matcher, gate, or sink ever has to understand. It covers null, boolean, arbitrary-precision signed and unsigned integers, byte strings, addresses, strings, arrays, and ordered maps. Arbitrary-precision integers are why a 256-bit token amount compares and serializes exactly rather than losing precision the way a JSON number would; each chain family’s decoder is responsible for encoding its own values into this model without two distinct native values ever colliding on the same canonical one.

Checkpoint

The resume point a network’s pipeline persists: a cursor plus whatever extra state the source that wrote it needs to verify a safe resume (a block hash for reorg linkage, for instance). It advances only past a fully finished prefix of events: one still-outstanding sink event anywhere in that prefix holds the whole checkpoint back, even if later events already finished. A positively detected invalidate is the one case that rewinds it, and only backward to the proven from cursor, only after retracts have finished; a rewind that cannot move the cursor backward (the invalidation’s cursor is not behind the stored checkpoint, no checkpoint is stored, or the post-drain read failed) is refused and counted as blockwatcher_rewinds_refused_total, leaving the stored checkpoint unmoved and an absent one absent rather than fabricated. A gate Retain/Discard does not leave outstanding work; only an emitted match/digest does. Held hits persist in gate_hits without holding the checkpoint. Every checkpoint also records which source module wrote it, and resuming it under a different module is refused outright, because a cursor’s two numbers mean nothing outside the module that produced them.

Contract spec

A resource pairing a chain identifier with that chain’s decode artifact (for the evm chain family, a Solidity ABI) as an opaque payload. Writing a spec triggers its chain’s decoder to compile that artifact once into chain-agnostic event and function schemas; every selector and predicate that references the spec afterward works from those compiled schemas, never from the raw payload again. A spec has no module field of its own: its chain field alone determines which decoder compiles it.

Cursor

A pair of numbers a source uses to mark its own position in its feed, meaningful only inside the module that wrote them. The evm-rpc source uses the first number as a block number and packs the second to order a block’s transactions ahead of its logs; evm-mempool uses the first number as a monotonic arrival count and leaves the second at zero. Core code only ever compares and orders cursors: it never interprets what the numbers mean.

Dead letter

The record blockwatcher keeps of a sink event that could not be delivered after its sink’s retry budget ran out. It carries the match’s id, which monitor and sink produced it, the cursor it traces back to, how many attempts were made, why the last one failed, and, for events produced after this field was added, the original SinkEvent payload, which lets a Match or a Digest be replayed through the API. Legacy rows stored as a bare Match object still load as SinkEvent::Match. A retraction payload cannot be replayed. Recording a dead letter, rather than discarding the event outright, is what lets the checkpoint move past work that genuinely could not be delivered without pretending nothing happened.

Decoder

The port that turns a source’s raw payload into the canonical value model, one implementation per chain family. Nothing about the port boundary limits a deployment to one chain family; evm is simply the only one in the shipped module catalog today. A decoder compiles a contract spec’s payload into reusable schemas once, at write time, and decodes every later occurrence against those schemas rather than re-parsing the original artifact each time.

Delivery journal

The bounded per-network log of Match ids that were delivered or dead-lettered, used to emit Retracted events after an invalidate. Depth is instance config journal_depth (default 1024, in cursor primary units). Rows older than that window are self-pruned on the next write; there is no operator prune API. A successful retract forgets its row; a dead-lettered retract keeps it so a later invalidate can re-offer. See Delivery guarantees § The delivery journal.

Gate

A port that decides, for one monitor, whether a decoded occurrence that already passed selector and predicate becomes a match (or digest), given an engine-owned journal of earlier hits. threshold and max_once key windows on block.timestamp. Omit the monitor’s gate field for passthrough. See Gates. Distinct from sink throttle / aggregate, which run after a match exists.

Invalidate

A typed source outcome, SourceOutcome::Invalidated { from }, meaning everything with cursor > from previously implied by this source is on a dead fork and must not be resumed past from. It is not a SourceError. from is a cursor the source can defend with evidence: a proven fork point, or, when every tracked ancestor is refuted, just below the oldest tracked height (the minimal rewind that evidence permits). A source that holds no such evidence does not return this outcome. Core never names a chain or a reorg; only the source module decides when to return this. The engine drains that network, prunes gate_hits with cursor > from (rows with cursor ≤ from stay), retracts journaled deliveries after from (to that network’s sinks, or after delete to the sinks recorded on its runtime tombstone), rewinds the checkpoint when from is behind it (leaving it unmoved otherwise), then restarts. See Delivery guarantees § Shallow vs deep invalidation.

Match

One occurrence where a monitor’s selectors picked up a decoded event and its predicate, if it has one, accepted it. A match carries a deterministic id derived from the network, the monitor, the decoded event, and an index among the matches that one event produced, so a consumer that receives the same match twice under at-least-once delivery can recognize the duplicate by id, except from evm-mempool, where the same pending transaction can carry two different ids across a restart because its cursor is not a stable position.

Matcher

The port that evaluates a compiled predicate against a decoded event and reports whether it counts as a match. The shipped implementation, expr, is selected once for the whole instance in blockwatcher.toml rather than per monitor, and does not become per-monitor because Gate exists. The port boundary means an alternative predicate engine is a possible module, not a hardcoded choice.

Mempool

The set of transactions a node has received but has not yet included in a mined block. blockwatcher’s evm-mempool source subscribes to one node’s pending-transaction feed and fetches each transaction’s full data as its hash arrives. Because nothing here has settled yet, a pending transaction may never be mined at all: the trade-off this source makes for seeing activity before evm-rpc ever could.

Module

Any of the six swappable behaviors blockwatcher selects by name plus a module-specific config object: source, decoder, matcher, gate, sink, and storage backend. Every module implements exactly one port trait, and a module name that was never registered into the running binary is refused at write or boot time with the list of names that were.

Monitor

A resource that ties everything else together: it names one network to watch, one or more selectors describing what to decode from it, an optional predicate to filter what those selectors decode, an optional gate that decides whether a predicate-true hit becomes a match at all, and the sink ids a resulting match should reach. A monitor is compiled and schema-checked the moment it is written, and edits to it hot-swap into its running pipeline without a process restart.

Network

A resource naming a chain and a source module plus that module’s own config (where blockwatcher looks and how it ingests from there). Exactly one pipeline runs per network, and changing which source module a network uses restarts only that one pipeline, resuming from its persisted checkpoint as long as the checkpoint’s recorded module still matches.

Port

An object-safe Rust trait defining one axis of swappable behavior. blockwatcher defines six: Source, Decoder, Matcher, Gate, Sink, and Storage. Core code depends only on these trait definitions, never on any concrete module’s implementation: the reason a chain family’s code never has to be linked into, or even known by, the engine that drives it.

Predicate

A boolean expression, written in blockwatcher’s own small expression language, evaluated over a decoded event’s fields to decide whether it produces a match. It supports field access by namespace (args.* for decoded event fields; chain-specific namespaces such as tx.* and block.*), comparisons, boolean logic, arithmetic, and chain-native literals like hex addresses and token-decimal numbers. A predicate is type-checked against its monitor’s selector schemas the moment it is written, and at evaluation time a field an occurrence simply doesn’t carry resolves to an explicit “unknown” value rather than raising an error.

Seed

The one-time load of resources (networks, specs, sinks, monitors) from a directory of JSON files into empty storage, run at first boot via a CLI flag. It only ever populates a storage backend that has nothing in it yet; once a backend holds any resource, seeding again is a no-op, and afterward resources are managed exclusively through the REST API.

Selector

Part of a monitor: which addresses, which contract spec, and which of that spec’s events and/or functions to decode. A monitor’s selector entries are OR’d, and each entry is self-contained, so which spec governs which address is never ambiguous. What a given selector can actually produce depends on the network’s source: an events selector has nothing to decode on evm-mempool, which only ever delivers pending transactions, never logs.

Sink

The port responsible for delivering one sink event somewhere: a webhook, an operator-run script, or a log line are the modules in the shipped module catalog. Retry behavior on a failed delivery (how many attempts, backoff timing, when to give up and record a dead letter) is owned entirely by the engine and applied identically regardless of which sink module is delivering; a sink module itself never retries on its own. Every sink event variant shares that path. A sink whose SinkDef sets throttle or aggregate holds matches in a window before delivering, rather than delivering each one as it arrives; see Delivery guarantees § Aggregation. A gate digest is the same Digest wire shape aggregation already defined.

Sink event

What a sink receives: Match (apply this occurrence), Retracted { match_id } (undo the occurrence that id named), or Digest { matches } (apply several matches an aggregate window bundled together; see Sink). A closed set, serialized on the wire as tagged JSON (type: match, type: retracted, or type: digest). All three are at-least-once. See Delivery guarantees § What a sink receives.

Source

The port that pulls raw activity into a pipeline, whether by polling an RPC endpoint for blocks and logs or by subscribing to a node’s pending-transaction feed. A source owns the meaning of its own cursor, must emit events in non-decreasing cursor order, and is the only module family configured on the network resource rather than on a resource of its own. Source::run returns a SourceOutcome (Ended or Invalidated { from }) or a SourceError; a positively detected deep invalidate is the outcome, not the error.

Storage

The port responsible for every resource ever written (versioned, with optimistic concurrency), each network’s checkpoint, its dead-letter queue, its delivery journal, and each monitor’s gate_hits / gate_meta. Unlike the other five ports, a deployment picks exactly one storage backend for the whole process rather than one per resource: memory (no persistence, gone on restart) or sqlite (durable, single-writer) are the two in the shipped module catalog.

Getting started

This part gets blockwatcher running on your machine: building or installing it, watching one real pipeline end to end, driving that same pipeline through a browser instead of the command line, or embedding the engine in another process. It is for a reader about to run blockwatcher for the first time, not yet for one auditing its internals.

• Quickstart: one pipeline running end to end against a live testnet contract.
• Installation and building: GitHub Release archive, GHCR images, building from source, or a local Docker build.
• Your first monitor: the quickstart’s example walked more slowly, then changed.
• The dashboard: the same operations through a browser instead of the API and the CLI.
• Embedding in a host process: run the engine inside another binary, with a host-owned sink and shutdown.

Quickstart

Get one blockwatcher pipeline running end to end: watching a live testnet contract and printing every occurrence it matches as JSON. This page assumes you already have a rough sense of what a network, a monitor, and a sink are; see What is blockwatcher? first if not. This page is about getting bytes moving; Your first monitor explains each piece in more depth, and the Concepts pages explain why the pieces are shaped the way they are.

The steps below all feed one running pipeline:

flowchart LR
    seed["seed/<br/>network, spec, sink, monitor"] --> check["blockwatcher check ./seed"]
    check -->|"ok: N networks..."| run
    cfg["blockwatcher.toml"] --> run["blockwatcher --config ... --seed ..."]
    seed --> run
    run --> pipe["running pipeline"]
    pipe --> out["stdout: one match per line"]

Prerequisites

• A Rust toolchain at or above the workspace minimum, rust-version = "1.85". Tagged releases publish a Linux x86_64 archive and GHCR images (see Installation and building); building from source is the path this walkthrough uses.
• An RPC endpoint URL for an EVM testnet. The example below uses Sepolia; a free key from Infura, Alchemy, or dRPC works.

From a clone of the repository, build the release binary once:

cargo build --release -p blockwatcher

The binary lands at target/release/blockwatcher.

1. Write an instance config

blockwatcher has two separate configuration planes. Instance config (blockwatcher.toml) is read once at process boot and controls process wiring: the API listener, the storage backend, engine tunables. It is not where you say what to watch; that’s a separate plane, managed at runtime (see Configuration later in this wiki). A minimal instance config:

[api]
enabled = true
listen = "127.0.0.1:8080"

[[auth.tokens]]
label = "operator"
scope = "admin"
secret = "env:BLOCKWATCHER_API_TOKEN"

[storage]
module = "sqlite"
config = { path = "blockwatcher.db" }

Each token’s secret is a reference (env:BLOCKWATCHER_API_TOKEN), never a literal token: boot refuses, naming the variable, if it is unset. GET /health answers without a token; everything else requires one.

2. Write a seed directory

blockwatcher loads its resources (what to watch, decode, and deliver) from a directory of JSON files on first boot only. Once the store holds anything at all, seeding is skipped and the resources plane is managed exclusively through the REST API from then on. A minimal seed needs one of each resource kind, one file per resource, sorted into kind subdirectories that the loader recognizes by name:

seed/
├── networks/sepolia.json
├── specs/usdc-erc20.json
├── sinks/log-sink.json
└── monitors/usdc-sepolia-transfers.json

networks/sepolia.json: where to look and how to ingest. start_block is an absolute block number rather than head-relative, so a restart resumes from exactly this point instead of silently skipping whatever passed while the process was down:

{
  "id": "sepolia",
  "chain": "evm",
  "source": {
    "module": "evm-rpc",
    "config": {
      "start_block": 9000000,
      "endpoints": [
        { "name": "primary", "url_secret": "env:SEPOLIA_RPC_URL" }
      ]
    }
  }
}

Set start_block near the chain’s current head: a value from months ago means a long wait before the first match arrives.

specs/usdc-erc20.json: the decode artifact. For "chain": "evm" this is a Solidity ABI fragment list; the decoder compiles it once into a schema everything downstream works from:

{
  "id": "usdc-erc20",
  "chain": "evm",
  "payload": [
    {
      "type": "event",
      "name": "Transfer",
      "anonymous": false,
      "inputs": [
        { "name": "from", "type": "address", "indexed": true },
        { "name": "to", "type": "address", "indexed": true },
        { "name": "value", "type": "uint256", "indexed": false }
      ]
    }
  ]
}

sinks/log-sink.json: one JSON line per match to stdout, no configuration of its own:

{
  "id": "log-sink",
  "module": "log",
  "config": {},
  "retry": { "max_attempts": 1, "initial_backoff_ms": 100, "max_backoff_ms": 1000 }
}

monitors/usdc-sepolia-transfers.json: ties the others together: watch this address on sepolia, decode Transfer against usdc-erc20, send every match to log-sink:

{
  "id": "usdc-sepolia-transfers",
  "network": "sepolia",
  "selectors": [
    {
      "addresses": ["0x1c7D4B196Cb0C7B01d743Fbc6116a902379C7238"],
      "spec": "usdc-erc20",
      "events": ["Transfer"]
    }
  ],
  "predicate": "args.value > 0",
  "actions": ["log-sink"]
}

3. Validate offline, then run

export SEPOLIA_RPC_URL=https://sepolia.infura.io/v3/YOUR_KEY

./target/release/blockwatcher check ./seed

check takes only a directory (it never reads blockwatcher.toml), but it constructs every module the seed references, including resolving each url_secret, so SEPOLIA_RPC_URL has to be set (to a real http/https URL: the check inspects its scheme, it does not need the endpoint to be reachable). BLOCKWATCHER_API_TOKEN isn’t needed yet; nothing about check touches the instance config that reads it.

A passing check prints a one-line summary to stdout and exits 0:

ok: 1 networks, 1 specs, 1 sinks, 1 monitors

Now export the API token and run for real:

export BLOCKWATCHER_API_TOKEN=quickstart-token

./target/release/blockwatcher --config ./blockwatcher.toml --seed ./seed

--seed only loads resources into a store that has nothing in it yet; once it has run once, blockwatcher.db is authoritative and the seed directory is ignored on every later boot.

What success looks like

Startup lines and warnings go to stderr; stdout carries nothing but tagged sink-event JSON ("type": "match", "type": "retracted", or "type": "digest" for a sink configured with aggregate), one line per delivery: nothing else is ever mixed in, so a consumer piping stdout into a parser never has to filter anything out. Within a few seconds of a matching Transfer on Sepolia, a line like this appears:

{
  "type": "match",
  "id": "a3f8b21c9d...",
  "monitor": "usdc-sepolia-transfers",
  "network": "sepolia",
  "event": {
    "kind": "event",
    "name": "Transfer",
    "fields": {
      "map": {
        "args": {
          "map": {
            "from": { "address": "0xff30fb28e1794bb91d5bceb7d66b731d0c61af8e" },
            "to": { "address": "0x7a3f2b16924f0e5c8f6a1c3d9e0b5a2f8c4d7e1a" },
            "value": { "uint": "13000000" }
          }
        },
        "tx": {
          "map": {
            "hash": { "bytes": "0x5f71ab..." },
            "index": { "uint": "26" },
            "status": { "uint": "1" }
          }
        },
        "block": {
          "map": {
            "number": { "uint": "11424039" },
            "hash": { "bytes": "0x9ab2cd..." },
            "timestamp": { "uint": "1785929472" }
          }
        },
        "log": {
          "map": {
            "address": { "address": "0x1c7d4b196cb0c7b01d743fbc6116a902379c7238" },
            "index": { "uint": "54" }
          }
        }
      }
    },
    "cursor": { "primary": 11424039, "secondary": 54 }
  }
}

Every scalar is tagged with its own type (address, uint, bytes), which is why a 256-bit integer like value survives as an exact decimal string rather than a lossy JSON number. USDC has 6 decimals, so 13000000 is 13 USDC. The top-level id is deterministic: the same occurrence always derives the same id, which is what lets a consumer recognize a redelivered match after a restart as a duplicate rather than as new activity.

Next: Your first monitor walks through this same shape of deployment file by file, and shows how to narrow what it matches.

Installation and building

A vX.Y.Z tag publishes a Linux x86_64 archive to GitHub Releases and two images to GHCR (ghcr.io/thethirdorigin/blockwatcher and ghcr.io/thethirdorigin/blockwatcher-ui). Building from source, or building the Docker image locally, remain the paths for anything that archive does not cover. The tag-triggered process is in CONTRIBUTING.md.

GitHub Release archive

gh release download vX.Y.Z --pattern "blockwatcher-*.tar.gz"
tar -xzf blockwatcher-X.Y.Z-x86_64-unknown-linux-gnu.tar.gz
./blockwatcher --version

The matching images are ghcr.io/thethirdorigin/blockwatcher:X.Y.Z and ghcr.io/thethirdorigin/blockwatcher-ui:X.Y.Z (also tagged vX.Y.Z). The UI image tag is the release it shipped with, not ui/server’s own crate version. Building the same images locally is Running with Docker.

Building from source

The workspace’s minimum Rust version is 1.85. Build the whole workspace:

cargo build --workspace

or just the binary crate:

cargo build --release -p blockwatcher

The resulting binary is target/release/blockwatcher (or target/debug/blockwatcher without --release).

Every commit is expected to pass the same gates CI runs on Linux:

cargo fmt --all --check
cargo check --workspace --locked
cargo clippy --workspace --all-features --all-targets -- -D warnings
cargo test --workspace --all-features
./scripts/check-dep-graph.sh
./scripts/check-release-version.test.sh
./scripts/changelog-release-notes.test.sh

CI additionally checks that every pairwise combination of the blockwatcher binary’s feature flags still builds:

cargo hack check --workspace --feature-powerset --depth 2 --locked

Worth knowing before reaching for a custom feature combination in production: the combinations that ship are the ones that are actually tested.

Feature flags

The blockwatcher binary crate defines optional features, all on by default:

[features]
default = ["evm", "expr", "sinks"]
evm = ["blockwatcher-embed/evm"]
expr = ["blockwatcher-embed/expr"]
sinks = ["blockwatcher-embed/sinks"]

Each forwards onto the matching blockwatcher-embed feature, which is what actually links the family’s crate: a family that isn’t linked can never be selected by name in config, however the config is written. Storage (memory, sqlite) is not behind a feature: blockwatcher-storage is a plain, non-optional dependency, so both storage modules are always present.

Building a custom binary that carries only what an operator’s deployment needs (say, evm and sinks without expr, if a different matcher module were added later) uses Cargo’s usual --no-default-features --features combination:

cargo build --release -p blockwatcher --no-default-features --features evm,sinks

A binary built this way still needs some matcher configured in [engine].matcher, and boot refuses at startup if the named module isn’t one this build actually carries: the refusal names what is available, not just what was asked for.

The check command and its exit codes

check runs the same construction and compilation path a real boot would, without ever starting a pipeline:

flowchart LR
    dir["seed directory"] --> load["load every JSON file"]
    load --> construct["construct every module<br/>resolves env: secrets"]
    construct --> compile["type-check every<br/>selector and predicate"]
    compile -->|"ok"| pass["print ok: N counts<br/>exit 0"]
    compile -->|"refused"| fail["print reason to stderr<br/>exit 1"]
    construct -->|"refused"| fail

blockwatcher check <dir> validates a seed directory offline: it loads every JSON file the directory holds, then runs the exact same construction and compilation path a real boot would run (every module gets built, every selector and predicate gets type-checked against its spec’s schema) without starting a pipeline or opening a listener. It takes only the directory argument; it never reads an instance config file, so nothing in blockwatcher.toml (the API token, the storage backend) plays any part in whether a check passes.

Because check constructs every module, any secret a seed’s configs reference through env:NAME must already be present in check’s own environment: a sink whose config resolves a secret at construction fails the same way it would at boot.

A pass prints a summary to stdout and returns 0:

ok: 3 networks, 5 specs, 2 sinks, 8 monitors

A failure prints the refusal to stderr and returns 1, naming the file and the reason: a malformed resource, a duplicate id within one kind, a selector referencing a spec that isn’t in the seed, a url_secret pointing at an unset variable.

The full set of exit codes the binary can return, useful for a supervisor’s restart policy:

check itself only ever returns 0 or 1: codes 2 and 64 belong to blockwatcher’s normal run mode and to argument parsing, respectively, neither of which check goes through.

Docker

docker/Dockerfile.blockwatcher builds the binary in a rust:1.88-bookworm stage (cargo build --release -p blockwatcher, with the default feature set: the Dockerfile passes no --features flags) and copies only the resulting binary into a debian:bookworm-slim runtime stage alongside ca-certificates and curl (the latter for the container healthcheck). The entrypoint is the binary itself, so container arguments are blockwatcher’s own CLI arguments:

docker build -f docker/Dockerfile.blockwatcher -t blockwatcher .
docker run --rm -p 127.0.0.1:8080:8080 \
  -e BLOCKWATCHER_API_TOKEN=change-me -e SEPOLIA_RPC_URL=https://sepolia.infura.io/v3/YOUR_KEY \
  -v "$(pwd)/blockwatcher.toml:/etc/blockwatcher/blockwatcher.toml:ro" \
  blockwatcher --config /etc/blockwatcher/blockwatcher.toml

(SEPOLIA_RPC_URL is only needed if the instance’s seeded or API-created networks reference it via url_secret: the binary itself doesn’t know that name, only whichever resources you give it do.)

For a full local stack (blockwatcher plus the companion dashboard), docker compose from the repository root is the supported path; see The dashboard.

Your first monitor

Quickstart got a pipeline running from files typed by hand. This page walks the same ground more slowly, using the runnable example shipped in the repository at examples/source-rpc-monitor/, and ends by changing what the monitor matches and watching the behavior change. The Concepts pages explain why each resource is shaped the way it is; this one is about doing it once, end to end.

What’s in the example

Eight tracked files (.env isn’t one of them: you create it from .env.example in a moment, and it’s git-ignored so your key never gets committed):

examples/source-rpc-monitor/
├── .env.example
├── README.md
├── blockwatcher.toml
├── setup.sh
└── resources/
    ├── monitors/usdc-sepolia-transfers.json
    ├── networks/sepolia.json
    ├── sinks/log-sink.json
    └── specs/usdc-erc20.json

Two of these aren’t blockwatcher resources at all: blockwatcher.toml is the instance config covered in Quickstart, and setup.sh is a one-off helper this page runs below: neither is read by check or seeded into storage. The other four are exactly the seed directory’s four resource kinds, one file each. Below, they’re discussed in the order they reference each other (network, then spec, then sink, then the monitor that ties all three together) rather than the tree’s alphabetical order.

The four files wire together into one running pipeline like this:

flowchart LR
    net["network<br/>sepolia.json"] --> mon
    spec["spec<br/>usdc-erc20.json"] --> mon
    sink["sink<br/>log-sink.json"] --> mon
    mon["monitor<br/>usdc-sepolia-transfers.json"] --> pipe["running pipeline<br/>on sepolia"]
    net --> pipe

resources/networks/sepolia.json names the evm-rpc source and tunes it beyond the defaults: 12 confirmations before a block is trusted, a starting eth_getLogs window of 1000 blocks growing to 5000, and a 30-second head-probe interval. None of these are required fields (the quickstart network file left every one of them at its default and still worked), but they’re realistic values for a testnet feed, not just illustration.

resources/specs/usdc-erc20.json is a Solidity ABI fragment list covering both Transfer and Approval events. Only Transfer is selected by the monitor below, but nothing stops a spec from carrying more than one monitor ever asks it to decode: the fragments a monitor doesn’t reference are simply unused.

resources/sinks/log-sink.json selects the log sink with a max_attempts of 1: one delivery attempt, dead-letter on failure, no retries. That’s a reasonable choice for a sink with nothing to fail on other than a broken stdout pipe.

resources/monitors/usdc-sepolia-transfers.json is the monitor: it watches that one address on the sepolia network, decodes only Transfer against usdc-erc20, keeps every occurrence where args.value > 0 (which is to say, everything a Transfer event can carry: a zero-value transfer is legal but rare), and sends every match to log-sink.

Running it

From the repository root:

cd examples/source-rpc-monitor
cp .env.example .env

Edit .env and replace YOUR_API_KEY in SEPOLIA_RPC_URL with a real Sepolia endpoint (a free key from Infura, Alchemy, or dRPC works). The key never gets written into any resource file: the network resource only names the SEPOLIA_RPC_URL variable, and blockwatcher resolves it from its own environment at construction time.

./setup.sh
set -a; . ./.env; set +a

setup.sh reads the current chain head over eth_blockNumber and rewrites resources/networks/sepolia.json’s start_block to a value just behind it. start_block is deliberately absolute, with no default: a head-relative start would derive a different block on every restart and silently skip whatever passed in between, which a monitor must never do. The trade-off is that a fresh network needs a block number named for it, and naming one from months ago means a long catching_up wait, visible on the GET /status endpoint, before anything is delivered.

cargo run --bin blockwatcher -- check ./resources
cargo run --bin blockwatcher -- --config ./blockwatcher.toml --seed ./resources

check constructs every module the seed references (including the evm-rpc source, which resolves SEPOLIA_RPC_URL), so it only passes with the environment loaded. A pass prints:

ok: 1 networks, 1 specs, 1 sinks, 1 monitors

The run command boots the engine, seeds resources/ into the empty blockwatcher.db it just created, and starts the pipeline. Diagnostics (the boot line, warnings) go to stderr; stdout carries nothing but match JSON, one line per delivery. Within a few seconds, matching transfers start arriving:

{"id":"a3f8b21c9d...","monitor":"usdc-sepolia-transfers","network":"sepolia","event":{"kind":"event","name":"Transfer","fields":{"map":{"args":{"map":{"from":{"address":"0xff30fb28e1794bb91d5bceb7d66b731d0c61af8e"},"to":{"address":"0x7a3f2b16924f0e5c8f6a1c3d9e0b5a2f8c4d7e1a"},"value":{"uint":"20000000"}}},"tx":{"map":{"hash":{"bytes":"0x5f71ab..."},"index":{"uint":"26"},"status":{"uint":"1"}}},"block":{"map":{"number":{"uint":"11424039"},"hash":{"bytes":"0x9ab2cd..."},"timestamp":{"uint":"1785929472"}}},"log":{"map":{"address":{"address":"0x1c7d4b196cb0c7b01d743fbc6116a902379c7238"},"index":{"uint":"54"}}}}},"cursor":{"primary":11424039,"secondary":54}}}

Pipe through jq to read it comfortably, or reduce to one line per transfer:

cargo run --bin blockwatcher -- --config ./blockwatcher.toml --seed ./resources \
  | jq -r '.event.fields.map.args.map | "\(.from.address) → \(.to.address)  \(.value.uint)"'

Narrowing what it matches

The seeded monitor’s predicate, args.value > 0, accepts every nonzero transfer. Suppose you only care about large ones, say, anything over 1000 USDC (six decimals: 1_000e6). The monitor is already running and its store already holds it, so re-seeding won’t touch it: --seed only loads into a store that has nothing in it yet. The way to change a live monitor is the same way you’d change any resource (the REST API), and the change takes effect on the running pipeline immediately, no restart.

Get the monitor’s current version (ETag), then PUT it back with the predicate changed:

export TOKEN=$BLOCKWATCHER_API_TOKEN   # from .env

etag=$(curl -s -o /dev/null -w '%header{etag}' \
  localhost:8080/monitors/usdc-sepolia-transfers \
  -H "Authorization: Bearer $TOKEN")

curl -s -X PUT localhost:8080/monitors/usdc-sepolia-transfers \
  -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
  -H "If-Match: $etag" \
  -d '{
    "id": "usdc-sepolia-transfers",
    "network": "sepolia",
    "selectors": [{
      "addresses": ["0x1c7D4B196Cb0C7B01d743Fbc6116a902379C7238"],
      "spec": "usdc-erc20",
      "events": ["Transfer"]
    }],
    "predicate": "args.value > 1_000e6",
    "actions": ["log-sink"]
  }'

If-Match carries the resource’s current version so the API can refuse a write that would clobber a change made since you last read it; omitting it on a PUT to a resource that already exists returns 409 Conflict (already_exists), and a stale If-Match value returns 412 Precondition Failed (version_conflict). 1_000e6 is blockwatcher’s token-decimal literal syntax (one thousand at six decimals), evaluated against args.value as an arbitrary-precision integer, so the comparison is exact even though value itself arrived as a decimal string, not a JSON number.

The behavior change is immediate: transfers under 1000 USDC that used to print a line stop appearing entirely, while nothing about the pipeline restarts or re-scans: the predicate is recompiled and hot-swapped into the running monitor set, and only events decoded from here on are evaluated against the new one. Widen the predicate back, or remove it entirely (an absent predicate matches everything the selector decodes), the same way: another PUT with a fresh If-Match.

Next: The dashboard covers a web UI that does the same resource CRUD and log-watching this page just did by hand, and Concepts covers what the predicate language can express beyond a threshold. To require N transfers in a block-time window before alerting, see Gates.

The dashboard

Everything so far has driven blockwatcher by hand: editing JSON files, calling the REST API with curl. blockwatcher ships an optional local web UI that does the same operations (resource CRUD, watching matches arrive) through a browser instead.

What it shows

• A monitors fleet table: every monitor, active or paused, with a live summary of its selectors and uptime.
• Per-monitor stats: pause/resume, a dry-run against the live decoder and matcher (either raw test payloads or a bounded history scan), and stat cards for the monitor’s counters.
• A virtualised match/event log per monitor, built to stay responsive even once a monitor has produced a large number of matches.
• CRUD for every resource kind: networks, specs, sinks, and monitors, as either a structured form (for monitors, a selector editor with a schema helper) or raw JSON.
• A networks page with per-network operational state: source status, lag, queue depth, cursor, pause/resume, skip-to-tip, and dead letters.

Two services, not one

The dashboard is not a feature you turn on inside the blockwatcher binary: it’s a second, independent process that sits in front of it. blockwatcher itself keeps running exactly as described everywhere else in this wiki: same REST API, same pipelines, same SQLite store, blockwatcher.db. What changes is that nothing outside its own container ever talks to it directly: its listen port isn’t published on the host at all. A second binary, blockwatcher-ui-server, built from Rust (Axum) and serving a compiled React single-page app, is the only thing that binds a host-reachable port (127.0.0.1:8080), and it holds its own SQLite database, ui.db, entirely separate from blockwatcher’s.

That server does two unrelated jobs behind the one port: it forwards every API call the browser makes through to the real blockwatcher API and relays the answer back, and it exposes an ingest endpoint that blockwatcher itself calls into to hand over matches. The first job is why a browser never needs to know blockwatcher’s address; the second is covered next.

flowchart LR
    browser["Browser"]

    subgraph ui_svc["ui service (127.0.0.1:8080, host-reachable)"]
        spa["React SPA"]
        srv["Companion server (Axum)"]
        uidb[("ui.db")]
    end

    subgraph ob_svc["blockwatcher service (internal only)"]
        api["REST API"]
        eng["Engine · pipelines"]
        obdb[("blockwatcher.db")]
    end

    browser --> spa
    spa --> srv
    srv -->|"proxy /api/blockwatcher/..."| api
    api --- eng
    eng -->|"webhook POST /ingest"| srv
    srv --> uidb
    eng --> obdb

Two consequences follow from that shape:

• Every API call the browser makes is really two hops. GET /api/blockwatcher/networks from the browser reaches the companion server first, which forwards it to blockwatcher’s real GET /networks and relays the response back: blockwatcher is never addressable directly.
• Matches reach the dashboard the same way any other consumer would: as a sink. The companion server registers a webhook sink named ui-ingest in blockwatcher automatically on boot. Any monitor whose actions include ui-ingest has its matches delivered (over HTTP, retried and dead-lettered by the engine exactly like any other webhook sink) to the companion’s /ingest endpoint, which stores them in ui.db for the log view. A monitor that omits ui-ingest from its actions runs and delivers to its other sinks as normal, but never shows up in the dashboard’s log.

Running it

From the repository root, copy and fill in the compose environment file:

cp docker/.env.example docker/.env

Edit docker/.env: set BLOCKWATCHER_API_TOKEN, UI_OPERATOR_SECRET, and UI_INGEST_SECRET (the last two must not be the same value). Set an RPC variable for each network you plan to create (SEPOLIA_RPC_URL if you follow the examples in this wiki). Every url_secret a network resource names must resolve to a variable present here, because blockwatcher resolves secrets inside its own container.

Then start the stack:

docker compose -f docker/compose.yaml up --build

The first run builds two images: one for blockwatcher itself, one for the companion server plus the built SPA. The ui container waits for blockwatcher’s health check (GET /health, no token) before serving. Open http://127.0.0.1:8080 and sign in with UI_OPERATOR_SECRET.

From there, create a network under Resources → networks, a spec if your selectors need ABI decoding, and a monitor with ui-ingest checked under Actions: matches appear in the monitor’s log within a few seconds of qualifying on-chain activity.

What the companion does and does not do

Operators sign in at /login with UI_OPERATOR_SECRET and hold an HttpOnly; SameSite=Strict session. The engine’s webhook authenticates with UI_INGEST_SECRET on x-blockwatcher-ingest; the two secrets must not be the same value. Compose still binds the dashboard to 127.0.0.1 because loopback is what keeps the login page off the wider network; re-point that bind to 0.0.0.0 only behind TLS (UI_COOKIE_SECURE=true) and a deployment story that is not “the operator’s laptop”. The engine API’s GET /health remains unauthenticated so Compose can probe it. The trust boundary is written out in the repository’s docs/threat-model.md.

Where the data actually lives

Two SQLite files back the two services, kept in named volumes rather than bind mounts so container recreation doesn’t lose them: blockwatcher’s own blockwatcher.db (the blockwatcher-data volume) holds resources, checkpoints, and dead letters, and pause state, exactly as it would outside Docker; the companion’s ui.db (the ui-data volume) holds only what it has ingested: match history. Pause state lives entirely in blockwatcher.db; the companion reads it from /status rather than keeping its own copy. Neither service reads the other’s file; the only thing that crosses between them is whatever the ui-ingest webhook forwards. docker compose down leaves both volumes in place for the next up; only down -v removes them, and that removal is permanent: every resource, checkpoint, and logged match is gone, with no confirmation beyond the flag itself. Because match bodies can carry on-chain data an operator considers sensitive, that risk applies to ui.db the same way it applies to any consumer that stores what a webhook sink delivers.

Embedding blockwatcher in a host process

The blockwatcher binary is one way to run the engine: it owns the CLI, instance config, HTTP control plane, metrics listener, and SIGTERM/SIGINT drain. A host that already has a process — a ledger-sync worker, a service that applies matches to its own store — can run the same engine in-process without standing any of that up.

That path is blockwatcher-embed. It is the one place compiled-in module families are registered (crates/blockwatcher-embed/src/catalog.rs), and it re-exports the engine types a host needs to boot and stop. The binary is a client of the same crate for catalog construction; it does not keep a second registrar.

flowchart LR
    subgraph host["host process"]
        embed["blockwatcher-embed<br/>build_catalog · Engine::start"]
        ports["host port impls<br/>Sink · Storage · Source<br/>Decoder · Matcher · Gate"]
        stop["host-owned shutdown"]
        embed --> ports
        stop --> embed
    end

    subgraph bin["blockwatcher binary"]
        embed2["blockwatcher-embed<br/>(same catalog + start)"]
        extra["CLI · config · seed<br/>API · metrics · SIGTERM"]
        extra --> embed2
    end

Key takeaways

• Depend on blockwatcher-embed, not on the blockwatcher binary crate, to run the engine in-process.
• build_catalog is the one registration path; Engine::start boots from a catalog, a storage backend, and an EngineConfig.
• A host can supply an implementation of any of the six ports, not just Sink: register it on the catalog before Engine::start, or, for storage, hand it straight to EngineDeps.
• A host sink implements Sink::deliver(&SinkEvent) in-process and must treat both Match and Retracted as at-least-once (idempotent on match_id).
• Each port’s own obligations are the same in-process as in a shipped module; Extending blockwatcher § What each port asks of a module states them per port.
• The host owns shutdown: call Engine::shutdown. Embed does not wire SIGTERM or SIGINT.
• Embed does not serve the REST API or the Prometheus scrape endpoint. Those stay process concerns of the binary.

The host recipe

Add blockwatcher-embed to the host’s Cargo.toml. Default features (evm, expr, sinks) mirror the binary: they fold the same module families into the catalog. Storage (memory, sqlite) and gate modules (threshold, max_once) are always registered. A host-supplied gate uses the same compile/on_hit contract and still must not open sqlite. A host that implements its own Sink also depends on blockwatcher-ports and blockwatcher-types.

Boot is four steps: build the catalog, construct storage, start the engine, stop it yourself.

#![allow(unused)]
fn main() {
use blockwatcher_embed::{build_catalog, Engine, EngineConfig, EngineDeps};

let catalog = build_catalog()?;
let storage_factory = catalog.storage("memory")?;
let storage = storage_factory(serde_json::json!({}))
    .await
    .expect("memory storage constructs from empty config");

let config: EngineConfig = serde_json::from_value(serde_json::json!({
    "matcher": { "module": "expr", "config": {} }
}))?;

let engine = Engine::start(EngineDeps {
    storage,
    catalog,
    config,
})
.await?;

// Host-owned shutdown: embed does not install signal handlers.
let _report = engine.shutdown().await;
}

EngineConfig has no overall default because it refuses to guess a matcher module. Deserializing a JSON object that names expr (when that feature is on) fills every other tunable from core’s own defaults, including journal_depth = 1024. See Configuration reference § [engine].

Resources (networks, specs, sinks, monitors) still live in storage. A host either writes them through ControlHandle (re-exported from embed) or persists them into the backend before Engine::start, the same shapes the binary’s seed path writes. Embed does not load a seed directory of its own.

Any of the six ports can be the host’s own

A sink is the common case, not the only one. build_catalog returns a catalog the host still owns until EngineDeps takes it by value, and every register_* method on it is public, so a host can add its own module to any port family before start:

Two things do not change by being in-process. Each port’s contract is identical to a shipped module’s, so an in-process source still owes the engine non-decreasing cursor order and an in-process matcher still owes it a bounded compile; Extending blockwatcher § What each port asks of a module states those obligations per port, and they are worth reading before writing the impl rather than after. And a duplicate name within one family is EngineError::DuplicateModule from register_*, reported rather than panicked precisely so a host with no main of its own can recover.

A host that supplies its own module for a family usually also wants to stop linking the shipped one. blockwatcher-embed’s default features are evm, expr, and sinks; turning them off with default-features = false sheds those families, leaving storage (always registered) and whatever the host registers itself. What a build can select is then exactly what it linked, so a name the host never registered is refused at write or boot time listing only the names actually present.

An in-process sink

Shipped sink modules (webhook, script, log) still work: register a sink resource that names one of them, the same as in the binary. A host that wants to apply and undo by MatchId in the same process implements the Sink port itself and registers it on the catalog before start.

#![allow(unused)]
fn main() {
use async_trait::async_trait;
use blockwatcher_ports::{Sink, SinkError};
use blockwatcher_types::SinkEvent;

struct LedgerSink;

#[async_trait]
impl Sink for LedgerSink {
    async fn deliver(&self, event: &SinkEvent) -> Result<(), SinkError> {
        match event {
            SinkEvent::Match(m) => {
                // Apply by m.id. At-least-once: a duplicate id is the same
                // occurrence, not a new one.
                let _ = m;
                Ok(())
            }
            SinkEvent::Retracted { match_id } => {
                // Undo by match_id. Also at-least-once: must be idempotent.
                let _ = match_id;
                Ok(())
            }
        }
    }
}
}

(Sink is one method, crates/blockwatcher-ports/src/sink.rs. The engine owns retry, backoff, and dead-lettering; a sink that retries internally is a bug, the same contract as every shipped module.)

Retracted is how a consumer undoes work that a deep invalidate proved was on a dead fork. Register the impl on the catalog before start, under a name a sink resource can select:

#![allow(unused)]
fn main() {
let mut catalog = build_catalog()?;
catalog.register_sink("ledger", |_config| {
    Box::pin(async move { Ok(Arc::new(LedgerSink) as Arc<dyn Sink>) })
})?;
}

A stored sink resource with "module": "ledger" then constructs this type the same way a catalog sink does. Delivery guarantees covers when retracts fire, the bounded journal they read from, and the ordering that every retract for a sink finishes (delivered or dead-lettered) before any post-restart Match from that network is offered. A host that ignores Retracted will keep orphan rows it can no longer identify by id.

Classify failures through SinkError::Delivery { message, class: ErrorClass::… } exactly as a shipped sink does. The engine’s retry loop reads that class; it does not know this sink is in-process.

Host-owned storage

Storage is the one port a host does not have to register at all. EngineDeps.storage is an Arc<dyn Storage>, so a host that already has a database hands its own backend straight in, and every resource, checkpoint, dead letter, and journal row lands in the host’s store instead of a sqlite file beside the process:

#![allow(unused)]
fn main() {
use std::sync::Arc;
use blockwatcher_ports::Storage;

let storage: Arc<dyn Storage> = Arc::new(LedgerStore::new(pool));

let engine = Engine::start(EngineDeps {
    storage,
    catalog: build_catalog()?,
    config,
})
.await?;
}

catalog.register_storage exists for symmetry, but nothing in the engine resolves storage by name; only the binary does, from [storage] in blockwatcher.toml. A host registers a storage module only if its own configuration should be able to name one.

Storage is also the widest port and the only one with a shared behavioural contract. Take blockwatcher-testkit as a dev-dependency and run exercise_storage_contract against the host backend: it is the same sequence memory and sqlite pass, so passing it is what makes the host’s store equivalent to them rather than merely compiling (blockwatcher-testkit § The storage contract). The rules it checks, optimistic concurrency on every write, unpaginated resource listings, and pruning the journal inside record_delivery, are stated in blockwatcher-storage § The Storage port contract.

A host-owned source

A host that already streams a chain, or that reads from an internal bus a shipped source knows nothing about, implements Source and registers it under a name a network resource can select:

#![allow(unused)]
fn main() {
use blockwatcher_ports::{Source, SourceError};

catalog.register_source("ledger-feed", |config| {
    Box::pin(async move {
        let config: FeedConfig =
            serde_json::from_value(config).map_err(|e| SourceError::InvalidConfig {
                message: format!("invalid ledger-feed config: {e}"),
            })?;
        Ok(Arc::new(LedgerFeed::new(config)) as Arc<dyn Source>)
    })
})?;
}

A stored network resource with "source": { "module": "ledger-feed", … } then gets one instance per network, exactly as evm-rpc would. The obligations are unchanged by the source being local: events go out through ctx.events in non-decreasing cursor order from a single loop, ctx.cancel means return SourceOutcome::Ended, a detected rewind is SourceOutcome::Invalidated { from } rather than an error, and the cursor mapping is the host’s to choose and document. scan and confirmed_tip default to Unsupported, so a streaming-only feed implements neither.

blockwatcher-testkit’s recv and stop are worth pulling in as a dev-dependency here too: they drive the source under a deadline and fail a run that ignores its cancellation token instead of hanging the host’s test suite.

A host-owned decoder

A decoder is the one module a host registers but never names. The engine constructs every registered decoder at boot with an empty config object and indexes them by the ChainKind each chain() returns, so a host decoder becomes reachable purely by storing Spec resources whose chain matches:

#![allow(unused)]
fn main() {
use blockwatcher_ports::{Decoder, SpecError};

catalog.register_decoder("ledger-wire", |_config| {
    Box::pin(async move { Ok(Arc::new(LedgerDecoder) as Arc<dyn Decoder>) })
})?;
}

Two registered decoders claiming the same ChainKind is EngineError::DuplicateChainDecoder at boot, so a host adding a decoder for a chain the build already carries should drop that family’s feature first. The compile-time and hot-path obligations, resolving only the specs a selector names, matching declared field names to the shapes actually decoded, and keeping decode output order deterministic, are where a host decoder most often goes quietly wrong; Extending blockwatcher § Decoder states each with the failure it produces.

A host-owned matcher

A host with its own predicate language implements Matcher, registers it, and names it in EngineConfig instead of expr:

#![allow(unused)]
fn main() {
catalog.register_matcher("host-lang", |_config| {
    Box::pin(async move { Ok(Arc::new(HostMatcher) as Arc<dyn Matcher>) })
})?;

let config: EngineConfig = serde_json::from_value(serde_json::json!({
    "matcher": { "module": "host-lang", "config": {} }
}))?;
}

One matcher runs per process, which is why EngineConfig has no overall default and refuses to guess one. A host that replaces expr should build embed without the expr feature; DEFAULT_MATCHER exists only under that feature, and nothing else fills the field. The obligation most easily missed is that compile takes operator-controlled source: the engine bounds the length it passes in, but the module must bound the work and memory one call can consume, because a compile that can be driven to exhaust a stack or heap is a write-path denial of service.

What embed is not

• Not a second engine API. Engine::start, Engine::shutdown, and ControlHandle are the same types the binary calls. Embed does not wrap them.
• Not a signal handler. The host decides when to drain. Engine::shutdown is the same deadline-bounded drain the binary maps onto its exit codes; without a host call, nothing asks the engine to stop.
• Not the HTTP or metrics stack. blockwatcher-api and blockwatcher-metrics are not dependencies of embed. A host that wants those listeners still runs the binary, or wires those crates itself.
• Not a second registration path for workspace modules. A module that ships in the workspace still becomes selectable through one get_all() line in its own crate, which build_catalog folds, and the binary must not grow its own catalog beside embed’s. A host registering a type it owns is the separate case above: it adds to the catalog build_catalog returned rather than replacing how that catalog is built.

Extending blockwatcher walks a shipped module through registration step by step and states what each of the six ports asks of the module behind it, in § What each port asks of a module; its § Extending to a new network covers the chain-family case, which a host reaches by registering a Source and a Decoder together rather than by adding a crate to the workspace. blockwatcher-embed is the crate page: features, allowlist, and what it re-exports.

Concepts

This part opens up the model underneath the pipeline: what a resource is, how a predicate turns a decoded occurrence into a yes-or-no answer, what blockwatcher guarantees about delivery, and where the boundary sits that keeps the engine from ever needing to know which chain it is watching. It is for a reader who has already run blockwatcher and now wants to understand why it behaves the way it does.

flowchart LR
    rpc["RPC endpoints<br/>(external chains)"]
    sources["Sources<br/>evm-rpc · evm-mempool"]
    sinks["Sinks<br/>webhook · script · log"]
    storage["Storage<br/>checkpoints · dead letters · resources"]
    api["REST API"]
    metrics["Metrics"]
    engine["engine<br/>bounded channels · checkpoints"]

    subgraph pipeline["Engine pipeline"]
        direction LR
        decoder["Decoder"]
        matcher["Matcher<br/>predicates"]
        gate["Gate<br/>threshold · max_once"]
        decoder --> matcher
        matcher --> gate
    end

    rpc --> sources
    sources -->|"decode and match"| decoder
    gate --> sinks
    api -->|"manages resources"| storage
    storage <--> engine
    engine -->|"drives"| pipeline
    engine -.->|"reports"| metrics

    classDef module fill:none,stroke:#a9a3e3
    classDef core fill:none,stroke:#8a8d86,stroke-dasharray: 5 5

    class rpc,sources,sinks,decoder,matcher,gate module
    class engine,api,storage,metrics core
classDef dim fill:none,stroke:#999999,color:#999999,opacity:0.35
classDef focus fill:#ffd43b,stroke:#d9480f,stroke-width:3px,color:#1a1a1a
class rpc,sources,sinks,storage,api,metrics dim
class engine,decoder,matcher,gate focus
click sources "selectors.html"
click decoder "chain-agnosticism.html"
click matcher "predicates.html"
click gate "gates.html"
click sinks "delivery.html"
click storage "resources.html"
click api "../reference/http-api.html"
click metrics "../reference/observability.html"
click engine "pipeline.html"

Key takeaways

• Concepts pages explain the model behind the pipeline, not how to run it: resources, predicates, gates, delivery, and the chain-agnostic boundary.

• The pipeline page covers which task does what and where the bounded queues sit; the resources page covers the resource kinds blockwatcher watches and delivers to.

• Predicates, gates, selectors, and delivery guarantees together describe how a decoded event becomes a match and reaches a sink.

• Chain-agnosticism and modules describe the boundaries that keep the engine chain-unaware and every axis of behavior swappable.

• The pipeline: which task does what, and where the bounded queues sit.

• Resources: the resource kinds blockwatcher watches and delivers to.

• Predicates and the expression language: how one decoded occurrence becomes a match.

• Gates: when a predicate-true hit becomes a delivery, across other hits of the same monitor.

• Selectors: how a monitor points at a contract spec.

• Delivery guarantees: what “done” means, and what happens when delivery fails.

• Chain-agnosticism: the boundary that keeps the engine chain-unaware.

• Modules and trade-offs: the swappable pieces behind each port, and what each choice costs.

The pipeline

How it works in one picture sketched a single pipeline. This page opens it up: which task does what, which crate implements it, where the bounded queues sit, and (the part that only reading crates/blockwatcher-core/src/pipeline/ settles) exactly what is shared across a deployment and what is private to one network.

flowchart LR
    rpc["RPC endpoints<br/>(external chains)"]
    sources["Sources<br/>evm-rpc · evm-mempool"]
    sinks["Sinks<br/>webhook · script · log"]
    storage["Storage<br/>checkpoints · dead letters · resources"]
    api["REST API"]
    metrics["Metrics"]
    engine["engine<br/>bounded channels · checkpoints"]

    subgraph pipeline["Engine pipeline"]
        direction LR
        decoder["Decoder"]
        matcher["Matcher<br/>predicates"]
        gate["Gate<br/>threshold · max_once"]
        decoder --> matcher
        matcher --> gate
    end

    rpc --> sources
    sources -->|"decode and match"| decoder
    gate --> sinks
    api -->|"manages resources"| storage
    storage <--> engine
    engine -->|"drives"| pipeline
    engine -.->|"reports"| metrics

    classDef module fill:none,stroke:#a9a3e3
    classDef core fill:none,stroke:#8a8d86,stroke-dasharray: 5 5

    class rpc,sources,sinks,decoder,matcher,gate module
    class engine,api,storage,metrics core
classDef dim fill:none,stroke:#999999,color:#999999,opacity:0.35
classDef focus fill:#ffd43b,stroke:#d9480f,stroke-width:3px,color:#1a1a1a
class rpc,sources,sinks,storage,api,metrics,decoder,matcher,gate dim
class engine focus
linkStyle 6,7,8 stroke:#d9480f,stroke-width:3px
click sources "selectors.html"
click decoder "chain-agnosticism.html"
click matcher "predicates.html"
click gate "gates.html"
click sinks "delivery.html"
click storage "resources.html"
click api "../reference/http-api.html"
click metrics "../reference/observability.html"
click engine "pipeline.html"

Key takeaways

• spawn turns one network’s PipelineSpec into four task kinds (source, processor, sink worker, checkpoint writer) linked end to end by channels.
• The decoder is shared per chain and the matcher is one instance for the whole engine; only the source, monitor set, and sink workers are per-network.
• The two bounded channels never drop a message: a full channel stalls the pipeline at the point of congestion rather than losing anything.
• A raw event registers with the progress tracker exactly once, after dispatch, for however many matches it produced, even zero.
• After a predicate-true decode, an optional per-monitor gate may Retain or Discard the hit with outstanding 0, or Emit a match/digest. Holding a hit does not stall the checkpoint.
• A cursor only ever persists once every completion guard the event handed out, across every monitor and sink it touched, has completed.

One pipeline per network

Engine::start builds one PipelineSpec per stored network and spawns it once, and every later network write goes through the same spawn call by way of a restart (crates/blockwatcher-core/src/pipeline/mod.rs). A PipelineSpec bundles that network’s already-constructed source, its chain’s decoder, the engine’s one matcher instance, the network’s compiled monitor set, and every sink any of those monitors names (crates/blockwatcher-core/src/pipeline/mod.rs). spawn turns that bundle into the task kinds below, wired together by channels it owns end to end.

flowchart LR
    src["Source task<br/>blockwatcher-evm"] -->|"events channel<br/>(bounded)"| proc
    proc["Processor task<br/>decode + match + gate<br/>blockwatcher-core + blockwatcher-evm + blockwatcher-expr"]
    proc -->|"sink channel<br/>(bounded, per sink)"| sw["Sink worker task<br/>blockwatcher-core"]
    proc -.->|register cursor| prog["Progress tracker<br/>blockwatcher-core"]
    sw -.->|complete / dead-letter| prog
    prog -->|checkpoint| cpw["Checkpoint writer task<br/>blockwatcher-core"]
    sw -->|on exhaustion| dl[("dead letters<br/>blockwatcher-storage")]
    cpw --> store[("checkpoint<br/>blockwatcher-storage")]

    classDef task fill:none,stroke:#a9a3e3
    class src,proc,sw,cpw task

Each task kind, and the crate doing the real work inside it:

• Source task: one per network, running whatever module the network’s source.module names (evm-rpc or evm-mempool, both in crates/blockwatcher-evm). It owns its own notion of position, its cursor, and must hand events to the next stage in non-decreasing cursor order, the contract Processor::run states for its receiver; nothing downstream reorders what it reads (crates/blockwatcher-core/src/pipeline/processor.rs).
• Processor task: one per network, decoding, matching, and gating in the same loop. It calls the network chain’s Decoder (the evm decoder in crates/blockwatcher-evm, the only one shipped) once per active, unpaused monitor’s selector, then evaluates that monitor’s compiled predicate through the engine’s single Matcher instance (blockwatcher-expr’s expr matcher, unless a deployment swaps it); see Processor::process_one (crates/blockwatcher-core/src/pipeline/processor.rs). Gate arithmetic lives in pipeline/gate.rs, not inlined in processor.rs.
• Sink worker task: one per sink a network’s monitors reference, running the sink module (webhook, script, or log) named by that sink’s SinkDef. It owns delivery retry and backoff and dead-lettering, identical regardless of which module is attached, per SinkWorker’s own doc comment (crates/blockwatcher-core/src/pipeline/sink_worker.rs).
• Checkpoint writer task: one per network, persisting whatever the progress tracker publishes and refusing to persist anything older than what it already wrote this run, in CheckpointWriter’s persist method (crates/blockwatcher-core/src/pipeline/checkpoint_writer.rs).

The Progress tracker (crates/blockwatcher-core/src/progress.rs) is not a task; it is a shared, lock-protected structure the processor registers events into and sink workers complete guards against, read by the checkpoint writer. It is covered in full on Delivery guarantees; this page treats it as the thing that turns “every dispatched match done” into “safe to persist this cursor.”

What is shared, and what is per-network

The pipeline diagram makes every task look network-scoped, but two of the module families it touches — the decoder and the matcher — are actually shared engine-wide, constructed once at boot and reused rather than rebuilt per network or per pipeline restart:

The two easiest to get wrong: the decoder is per chain, not per network (two evm networks decode through the exact same Arc<dyn Decoder>), and the matcher is a single engine-wide instance, not something a monitor or a network picks; blockwatcher.toml names it once for the entire process, as Engine’s matcher field doc comment states (crates/blockwatcher-core/src/engine.rs). A sink’s module instance, by contrast, is rebuilt every time a pipeline is (re)constructed: it is not shared even across two restarts of the same network, though this is invisible in practice because a sink module carries no state beyond its config.

Backpressure: two bounded channels, no drops

Between source and processor sits one tokio::sync::mpsc channel of capacity event_channel_capacity (default 256); between processor and each sink worker sits one of capacity sink_channel_capacity (default 64), both configured once in blockwatcher.toml’s engine section and applied to every pipeline alike, EngineConfig’s fields, read by spawn (crates/blockwatcher-core/src/config.rs, crates/blockwatcher-core/src/pipeline/mod.rs). Neither channel ever drops a message when full. A source’s send and the processor’s dispatch are plain awaited .send() calls, in Processor::process_one (crates/blockwatcher-core/src/pipeline/processor.rs); when a channel is at capacity, that await simply doesn’t resolve until a slot opens, which means a slow sink worker’s channel filling up stalls the processor’s dispatch loop, which in turn stalls the source’s own send: the whole pipeline pauses at the first point of congestion rather than losing anything. The two channels close only when their sending side is dropped, which is what a graceful shutdown or restart uses to drain: cancel the source, its sender drops, the processor’s receive loop ends and drops the sink senders, and each sink worker drains whatever is already queued before it exits, the cascade spawn’s own doc comment states (crates/blockwatcher-core/src/pipeline/mod.rs).

Escalation, abort, and the storage fence

Every drain (a whole-engine shutdown, and a single network’s stop or restart through the control plane) is bounded by drain_deadline_ms. Missing that deadline escalates: a hard_cancel signal fires, tasks that are still running get a brief grace window to exit on their own, and anything still not done after that is stopped with a forced abort() (crates/blockwatcher-core/src/engine/drain.rs). An escalation like this is always loud (a warning naming the network) and always counted (blockwatcher_pipelines_aborted_total).

An abort() can leave more behind than a stopped task: a storage backend whose operations run independent of the caller that started them (sqlite’s own connection lives behind a tokio::task::spawn_blocking call, for example) keeps running whatever it was asked to do at the moment of the abort, and that call still lands later, on its own schedule. Without accounting for it, a pipeline restarted right after such an escalation could read its resume point, or its gate journal, before that residual write lands, then start delivering against what it read, only for the residual write to land afterward and overwrite what the restart’s own work just persisted: a gap, not a duplicate.

Storage::quiesce closes that window: after any escalated drain, the engine awaits it, bounded by its own budget, before treating the network as safely stopped. It resolves once every storage operation an aborted task had in flight has actually completed. One function, stop_pipeline, is where every single-network stop and restart drains through, so this quiescing, its budget, and its counting all live in that one place regardless of which control-plane path called it. Three paths read what it decides before touching storage themselves, and all three refuse in the same shape when it times out, leaving the network abandoned (visible in /status the same way a failed crash-restart already is) until an operator retries or the process restarts:

• a restart, before re-reading the checkpoint and gate outbox it would otherwise redeliver or overwrite against;
• a gate envelope change or a gated monitor’s delete, before dropping that monitor’s held journal, which a residual write could otherwise resurrect after the drop; and
• a reorg invalidation, before pruning gate holds, retracting journaled deliveries, or rewinding the checkpoint, any of which a residual write could otherwise race.

Every one of these refusals is counted under blockwatcher_pipelines_quiesce_timeout_total exactly once, at stop_pipeline itself, no matter which of the three paths called it or how many times. A timeout is not forgiven just because a later attempt on the same network finds no pipeline left running to escalate: the engine remembers the debt and re-attempts the same quiesce, under the same budget, on that later attempt too, refusing again for as long as the residual write stays outstanding and clearing the debt only once some attempt actually waits it out.

A whole-engine shutdown facing the same timeout only warns and counts it instead of refusing anything, since the process is exiting either way and a later process starts with its own storage connections, which the residual write from this one cannot reach.

From one raw event to zero or more deliveries

Inside Processor::process_one, run once per raw event (crates/blockwatcher-core/src/pipeline/processor.rs):

1. An event addressed to a different network is dropped and counted (misrouted) rather than processed: a pipeline never touches an event that isn’t its own, because attributing it would corrupt this pipeline’s own cursor ordering.
2. The current monitor set is snapshotted once for the whole event, not once per monitor, so a hot-swapped monitor set landing mid-event can never split that event’s processing between an old and a new set.
3. For every active, unpaused monitor, the decoder runs against that monitor’s compiled selector. Zero, one, or several decoded events can result (an EVM log’s Transfer and a batched transaction’s several decoded calls are both possible); each one that decodes is evaluated against the monitor’s predicate (or accepted outright if it has none).
4. Every monitor that accepts a decoded event offers that occurrence to its gate (or, with gate omitted, treats it as Emit of this hit). Retain / Discard mint nothing. Emit mints one match per chosen journal index (one SinkEvent::Match, or one Digest if two or more) and queues one work item per sink that monitor’s actions name.
5. Once every pair from this one raw event is known, the event is registered with the progress tracker for exactly that many outstanding completions: registered exactly once per event, after dispatch, never before and never twice. A gate that retained or discarded adds zero to that count for that monitor (crates/blockwatcher-core/src/pipeline/processor.rs).

An event that matched nothing (no selector, or predicate false, or every gate quiet) still gets registered, with zero outstanding, which is what lets its cursor advance immediately rather than wait on deliveries that were never dispatched.

Delivery, retry, and the checkpoint’s dependency on it

Each sink worker pulls one work item at a time and calls deliver_with_retry (crates/blockwatcher-core/src/pipeline/delivery.rs): a transient or rate-limited failure sleeps a doubling backoff and tries again, up to the sink’s configured max_attempts; a permanent or range-narrower failure skips straight to giving up. Giving up means recording a dead letter in storage (retried forever on its own backoff if that write itself fails), because a checkpoint may only move past a loss storage has durably recorded, never past one it merely intends to record, the ordering SinkWorker::dead_letter’s own doc comment states (crates/blockwatcher-core/src/pipeline/sink_worker.rs). Only once delivery succeeds or the dead letter is durably written does the sink worker complete its guard. The checkpoint writer, in turn, only ever persists a cursor once every guard the processor handed out for it (across every monitor and every sink that event’s matches touched) has completed, via Progress and the CompletionGuard it hands out (crates/blockwatcher-core/src/progress.rs). Delivery guarantees covers this mechanism itself in depth, along with the one shipped source, evm-mempool, whose cursor is not a chain position and so cannot make the same promise across a crash.

Resources

Everything blockwatcher watches and everywhere it delivers to is one of its resource kinds, each its own Rust struct in crates/blockwatcher-types/src/resource.rs, each rejecting an unrecognized field at write time (#[serde(deny_unknown_fields)], one exception noted below). This page covers what each kind actually contains, how a monitor references the others, how a resource gets into storage in the first place, and exactly what gets checked before a write is allowed to land. The field-level tables (every key, its type, its default, and the exact refusal a bad value gets) live in the Resource reference, one page per kind.

flowchart LR
    rpc["RPC endpoints<br/>(external chains)"]
    sources["Sources<br/>evm-rpc · evm-mempool"]
    sinks["Sinks<br/>webhook · script · log"]
    storage["Storage<br/>checkpoints · dead letters · resources"]
    api["REST API"]
    metrics["Metrics"]
    engine["engine<br/>bounded channels · checkpoints"]

    subgraph pipeline["Engine pipeline"]
        direction LR
        decoder["Decoder"]
        matcher["Matcher<br/>predicates"]
        gate["Gate<br/>threshold · max_once"]
        decoder --> matcher
        matcher --> gate
    end

    rpc --> sources
    sources -->|"decode and match"| decoder
    gate --> sinks
    api -->|"manages resources"| storage
    storage <--> engine
    engine -->|"drives"| pipeline
    engine -.->|"reports"| metrics

    classDef module fill:none,stroke:#a9a3e3
    classDef core fill:none,stroke:#8a8d86,stroke-dasharray: 5 5

    class rpc,sources,sinks,decoder,matcher,gate module
    class engine,api,storage,metrics core
classDef dim fill:none,stroke:#999999,color:#999999,opacity:0.35
classDef focus fill:#ffd43b,stroke:#d9480f,stroke-width:3px,color:#1a1a1a
class rpc,sources,sinks,metrics,engine,decoder,matcher,gate dim
class storage,api focus
click sources "selectors.html"
click decoder "chain-agnosticism.html"
click matcher "predicates.html"
click gate "gates.html"
click sinks "delivery.html"
click storage "resources.html"
click api "../reference/http-api.html"
click metrics "../reference/observability.html"
click engine "pipeline.html"

Key takeaways

• blockwatcher’s resource kinds are exactly Network, Spec, SinkDef, and Monitor, each rejecting an unrecognized field except Monitor’s selectors, checked later by the decoder.
• A monitor is the only resource that references any other: it names one network, one spec per selector, and one or more sinks.
• Every resource is a versioned record, written through the REST API’s ETag/If-Match concurrency or, once, through a first-boot seed directory.
• What restarts on a write differs by kind: a monitor write hot-swaps in place, a network write always restarts, and a spec or sink write restarts every network it affects.
• Every write validates before it reaches storage, and every validation rejection surfaces on the wire as 422.

The kinds and their fields

Network (Network, crates/blockwatcher-types/src/resource.rs): an id, a chain tag, and a source module selection (ModuleSel: a module name plus that module’s own opaque config object). Nothing else. It is the only resource whose module family (Source) is configured on the resource itself rather than referenced from elsewhere.

Spec (Spec): an id, a chain tag, and a payload that core never interprets: for the evm chain family this is a Solidity JSON ABI array, meaningless to anything outside that one chain’s decoder. The write itself is what runs EvmDecoder::compile_spec (crates/blockwatcher-evm/src/decoder/mod.rs), turning the ABI into a SchemaSet and a selector-keyed dispatch table exactly once; a selector or predicate written against this spec afterward resolves against that already-built result, so re-parsing the ABI text is a write-time cost this resource pays only once, not something every later selector compile repeats.

SinkDef (SinkDef): an id, a module name, that module’s config, an optional retry policy (DeliveryRetry { max_attempts, initial_backoff_ms, max_backoff_ms }), an optional throttle policy (Throttle { max_deliveries, window_ms }), and an optional aggregate policy (Aggregate { window_ms, max_batch }). All of them are siblings of config rather than nested inside it: a sink module never sees or interprets its own retry, throttle, or aggregate policy, the engine owns retry, dead-lettering, throttling, and aggregation identically for every module. A SinkDef with no retry falls back to the engine-wide default from blockwatcher.toml; a SinkDef with no throttle is not throttled at all, since throttling is opt-in; a SinkDef whose throttle field is present fills any field left out of it with the default, 60 deliveries per 60,000ms; a SinkDef with no aggregate delivers every match on its own, unbatched. See Delivery policies for what enforcing each one means, and for a complete SinkDef JSON example carrying all three beside a webhook module config.

Monitor (Monitor): an id, the network it watches, one or more selectors, an optional predicate, an optional gate (ModuleSel: module + config; omitted means passthrough), and actions (a list of sink ids). gate is this monitor’s decision rule (like predicate), not a shared resource id (like actions: SinkId[]). Journals are per (pipeline, monitor). RawSelector is deliberately the one resource shape without deny_unknown_fields: it carries a core-readable spec field plus a #[serde(flatten)] bag of decoder-owned keys (events, functions, addresses for the evm decoder), and serde cannot combine flatten with deny_unknown_fields on the same struct. An unrecognized selector key is still rejected, just later, by the chain’s decoder at compile time rather than by serde at parse time, in compile’s own unknown-key check (crates/blockwatcher-evm/src/decoder/selector.rs).

How they reference each other

A monitor is the only resource that names any of the others, and it names all of them:

erDiagram
    Network {
        NetworkId id
        ChainKind chain
        ModuleSel source
    }
    Spec {
        SpecId id
        ChainKind chain
        Json payload
    }
    SinkDef {
        SinkId id
        string module
        DeliveryRetry retry
        Throttle throttle
        Aggregate aggregate
    }
    Monitor {
        MonitorId id
        NetworkId network
        RawSelector[] selectors
        string predicate
        ModuleSel gate
        SinkId[] actions
    }
    Monitor }o--|| Network : "network"
    Monitor }o--o{ Spec : "selectors[].spec"
    Monitor }o--o{ SinkDef : "actions"

Monitor.gate is optional; mermaid cannot show Option cleanly, so omitted means passthrough, as Gates describes. Monitor.network names exactly one Network; each entry in Monitor.selectors names exactly one Spec through its spec field; and Monitor.actions names one or more SinkDefs. Network, Spec, and SinkDef never reference each other or point back at a monitor: the reference graph is a star with Monitor at the center, one level deep. A Spec and the Networks that use it are connected only indirectly, through whichever monitors select that spec on that network; spec_set_for_chain (crates/blockwatcher-core/src/compile.rs) compiles every spec sharing a network’s chain together, which is why a spec edit can affect a network that no monitor explicitly ties to it by name (see the write-time checks below).

Lifecycle: how a record is created, versioned, and reloaded by kind

Every resource is a VersionedRecord in storage: an id, a version for optimistic concurrency, and the JSON value (crates/blockwatcher-types/src/resource.rs). A record gets there by exactly the routes below, and by no other:

• The REST API, PUT /{kind}/{id}. Without an If-Match header the write is a create: it fails with 409 Conflict if the id already exists. With If-Match: "<version>" it is an update: it fails with 412 Precondition Failed (naming the actual current version) if the version doesn’t match, or 404 if the record is gone, per Storage::put’s contract and the write handler’s call to if_match (crates/blockwatcher-ports/src/storage.rs, crates/blockwatcher-api/src/routes/resources.rs). A successful write’s response carries the resulting version in its own ETag, which is what the next conditional write is expected to send back as If-Match.
• A seed directory, read once at boot via the binary’s --seed <dir> flag. It expects exactly one subdirectory per kind (networks/, specs/, sinks/, monitors/), one JSON file per resource, and it is strictly first-boot: seeding runs the whole bundle through the same validation Engine::start itself would run, then writes it with create-only semantics, and once storage holds any resource of any kind, seeding is refused as a no-op on every later boot, per validate and store_is_empty (crates/blockwatcher/src/seed.rs). After that first boot, every resource in that deployment is managed exclusively through the API.

What happens to a running pipeline differs by kind, and this is the one place “hot-reloaded” oversimplifies:

• Writing a Monitor usually restarts nothing. It recompiles that monitor’s network’s whole MonitorSet from storage and publishes it to the already-running pipeline through a watch channel: the same source instance keeps running, its checkpoint keeps advancing straight through the swap, in ControlHandle::put_monitor (crates/blockwatcher-core/src/control/writes.rs). Two things turn that swap into a restart of the one network:

  • Changing gate (module or config) is a new compiled artifact and drops that monitor’s gate journal. Holds belong to one envelope, and a processor keeps the journal of every gated monitor it is running in memory, so the drop may only happen with nothing processing for that network: the pipeline is stopped first, the holds dropped, and the pipeline brought back, in wipe_gate_state_while_stopped (crates/blockwatcher-core/src/control/mod.rs). Deleting a gated monitor takes the same path, per delete_monitor (crates/blockwatcher-core/src/control/deletes.rs). The wipe drops holds (undecided accumulation), never committed emissions: an outbox row the old envelope emitted but had not yet delivered survives and is delivered (or dead-lettered) when the pipeline comes back, per the gate delivery guarantee.
  • If the recompiled set now names a sink the running pipeline never spawned a worker for, the write escalates to a full restart, because a sink worker cannot be added to a pipeline after the fact, per hot_swap_monitors’s own doc comment (crates/blockwatcher-core/src/engine/control.rs).

  Both cost only what any restart costs: the source resumes from the checkpoint the drain left behind. A network with no running pipeline is never brought up by either: there is nothing to protect, and starting one would undo whatever stopped it.

• Writing a Network always restarts that network’s pipeline: the old one is drained, a fresh one is spawned, and it resumes from whatever checkpoint the drain left behind, in put_network (crates/blockwatcher-core/src/control/writes.rs). A restart is cheap specifically because the checkpoint survives it; it is not a re-scan from the beginning.

• Writing a SinkDef restarts every network whose stored monitors currently name that sink id, not every network in the deployment, per put_sink (crates/blockwatcher-core/src/control/writes.rs).

• Writing a Spec restarts every network on either the spec’s new chain or (on a reassignment) its prior chain, because spec_set_for_chain compiles every spec sharing a chain together: a network that never named this spec by id can still be running a compiled set built partly from it, per put_spec (crates/blockwatcher-core/src/control/writes.rs).

A restart that fails part-way through a multi-network fan-out (a SinkDef or Spec write affecting several networks) does not fail the request or stop the ones after it: the resource is already durably stored either way, and each network’s own restart failure is only logged, in restart_affected (crates/blockwatcher-core/src/control/writes.rs).

Whatever triggers it, a restart has the same observable side effects: a network write, a spec or sink write, a gate-envelope change, a gated monitor’s delete, and reorg recovery all go through this same replacement. Each one resets the /status counter snapshot for that network and rebuilds its sink workers, so in-memory throttle and aggregate windows start fresh; the Prometheus counters, checkpoints, journals, and outbox are unaffected, since none of them live in the pipeline instance a restart replaces.

Write-time validation

Every write validates before it ever reaches storage, and every rejection is 422 Unprocessable Entity on the wire: classify, in crates/blockwatcher-api/src/error.rs, classifies exactly this set of engine refusals that way, distinguishing them from a version conflict (412), a missing reference lookup (404), or a genuine server fault (500, and never with internal detail on the wire). What gets checked differs by kind:

• Monitor: its network must exist; every id in actions must name an existing sink; every selector’s spec must exist and share the network’s chain; and the whole monitor (selectors, predicate, and gate) must compile against the live decoder, matcher, and gate catalog before the write is persisted, in put_monitor (crates/blockwatcher-core/src/control/writes.rs). An unknown gate module, unknown config keys, out-of-range window_ms/count, or missing block.timestamp is 422: gate requires 'block.timestamp'; this monitor's selectors do not expose it.
• Network: its chain must have a loaded decoder, and its source module must actually construct with the given config, not merely be a name the catalog recognizes. Updating an existing network additionally recompiles every monitor already stored for it against the incoming chain, refusing a chain reassignment that would strand them, in put_network (writes.rs).
• SinkDef: its module must actually construct with the given config. Nothing about existing monitors is re-validated: a sink id, once it exists, is a stable dependency by name, per put_sink (writes.rs).
• Spec: its chain must have a loaded decoder, and the payload must compile standalone; then every monitor on every network that would be affected (per the chain-sharing rule above) is recompiled against a hypothetical spec set with this write already applied, so a reassignment that would break one of them is refused before it is stored, in put_spec (writes.rs).
• Delete, for Network/Spec/SinkDef, refuses outright while any stored monitor still references the id (checked before storage is touched at all), because a monitor left pointing at nothing would not fail on its own; it would fail the next boot for the whole deployment, per refuse_if_referenced (crates/blockwatcher-core/src/control/deletes.rs).

The did-you-mean suggestion

A 422 for an unresolvable name (an event or function a selector names that the spec doesn’t declare) carries a suggestion when one is available. For a selector’s events/functions list, the suggestion is simply the first event or function of that kind the spec declares, kind_suggestion (crates/blockwatcher-evm/src/decoder/selector.rs); it is not an edit-distance match. A predicate’s unknown field or namespace, covered on Predicates, gets a real bounded edit-distance suggestion instead: the mechanisms are deliberately different, one per crate that owns the vocabulary being checked. Either way the message shape is the same: SelectorError::UnknownField and PredicateError::UnknownField both render as unknown field '{field}', followed by a did you mean '{suggestion}'? suffix when one exists (crates/blockwatcher-ports/src/error.rs); for example, naming "Transfr" in a selector against a spec that declares Transfer and Approval rejects with unknown field 'Transfr' plus did you mean 'Approval'? (the suggestion is the spec’s first declared event of that kind, not the closest by spelling).

Predicates and the expression language

Every predicate a monitor carries reduces one decoded occurrence to a single yes-or-no answer, deciding whether it becomes a match. The whole language it’s written in lives in crates/blockwatcher-expr: a lexer, a recursive-descent parser, a type-checker that runs at write time against a monitor’s selector schemas, and a total evaluator that runs at decode time and can never error. This page enumerates the grammar from the parser itself, explains the type system it checks against, and walks compilation and evaluation end to end. Every predicate shown below was checked against the parser and type-checker described here, against a schema this page defines in The example schema below.

flowchart LR
    rpc["RPC endpoints<br/>(external chains)"]
    sources["Sources<br/>evm-rpc · evm-mempool"]
    sinks["Sinks<br/>webhook · script · log"]
    storage["Storage<br/>checkpoints · dead letters · resources"]
    api["REST API"]
    metrics["Metrics"]
    engine["engine<br/>bounded channels · checkpoints"]

    subgraph pipeline["Engine pipeline"]
        direction LR
        decoder["Decoder"]
        matcher["Matcher<br/>predicates"]
        gate["Gate<br/>threshold · max_once"]
        decoder --> matcher
        matcher --> gate
    end

    rpc --> sources
    sources -->|"decode and match"| decoder
    gate --> sinks
    api -->|"manages resources"| storage
    storage <--> engine
    engine -->|"drives"| pipeline
    engine -.->|"reports"| metrics

    classDef module fill:none,stroke:#a9a3e3
    classDef core fill:none,stroke:#8a8d86,stroke-dasharray: 5 5

    class rpc,sources,sinks,decoder,matcher,gate module
    class engine,api,storage,metrics core
classDef dim fill:none,stroke:#999999,color:#999999,opacity:0.35
classDef focus fill:#ffd43b,stroke:#d9480f,stroke-width:3px,color:#1a1a1a
class rpc,sources,sinks,storage,api,metrics,engine,decoder,gate dim
class matcher focus
click sources "selectors.html"
click decoder "chain-agnosticism.html"
click matcher "predicates.html"
click gate "gates.html"
click sinks "delivery.html"
click storage "resources.html"
click api "../reference/http-api.html"
click metrics "../reference/observability.html"
click engine "pipeline.html"

Key takeaways

• A predicate compiles once, at write time, into an AST checked against the monitor’s schema; evaluation at decode time is total and can never error.
• The type system groups values into families (Int, Str, address/bytes, Bool) rather than exact types, so a heterogeneous field can carry more than one family at once.
• Evaluation is three-valued (True, False, Unknown) with Kleene logic for &&/||, and a predicate treats Unknown the same as False.
• An unknown field or namespace gets a bounded edit-distance suggestion within distance 3; beyond that there is no suggestion at all.
• expr is the one matcher this repository ships, selected once for the whole process rather than per monitor.
• A predicate is per-event. Counting, windows, and “already alerted” are a gate, not an expression.

Compilation: what happens when a monitor is written

Compilation happens once, at write time; evaluation happens on every decoded event, against the artifact compilation already produced:

flowchart LR
    subgraph write["write time, once"]
        text["predicate text"] --> parse["parser::parse"] --> ast["Ast"]
        ast --> check["typecheck::check<br/>against SchemaSet"]
        check --> compiled["compiled Predicate"]
    end
    subgraph hot["decode time, every event"]
        compiled --> eval["eval::evaluate(ast, event)"]
        eval --> truth["True / False / Unknown"]
    end

Predicate::compile (crates/blockwatcher-expr/src/lib.rs) does exactly two things, in order: parser::parse turns the source text into an Ast, and typecheck::check walks that tree against the monitor’s SchemaSet, resolving every field path and admitting or rejecting every operator’s operand types. Either stage can fail with a PredicateError, and either failure is what a monitor write rejects with. A predicate never reaches storage unparsed or untyped. A Predicate that compiled successfully cannot fail to evaluate: matches (lib.rs) is eval::evaluate(&self.ast, event) == Truth::True, a total function over the type-checked tree. This is also why the same text can mean two different, unrelated things depending on which monitor it’s attached to: the schema it type-checks against comes entirely from that monitor’s own selectors, never from a shared global vocabulary. The expr module (crates/blockwatcher-expr/src/matcher.rs) is the one Matcher this repository ships, and blockwatcher.toml’s matcher field names it for the entire process at boot: one choice for every network and every monitor the deployment runs, not a per-monitor setting. Nothing about the port boundary itself limits a deployment to this one engine; it is simply the only implementation in the module catalog blockwatcher ships (see Modules). The pipeline covers where in a running pipeline this compiled predicate actually runs.

The example schema

Every predicate example on this page type-checks against one small schema: a spec declaring two ERC-20-style events plus one made-up event for string and array examples, alongside the tx/block/log namespaces the evm decoder’s namespaces function attaches to every compiled spec (crates/blockwatcher-evm/src/decoder/compile.rs):

args (from the selected event)
  Transfer:  from address, to address, value uint256
  Approval:  owner address, spender address, value uint256
  Registered: owner address, name string, tags string[]
tx:    hash bytes, index uint, status uint, from address, to address, value uint
block: number uint, hash bytes, timestamp uint
log:   address address, index uint

args.* is always derived from whichever event or function the monitor’s selectors decoded: it is never declared separately. tx.*, block.*, and log.* are the same three namespaces on every evm spec, and which of their fields an occurrence actually carries depends on the selector that produced it: a log-decoded event’s tx.status is always the constant 1 (a log exists only in a transaction that succeeded) and it never carries tx.from/tx.to/tx.value, while a function-call-decoded occurrence carries those but only carries tx.status/tx.index/block.* once the transaction is mined, never on evm-mempool. A predicate reading a field this occurrence doesn’t carry resolves Unknown, covered under Evaluation below.

Syntax by example

args.value > 1_000_000e6

args.value is declared uint256, so it’s an integer; 1_000_000e6 is token-decimal notation: one million at six decimals, expanded exactly to 1000000000000 at parse time by expand_token_decimal (crates/blockwatcher-expr/src/lexer.rs). This reads “more than one million whole units of a six-decimal token.”

tx.from != 0x0000000000000000000000000000000000000000 && args.value > 0

tx.from is declared address; the hex literal is a byte string compared byte-for-byte, regardless of length: lex_hex (lexer.rs) doesn’t require 20 bytes or any other specific length, only an even digit count. && requires both sides to admit Bool, which both comparisons do.

"promo" in args.tags

args.tags is declared string[]; in’s left operand is a scalar, its right operand here is a path resolving to an array whose element type (Str) intersects the left operand’s family.

args.name contains "USDC" || args.name starts_with "test-"

Both contains and starts_with require Str on both sides; args.name is declared string.

args.value % 1_000_000 == 0

% requires both operands to admit Int; 1_000_000 is a plain integer literal (no exponent), and dividing or taking the modulus of a literal zero is rejected at write time by bin_type_arithmetic (crates/blockwatcher-expr/src/typecheck/mod.rs). A runtime zero divisor, by contrast, resolves to Unknown rather than panicking (covered below).

One rejection worth showing alongside the working examples, because it’s the language’s one surprising parse rule: comparison operators don’t chain.

1 < args.value < 100

This is refused at parse time with comparisons do not chain; parenthesize (crates/blockwatcher-expr/src/parser.rs:262-282) rather than silently reading as either (1 < args.value) && (args.value < 100) or the mathematically different (1 < args.value) < 100. Write it as args.value > 1 && args.value < 100 instead.

Grammar and precedence

The parser is recursive descent, one function per precedence level, from loosest binding to tightest (crates/blockwatcher-expr/src/parser.rs):

1. ||
2. &&
3. ! (prefix; its operand is itself parsed at this level, so ! chains, and it wraps an entire comparison rather than binding inside one: !a.x == 1 parses as !(a.x == 1), not (!a.x) == 1)
4. Comparisons: == != < <= > >= in starts_with ends_with contains (non-chaining, exactly as shown above)
5. + -
6. * / %
7. Unary -
8. Atoms: integer/hex/string/boolean literals, field paths, (...)

in’s right-hand side is special-cased in the grammar itself, inside parse_in_rhs (parser.rs): it accepts only a literal list ([...], elements literals-only, one level of nesting so an address allowlist of any width never counts as “deep”) or a bare field path, never an arbitrary expression, so args.x in [1 + 2] is rejected as a list-syntax error before type-checking ever runs.

Operators and functions

This table is exhaustive against crates/blockwatcher-expr/src/parser.rs’s BinOp enum (Or, And, Eq, Ne, Lt, Le, Gt, Ge, In, StartsWith, EndsWith, Contains, Add, Sub, Mul, Div, Mod) plus the unary node kinds, Not and Neg. There is no function call syntax, no user-defined name, and no loop construct anywhere in the grammar. starts_with/ends_with/contains/in are keywords lex_ident recognizes, not calls (lexer.rs), which is why the language has no general extensibility surface beyond what the parser hard-codes.

Literal forms

Field paths

A path is namespace.field, optionally followed by more .segments, parsed by parse_path (parser.rs). Two things resolve past the first dot, in order, inside SchemaSet::resolve (crates/blockwatcher-types/src/schema.rs):

• A dotted flattened name, matched whole. A decoder that flattens a nested ABI parameter into order.maker is what makes args.order.maker resolve: the whole remainder after the namespace is one field name, checked as a unit, not traversed component by component.
• A trailing run of digit segments, derived through a declared array type. args.tags.0 needs no declaration of its own; it resolves through tags’s declared Array(Str) to Str, one array layer unwrapped per digit segment. A digit segment too large to fit a usize is rejected: such an index could never be read back at evaluation time, so admitting it would compile a predicate that resolves to Unknown forever.

A flat declaration always wins over derivation when both would apply to the same spelling: a decoder’s explicit word for what it emits takes priority.

The type system: families, not exact types

The type-checker’s own vocabulary, the Family enum (crates/blockwatcher-expr/src/typecheck/mod.rs), is coarser than the canonical value model: Int and Uint collapse into one family, Int, because every arithmetic and ordering operator treats them identically; Address and Bytes collapse into one family too, because a predicate only ever compares them as raw bytes. A path can carry more than one family at once when heterogeneous selectors declare the same name at different types: a monitor with two selectors, one where args.id is a uint256 and another where it’s a bytes32, type-checks an operation against args.id if either declared type admits it.

Two consequences worth knowing:

• A negative literal against an all-Uint field never gets as far as running. args.value == -1 against a field declared only uint256 fails to compile with 'args.value' is unsigned and can never satisfy this comparison (checked in crates/blockwatcher-expr/src/typecheck/mod.rs:182-204, the message itself built by diagnostics.rs:282-293). An unsigned value can never equal or be less than a negative number, so the predicate would never fire, and that’s caught before it’s ever stored rather than discovered later as a monitor that silently never matches. The opposite direction (args.value > -1, always true for a Uint) compiles: it’s loud in a different way, matching everything, which is visible, unlike a predicate that matches nothing.
• A type error names the schema field it’s about whenever one side of the failure is a path, and otherwise names the computed families on both sides. args.name + 1 (a Str field used in arithmetic) reports 'args.name' is declared as [Str], which does not support arithmetic; 1 > "x" (neither side a field) reports ordering requires an integer, but a string was given.

Evaluation: three-valued, not two

Evaluation (crates/blockwatcher-expr/src/eval.rs) never errors and never panics. Every subexpression reduces to one of three truth values (True, False, or Unknown), and Unknown has exactly three sources: a field the occurrence doesn’t carry at all, an explicit null, or a runtime value that the operation in question cannot digest (possible only through a heterogeneous field, since the type-checker already confirmed some declared type admits the operation). matches() collapses Unknown the same way it collapses False (no match), which is what makes a predicate over a field only some of a monitor’s selectors produce safe to write: an event that never carries the field simply never satisfies that clause, on either side of a negation.

&& and || follow the standard three-valued (Kleene) tables rather than treating Unknown as either extreme:

The practical effect: False && Unknown is False (nothing on the unresolved side could rescue a conjunct that already failed), but True || Unknown is True for the same reason in reverse. Neither of those outcomes requires evaluating the Unknown side at all, so evaluate short circuits exactly where the table already has an answer (eval.rs).

Integer comparisons are exact at every width the canonical value model carries: Value::Int/Value::Uint wrap num-bigint’s arbitrary-precision types, so a comparison against a number one above u128::MAX is not approximated: it is pinned directly in blockwatcher-expr’s own test suite (comparisons_are_exact_past_u128, eval.rs), and Int and Uint compare mathematically regardless of which one a decoder happened to spell a value as. Division and modulus truncate toward zero; a runtime zero divisor (reachable only through a heterogeneous field or an arithmetic expression whose value depends on the event) resolves to Unknown, never a panic.

Errors and the did-you-mean suggestion

Every PredicateError variant carries an operator-facing message, and two (UnknownField and UnknownNamespace) carry a suggestion when one exists. The mechanism is suggestion’s bounded edit-distance search (crates/blockwatcher-expr/src/typecheck/diagnostics.rs): every field the resolved namespace actually declares is compared against what was typed, a two-row dynamic-programming pass that bails out the moment a candidate provably can’t come in under the threshold, and the closest candidate within distance 3 is offered. Ties keep whichever declared name comes first. Given this page’s example schema, args.vlaue > 100 (a five-character transposition, edit distance 2 from args.value) is rejected at write time with unknown field 'args.vlaue' — did you mean 'args.value'? (the em-dash is part of the literal error text, verbatim from the code that emits it). Beyond distance 3 there is no suggestion at all: a wildly wrong name gets a plain rejection rather than a misleading nudge. A namespace typo (argss.value) gets the list of every namespace this monitor’s schema actually declares instead of a single guess, since there’s no declared spelling to measure distance against: the operator wrote a namespace, not a typo of a field.

This is a different mechanism from the one Resources describes for an unknown event or function name in a selector: that suggestion is simply the spec’s first declared name of the right kind, not an edit-distance match. The two live in different crates (blockwatcher-expr for predicates, blockwatcher-evm for selectors) because they check different vocabularies against different failure shapes.

Gates

A predicate answers a per-event yes-or-no. A gate answers a question that depends on other hits of the same monitor: did Transfer occur three times in the last hour of event time? Has this monitor already alerted in this window? The engine asks that question after selector and predicate succeed, and before a match exists.

Omit gate and the engine behaves as it does today: every predicate-true decode is a match. Name a gate module the same way a sink names its module ({ "module": "threshold", "config": { … } }), and the engine offers each hit to that compiled module together with an engine-owned journal of earlier holds.

Delivery guarantees still describe what happens after a match (or digest) is minted. This page is the seam before that mint. Sink throttle and aggregate are the wrong layer for “count three Transfers”: they run after a Match already exists, and aggregate stalls the checkpoint for the whole window.

flowchart LR
    rpc["RPC endpoints<br/>(external chains)"]
    sources["Sources<br/>evm-rpc · evm-mempool"]
    sinks["Sinks<br/>webhook · script · log"]
    storage["Storage<br/>checkpoints · dead letters · resources"]
    api["REST API"]
    metrics["Metrics"]
    engine["engine<br/>bounded channels · checkpoints"]

    subgraph pipeline["Engine pipeline"]
        direction LR
        decoder["Decoder"]
        matcher["Matcher<br/>predicates"]
        gate["Gate<br/>threshold · max_once"]
        decoder --> matcher
        matcher --> gate
    end

    rpc --> sources
    sources -->|"decode and match"| decoder
    gate --> sinks
    api -->|"manages resources"| storage
    storage <--> engine
    engine -->|"drives"| pipeline
    engine -.->|"reports"| metrics

    classDef module fill:none,stroke:#a9a3e3
    classDef core fill:none,stroke:#8a8d86,stroke-dasharray: 5 5

    class rpc,sources,sinks,decoder,matcher,gate module
    class engine,api,storage,metrics core
classDef dim fill:none,stroke:#999999,color:#999999,opacity:0.35
classDef focus fill:#ffd43b,stroke:#d9480f,stroke-width:3px,color:#1a1a1a
class rpc,sources,sinks,storage,api,metrics,engine,decoder,matcher dim
class gate focus
click sources "selectors.html"
click decoder "chain-agnosticism.html"
click matcher "predicates.html"
click gate "gates.html"
click sinks "delivery.html"
click storage "resources.html"
click api "../reference/http-api.html"
click metrics "../reference/observability.html"
click engine "pipeline.html"

Key takeaways

• A gate is a sixth port: one compiled decision per monitor over an engine-owned hit journal. The module never opens sqlite.
• Retain / Discard complete with zero outstanding: holding a hit does not stall the checkpoint. Only an Emit journals a delivery.
• Window membership uses decoded block.timestamp (unix seconds), never wall-clock time. Catch-up does not dump history into “now.”
• On invalidate the engine prunes journal rows with cursor strictly after from. It does not drain the whole bag. A tip reorg must not wipe holds from before the fork.
• One gate per monitor. Compose behaviours with two monitors. Per-event filters stay in the predicate.

Assumptions (what the operator is agreeing to)

Guarantees (what the engine actually promises)

Contrast operators will get wrong

Where it sits

flowchart LR
    src[Source] --> dec[Decode]
    dec --> pred[Predicate]
    pred --> gate[Gate]
    gate --> sink[Sink]

Inside Processor::process_one (crates/blockwatcher-core/src/pipeline/processor.rs), after a monitor accepts a decoded event:

1. If the monitor has no gate, mint a match as today.
2. If it has a gate, load that monitor’s persisted hits, append this hit, call Gate::on_hit, persist the resulting journal, then either stay quiet or mint SinkEvent::Match / Digest from the indices the module returned.
3. Quiet (Retain or Discard) registers outstanding 0 for this monitor’s sinks. Other monitors on the same raw event still add theirs.
4. The raw event still registers with Progress exactly once, after every monitor has been considered, for the total outstanding across all of them, including zero.

A hit the gate kept is not a match. Sinks never see it. Dead-letter replay cannot resurrect it. The only durable trace is the gate_hits row (and the metrics on Observability).

Resource shape

On Monitor, sibling of predicate (crates/blockwatcher-types/src/resource.rs):

{
  "id": "usdc-burst",
  "network": "eth-mainnet",
  "selectors": [{ "spec": "erc20", "events": ["Transfer"] }],
  "predicate": "args.value > 0",
  "gate": {
    "module": "threshold",
    "config": { "count": 3, "window_ms": 3600000 }
  },
  "actions": ["alerts"]
}

"gate": { "module": "max_once", "config": { "window_ms": 3600000 } }

gate is this monitor’s decision rule (like predicate), not a shared resource id (like actions: SinkId[]). Journals are per (pipeline, monitor). The envelope is ModuleSel-shaped: module plus config object, deny_unknown_fields on the envelope. The named module validates config at write (Modules, P9). An unknown module name is 422, listing catalog names, the same as an unknown sink module.

gate omitted: no module is constructed; the engine emits the hit.

There is no gates array. Order of stacked gates would be a footgun (max_once before threshold never reaches count 3). Two monitors with the same selectors and different gates are how an operator composes behaviours.

Decisions: Retain, Discard, Emit

Gate::on_hit (crates/blockwatcher-ports/src/gate.rs) is total after a successful compile, the same way Matcher::matches is total after compile. It sees the journal including this hit as the last element and returns exactly one of:

The module returns indices, never a payload it invented. Core calls existing Match::new on each chosen DecodedEvent.

A buggy on_hit after compile is counted (blockwatcher_gate_decision_invalid_total) and treated as Discard of this hit only. It does not stall the pipeline.

Event time

Time-window gates require block.timestamp in the compiled schema (unsigned or non-negative int). Otherwise the write refuses:

gate requires 'block.timestamp'; this monitor's selectors do not expose it

Window membership:

(newer_ts.saturating_sub(older_ts) as u128) * 1000 <= window_ms

window_ms is 1..=86_400_000 (one day, inclusive). threshold.count is 2..=10_000.

A predicate-true hit that still has no usable timestamp at runtime is not inserted, does not stall, and increments blockwatcher_gate_untimestamped_total. That counter is a decoder hole the write-time check is supposed to have closed; it is not a third way to configure a gate.

Shipped modules

Both crates depend on types + ports only. Catalog family gate, same register path as sinks (build_catalog in crates/blockwatcher-embed/src/catalog.rs).

threshold: session digest

Config: { "count": u32, "window_ms": u64 }.

On a hit, drop a prefix until the remaining span fits window_ms (those drops are internal, not operator-visible, not dead letters). If len >= count, Emit the oldest count indices. Else Retain.

Worked example, count: 3, one-hour window:

• T1@5m, T2@20m, T3@30m → one digest [1, 2, 3]. Journal empty of them.
• T4@45m, T5@50m, T6@55m → second digest [4, 5, 6].

That is reset-on-fire, not “wait out the original hour,” and not a cooldown that would drop T4–T6. Leftover hits that were not part of the emitted N stay in the journal.

max_once: first hit per window

Config: { "window_ms": u64 }.

If last_emit_ts is none, or this event_ts is outside the window from last_emit_ts, Emit([this]) and record last_emit_ts. Else Discard([this]). The bag does not grow.

The payload is the first hit as a Match, not a digest of everything seen in the hour. Later in-window hits are gone: not dead-lettered, not replayable. If the operator needs those hits later, that is a different product (follow-up), not throttle.

Passthrough

Omit gate. Tests also ship an in-memory fake that always Emit([this]); it is not an operator-facing module in a default binary the way threshold is.

Persistence and crash

Storage (sqlite schema v4 and every Storage impl, including memory and the ports fake) keeps:

• gate_hits: (pipeline, monitor, cursor_primary, cursor_secondary, event_index) plus event_ts and the decoded event JSON.
• gate_meta: (pipeline, monitor) → last_emit_ts, last_emit_cursor, optional opaque aux blob.

Identity is those columns, never a string-joined key.

After insert, if that monitor has more than 10_000 rows, the oldest are dropped and blockwatcher_gate_hits_dropped_total ticks.

Persist the new journal before completing the event. An Emit writes the journal and meta in one storage transaction so a crash cannot drop the cooldown after the emitted rows are already gone. A persist failure begins Progress with outstanding 1 and does not complete the guard: the checkpoint stays on this cursor, same as a closed sink channel. Skipping begin would let a later cursor complete and strand this one behind it.

Restart: sqlite still has the holds; the next hit sees them. Memory storage does not survive a process exit, matching its own contract.

Delete or rewrite a monitor (including changing gate.module or config): delete that monitor’s gate_hits and gate_meta. Holds are not migrated across compiled artifacts.

Invalidate

After drain, before rewind/restart, for that pipeline:

1. Delete gate_hits rows whose cursor is strictly after from.
2. Clear last_emit_ts / last_emit_cursor only when last_emit_cursor is strictly after from. An empty journal is not stale: after Emit those rows are gone on purpose, and a tip reorg must not reset max_once. A reorg of the emitted hit itself (cursor after from) does clear the cooldown so the replacement on the new fork can fire.
3. Existing delivery-journal retract: Retracted for already-emitted Matches after from, as Delivery guarantees already describes.

Replay from from feeds on_hit again. Duplicate Emit is at-least-once; Match ids are deterministic.

Forbidden: DELETE FROM gate_hits WHERE pipeline = ? with no cursor predicate. A two-block tip reorg must not wipe holds with cursor ≤ from. Replay only re-emits after from; those earlier holds would never come back, and a later threshold would under-fire.

Gate::on_invalidate exists for follow-up aux (e.g. forever-distinct keys). threshold and max_once do not need it. The engine has already pruned holds before that hook runs; a module must not use the hook to drain the journal.

Dry-run

POST /monitors/{id}/test stays inert: no persist, no sink, no checkpoint. It constructs a fresh empty journal for the request and feeds payloads or fetch events in order. It does not read stored gate_hits. A three-payload body can fire a threshold digest. A single payload against threshold with count: 3 reports a retained hit and an empty matched list.

TestReport reports flattened emitted Matches (existing clients still see the events that would have been delivered) plus held (Retain decisions) and dropped (Discard decisions) without emitting.

What a gate is not

• Not a predicate. “Odd block number” / “before noon UTC” belong in predicate (block.number % 2, block.timestamp % 86400).
• Not sink throttle. Throttle caps deliveries after a Match exists and dead-letters the rest (replayable). max_once drops later hits.
• Not sink aggregate. Aggregate batches Matches and stalls the checkpoint until the window closes. A 60-minute aggregate window is a 60-minute checkpoint stall. A 60-minute threshold window is not.
• Not on_tick. A gate cannot fire because “an hour passed with no hit.” That needs a timer-driven path and a checkpoint-without-cursor spec, which this version does not have.
• Not module-owned storage. A gate crate that opens sqlite, or that clears all RAM on any reorg, is not this port.

Chain-agnosticism

The gate sees Cursor and canonical Value only. event_ts is the block.timestamp path the decoder already put in the tree. Chain-agnosticism is unchanged: core does not learn a chain to “do windows.” A decoder that does not expose block.timestamp cannot use a time-window gate; the write fails closed.

Sources must provide event timestamps

A time-window gate windows on decoded block.timestamp, which presumes a block: a feed of not-yet-mined data has none to stamp. Each source module declares SourceCaps { event_timestamps: bool } (crates/blockwatcher-ports/src/source.rs). evm-rpc declares true; evm-mempool declares false, since a pending transaction has not been included in a block yet.

PUT /monitors/{id} checks the target network’s source caps against what the compiled gate needs, the same 422 compile_failed class every other compile refusal in this document uses. A gated monitor is invalid configuration on a network whose source does not declare event_timestamps, not a monitor that silently never fires. The same check runs the other way: PUT /networks/{id} refuses to swap a network’s source module out from under an existing gated monitor, because doing so would leave that monitor pointed at a feed its gate can no longer window against. Neither route lets the pairing exist for even one write.

Delivery guarantee

An emission’s durability starts inside the same storage transaction that consumes the journal members it was built from: Emit does not exist as a decision without also existing as a row a sink can still be delivered from. The sink worker deletes that row only once the delivery outcome itself is durable: landed in the delivery journal, or recorded as a dead letter, never before. A crash between those two points redelivers on the next start rather than losing the emission: duplicates are possible, gaps are not. A pending emission outlives the monitor that minted it: deleting the monitor leaves its outbox row in place, and the row is still delivered, or dead-lettered when its sink no longer has a worker, on the next start. Deleting the emission’s network dead-letters the row right away instead, because a deleted id gets no next start for the drain to run at: each contained match becomes a dead letter for the pipeline id, and the dead-letter surface keeps answering for any id that still holds letters, so they stay listable, replayable, and discardable while the network stays deleted. Whatever that delete-time sweep could not finish (a storage fence that did not resolve, a lapsed budget, a failed write) is dead-lettered by the next boot’s orphan sweep, which also covers rows stranded before the sweep existed. Every pipeline’s pending rows are counted by /status as gate_outbox_depth, read fresh from storage, so an emission parked behind a paused network stays visible until a resume delivers it. See Delivery guarantees for the outbox’s shape and the startup drain that sweeps whatever a crash left behind. Invalidate reaches into a pending emission’s row the same way it reaches into the hit journal: reorged-out members of the row are pruned at invalidation, and their canonical replacements re-accumulate through the gate on replay.

Selectors

Every entry in a monitor’s selectors list is a selector: a reference to one contract spec, an optional address restriction, and a choice of what that spec has on offer to actually decode. But naming something in a selector is only half the story: what it can produce also depends on which source module happens to be feeding its network, since evm-rpc and evm-mempool hand the decoder two very different kinds of raw material. This page walks both halves: the compile-time selection rule in crates/blockwatcher-evm/src/decoder/selector.rs, then what each source in crates/blockwatcher-evm/src/source/ actually supplies for it to work with.

flowchart LR
    rpc["RPC endpoints<br/>(external chains)"]
    sources["Sources<br/>evm-rpc · evm-mempool"]
    sinks["Sinks<br/>webhook · script · log"]
    storage["Storage<br/>checkpoints · dead letters · resources"]
    api["REST API"]
    metrics["Metrics"]
    engine["engine<br/>bounded channels · checkpoints"]

    subgraph pipeline["Engine pipeline"]
        direction LR
        decoder["Decoder"]
        matcher["Matcher<br/>predicates"]
        gate["Gate<br/>threshold · max_once"]
        decoder --> matcher
        matcher --> gate
    end

    rpc --> sources
    sources -->|"decode and match"| decoder
    gate --> sinks
    api -->|"manages resources"| storage
    storage <--> engine
    engine -->|"drives"| pipeline
    engine -.->|"reports"| metrics

    classDef module fill:none,stroke:#a9a3e3
    classDef core fill:none,stroke:#8a8d86,stroke-dasharray: 5 5

    class rpc,sources,sinks,decoder,matcher,gate module
    class engine,api,storage,metrics core
classDef dim fill:none,stroke:#999999,color:#999999,opacity:0.35
classDef focus fill:#ffd43b,stroke:#d9480f,stroke-width:3px,color:#1a1a1a
class rpc,sinks,storage,api,metrics,engine,decoder,matcher,gate dim
class sources focus
click sources "selectors.html"
click decoder "chain-agnosticism.html"
click matcher "predicates.html"
click gate "gates.html"
click sinks "delivery.html"
click storage "resources.html"
click api "../reference/http-api.html"
click metrics "../reference/observability.html"
click engine "pipeline.html"

Key takeaways

• A selector’s events and functions keys are independent dispatch tables; leaving both unset selects everything the spec declares.
• events matches a decoded log (full block/log fields, never tx.from/to/value); functions matches decoded calldata (tx fields present immediately, tx.status and block.* only once mined).
• Both selector kinds compile the same way regardless of source; what differs is only what raw material a given source ever supplies to decode.
• evm-rpc can satisfy both events and functions in the same poll cycle; evm-mempool only ever produces functions, and only unmined fields.
• evm-mempool’s cursor is a per-run arrival counter, not a chain position, so it cannot support at-least-once delivery or history fetches the way evm-rpc can.

Which combination of selector kind and source actually produces something for a predicate to see comes down to one decision tree:

flowchart TD
    start{"selector kind"} -->|"events"| ev{"source?"}
    start -->|"functions"| fnq{"source?"}

    ev -->|"evm-rpc"| evlog["decodes a log<br/>full args, tx, block, log"]
    ev -->|"evm-mempool"| evnone["no-op: never fires,<br/>this source has no logs"]

    fnq -->|"evm-rpc"| fnmined["decodes a mined tx<br/>tx/block fields once mined"]
    fnq -->|"evm-mempool"| fnpending["decodes a pending tx<br/>no tx.status, no block.*"]

    evlog --> pred["predicate evaluates"]
    fnmined --> pred
    fnpending --> pred

events and functions: one presence rule

A selector body carries optional keys (events, functions, addresses) and nothing else; an unrecognized key is rejected by name in compile (crates/blockwatcher-evm/src/decoder/selector.rs). What actually gets selected turns on one boolean the compiler computes once per entry, select_all, which is true exactly when neither events nor functions appears in the body at all (selector.rs). With select_all false, each key resolves independently against whatever names it carries: writing only events: ["Transfer"] builds a dispatch table with that one event and zero functions; it never falls back to “every function” for the side you didn’t mention. With select_all true, both tables are filled from the entire vocabulary the spec exposes: nothing past that one spec’s boundary, so two specs on the same chain never bleed into each other’s “select everything,” which is why adding a new declaration to a spec silently widens an existing catch-all monitor’s reach the moment the deployment next recompiles it, without anyone touching the monitor. One spelling is refused outright rather than given either meaning: a key present but pointed at an empty array, rejected in name_list (selector.rs). “Present but empty” and “never written” would otherwise both have to mean something, and letting the empty spelling default to “select nothing” would compile an entry nobody could ever notice never fires.

Naming an event or function the referenced spec doesn’t declare rejects with a suggestion from kind_suggestion: the spec’s first declared name of that kind, not an edit-distance match (selector.rs; see Resources for why that differs from a predicate’s unknown-field suggestion). events decodes against a log’s topic0; functions decodes against a transaction’s first four bytes of calldata: two independent dispatch tables on the Entry struct inside one compiled selector (selector.rs), which is what lets one selector entry watch both at once.

What each kind decodes, and what it hands the predicate

events matches a decoded log. Its data reaches a predicate as:

• args.*: the event’s own declared parameters.
• tx.hash, tx.index: always present; every log carries its transaction’s envelope.
• tx.status: always present and always 1: a log exists only inside a transaction that succeeded, so this is a constant stamped at decode time by assemble_fields, never a receipt fetch (crates/blockwatcher-evm/src/decoder/decode.rs).
• tx.from, tx.to, tx.value: never present on a log-decoded occurrence; nothing in a log’s own envelope carries them, and fetching them would mean a receipt/transaction lookup the log-only path is designed to avoid, as documented on namespaces (crates/blockwatcher-evm/src/decoder/compile.rs).
• block.number, block.hash, block.timestamp, log.address, log.index: always present; every log’s own envelope carries all five.

functions matches a decoded transaction’s calldata. Its data reaches a predicate as:

• args.*: the function’s own declared parameters, decoded regardless of whether the call reverted: calldata decodes independent of the receipt.
• tx.hash, tx.from, tx.value: always present; every transaction carries them by definition.
• tx.to: present unless the transaction is a contract creation, which carries no to at all.
• tx.index, tx.status, and every block.* field: present only once the transaction is mined. A pending transaction has none of them yet, since they are TxEnvelope’s optional fields, assembled by assemble_call_fields (crates/blockwatcher-evm/src/decoder/decode.rs), and a predicate reading one before that resolves Unknown, never an error, never a fabricated value.
• No log.* namespace at all: nothing about a function-call decode came from a log, as documented on assemble_call_fields (decode.rs).

Because a log only exists when its transaction succeeded, an events selector has no way to notice a revert at all: the occurrence it would decode is never emitted in the first place. A functions selector sees the call regardless of how it ended, since decoding calldata never touches the receipt; only its tx.status field carries the outcome, and only once the transaction is mined. Watching for failed calls is therefore a functions concern exclusively.

The source: what raw material even reaches the selector

Both selector kinds compile the same way regardless of source. What differs is what a given source ever calls decode with, and decode itself dispatches purely on the shape of that payload: an object carrying topics is a log, one carrying input (and no topics) is a transaction, and nothing ever carries both (crates/blockwatcher-evm/src/decoder/decode.rs).

evm-rpc scans confirmed blocks: it fetches full block bodies (headers plus every transaction) and eth_getLogs results in the same poll cycle, so one network running this source can satisfy both selector kinds at once: whatever a selector’s events half names comes from that cycle’s logs, whatever its functions half names comes from that cycle’s mined transactions, carried on one non-decreasing cursor stream, packed by pack_secondary and emitted by emit_verified_leaf, that always walks a block’s transaction list to completion before touching that same block’s logs (crates/blockwatcher-evm/src/source/rpc/emit.rs). A selector written with only one of the two keys simply gets fed from only the matching half of that cycle. Whether the source bothers fetching full transaction bodies at all is itself interest-driven: a pipeline with no monitor watching any functions selector never asks for them, in scan_range (crates/blockwatcher-evm/src/source/rpc/emit.rs, mirrored in the streaming path).

evm-mempool, by contrast, watches a single node’s stream of not-yet-mined transactions and never sees a log in its entire lifetime: mining is the event that produces a receipt, and a log lives inside one, so an occurrence this source hands the decoder is calldata or nothing. That means the only payload shape it ever produces is one decode routes down the functions path, as documented on EvmMempoolSource (crates/blockwatcher-evm/src/source/mempool/run.rs). A selector’s events half compiles cleanly against this source’s network too (selection is checked against the spec, not the source that will feed it), but can never contribute a single match, because there is nothing here for it to decode. Only the functions half of any selector on such a network ever does anything, and if the spec behind it has no functions in it to begin with, that selector on this source is a complete no-op, matching nothing ever. The source itself checks whether any monitor on its pipeline watches functions before paying for a lookup; with no function interest published, it skips the eth_getTransactionByHash round trip entirely and forwards nothing, in run (source/mempool/run.rs).

The position problem: why evm-mempool can’t make the same promises

Every other part of a compiled selector is source-independent: the same schema, the same dispatch table, the same predicate. What genuinely differs between evm-rpc and evm-mempool is what each one’s cursor is even counting. evm-rpc answers a question a mined chain can always answer, “where in the chain is this?”, with a block number and an in-block ordering bit that puts every transaction ahead of every log it shares a block with. evm-mempool has no such question to answer: nothing pending has a place in the chain yet, so its cursor counts something else entirely: how many occurrences this one process has forwarded since it started, tracked by run’s arrival counter (crates/blockwatcher-evm/src/source/mempool/run.rs).

This is not a detail that stays contained inside the source. Because a checkpoint is exactly that cursor plus enough state to verify a resume (crates/blockwatcher-core/src/pipeline/checkpoint_writer.rs), evm-mempool’s persisted checkpoint is a dedupe watermark for this one process’s lifetime, never a replayable position: whatever was pending in the node’s mempool while the process was down is simply gone on restart, and nothing resumes it, unlike evm-rpc’s checkpoint, which always names a real block to continue scanning from. scan and confirmed_tip are therefore unsupported on this source outright: there is no history to fetch and no confirmed tip to report (source/mempool/run.rs).

That instability reaches all the way into how a match is identified. Restart this source and its arrival counter starts over from wherever the fresh checkpoint left off, so the identical pending call, seen again after the gap, is minted a different number and therefore a different match id. There is no way to prevent this, because nothing about a hash-derived position can be made to behave monotonically across a process boundary, which is exactly what the Source port requires of a cursor. The same drift can happen inside one run, with no restart at all: a transaction the subscription already announced can get mined while its hydration lookup is still outstanding, and the copy that comes back then carries a real transactionIndex it didn’t have a moment before, changing the decoded fields (and the id derived from them) out from under it, as documented on EvmMempoolSource (crates/blockwatcher-evm/src/source/mempool/run.rs). The fix lives with whatever consumes these matches, not with the source itself: read the transaction’s own hash back out of the delivered payload and deduplicate on that, since it is the one thing two sightings of the same pending call are guaranteed to agree on. It’s exactly this instability that keeps evm-mempool outside the reach of at-least-once delivery: every other shipped source can promise it, this one cannot.

Comparison table

Predicates covers how a predicate reads any of this data once it’s decoded; The pipeline covers where decoding and matching actually run and what’s per-network versus shared.

Delivery guarantees

The pipeline named the Progress tracker as the thing that turns “every dispatched match done” into “safe to persist this cursor.” This page opens that mechanism up: what “done” means precisely, what happens when it never becomes true, and what a crash actually costs a consumer versus what it never costs them. Delivery scenarios is this page’s companion for a different question: given a monitor, a sink, and whatever policies are configured on it, what actually happens in the situation in front of you right now, walked as flow diagrams rather than field-by-field prose.

flowchart LR
    rpc["RPC endpoints<br/>(external chains)"]
    sources["Sources<br/>evm-rpc · evm-mempool"]
    sinks["Sinks<br/>webhook · script · log"]
    storage["Storage<br/>checkpoints · dead letters · resources"]
    api["REST API"]
    metrics["Metrics"]
    engine["engine<br/>bounded channels · checkpoints"]

    subgraph pipeline["Engine pipeline"]
        direction LR
        decoder["Decoder"]
        matcher["Matcher<br/>predicates"]
        gate["Gate<br/>threshold · max_once"]
        decoder --> matcher
        matcher --> gate
    end

    rpc --> sources
    sources -->|"decode and match"| decoder
    gate --> sinks
    api -->|"manages resources"| storage
    storage <--> engine
    engine -->|"drives"| pipeline
    engine -.->|"reports"| metrics

    classDef module fill:none,stroke:#a9a3e3
    classDef core fill:none,stroke:#8a8d86,stroke-dasharray: 5 5

    class rpc,sources,sinks,decoder,matcher,gate module
    class engine,api,storage,metrics core
classDef dim fill:none,stroke:#999999,color:#999999,opacity:0.35
classDef focus fill:#ffd43b,stroke:#d9480f,stroke-width:3px,color:#1a1a1a
class rpc,sources,api,metrics,engine,decoder,matcher,gate dim
class sinks,storage focus
click sources "selectors.html"
click decoder "chain-agnosticism.html"
click matcher "predicates.html"
click gate "gates.html"
click sinks "delivery.html"
click storage "resources.html"
click api "../reference/http-api.html"
click metrics "../reference/observability.html"
click engine "pipeline.html"

Key takeaways

• A checkpoint only ever advances past the longest contiguous run of fully-done events from the front; one stuck event holds back everything behind it.
• Overload is backpressure, not data loss: bounded channels stall the pipeline rather than dropping a message.
• A crash can only repeat work, never skip it, because a cursor only advances once every match tied to it is finished.
• Deterministic match ids let a consumer treat a redelivered duplicate as the same occurrence rather than a new one.
• A delivery that exhausts its retry budget becomes a durably recorded dead letter, and the guard only completes once that write lands.
• Sinks consume a closed SinkEvent set: Match to apply, Retracted to undo by match_id, or Digest to apply several matches bundled by aggregation at once. All three are at-least-once; retracts and digests share the same retry and dead-letter path as a plain match.
• A deep invalidate is a typed source outcome, not a failure: the engine drains, retracts journaled ids, rewinds, then restarts. Retracts for a sink finish before any replacement Match from that network.
• A monitor gate decides whether a predicate-true hit becomes a match at all. Quiet gate hits are not deliveries, not dead letters, and do not hold the checkpoint. Sink throttle / aggregate still apply only after a match exists.

The contract, precisely

A network’s checkpoint is a cursor plus whatever the source needs to verify a resume. Progress (crates/blockwatcher-core/src/progress.rs) is the structure that decides when a cursor is safe to hand the checkpoint writer: it tracks every registered event as a Slot in arrival order, each carrying a remaining count of outstanding matches, and only ever publishes the cursor of the longest contiguous run of fully-done slots starting from the front, in advance_locked (progress.rs). “Fully done” means every match dispatched from that event has either reached its sink or been recorded as a dead letter. There is no third outcome. One still-open slot anywhere in that prefix holds every slot behind it back, even ones that finished long ago (CompletionGuard::complete, progress.rs, exercised directly in middle_event_finishing_last_gates_the_prefix, progress.rs).

Three consequences follow directly from that rule:

• Checkpoints only ever move forward past finished work. A CompletionGuard dropped without calling complete() (a worker task dying mid-delivery, say) permanently stalls the prefix at that slot rather than letting it expire or time out (progress.rs; see a_dropped_guard_stalls_the_checkpoint, progress.rs). A stall is visible in the status API as in_flight_events; nothing here manufactures a false “done.”
• Overload is backpressure, not data loss. The channels between source, processor, and each sink worker are bounded and never drop a message on the full side: a slow sink fills its own channel, which stalls the processor’s dispatch loop, which stalls the source’s own send. Nothing downstream of a full queue is asked to discard anything; the entire pipeline just waits.
• A crash can only repeat work, never skip it. Resuming means reading the last persisted checkpoint and continuing from there. Anything that cursor already covers is gone for good from the source’s perspective, but by the rule above that only happens once every match tied to it is finished, so the only way a crash surfaces to a consumer is by re-delivering an event whose checkpoint write hadn’t landed yet, never by silently dropping one.

Deterministic match IDs are what make that repetition harmless. MatchId::derive hashes the network, monitor, cursor, an index within the event, and a structural walk of the decoded event’s own field tree (crates/blockwatcher-types/src/id.rs): nothing about it depends on wall-clock time, retry count, or which run produced it. Re-processing the same on-chain event after a restart decodes the same bytes at the same slot and derives the exact same id (match_id_re_derivation_from_the_same_content_is_stable, id.rs), which is what lets a consumer treat two deliveries of the same id as one occurrence rather than two.

Happy path

One raw event, one match, one sink, nothing goes wrong:

sequenceDiagram
    participant Source as Source task
    participant Processor as Processor task
    participant Progress as Progress tracker
    participant Sink as Sink worker task
    participant CPW as Checkpoint writer task
    participant Store as checkpoint storage

    Source->>Processor: event at cursor C
    Processor->>Processor: decode + match (1 match)
    Processor->>Progress: begin(C, outstanding: 1)
    Processor->>Sink: dispatch match M
    Sink->>Sink: deliver Match(M) (success)
    Sink->>Progress: guard.complete()
    Progress->>Progress: prefix fully done, watermark = C
    Progress->>CPW: publish checkpoint(C)
    CPW->>Store: persist checkpoint(C)

An event with zero matches is registered too, with zero outstanding: it completes the moment begin runs, which is why an event that matched nothing still lets the cursor advance immediately rather than wait on deliveries that were never dispatched. An event whose monitors all retained or discarded at the gate is registered with zero outstanding the same way.

Crash after partial delivery, then resume

Two matches from one event, going to two different sinks. One delivery lands before the crash; the other is still retrying when the process dies. Because the checkpoint requires both guards, cursor C never got persisted. The last durable checkpoint is still whatever preceded it:

sequenceDiagram
    participant Source as Source task
    participant Processor as Processor task
    participant Progress as Progress tracker
    participant SinkA as Sink worker task A
    participant SinkB as Sink worker task B
    participant Store as checkpoint storage

    Note over Store: persisted checkpoint = C-1
    Source->>Processor: event at cursor C
    Processor->>Progress: begin(C, outstanding: 2)
    Processor->>SinkA: dispatch match M1
    Processor->>SinkB: dispatch match M2
    SinkA->>SinkA: deliver M1 (success)
    SinkA->>Progress: guard.complete() (M1)
    Note over Progress: M2 still outstanding, watermark holds at C-1
    Note over SinkB: process crashes mid-retry of M2
    Note over Store: restart resumes from C-1 (never advanced)

    Source->>Processor: re-emit event at cursor C
    Processor->>Progress: begin(C, outstanding: 2)
    Processor->>SinkA: dispatch match M1 (same MatchId as before)
    Processor->>SinkB: dispatch match M2 (same MatchId as before)
    SinkA->>SinkA: deliver M1 again (duplicate)
    Note over SinkA: consumer recognizes the duplicate by M1's id
    SinkA->>Progress: guard.complete() (M1)
    SinkB->>SinkB: deliver M2 (success this time)
    SinkB->>Progress: guard.complete() (M2)
    Progress->>Progress: prefix fully done, watermark = C
    Progress->>Store: checkpoint(C) persisted

Nothing about which sink had already succeeded survives the crash: the in-memory Progress state is gone, and the only source of truth on restart is the last persisted checkpoint. Re-reading from C-1 means the whole event at C is decoded and matched again, M1 included, which is exactly the duplicate a consumer’s dedup-by-id exists for.

Dead letters

When deliver_with_retry (crates/blockwatcher-core/src/pipeline/delivery.rs) exhausts a sink’s retry budget (every attempt spent on a transient or rate-limited failure, or a permanent/retry-narrower error that skips retry entirely), the sink worker does not drop the event. It builds a DeadLetter carrying the event’s match id, which monitor and sink produced it, the cursor it traces back to, the attempt count, a reason string, and the original SinkEvent payload, and writes it through the network’s storage backend, in dead_letter (crates/blockwatcher-core/src/pipeline/sink_worker.rs). Only after that write durably lands, and, for a Match, only after the delivery journal row is written, does the worker complete its guard, in dead_letter’s success branch (sink_worker.rs); if the write itself fails, the worker retries the write forever on its own doubling backoff rather than completing early, in dead_letter’s retry loop (sink_worker.rs): the same “never advance past an unrecorded loss” rule as everywhere else in this contract, just applied one layer further out. A dead-lettered Match stays in the journal so a later invalidate can still retract it; a dead-lettered Retracted does not.

An operator interacts with a network’s dead-letter queue through these routes, in router (crates/blockwatcher-api/src/routes/networks_ops.rs):

• GET /networks/{id}/dead-letters pages the queue in insertion order: a pure storage read, in list_dead_letters (crates/blockwatcher-core/src/control/dead_letters.rs).
• POST /networks/{id}/dead-letters/{match_id}/replay re-runs the same deliver_with_retry path against the sink the letter names, rebuilt fresh from its stored config. Success deletes the letter (204); exhaustion bumps its attempts and reason in place and leaves it queued, in replay_dead_letter (dead_letters.rs). A letter recorded before payload storage was added has no payload to replay and is refused outright rather than silently no-op’d (dead_letters.rs, message NO_REPLAY_PAYLOAD). A letter whose payload is SinkEvent::Retracted is likewise refused (NO_REPLAY_RETRACTED): only match events can be replayed.
• DELETE /networks/{id}/dead-letters/{match_id} and DELETE /networks/{id}/dead-letters (the latter with optional sink/monitor filters, returning the count discarded) discard letters outright, without attempting delivery, in discard_dead_letter/discard_dead_letters (dead_letters.rs). Replaying is the only route that resends a letter; discarding is permanent.

The sqlite storage backend is what makes a dead-letter queue survive a restart at all; the memory backend does not, matching its own no-persistence contract.

Delivery policies

Everything on this page is configured per sink, on the SinkDef resource (a JSON document written through PUT /sinks/{id} or a seed directory, hot-reloaded like every resource): nothing here is hardcoded or instance-config. One complete sink showing every policy field beside the module’s own config:

{
  "id": "alerts",
  "module": "webhook",
  "config": {
    "url_secret": "env:ALERTS_WEBHOOK_URL",
    "headers": { "x-source": "blockwatcher" },
    "timeout_ms": 10000,
    "body_template": "{% if type == \"digest\" %}{{ matches | length }} matches{% else %}{{ type }}: {{ monitor }}{% endif %}"
  },
  "retry": { "max_attempts": 8, "initial_backoff_ms": 200, "max_backoff_ms": 30000 },
  "throttle": { "max_deliveries": 60, "window_ms": 60000 },
  "aggregate": { "window_ms": 30000, "max_batch": 100 }
}

config belongs to the named module (here webhook; its fields, including the optional body_template renderer, are that module’s own documentation). retry, throttle, and aggregate are the engine’s, identical for every module, each optional: omit retry for the engine-wide default, omit throttle for no throttling, omit aggregate for unbatched delivery. The values shown are each field’s defaults.

A retry whose initial_backoff_ms sits above its own max_backoff_ms is refused by the same validate call throttle and aggregate go through, at the same two points described below for aggregation: put_sink refuses the write, validate_and_build refuses to boot a stored row that fails the same check. DeliveryRetry::backoff_bounds (crates/blockwatcher-types/src/resource.rs) is what every retry loop reads its starting backoff and ceiling from, deliver_with_retry and the sink worker’s journal, dead-letter, and forget retries alike, so the floor against an out-of-range pair lives in one place rather than at each of those call sites.

A sink’s throttle field (SinkDef.throttle: Option<Throttle>, crates/blockwatcher-types/src/resource.rs) caps how often that sink admits successful deliveries, independent of the retry and dead-letter policy above. The fields that define it:

• max_deliveries: the number of successful Match deliveries the window admits before it starts refusing more, default 60.
• window_ms: the window’s length in milliseconds, default 60_000.

A zero in either field is refused at write time, the same “validate before constructing anything” discipline every resource write goes through: a sink’s throttle is either absent or fully valid, never partially so.

At runtime, SinkWorker (crates/blockwatcher-core/src/pipeline/sink_worker.rs) holds a fixed admission window per sink, anchored at the first admitted delivery and reset once window_ms has elapsed. The window belongs to one network’s pipeline, not to the sink id globally: a sink named by monitors on several networks gets one independent window per network, each with its own budget, rather than one shared budget across all of them. A Match that arrives inside the current window’s budget is delivered normally; a Match that would exceed it never reaches the sink at all. It is dead-lettered instead, through the same dead_letter path an exhausted retry budget uses, with a reason naming the policy that suppressed it:

permanent: throttled: policy allows 60 deliveries per 60000ms; replay this dead letter to deliver it

The permanent: class prefix is the same one every dead-letter reason carries; nothing about the throttle path is a distinct class of dead letter. A throttled match is exactly as listable and replayable as any other dead letter, through the same GET /networks/{id}/dead-letters and POST /networks/{id}/dead-letters/{match_id}/replay routes described above: replaying one delivers it directly, without re-checking the window it was suppressed under.

Only a delivered Match spends budget. A failed attempt already pays for itself through the ordinary retry-then-dead-letter path and must not also consume quota meant for the next real delivery. Retracted events skip the throttle check entirely, at the same call site that would otherwise apply it: a correctness signal outranks a fatigue policy, so an undo is never the thing a noisy sink drops on the floor.

The window is runtime-only state, held in memory beside the sink worker rather than persisted: a pipeline restart opens a fresh window with zero deliveries counted. An operator relying on the throttle to bound a sustained rate should not expect the count to survive a restart.

Aggregation

A sink’s aggregate field (SinkDef.aggregate: Option<Aggregate>, crates/blockwatcher-types/src/resource.rs) batches several matches into one digest delivery instead of sending each on its own. The fields that define it:

• window_ms: how long the batch stays open once the first match joins it, default 30_000, capped at 86_400_000 (one day).
• max_batch: the number of matches that closes the batch immediately, default 100, capped at 10_000.

A zero in either field is refused the same way a zero throttle field is, and so is a value above either cap: the caps bound what one open window can cost, since every match in it is held in memory and the checkpoint stays behind all of them until the window closes. max_batch prices that worst case in matches, window_ms prices it in time. Each route runs a SinkDef’s policy through the same validate call, but they enforce it at different points. put_sink (crates/blockwatcher-core/src/control/writes.rs) refuses to persist a SinkDef whose policy fails validation. validate_and_build (crates/blockwatcher-core/src/engine/boot.rs) runs the identical check against rows already in storage, a restored backup or a hand-edited row among them, and refuses to start the engine rather than refusing a write. A sink’s aggregation policy is either absent or fully valid, never partially so.

At runtime, AggregateBuffer (crates/blockwatcher-core/src/pipeline/aggregate.rs) holds one batch per sink worker, and only a Match ever joins it; the buffer flushes, delivering whatever it holds as a single event, on any of:

• the batch reaches max_batch;
• window_ms elapses since the first match joined the batch, not the most recent one, so a steady trickle still reaches the sink once per window instead of waiting forever for a batch that never fills;
• the sink worker’s channel closes, which is what a cooperative drain looks like from this worker’s side: a partial batch still in progress is flushed rather than dropped;
• a Retracted event arrives. The buffer flushes first and the retract is delivered after, so a retract can never overtake the match it retracts: per-sink delivery order is the only evidence a sink has of which of the two came last. Retracted events are never buffered themselves and never count toward max_batch, the same correctness-outranks-fatigue rule that exempts them from the throttle above.

A flush of exactly one match delivers a plain SinkEvent::Match: a quiet sink whose matches never overlap inside one window sees the identical wire format it would see with no aggregate field set at all. A flush of two or more delivers a single SinkEvent::Digest { matches }, rendered by canonical_body as:

{
  "type": "digest",
  "matches": [
    { "id": "…", "monitor": "usdc-sepolia-transfers", "network": "sepolia", "event": { } },
    { "id": "…", "monitor": "usdc-sepolia-transfers", "network": "sepolia", "event": { } }
  ]
}

Every buffered match holds its CompletionGuard uncompleted for as long as it sits in the batch, exactly as an in-flight delivery does. A buffered window stalls the checkpoint by design: the batch’s oldest match cannot pass the checkpoint until the whole batch it ends up part of is delivered or dead-lettered, even though nothing has been sent to the sink yet. A hard-cancel or crash drops those guards uncompleted rather than flushing them, the same safe-by-default direction a mid-delivery crash already takes elsewhere in this contract: on replay, the source re-emits the events those matches came from, they are re-matched, and re-buffered, and eventually re-delivered, deduplicated by MatchId exactly as any other redelivered match is. There is no persisted window state to resume from; the window itself is runtime-only, the same stance the throttle above takes.

On a successful digest delivery, every contained match is journaled at its own cursor, in arrival order, before any guard completes, the same journal-then-complete ordering a lone match’s delivery follows. On retry exhaustion, the digest does not dead-letter as one row: each contained match is dead-lettered individually, carrying the digest’s own failure reason, one match at a time and in arrival order. Per match, the record order is the same as a lone dead-lettered match’s: the dead-letter row is written first, then that match is journaled, then the next match in the digest follows the same two steps. This is the replay caveat to carry forward: a dead-lettered digest member replays as a single match, never as the digest it originally arrived in, because POST /networks/{id}/dead-letters/{match_id}/replay only ever knows how to resend one match at a time.

The delivery policies compose rather than interact specially: a digest counts as exactly one delivery against the sink’s throttle window, the same one unit a lone match would have spent, so aggregating a burst into fewer deliveries is one way an operator keeps a bursty sink under a tight throttle budget. A digest the throttle refuses is suppressed exactly like a suppressed lone match would be, except it dead-letters every contained match individually rather than one row, matching how a delivered digest is journaled.

Monitor gates versus sink policies

throttle and aggregate are fields on SinkDef. A gate is a field on Monitor. They are not interchangeable:

• threshold gate: N predicate-true hits whose block.timestamp values span ≤ window_ms mint one digest of those hits, then reset the session. Held hits persist in gate_hits and complete with outstanding 0, so a 60-minute threshold does not stall the checkpoint for 60 minutes.
• aggregate sink: already-minted Matches sit in memory and the checkpoint stays behind all of them until the window flushes. That is why a long aggregate window is operationally expensive.
• max_once gate: at most one alert per event-time window; later hits are discarded with no dead letter.
• throttle sink: later Matches are dead-lettered and remain replayable. Only successful deliveries spend budget. Restarts reset the in-memory window.

Full gate contract: Gates.

The evm-mempool exception

Everything above assumes a source whose cursor is a real position in a feed it can resume from. evm-mempool is the one shipped source where that assumption fails: its cursor is a per-run arrival counter, not a chain position, so its checkpoint is only a dedupe watermark for the current process’s lifetime: a restart cannot replay whatever was pending in the node’s mempool while the process was down, and that work is simply gone. The same instability reaches into identity: because the counter restarts from zero-relative-to-the-fresh-checkpoint on every process start, the identical pending call seen again after a gap is minted a different arrival number and therefore a different MatchId. Consumers of this source deduplicate on the transaction’s own hash, not on the match id. Selectors § The position problem covers the mechanics in full.

What a sink receives: SinkEvent

Sink::deliver takes a SinkEvent, not a bare match (crates/blockwatcher-ports/src/sink.rs):

#![allow(unused)]
fn main() {
pub enum SinkEvent {
    Match(Match),
    Retracted { match_id: MatchId },
    Digest { matches: Vec<Match> },
}
}

(crates/blockwatcher-types/src/event.rs)

That set is closed: core reasons about every variant exhaustively, the same way it reasons about the canonical value model. Digest is what aggregation delivers in place of several separate Match events; it only ever reaches a sink whose SinkDef sets aggregate. A sink module that only handles Match will silently keep work a later Retracted asked it to undo, or drop every match a later Digest bundled together.

Every variant travels the same deliver_with_retry path (crates/blockwatcher-core/src/pipeline/delivery.rs): the same retry budget, the same backoff, the same dead-letter write when the budget runs out, and the same CompletionGuard (one per contained match, for a Digest). A failed undo can stall the checkpoint or land in the dead-letter queue; it is never dropped on the floor. Consumers must treat a redelivered Retracted as idempotent on match_id, just as they already treat a redelivered Match as idempotent on Match.id, and a redelivered Digest’s contents as idempotent per contained Match.id for the same reason.

Built-in sinks (webhook, script, log) render the event as tagged JSON through one helper, canonical_body (crates/blockwatcher-sinks/src/lib.rs):

{ "type": "match", "id": "…", "monitor": "…", "network": "…", "event": { } }

{ "type": "retracted", "match_id": "…" }

{
  "type": "digest",
  "matches": [
    { "id": "…", "monitor": "…", "network": "…", "event": { } },
    { "id": "…", "monitor": "…", "network": "…", "event": { } }
  ]
}

That tagging is a wire break against a bare Match object. A DeadLetter’s payload is Option<SinkEvent>: a row written before this shape existed, or written as a legacy Match object, still loads as SinkEvent::Match. Replay through the API only accepts a Match payload; a retraction payload is refused rather than re-sent (NO_REPLAY_RETRACTED, crates/blockwatcher-core/src/control/dead_letters.rs).

Shallow vs deep invalidation

A source whose cursor is a real chain position can see its feed fork. Whether sinks hear about that fork depends on which side of the confirmation barrier moved:

──── already emitted / checkpointed ────|──── unconfirmed window ────| head
                                        ^ confirmations barrier

• Shallow. Only the unconfirmed side moves. Those cursors were not durably offered to sinks (or a mid-scan race is retried in place, inside the source). No Retracted leaves the module.
• Deep. The fork cuts into work already emitted. Restarting the source alone would offer the new fork as new Matches (often with new MatchIds) and leave the consumer holding the orphan with no undo signal.

Core never names a chain, a reorg, or a confirmation depth. A source that has positively detected a deep invalidate returns SourceOutcome::Invalidated { from } from Source::run (crates/blockwatcher-ports/src/source.rs): a typed non-success, not a SourceError. The from cursor is the proven point; everything with cursor > from previously implied by this source is on a dead fork. A source whose entire tracked history is refuted (no tracked ancestor matches the live chain at all) proves divergence only at or below the oldest height it can still vouch for, bounded by the tracker’s own depth rather than any named fork block; when that depth reaches back to block 0, the bound is genesis, but that is the tracker’s own limit speaking, not a claim the fork itself sits there. The EVM RPC source is what translates a break against already-emitted work into Invalidated at the module boundary: a walk that finds a still-matching ancestor invalidates from that proven fork (beyond_confirmations); a walk that finds none invalidates from just below the oldest tracked height (beyond_window); journaled deliveries above it are retracted the same as any other invalidate (the journal-depth caveat below still applies), and the source restarts from the rewound checkpoint. Both are the same on a mid-run break as on resume. Mempool and other sources that have no finality never produce it.

The engine supervisor handles Invalidated as a control path, not a crash (crates/blockwatcher-core/src/engine/invalidate.rs):

1. Cooperative cancel and drain of that network’s pipeline (the same deadline drain any restart uses; no abort on this path unless the deadline is missed).
2. Prune gate_hits for this pipeline where cursor > from. Do not delete rows with cursor ≤ from.
3. list_deliveries_after(network, from) against the bounded delivery journal.
4. One-shot retract pass: a SinkEvent::Retracted { match_id } for each retained id, through the same deliver_with_retry live traffic uses.
5. Wait until every retract is delivered or dead-lettered. An unrecovered retract leaves the checkpoint unrewound and does not restart.
6. Rewind the checkpoint to from, unless no write could move the cursor strictly backward: from is not behind the stored checkpoint, no checkpoint is stored at all (writing one would fabricate delivery progress), or the post-drain checkpoint read failed. A rewind must move the cursor backward or not at all, so in each of those cases the write is refused instead and counted as blockwatcher_rewinds_refused_total.
7. Re-read the persisted checkpoint, then restart the source so replacement Matches can flow.

In-flight work on the old pipeline is dropped with it: it was never checkpointed, so it is not a silent skip. For a given sink, every Retracted from an invalidate finishes before any post-restart Match from that network is offered. Retracts go to the sinks that network’s pipeline was spawned with. After the network resource is deleted, those ids live on a runtime-only tombstone so a late Invalidated still notifies only those sinks, not every other network’s consumers. Sibling networks are not touched. One network’s invalidate does not abort the others.

sequenceDiagram
    participant Src as Source
    participant Sup as Supervisor
    participant Drain as Pipeline drain
    participant Journal as Delivery journal
    participant Sink as Sink worker
    participant Store as Checkpoint

    Src->>Sup: Invalidated { from }
    Sup->>Drain: cooperative cancel + drain
    Drain-->>Sup: outgoing work finished
    Sup->>Journal: list_deliveries_after(from)
    Journal-->>Sup: retained (cursor, match_id) rows
    loop each retained id, per sink
        Sup->>Sink: Retracted { match_id }
        Sink->>Sink: deliver_with_retry
        Note over Sink: success forgets the row,<br/>dead-letter keeps it
    end
    Note over Sink: all retracts delivered or dead-lettered
    Sup->>Store: rewind checkpoint to from
    Sup->>Src: restart (replacement Matches)

The delivery journal

A Match that is delivered or dead-lettered is recorded in a per-network journal before its completion guard completes (journal_delivery, crates/blockwatcher-core/src/pipeline/sink_worker.rs). That ordering is load-bearing: completing first would let the checkpoint pass a match the journal never durably saw, so a later invalidate could not retract it. Retracted is not journaled as a positive fact; a successful retract forgets the row. A retract that exhausts to a dead letter keeps the journal row so a later invalidate can re-offer it.

The journal is bounded by instance config journal_depth (default 1024, measured in cursor primary units: block number on EVM). On each record_delivery, rows whose cursor.primary is strictly behind recorded.primary.saturating_sub(journal_depth) are dropped as part of the same write. There is no operator prune API and no “disable journal” switch in this version; the depth is meant to sit well above typical confirmation windows (12–64).

When an invalidate’s from plus journal_depth does not cover the checkpoint high-water mark, rewind and restart still happen, but only ids still retained are retracted. The gap is counted (blockwatcher_journal_gap_total) and logged at error, naming the network, the invalidate cursor, the checkpoint, and the configured depth, never silent. See Observability and Troubleshooting. Deliveries at or below the invalidate’s own from bound are never retracted even when the true fork lies deeper than the source could prove: block from.primary itself is re-emitted in full on restart, so any already-delivered match inside it arrives again as a byte-identical duplicate under the at-least-once contract, not a gap.

A gate’s Emit carries the same durability shape one layer earlier: it rides an engine-owned outbox row per sink, written in the same storage transaction that consumes the hits its journal is built from, and held until that sink’s delivery outcome is itself durable. A crash between those two points loses nothing: the row survives to be drained and redelivered, at the cost of a possible duplicate, never a gap, the same trade this page’s dead-letter and journal contracts already make. Downstream consumers dedupe on Match.id, deterministic for the same reason a redelivered plain match already is. See Gates for the emission-to-outcome path in full.

Delivery scenarios

Delivery guarantees is the field-by-field reference: what each policy’s config keys do, what a Progress slot tracks, exactly which route replays a dead letter. This page is its companion for a different question: given a monitor, a sink, and whatever policies are configured on it, what actually happens, end to end, in the situation an operator is looking at right now? Every diagram here traces a real code path already described on that page; nothing here is a new mechanism, only a new view of the ones that exist.

Picking a scenario

The full path, one picture

Every match takes the same route from decode to checkpoint, whether or not any policy below is configured. Policies insert themselves as named branch points on this one path; they never create a second path:

flowchart TD
    A[Event decoded, matched] --> B["Progress::begin: guard created,<br/>cursor stalls until it completes"]
    B --> C{Sink has<br/>throttle?}
    C -->|admitted| D{Sink has<br/>aggregate?}
    C -->|refused| DL1["Dead letter<br/>(reason: throttled)"]
    D -->|no| E[Deliver attempt]
    D -->|yes| F["Buffer holds the match<br/>(guard stays uncompleted)"]
    F -->|flush trigger fires| G{Flush held<br/>one match or many?}
    G -->|one| E
    G -->|many| H[Deliver as one Digest]
    E --> I{Attempt<br/>result?}
    H --> I
    I -->|success| J[Journal the delivery]
    I -->|exhausted retries| K["Dead letter<br/>(per contained match, if a digest)"]
    J --> L["guard.complete()"]
    K --> L
    DL1 --> L
    L --> M["Progress advances the checkpoint<br/>once every guard in the prefix is done"]

Two things this picture makes visible that are easy to miss reading the policies one at a time: every exit except the throttle refusal passes through a delivery attempt first (aggregation defers when an attempt happens; it never skips one), and every exit reaches guard.complete(). There is no branch that drops a match without either delivering it, dead-lettering it, or leaving its guard open (which stalls the checkpoint rather than silently passing it). See The contract, precisely for why that third option doesn’t exist.

Retry and backoff

One delivery attempt, from dispatch to either success or exhaustion. This is the branch under “Deliver attempt” in the picture above:

flowchart TD
    A[Attempt delivery] --> B{Result?}
    B -->|success| Z[Journal + complete guard]
    B -->|classified error| C{Class?}
    C -->|Permanent| D[Skip remaining attempts]
    C -->|Transient / RateLimited| E{Attempts<br/>remaining?}
    C -->|RetryNarrower| E
    E -->|yes| F[Wait backoff, doubling<br/>from initial_backoff_ms<br/>up to max_backoff_ms]
    F --> A
    E -->|no| D
    D --> G["Dead letter<br/>(reason names the class)"]

A Permanent error (a config the sink will never accept, a 4xx a retry can’t fix) skips straight to dead-lettering rather than spending the rest of the attempt budget on a result that cannot change; see Dead letters for the exact field a dead letter carries and the API routes (GET .../dead-letters, POST .../dead-letters/{id}/replay, and the DELETE routes that discard one or many) that read, replay, or discard it. retry’s fields (max_attempts, initial_backoff_ms, max_backoff_ms) are what this diagram’s boxes read from; omitting retry on a SinkDef uses the engine-wide default from blockwatcher.toml, not “no retry at all.”

Throttle: admission and suppression

The branch under “Sink has throttle?” above. A window opens on the first admitted delivery and holds for window_ms:

flowchart TD
    A[Match arrives at the sink worker] --> B{Window still open,<br/>and budget left?}
    B -->|no window open yet, or window_ms elapsed| C[Open a fresh window]
    C --> D[Admit, spend one unit]
    B -->|open and budget remains| D
    D --> E[Deliver normally]
    B -->|open and budget exhausted| F["Dead letter<br/>(reason: throttled, attempts: 0)"]

Only a delivered match spends budget: a failed attempt already paid for itself through the retry path above and must not also cost throttle quota meant for the next real delivery. A throttled match is not silently lost: it’s dead-lettered through the exact same path an exhausted retry uses, so it’s listed and replayable identically (replaying delivers it directly, without re-checking the window it was suppressed under). See Delivery policies for the two config fields and their defaults.

Aggregation: hold and flush

The branch under “Sink has aggregate?” above. Unlike throttle, a held match is not resolved yet: it sits with its guard open until one of four things flushes the buffer:

flowchart TD
    A[Match arrives, sink has aggregate] --> B[Add to the open buffer]
    B --> C{Which flush trigger<br/>fires first?}
    C -->|buffer reaches max_batch| D[Flush now]
    C -->|window_ms since the FIRST<br/>held match elapses| D
    C -->|sink worker's channel closes<br/>= a cooperative drain| D
    C -->|a Retracted event arrives<br/>for this sink| E[Flush the buffer,<br/>THEN deliver the Retracted]
    D --> F{One match held,<br/>or several?}
    F -->|one| G[Deliver as a plain Match]
    F -->|several| H[Deliver as one Digest]

The window is anchored to the first match that joined it, not the most recent: a steady trickle of matches still reaches the sink once per window rather than waiting forever for a batch that never fills. A Retracted is never buffered and never counts toward max_batch: it flushes whatever is already held first, so a retract can never overtake the match it retracts (per-sink delivery order is a consumer’s only evidence of which came last). See Aggregation for the digest wire shape, the exhaustion-dead-letters-each-match-individually behavior, and why a dead-lettered digest member replays as a single match, never as the digest it arrived in.

The one behavior this diagram can’t show, because it’s an absence, not a branch: a hard-cancel or crash while matches are held drops those guards uncompleted rather than flushing them. Nothing here reads that as “lost”: the checkpoint simply doesn’t advance past them, and on restart the source re-emits the events those matches came from, which re-match, re-buffer, and eventually re-deliver, deduplicated by MatchId like any other redelivery. There’s no persisted window to resume from; the window is runtime-only, same as throttle’s.

Threshold gate, not aggregate

The branch sits before Progress::begin for a match. Three hits spanning ≤ window_ms produce one digest and complete outstanding for that emit; two hits complete with outstanding 0 and persist holds. Contrast the aggregate diagram on this page: every buffered match there holds a guard open. Full contract: Gates.

Composing policies

The two flow diagrams above never run in isolation on a sink that configures both. Four compositions are worth naming because each answers a question an operator will actually ask:

• A digest costs one throttle unit, not one per contained match. Aggregating a bursty sink into fewer, larger deliveries is a direct way to keep it under a tight throttle budget: ten matches held into one digest spend the same one unit of window budget a single match would.
• A throttled digest still dead-letters every contained match individually. The throttle refusal happens at the delivery-attempt boundary, after aggregation has already flushed a Digest; suppressing it doesn’t collapse it back into one dead-letter row, since replay still needs to resend each match on its own.
• Retract bypasses both policies, at the same call site each time. A Retracted skips the throttle check entirely and flushes (rather than joins) an aggregate buffer. This is deliberate in both cases: a correctness signal (undoing a match a consumer may have already acted on) outranks a fatigue policy meant for noisy matches, so an undo is never the thing a loud sink drops on the floor.
• A reorg’s retract pass ignores both policies too, for the same reason: see the sequence diagram in the next section. The retracts an invalidate sends ride the same deliver_with_retry path live traffic does, but they are never throttled or buffered; only the ordinary retry budget applies to them.

Reorg and invalidation

Whether a fork produces a Retracted at all depends on which side of the confirmation barrier it cuts into. This is a three-way branch, not two:

flowchart TD
    A[Source detects its tracked chain<br/>no longer matches the live head] --> B{Where does the<br/>fork land?}
    B -->|only in the unconfirmed window,<br/>never durably offered| C["Shallow: retried in place<br/>inside the source. No Retracted."]
    B -->|cuts into already-emitted work,<br/>a tracked ancestor still matches| D["Deep, proven fork:<br/>Invalidated { from: fork point }"]
    B -->|cuts deeper than the tracker<br/>can prove: no tracked ancestor matches| E["Deep, unproven depth:<br/>Invalidated { from: just below<br/>oldest tracked height }"]
    D --> F[Engine invalidate sequence]
    E --> F

The tracker’s own memory bounds how deep a fork can be proven: it keeps clamp(2 × confirmations, 8, 64) entries, so a reorg deeper than 64 blocks is a real limit of the design, not an oversight; the source still invalidates, just from the deepest point it can still vouch for rather than the true fork block. Both deep branches converge on the same sequence once Invalidated { from } is raised: this is the existing diagram from Delivery guarantees, repeated here because it’s the scenario this page’s readers are most often looking for:

sequenceDiagram
    participant Src as Source
    participant Sup as Supervisor
    participant Drain as Pipeline drain
    participant Journal as Delivery journal
    participant Sink as Sink worker
    participant Store as Checkpoint

    Src->>Sup: Invalidated { from }
    Sup->>Drain: cooperative cancel + drain
    Drain-->>Sup: outgoing work finished
    Sup->>Journal: list_deliveries_after(from)
    Journal-->>Sup: retained (cursor, match_id) rows
    loop each retained id, per sink
        Sup->>Sink: Retracted { match_id }
        Sink->>Sink: deliver_with_retry
        Note over Sink: success forgets the row,<br/>dead-letter keeps it
    end
    Note over Sink: all retracts delivered or dead-lettered
    Sup->>Store: rewind checkpoint to from
    Sup->>Src: restart (replacement Matches)

One gap worth knowing about explicitly: if from plus journal_depth doesn’t reach back far enough to cover the checkpoint’s high-water mark, the rewind and restart still happen, but only the ids the journal actually retained get a Retracted. That gap is never silent: it’s counted (blockwatcher_journal_gap_total) and logged naming the network, the invalidate cursor, the checkpoint, and the configured depth. See The delivery journal.

evm-mempool never produces Invalidated at all: its cursor has no chain position to fork against. See The evm-mempool exception.

What the guarantee actually is, by scenario

“At-least-once” is the contract everywhere, but what that means in practice (duplicates, gaps, and whether a consumer hears about it) differs by scenario:

“Gaps possible” only ever appears where the journal’s bounded depth couldn’t reach far enough, or on the one source that never promised a resumable position in the first place. Every other row’s answer is “no” because of the same one rule: a checkpoint only advances once every match tied to it has either reached its sink or become a durably recorded dead letter, so anything a crash or a restart repeats was, by construction, never durably finished.

Configuration cookbook

Named starting points, not the only correct answer for every case: each maps a goal to the fields that get you there, with the reasoning so you can move off the suggested numbers with the right things in mind.

“I don’t want duplicate delivery noise from one burst of activity.” Set aggregate with a window_ms wide enough to catch a typical burst and a max_batch above your usual burst size, so bursts land as one digest instead of many separate matches — HTTP batching of matches that already fired. Widening window_ms trades delivery latency for fewer messages; a buffered match holds the checkpoint for the whole window, so an unreasonably wide window (the cap is one day, 86_400_000 ms) is a real cost, not a free knob. If the burst should not fire at all until N predicate-true hits, that is the threshold-gate recipe below, not sink aggregate.

“Notify only if Transfer occurred more than 3 times in the last 60 minutes of block time, then start counting again.” Set the monitor’s gate to threshold with count: 3 and window_ms: 3600000. Do not use sink aggregate for this: aggregate would mint a Match per Transfer and stall the checkpoint for the whole window.

“At most one webhook per hour of block time; I do not want the rest queued.” Set gate to max_once. Do not use throttle if you intend those hits to vanish; throttle dead-letters them for replay.

“I want to bound how often a flaky or rate-limited downstream gets hit, without losing any match.” Set throttle to the rate your downstream can actually sustain. Nothing is lost: everything the window refuses becomes a replayable dead letter, so you’re trading “deliver immediately” for “deliver on demand, in bulk, via POST /dead-letters/{id}/replay” rather than trading away completeness.

“I need to know the alert reached the sink even through a real chain reorg, not just that it was attempted once.” This is a confirmations and journal_depth question, not a sink-policy one. confirmations (default 12) is a latency-for-safety trade: a deeper value makes an already-emitted match’s retraction by a reorg less likely, at the cost of how quickly it reaches the sink at all. Because the tracker that proves a fork’s exact depth caps at 64 blocks regardless of confirmations, a journal_depth (default 1024, measured in cursor-primary units: block number on EVM) that comfortably exceeds your confirmations value is what makes a proven-fork retract actually reach every affected match; a shallower journal risks the “unproven depth” row above.

“This monitor watches pending transactions and I need to understand what I’m giving up.” evm-mempool has no resumable cursor and no reorg signal at all: a restart loses whatever was pending, and identity is only stable within one process’s lifetime, so consumers must deduplicate on the transaction hash rather than MatchId. If your use case needs delivery guarantees, watch the confirmed chain with evm-rpc instead; evm-mempool is for latency-sensitive, best-effort visibility into what hasn’t landed yet, not for anything that must not be missed. See The evm-mempool exception.

“I want every dropped or suppressed event to show up somewhere I can alert on.” Nothing here is ever silently dropped, and that’s true without any config, but the observable surface differs by scenario: throttle and retry-exhaustion both produce dead letters (poll GET /networks/{id}/dead-letters or alert on its growth); a deep reorg’s retracts are visible in the Retracted events your sinks receive; a journal gap is a named metric (blockwatcher_journal_gap_total). See Observability for the full metric list.

Chain-agnosticism

Every other concept page so far describes blockwatcher’s engine in terms that never mention a specific chain: cursors, matches, selectors, checkpoints. That is not a writing choice; it reflects an actual boundary drawn in the dependency graph and enforced by tooling, not by convention. This page covers the two halves of that boundary: the shape everything normalizes into on the way past a decoder, and the mechanism that keeps a chain SDK from ever crossing into the crates that don’t need one.

flowchart LR
    rpc["RPC endpoints<br/>(external chains)"]
    sources["Sources<br/>evm-rpc · evm-mempool"]
    sinks["Sinks<br/>webhook · script · log"]
    storage["Storage<br/>checkpoints · dead letters · resources"]
    api["REST API"]
    metrics["Metrics"]
    engine["engine<br/>bounded channels · checkpoints"]

    subgraph pipeline["Engine pipeline"]
        direction LR
        decoder["Decoder"]
        matcher["Matcher<br/>predicates"]
        gate["Gate<br/>threshold · max_once"]
        decoder --> matcher
        matcher --> gate
    end

    rpc --> sources
    sources -->|"decode and match"| decoder
    gate --> sinks
    api -->|"manages resources"| storage
    storage <--> engine
    engine -->|"drives"| pipeline
    engine -.->|"reports"| metrics

    classDef module fill:none,stroke:#a9a3e3
    classDef core fill:none,stroke:#8a8d86,stroke-dasharray: 5 5

    class rpc,sources,sinks,decoder,matcher,gate module
    class engine,api,storage,metrics core
classDef dim fill:none,stroke:#999999,color:#999999,opacity:0.35
classDef focus fill:#ffd43b,stroke:#d9480f,stroke-width:3px,color:#1a1a1a
class rpc,sources,sinks,storage,api,metrics,engine,matcher,gate dim
class decoder focus
click sources "selectors.html"
click decoder "chain-agnosticism.html"
click matcher "predicates.html"
click gate "gates.html"
click sinks "delivery.html"
click storage "resources.html"
click api "../reference/http-api.html"
click metrics "../reference/observability.html"
click engine "pipeline.html"

Key takeaways

• Every decoder must normalize chain-specific data into one canonical Value enum; nothing past the decoder (a matcher, a gate, a sink) ever needs to know which chain produced it.
• Int/Uint are arbitrary-precision and Address is canonical bytes, so no chain’s values lose precision or collide with another chain’s.
• blockwatcher-core depends only on the port trait signatures (Source, Decoder, Matcher, Gate, Sink, Storage); it never imports a chain SDK, RPC client, or storage driver directly.
• check-dep-graph.sh enforces that boundary two ways: a per-crate allowlist of direct dependencies, and a transitive-tree denylist of chain/infra crate families.
• A new chain family arrives as a module implementing Decoder and Source; every existing predicate, matcher, and sink works against it unmodified.

One value model, every chain

crates/blockwatcher-types/src/value.rs defines the enum that a decoder is required to normalize into and that nothing past it (a matcher, a gate, a sink) ever has to unlearn:

#![allow(unused)]
fn main() {
pub enum Value {
    Null,
    Bool(bool),
    Int(BigInt),
    Uint(BigUint),
    Bytes(Vec<u8>),
    Address(Vec<u8>),
    Str(String),
    Array(Vec<Value>),
    Map(IndexMap<String, Value>),
}
}

(crates/blockwatcher-types/src/value.rs:30-48). Two design choices here carry the whole point of this page:

• Int and Uint hold arbitrary-precision integers, not a fixed-width native type. A 256-bit token amount off an EVM chain has nowhere to go in a JSON number or an i64 without losing bits; a BigInt/BigUint has no such ceiling, so the value that leaves a decoder is bit-for-bit the value a sink eventually serializes, all the way out to the wire as a decimal string rather than a JSON number, per Value’s own doc comment and the decimal module’s serialization rules (value.rs). This is also why match ids hash Value trees directly, in MatchId::derive’s eat_value step (crates/blockwatcher-types/src/id.rs) rather than some chain-specific encoding of them: one representation, used identically by every consumer of a decoded event.
• Everything else a chain family might invent still has to land in one of the shapes above. Address is canonical bytes rather than a chain’s native address type, and the decoder that produced those bytes is on the hook for keeping distinct address flavors from ever comparing equal: the doc comment on Value::Address calls out Stellar account versus contract addresses as the concrete case where two genuinely different values are the same byte width and would collide unless the decoder tags them apart (value.rs). The same discipline applies to Map keys: they are always String, so a chain whose native map keys aren’t strings needs an injective rendering rule from the decoder, because two distinct native keys landing on the same canonical string would make a predicate silently address the wrong field (value.rs). Canonicalization is squarely the decoder’s job: the model itself has no chain awareness to lean on.

block.timestamp is that same decoder vocabulary: unix seconds at that path on the canonical tree. A Stellar or Solana decoder maps close_time / blockTime onto it. A chain that cannot expose unix time cannot use time-window gates; the write is 422. The path is not an EVM-only leak.

Value::from_json (value.rs) is the one general-purpose constructor in this file, and it demonstrates part of the same rule from the other direction: JSON numbers that fit in u64/i64 become Uint/Int, and JSON null becomes Value::Null rather than being dropped or coerced into an empty string. A JSON number can’t carry a 256-bit token amount in the first place, so a decoder never gets there through from_json at all: it builds Value::Uint/Int directly from the chain’s own encoding (hex, decimal string, whatever the RPC returns) and reserves from_json for plain JSON-RPC payloads whose numbers already fit a native width. Numbers outside that range or non-integers passed to from_json fall back to Value::Str holding the literal text, so even that path never rounds through a lossy f64.

Core must not know chains

The value model is the shape of the boundary; the rule that nothing past the decoder needs to know which chain produced a value is the reason the boundary exists. blockwatcher states that rule as trait definitions (Source, Decoder, Matcher, Gate, Sink, Storage), all living in blockwatcher-ports, and blockwatcher-core, the crate that actually runs pipelines, tracks checkpoints, and serves the control API, depends on nothing but those trait signatures. It never imports an RPC client, a chain SDK, an ABI library, or a storage driver directly; every one of those lives one layer out, in a module crate that implements one port. crates/blockwatcher-evm is where an EVM-specific type ever appears in this codebase at all: nothing named alloy or ethers is reachable from blockwatcher-core’s own dependency list.

That separation is what a chain-agnostic decoder boundary actually buys: a second chain family arrives as a new crate implementing Decoder and Source, translating its own wire format into the same Value tree above, and every existing predicate, matcher, gate, and sink works against it unmodified, because none of them were ever written against the first chain’s types to begin with.

The direct edge a chain SDK would need to reach the core ring simply does not exist; the only path in is through a module’s decoder, onto the canonical value tree:

flowchart LR
    sdk["chain sdk<br/>e.g. alloy"] --> mod["module crate<br/>blockwatcher-evm"]
    mod -->|"decode: normalize"| val["value tree<br/>blockwatcher-types"]
    val --> core["core ring<br/>blockwatcher-core, blockwatcher-ports, blockwatcher-api"]
    mod -.->|"forbidden edge,<br/>blocked by check-dep-graph.sh"| core

The CI gate that proves it

Stating a rule in a crate’s Cargo.toml dependency list is easy to state and easy to quietly violate one dependency at a time. scripts/check-dep-graph.sh exists because “blockwatcher-core has no chain code” is a claim worth re-checking on every commit rather than trusting once at review time, and it checks it with two independent mechanisms layered on top of each other rather than one.

The first mechanism is a per-crate allowlist of direct dependencies: every workspace crate has a hand-maintained list of exactly what it may declare in [dependencies], and anything a crate’s manifest names that isn’t on its own list fails the check outright (allowlist_for and check_direct in check-dep-graph.sh). blockwatcher-core’s list, for instance, names blockwatcher-types, blockwatcher-ports, serde, tokio, tracing, and a small handful more: no RPC crate, no storage driver, nothing chain-specific (ALLOW_BLOCKWATCHER_CORE in check-dep-graph.sh). Because the list is explicit rather than inferred, adding any new direct dependency to a core crate is a decision someone has to make in the script itself, not a side effect of running cargo add.

The second mechanism runs over the whole transitive dependency tree (cargo tree, not just the manifest) and checks it against a denylist of family-name prefixes that are never allowed to appear anywhere underneath a core crate: alloy, reqwest, sqlx, rusqlite, several chain SDKs, and more (CHAIN_AND_INFRA and check_transitive in check-dep-graph.sh). This is the check that catches what the allowlist alone cannot: a dependency the manifest never names directly, pulled in indirectly by something the allowlist did approve. A handful of module and binary crates that legitimately need one of these families get a narrow, per-crate, per-family exemption from this second net only: blockwatcher-evm is allowed to carry alloy because it is an EVM module and that is its entire purpose, but every family it wasn’t exempted for still applies in full, and the first net (the direct-dependency allowlist) is never relaxed by an exemption at all.

Running both nets together is deliberate, not redundant. Blocking specific names is inherently reactive: a maintainer has to have heard of a chain SDK before they can add it to a denylist, so anything released after the list was last updated walks straight through. Requiring explicit approval flips that around: a crate is rejected by default the moment it shows up unlisted, regardless of whether anyone had ever heard of it, chain-related or not. Layering the transitive scan on top closes the one gap an allowlist alone leaves: a dependency an approved crate quietly starts pulling in one level down, which a scan of direct manifests never sees. Together they mean a core crate cannot gain a chain (or storage, or HTTP) dependency by any route without the gate failing the build, whether that route is a manifest edit or a transitive version bump three crates away.

Modules and trade-offs

Every other concept page treats one specific behavior as swappable: a source in The pipeline, a matcher in Predicates. This page steps back and treats “swappable” itself as the organizing idea: what a module actually is, how one gets into a running binary, and what it means for an operator that every axis of behavior in blockwatcher is one.

flowchart LR
    rpc["RPC endpoints<br/>(external chains)"]
    sources["Sources<br/>evm-rpc · evm-mempool"]
    sinks["Sinks<br/>webhook · script · log"]
    storage["Storage<br/>checkpoints · dead letters · resources"]
    api["REST API"]
    metrics["Metrics"]
    engine["engine<br/>bounded channels · checkpoints"]

    subgraph pipeline["Engine pipeline"]
        direction LR
        decoder["Decoder"]
        matcher["Matcher<br/>predicates"]
        gate["Gate<br/>threshold · max_once"]
        decoder --> matcher
        matcher --> gate
    end

    rpc --> sources
    sources -->|"decode and match"| decoder
    gate --> sinks
    api -->|"manages resources"| storage
    storage <--> engine
    engine -->|"drives"| pipeline
    engine -.->|"reports"| metrics

    classDef module fill:none,stroke:#a9a3e3
    classDef core fill:none,stroke:#8a8d86,stroke-dasharray: 5 5

    class rpc,sources,sinks,decoder,matcher,gate module
    class engine,api,storage,metrics core
classDef dim fill:none,stroke:#999999,color:#999999,opacity:0.35
classDef focus fill:#ffd43b,stroke:#d9480f,stroke-width:3px,color:#1a1a1a
class rpc,api,metrics,engine dim
class sources,decoder,matcher,gate,sinks,storage focus
click sources "selectors.html"
click decoder "chain-agnosticism.html"
click matcher "predicates.html"
click gate "gates.html"
click sinks "delivery.html"
click storage "resources.html"
click api "../reference/http-api.html"
click metrics "../reference/observability.html"
click engine "pipeline.html"

Key takeaways

• blockwatcher has exactly six ports (Source, Decoder, Matcher, Gate, Sink, Storage), and a module is one concrete implementation of exactly one of them.
• A module never straddles two ports, even when several modules live in the same crate, and configuration resolves a name against the running binary’s own module catalog.
• Two modules behind the same port can trade off completeness against something else, most visibly evm-rpc (at-least-once delivery, higher latency) against evm-mempool (pending visibility, no restart recovery).
• Switching modules is the operator’s actual lever for a trade-off: changing one module leaves every other port’s behavior untouched, since none of them are written against one module’s assumptions.

Everything an operator composes is a module

An blockwatcher deployment has exactly six axes along which behavior can differ: how it reads a chain, how it turns raw bytes into blockwatcher’s canonical shape, how it decides whether a decoded occurrence counts, when a predicate-true hit becomes a delivery, where a match goes, and where resources and operational state persist. Those six axes are the six ports blockwatcher defines (Source, Decoder, Matcher, Gate, Sink, Storage), and a module is, by definition, one concrete implementation of exactly one of them. Nothing in blockwatcher is configured by writing code against a bespoke integration point; every one of these six choices is made the same way, by naming a module in configuration and giving it whatever config object that module expects.

The six ports and every module registered behind each one, in this build, look like this:

flowchart TD
    source["Source"] --> rpc["evm-rpc"]
    source --> mempool["evm-mempool"]
    decoder["Decoder"] --> evm["evm"]
    matcher["Matcher"] --> expr["expr"]
    gate["Gate"] --> threshold["threshold"]
    gate --> maxonce["max_once"]
    sink["Sink"] --> webhook["webhook"]
    sink --> script["script"]
    sink --> log["log"]
    storage["Storage"] --> memory["memory"]
    storage --> sqlite["sqlite"]

    %% Layout only, no meaning: these invisible edges wrap the six ports
    %% into bands. Without them every port sits on one row, which renders
    %% too wide for the content column and shrinks the labels. Keep them.
    rpc ~~~ sink
    evm ~~~ storage
    expr ~~~ maxonce

ModuleCatalog (crates/blockwatcher-core/src/catalog.rs) is where that naming resolves: one name-keyed map per port family, each holding factory functions folded in from every compiled-in module’s own get_all() enumeration. A config that names a module absent from the catalog (a typo, or a module simply not compiled into this build) fails at boot or at write time with EngineError::UnknownModule, listing every alternative that actually is registered, via unknown and the family! macro’s lookup arm (catalog.rs), rather than panicking or silently no-op’ing that pipeline stage. Because the list comes from the catalog itself, the message always names what this particular binary carries, never a superset the workspace merely contains somewhere.

One module, one port, one name

A module never straddles two ports. crates/blockwatcher-evm, for instance, ships two Source implementations (evm-rpc, evm-mempool) and one Decoder (evm): three separate modules living in one crate, each registered under its own name and each satisfying exactly one port’s trait, never blending source and decode logic into a single type, in build_catalog (crates/blockwatcher-embed/src/catalog.rs, folding blockwatcher_evm::sources::get_all() and blockwatcher_evm::decoders::get_all() separately into the catalog). Which module handles which resource is itself fixed by the port: a network names its Source module, a spec’s chain determines its Decoder, a sink resource names its Sink module, blockwatcher.toml names the one Matcher and one Storage backend for the whole process. An operator never picks a module without also picking, structurally, which port it fills.

Trade-offs are a property of the module, not a setting

Because two modules behind the same port are interchangeable at the trait level, they are free to differ arbitrarily in the trade-offs they make. blockwatcher leans on that rather than trying to expose every axis as a tunable knob on one do-everything implementation. The clearest example ships on the Source port, between the EVM sources:

• evm-rpc scans confirmed blocks and logs over RPC. It only ever reports something once that something has a fixed position in the chain, which is exactly what lets it make at-least-once delivery’s promise: its checkpoint names a real block to resume from, so a crash costs at worst a duplicate, never a gap. The cost is latency and RPC quota: an event is only visible once it is mined and this source has polled far enough to see it.
• evm-mempool watches a node’s pending-transaction feed instead. It reports a transaction before it is ever mined, which is strictly faster, but at the cost of the one guarantee evm-rpc can make: a pending transaction may never be mined at all, and this source’s cursor is a per-run arrival counter rather than a chain position, so a restart cannot resume it: whatever was pending while the process was down is simply gone. Delivery guarantees covers exactly what that costs a consumer.

Choosing between them is choosing a point on a latency-vs-completeness trade-off, and blockwatcher does not try to collapse that choice into a single configurable source with a “mode” flag: the two behaviors are different enough, and the guarantee difference consequential enough, that they are two separate modules an operator picks between by name, each documenting its own side of the trade-off rather than one module documenting a matrix of settings. The same pattern of “trade-off lives in which module you picked, not in a shared config surface” carries across the other five ports as well: the webhook sink trades a network dependency for delivering anywhere HTTP reaches; the log sink trades reach for having none; sqlite storage trades a single-writer constraint for surviving a restart, memory storage trades restart survival for zero setup; threshold trades per-hit delivery for a session digest that does not stall the checkpoint; max_once trades later in-window hits for a single alert.

The module catalog

Every row below is a module registered into ModuleCatalog by one of the get_all() calls build_catalog (crates/blockwatcher-embed/src/catalog.rs) folds together. The table is exhaustive in both directions: every module compiled into this binary appears here, and every row names something a real registration call, cross-checked against each family’s own enumeration, actually produces.

Omit-gate is engine passthrough, not a catalog row an operator must name. Matcher is still one module for the whole process. Gate is per monitor, like sink.

Module selection is by name in configuration; a name the running binary never registered fails loudly, with the list of names it actually did register, rather than silently doing nothing.

Why this is the operator’s actual lever

Because a trade-off lives inside the module rather than in a shared knob, “choose your trade-offs” and “choose your modules” are the same action for an operator. Wanting faster visibility into pending activity, accepting that some of it will never be confirmed, means switching a network’s source from evm-rpc to evm-mempool, not flipping a setting on one source that tries to serve both goals at once. The six-port boundary is what makes that swap safe to make at all: switching sources changes nothing about how the decoder, the matcher, a gate, or any configured sink behaves, because none of them were ever written against one source’s assumptions to begin with.

Guides & reference

This part covers running and operating a deployed instance: the configuration surface, the HTTP API and observability endpoints, a Docker deployment, two fully worked examples, a symptom-first troubleshooting index, and a benchmark baseline. It is for an operator or integrator working against a running blockwatcher, not for someone deciding whether to adopt it.

flowchart LR
    rpc["RPC endpoints<br/>(external chains)"]
    sources["Sources<br/>evm-rpc · evm-mempool"]
    sinks["Sinks<br/>webhook · script · log"]
    storage["Storage<br/>checkpoints · dead letters · resources"]
    api["REST API"]
    metrics["Metrics"]
    engine["engine<br/>bounded channels · checkpoints"]

    subgraph pipeline["Engine pipeline"]
        direction LR
        decoder["Decoder"]
        matcher["Matcher<br/>predicates"]
        gate["Gate<br/>threshold · max_once"]
        decoder --> matcher
        matcher --> gate
    end

    rpc --> sources
    sources -->|"decode and match"| decoder
    gate --> sinks
    api -->|"manages resources"| storage
    storage <--> engine
    engine -->|"drives"| pipeline
    engine -.->|"reports"| metrics

    classDef module fill:none,stroke:#a9a3e3
    classDef core fill:none,stroke:#8a8d86,stroke-dasharray: 5 5

    class rpc,sources,sinks,decoder,matcher,gate module
    class engine,api,storage,metrics core
classDef dim fill:none,stroke:#999999,color:#999999,opacity:0.35
classDef focus fill:#ffd43b,stroke:#d9480f,stroke-width:3px,color:#1a1a1a
class rpc,sources,sinks,storage,engine,decoder,matcher,gate dim
class api,metrics focus
click sources "../concepts/selectors.html"
click decoder "../concepts/chain-agnosticism.html"
click matcher "../concepts/predicates.html"
click gate "../concepts/gates.html"
click sinks "../concepts/delivery.html"
click storage "../concepts/resources.html"
click api "http-api.html"
click metrics "observability.html"
click engine "../concepts/pipeline.html"

Key takeaways

• This part is for operating a running deployment, not for deciding whether to adopt blockwatcher.

• Configuration and the HTTP API cover how an instance is configured and controlled; observability covers how to watch it run.

• The two worked examples build the same monitor against confirmed blocks and against pending transactions.

• Troubleshooting indexes failure modes by symptom rather than by component.

• Configuration: the instance configuration file and its environment overrides.

• Resource reference: every resource kind’s fields, defaults, and write-time refusals, one page per kind.

• HTTP API: the REST control plane.

• Observability: the status endpoint and the Prometheus metrics.

• Running with Docker: the Compose stack and its own configuration.

• Monitoring ERC-20 transfers [RPC]: a worked example against confirmed blocks.

• Monitoring ERC-20 transfers [Mempool]: the same example against pending transactions instead.

• Troubleshooting: a symptom-first index into failure modes and their fixes.

• Benchmarks: what’s measured, how to reproduce it, and the baseline results.

Configuration reference

blockwatcher boots from exactly one instance configuration: an optional TOML file layered with BLOCKWATCHER_* environment overrides, read once at process start and immutable for the rest of the run (crates/blockwatcher/src/config.rs). It covers only the process’s own plumbing: the API listener, the metrics listener, the storage backend, and the engine’s runtime tunables. Everything else (networks, specs, sinks, and monitors) is resource configuration: storage-backed, mutated through the HTTP API or loaded once from a seed directory, and never read from blockwatcher.toml.

InstanceConfig (crates/blockwatcher/src/config.rs) and every section inside it reject an unrecognized field (#[serde(deny_unknown_fields)]), so a typo’d key fails the boot rather than being silently ignored. Every section also defaults, so an empty file (or no file at all, given at least one recognized environment override) boots a working instance: an in-memory store, no API, no metrics.

Loading a configuration

blockwatcher [--config <path>] [--seed <dir>]

Three layers, each one able to override the layer before it:

flowchart LR
    def["InstanceConfig::default()"] --> file["--config file<br/>TOML, if given"]
    file --> env["BLOCKWATCHER_* env vars<br/>win over both"]
    env --> final["final InstanceConfig"]

load (config.rs) builds the configuration in three layers:

1. InstanceConfig::default(): every section’s own default.
2. The TOML file at --config <path>, if given. A file that fails to read or parse refuses the boot naming the path, never the offending value.
3. BLOCKWATCHER_* environment variables, which win over both of the above.

Nested keys use a double underscore: BLOCKWATCHER_API__LISTEN, BLOCKWATCHER_STORAGE__MODULE. Only variables whose name contains __ after the BLOCKWATCHER_ prefix are read as overrides (env_provider, config.rs): a variable such as BLOCKWATCHER_API_TOKEN, the environment variable a token’s secret = "env:BLOCKWATCHER_API_TOKEN" reference names, has no __ in it and is never mistaken for a config key. The token table itself is file-based: Figment numbered env keys cannot spell a sequence, so [[auth.tokens]] rows are not overridden from the environment.

Two aliases exist for one deprecation cycle: BLOCKWATCHER_API_LISTEN and BLOCKWATCHER_METRICS_LISTEN, equivalent to BLOCKWATCHER_API__LISTEN and BLOCKWATCHER_METRICS__LISTEN respectively, applied after everything else so they still win when both an alias and its nested form are set (apply_listen_aliases, config.rs).

Neither a --config path nor a single recognized BLOCKWATCHER_* override is optional: at least one must be present, or the boot refuses with:

no configuration source: provide --config <path> or set BLOCKWATCHER_* variables
(e.g. BLOCKWATCHER_STORAGE__MODULE, BLOCKWATCHER_API__LISTEN)

This is deliberate: an empty file and “no configuration at all” would otherwise both boot the same all-defaults instance, and the second of those is far more likely to be a mistake than an intent.

[api]

The REST control plane. Disabled by default: a listener that appears without an operator having asked for one is a listener nobody remembered to put a token on.

GET /health on this listener is the only path that answers without a bearer token. Compose healthchecks curl it with no header. Every other path requires a token from [auth].

[auth]

The labelled bearer table. Required (and must be non-empty) when [api].enabled is true. Each row is one credential: a label audit records name, a scope (read, operate, or admin), and a secret that is an env:NAME reference, never a literal. Boot refuses an empty table, two rows that resolve to the same secret (naming the labels, not the value), an unresolvable reference, or a secret that is not of the env:NAME form.

read < operate < admin. The minimum is attached per MethodRouter, not by path prefix: GET of a resource is read; pause/resume/replay are operate; PUT/DELETE of networks, specs, sinks, and monitors are admin. Creating a script sink is therefore not the same privilege as reading the dashboard.

[metrics]

The Prometheus scrape listener. Disabled by default for the same reason as [api].

[storage]

Which storage module backs every resource and every checkpoint and dead letter. This section is a ModuleSel (a module name plus that module’s own opaque config object), the same envelope every other module family uses (crates/blockwatcher-types/src/resource.rs).

blockwatcher ships memory and sqlite storage modules. memory takes no config at all: it persists nothing, which is the floor that lets an instance boot with zero setup and never a durable deployment. sqlite takes:

[engine]

The core pipeline’s own tunables, as an operator writes them (EngineSection, crates/blockwatcher/src/config.rs). Every field here is optional; an unset one is left out of the value core’s own deserializer sees (engine_config, config.rs), so the default that applies is whatever blockwatcher-core’s EngineConfig itself defaults to. This binary never restates a number core owns. matcher is the one field this rule doesn’t quite cover: core has no sane default matcher to fall back to on its own, so when this key is unset the binary substitutes the matcher module it was compiled with (expr, when the expr feature is on) rather than leaving the boot to fail on a genuinely absent value.

default_retry is the same DeliveryRetry shape a SinkDef’s own retry field carries, validated at boot the same way a sink’s own retry override is validated at write time and at boot:

source_restart governs the supervisor that restarts a pipeline whose source task exits before its own cancellation fires, validated at boot on the same terms:

An invalid default_retry, source_restart, or dead_letter_retention fails Engine::start with InvalidEngineConfig, naming the offending field: process config, not a stored resource, so it carries no resource kind or id the way a rejected sink policy does.

matcher is a ModuleSel like [storage]: a module name and that module’s own config object. The shipped expr module takes no config.

Secrets: the env: indirection

A token’s secret is never a literal token: it is a reference of the form "env:NAME", parsed by SecretRef (crates/blockwatcher-types/src/secret.rs) and resolved by reading the named environment variable at the moment it’s needed, never cached and never logged. A value that isn’t of that form refuses the boot naming the required shape and never echoing what was actually written, because the likeliest mistake feeding this key is pasting the real token where a reference belongs. AuthTokenEntry’s own Debug implementation is hand-written for the same reason: it renders secret as <redacted> whenever one is set, so a panic message or a boot-failure log line can never carry it (config.rs). When the API is enabled, boot also refuses a label that is empty, a label used by more than one row, and a reference that resolves to an empty string: audit names the label, and an empty secret would start a listener that can never authenticate.

This indirection isn’t unique to the token table: it’s the general shape blockwatcher uses anywhere a secret has to live in a config object: a webhook sink’s url_secret field, and an evm-rpc or evm-mempool network’s endpoint URLs, are all env:NAME references resolved the same way, per use, by the same SecretRef type. The instance config surfaces those fields on [[auth.tokens]], because they are the secrets this file itself carries; every other env: reference lives in resource configuration (network, sink), set through the API or a seed directory rather than blockwatcher.toml.

CLI flags

Parsed by hand, not through a flag-parsing crate, in parse (crates/blockwatcher/src/cli.rs):

Three commands, no explicit subcommand name for the default one:

blockwatcher [--config <path>] [--seed <dir>]      # boot and run
blockwatcher check <dir>                            # validate a seed directory
blockwatcher prune-checkpoints --config <path> [--dry-run]
blockwatcher --help      # or -h
blockwatcher --version   # or -V

check <dir> validates a seed directory against the modules actually compiled into the binary and constructs every one of them, so every secret a seed’s configs reference through env:NAME must be present in check’s own environment, exactly as it would need to be at a real boot. prune-checkpoints deletes checkpoint rows whose network resource is gone; deleting a network deliberately leaves its checkpoint behind so a re-created id can resume, and this command is the offline sweep for orphans that will never come back.

Exit codes (cli.rs):

RUST_LOG sets the log filter; unset, it defaults to info rather than silence, so the shutdown report, seeding warnings, and restart notices are still visible by default (init_tracing, crates/blockwatcher/src/lib.rs). Every diagnostic goes to stderr, never stdout. Stdout, when running with no [api] at all, is reserved for sink-event JSON (tagged type: match or type: retracted).

Full annotated example

# blockwatcher instance configuration. Resources (networks, specs, sinks,
# monitors) are managed through the HTTP API or a --seed directory, never
# through this file.

[api]
enabled = true
listen = "0.0.0.0:8080"

[[auth.tokens]]
label = "operator"
scope = "admin"
# A reference, never the token itself: resolved from the environment at
# each request's auth check.
secret = "env:BLOCKWATCHER_API_TOKEN"

[metrics]
enabled = true
listen = "0.0.0.0:9090"

[storage]
module = "sqlite"
config = { path = "/data/blockwatcher.db", busy_timeout_ms = 5000 }

[engine]
event_channel_capacity = 256
sink_channel_capacity = 64
drain_deadline_ms = 10000
journal_depth = 1024
# omit dead_letter_retention to keep every letter; a positive value is the cap
default_retry = { max_attempts = 8, initial_backoff_ms = 200, max_backoff_ms = 30000 }
source_restart = { initial_backoff_ms = 500, max_backoff_ms = 60000, reset_after_ms = 300000 }
matcher = { module = "expr", config = {} }

Every value shown above under [engine] is also that key’s own default: the section could be omitted entirely and the instance would boot identically, apart from matcher, which still resolves to expr as long as the binary was built with that feature.

Resource reference

Everything blockwatcher watches and everywhere it delivers to is one of four resource kinds, each its own Rust struct in crates/blockwatcher-types/src/resource.rs. This section is the field-level reference for an operator writing one: every key, its type, its default, and the exact refusal a bad value gets. What the kinds mean and how they relate lives on Resources; this section owns the tables.

One page per kind:

• Network: a feed to watch, including the evm-rpc and evm-mempool source module configs.
• Spec: a chain-tagged decode artifact, including the evm payload shape.
• Sink: a delivery destination and its engine-owned policies, including the webhook, script, and log module configs.
• Monitor: the rule tying the other three together, optionally gated.

The two configuration planes

blockwatcher splits configuration by lifetime, not by topic:

• Instance configuration (blockwatcher.toml plus BLOCKWATCHER_* environment overrides): the process’s own plumbing, read once at boot and immutable for the run. See the Configuration reference.
• Resource configuration (this section): networks, specs, sinks, and monitors, stored in the storage backend, mutated through the HTTP API or loaded once from a seed directory, and never read from blockwatcher.toml.

Write routes

Every kind answers on the identical CRUD shape (crates/blockwatcher-api/src/routes/resources.rs):

Optimistic concurrency runs on ETag / If-Match:

• A PUT with no If-Match is a create: 201 with the new version in ETag, or 409 already_exists if the id is already there.
• A PUT with If-Match: "<version>" is a conditional update: 200 on success, 412 version_conflict (the body carries actual_version) if the version has moved on, 404 not_found if the record is gone.
• A DELETE without If-Match is 428 precondition_required.

The full route contract, including error bodies and the If-Match format, is on the HTTP API reference.

Seed directories

The binary’s --seed <dir> flag loads resources once, at first boot only (crates/blockwatcher/src/seed.rs):

• The directory holds exactly one subdirectory per kind: networks/, specs/, sinks/, monitors/, one JSON file per resource.
• The bundle runs through the same validation Engine::start itself runs, then writes with create-only semantics.
• Once storage holds any resource of any kind, seeding is refused as a no-op on every later boot: after the first boot, resources are managed exclusively through the API.
• blockwatcher check <dir> validates a seed directory offline, against the modules compiled into the binary, constructing every one of them; every env:NAME secret the configs reference must be present in check’s own environment.

What a write does to a running pipeline

Hot-reload granularity differs by kind, per crates/blockwatcher-core/src/control/writes.rs and, for the delete noted below, crates/blockwatcher-core/src/control/deletes.rs:

The mechanics, including what a failed fan-out restart does, are on Resources § Lifecycle.

Validation and deletes

Every write validates before it reaches storage, and every validation rejection surfaces on the wire as 422 Unprocessable Entity; the per-kind checks are on Resources § Write-time validation. The refusal messages quoted throughout this section are those checks’ actual output.

A DELETE of a network, spec, or sink refuses outright while any stored monitor still references the id, per refuse_if_referenced (crates/blockwatcher-core/src/control/deletes.rs): delete the monitors first.

Network

A network names one feed to watch: an id, a chain tag, and a source module selection (Network, crates/blockwatcher-types/src/resource.rs). It is written through PUT /networks/{id} on the HTTP API or as one JSON file under a seed directory’s networks/ subdirectory. Writing a network always restarts that network’s pipeline; the restart is cheap because the checkpoint survives it, per Resources § Lifecycle.

A complete network resource. The source.config object is registry_examples/evm_rpc.json from crates/blockwatcher-evm/src/, copied verbatim from the same file the crate’s family-completeness test constructs (the resource envelope around it precludes a literal include); any change to that file updates this example in the same change, per the wiki-parity rule:

{
  "id": "eth-mainnet",
  "chain": "evm",
  "source": {
    "module": "evm-rpc",
    "config": {
      "start_block": 18000000,
      "endpoints": [
        {
          "name": "alchemy",
          "url_secret": "env:BLOCKWATCHER_EXAMPLE_EVM_RPC_URL",
          "priority": "high",
          "rate_limit": { "rps": 25 }
        },
        {
          "name": "public",
          "url_secret": "env:BLOCKWATCHER_EXAMPLE_EVM_RPC_URL_FALLBACK",
          "priority": "low"
        }
      ],
      "confirmations": 12
    }
  }
}

Fields

Network rejects an unrecognized field (#[serde(deny_unknown_fields)]), so a typo’d key fails the write rather than being silently ignored.

source is the universal module envelope (ModuleSel, resource.rs):

Write-time validation constructs the named module with the given config, so every refusal quoted below is raised at the write (as a 422) or at seed validation, never discovered later at runtime. Updating an existing network additionally recompiles every monitor already stored for it against the incoming chain, refusing a chain reassignment that would strand them, per put_network (crates/blockwatcher-core/src/control/writes.rs).

Endpoints: the shared pool vocabulary

Both source modules name a pool of HTTP JSON-RPC endpoints with the same shape (EndpointDef, crates/blockwatcher-evm/src/source/endpoint.rs). Each endpoint object:

evm-rpc

Polls confirmed blocks over HTTP JSON-RPC: events via eth_getLogs, and, only while some monitor watches functions, full transaction bodies too. Its config (EvmRpcConfig, crates/blockwatcher-evm/src/source/rpc/config.rs) rejects unrecognized fields, including a network key: which network a source’s events belong to is the engine’s to supply, never the config’s.

The annotated example above is the module’s complete example config. The tunables:

logs_window (LogsWindow, same file):

evm-mempool

Subscribes to pending transaction hashes over WebSocket and hydrates each against an HTTP pool: a notification carries a hash and nothing else, so every candidate costs one eth_getTransactionByHash. Its config (EvmMempoolConfig, crates/blockwatcher-evm/src/source/mempool/config.rs) has no start_block and no confirmations: a pending stream has no history to begin from and no reorg barrier to honor, and a start_block copied over from an evm-rpc network is refused by name as an unknown field rather than silently ignored. What that difference means for delivery guarantees is on Selectors § The position problem.

The module’s complete example config, registry_examples/evm_mempool.json from crates/blockwatcher-evm/src/, verbatim, the same file the crate’s family-completeness test constructs:

{
  "ws_url_secret": "env:BLOCKWATCHER_EXAMPLE_EVM_MEMPOOL_WS_URL",
  "endpoints": [
    {
      "name": "primary",
      "url_secret": "env:BLOCKWATCHER_EXAMPLE_EVM_RPC_URL"
    }
  ]
}

idle_policy (IdlePolicyDef, same file, the wire form of ws::IdlePolicy):

Spec

A spec is a chain-tagged decode artifact: an id, a chain tag, and a payload that core never interprets (Spec, crates/blockwatcher-types/src/resource.rs). It is written through PUT /specs/{id} on the HTTP API or as one JSON file under a seed directory’s specs/ subdirectory. The write itself compiles the payload (EvmDecoder::compile_spec, crates/blockwatcher-evm/src/decoder/mod.rs), so every refusal quoted below is raised at that write, as a 422, never later; the same compilation also reruns at boot and on pipeline restart, against payloads the write already proved valid. Writing a spec restarts every network on the spec’s chain, and on a chain reassignment its prior chain too, because every spec sharing a chain compiles together, per Resources § Lifecycle.

A complete spec resource. The payload is the two-declaration ERC-20 ABI the decoder’s own tests compile (erc20_spec_with_function, crates/blockwatcher-evm/src/decoder/mod.rs):

{
  "id": "erc20",
  "chain": "evm",
  "payload": [
    {
      "type": "event",
      "name": "Transfer",
      "anonymous": false,
      "inputs": [
        { "name": "from", "type": "address", "indexed": true },
        { "name": "to", "type": "address", "indexed": true },
        { "name": "value", "type": "uint256", "indexed": false }
      ]
    },
    {
      "type": "function",
      "name": "transfer",
      "stateMutability": "nonpayable",
      "inputs": [
        { "name": "to", "type": "address" },
        { "name": "amount", "type": "uint256" }
      ],
      "outputs": [{ "name": "", "type": "bool" }]
    }
  ]
}

Fields

Spec rejects an unrecognized field (#[serde(deny_unknown_fields)]).

The evm payload

The evm decoder parses payload as a Solidity JSON ABI array. A payload that does not parse refuses with spec '<id>': payload is not a JSON ABI array: ..., and a spec whose chain does not match the decoder is refused as an unsupported chain.

What the ABI’s declarations become:

• Events: every named event overload becomes a schema and a decode plan keyed by the keccak256 hash of its signature (the log’s first topic).
• Functions: every function overload becomes a schema and a decode plan keyed by the 4-byte selector its calldata leads with.

A monitor’s selector then names these declarations by name; see Monitor for the selector keys and Selectors for what each kind decodes.

What the decoder rejects

All of these refuse the write before a CompiledSpec is ever constructed:

• An anonymous event: spec '<id>' event '<name>' is anonymous: this decoder identifies events by the keccak256 hash of their signature in the log's first topic, which an anonymous event's log never carries, so it could never be matched at decode time.
• Two events whose selectors collide: spec '<id>': events '<a>' and '<b>' both hash to the same selector; a decoder cannot tell them apart at decode time, so only one may be declared. Two colliding functions refuse with the same message shape.
• A payload declaring nothing: spec '<id>' declares zero events and zero functions, so no selector against it could ever match anything.
• A tuple parameter with no components: tuple field '<name>' has no components, so it would disappear from the schema instead of declaring anything; give it at least one component or remove it from the ABI.
• Two components of one tuple resolving to the same name: tuple field '<name>': the component at position <n> is named '<x>', which another component of the same tuple already carries; give each component in the tuple a distinct name. When the colliding component is unnamed, the message instead reads ... the unnamed component at position <n> would default to '_<n>', which another component of the same tuple already carries; rename that other field to something other than '_<n>'.

The dotted-name flattening contract

Field names in the compiled schema come from the ABI’s parameters, flattened per flatten_field (crates/blockwatcher-evm/src/decoder/compile.rs):

• A bare tuple parameter flattens into one field per leaf, dotted by component name (name.component, recursively through nested tuples).
• An unnamed tuple component is named _<position> (_0, _1, …).
• Every other shape is one field. An array of tuples (tuple[]) never flattens: it becomes a single array-of-map field, so a predicate can address it as a whole but never a named component inside one element.

Each leaf’s ABI type maps to a canonical value family (address, unsigned or signed integer, bytes, bool, string, array, map); how a predicate types against those families is on Predicates § The type system.

The namespaces every evm spec carries

Alongside its declared events, every compiled evm spec carries the same three predicate namespaces (namespaces, crates/blockwatcher-evm/src/decoder/compile.rs). A field a particular occurrence does not carry (tx.from on a log-decoded event, tx.status on a pending transaction) resolves Unknown at predicate time, never an error:

Sink

A sink is a delivery destination: an id, a module name, that module’s config, and optional engine-owned policies (SinkDef, crates/blockwatcher-types/src/resource.rs). It is written through PUT /sinks/{id} on the HTTP API or as one JSON file under a seed directory’s sinks/ subdirectory. Writing a sink restarts every network whose stored monitors name that sink id, per Resources § Lifecycle.

A complete sink carrying every policy field beside a webhook module config, the same example Delivery guarantees § Delivery policies walks through (the policy values shown are each field’s defaults):

{
  "id": "alerts",
  "module": "webhook",
  "config": {
    "url_secret": "env:ALERTS_WEBHOOK_URL",
    "headers": { "x-source": "blockwatcher" },
    "timeout_ms": 10000,
    "body_template": "{% if type == \"digest\" %}{{ matches | length }} matches{% else %}{{ type }}: {{ monitor }}{% endif %}"
  },
  "retry": { "max_attempts": 8, "initial_backoff_ms": 200, "max_backoff_ms": 30000 },
  "throttle": { "max_deliveries": 60, "window_ms": 60000 },
  "aggregate": { "window_ms": 30000, "max_batch": 100 }
}

Fields

SinkDef rejects an unrecognized field (#[serde(deny_unknown_fields)]). retry, throttle, and aggregate are siblings of config, never nested inside it: a sink module never sees or interprets its own policies, so enforcement is identical for every module. These policies run after a match exists. Counting hits or capping alerts before a match is minted is a monitor gate, not a sink field. throttle dead-letters refused matches (replayable); max_once discards them. aggregate stalls the checkpoint for the window; threshold does not. What enforcing each policy means at runtime is documented on Delivery guarantees § Delivery policies; this page owns the fields and their constraints.

retry, throttle, and aggregate are each validated by the same validate call at two points: put_sink (crates/blockwatcher-core/src/control/writes.rs) refuses the write, and validate_and_build (crates/blockwatcher-core/src/engine/boot.rs) runs the identical check against rows already in storage at boot, refusing to start the engine instead. A policy is either absent or fully valid, never partially so.

retry (DeliveryRetry)

throttle (Throttle)

aggregate (Aggregate)

The caps bound what one open window can cost: every buffered match is held in memory and the checkpoint stays behind all of them until the window closes.

Sink modules

Each module’s example below is copied verbatim from that crate’s registry_examples/*.json file, the same file the crate’s family-completeness test constructs; any change to those files updates these examples in the same change, per the wiki-parity rule.

webhook

POSTs each sink event as JSON to a secret-referenced URL (crates/blockwatcher-sinks/src/webhook.rs). Redirects are never followed (a 3xx fails permanently). Header values come from two maps: headers is plain configuration, and header_secrets names secret references (env:NAME) resolved fresh at each delivery, the same way url_secret is — so a secret Authorization header is supported without ever holding a resolved copy past the request that carries it. A header named in both maps has no defined winner, so the write refuses rather than guessing one. The example config, registry_examples/webhook.json:

{
  "url_secret": "env:BLOCKWATCHER_EXAMPLE_WEBHOOK_URL",
  "headers": { "x-blockwatcher-monitor": "treasury" },
  "timeout_ms": 10000
}

script

Runs an operator-authored program per event with the canonical sink-event JSON on stdin (crates/blockwatcher-sinks/src/script.rs). The example config, registry_examples/script.json:

{
  "command": "/usr/local/bin/blockwatcher-notify",
  "args": ["--channel", "ops"],
  "timeout_ms": 30000
}

The exit-status contract (sysexits):

• 0: delivered. A script that exits 0 without reading stdin has still delivered; the exit status is the acknowledgement.
• 75 (sysexits EX_TEMPFAIL): transient, retried by the engine under the sink’s retry policy.
• Any other exit code: permanent, straight to the dead-letter queue.
• Death by signal, and the module’s own timeout: transient.

The last 4 KiB of the script’s stderr ride the error message; stdout is ignored.

log

Writes one line of canonical sink-event JSON to the process’s stdout per delivery (crates/blockwatcher-sinks/src/log.rs). It takes no required configuration; the example config, registry_examples/log.json, is the empty object, and any key other than body_template refuses the write:

{}

Monitor

A monitor is the rule that ties the other three kinds together: an id, the network it watches, one or more selectors, an optional predicate, an optional gate, and actions naming the sinks it delivers to (Monitor, crates/blockwatcher-types/src/resource.rs). It is written through PUT /monitors/{id} on the HTTP API or as one JSON file under a seed directory’s monitors/ subdirectory. Writing a monitor usually restarts nothing: the running pipeline hot-swaps the recompiled monitor set in place. Two things restart that one network instead: changing gate, whose previous envelope’s holds are dropped with the pipeline stopped and the pipeline then brought back (deleting a gated monitor takes the same path), and naming a sink the pipeline never spawned a worker for, per Resources § Lifecycle. Dropping holds never drops an already-emitted digest: an emission the old envelope committed but had not yet delivered survives the wipe and is delivered (or dead-lettered) when the pipeline comes back, see Gates § Delivery guarantee.

Monitor
  id         MonitorId
  network    NetworkId
  selectors  RawSelector[]
  predicate  string?          // this monitor's filter, not a shared id
  gate       ModuleSel?       // this monitor's decision rule, like predicate
  actions    SinkId[]

A complete monitor resource, selecting against the erc20 spec from the Spec page:

{
  "id": "usdc-transfers",
  "network": "eth-mainnet",
  "selectors": [
    {
      "spec": "erc20",
      "events": ["Transfer"],
      "addresses": ["0xA0b86991c6218b36c1d19D4a2e9Eb0cE3606eB48"]
    }
  ],
  "predicate": "args.value > 1000000",
  "actions": ["alerts"]
}

The same monitor with a threshold gate; omitting gate (the first example) remains valid passthrough:

{
  "id": "usdc-burst",
  "network": "eth-mainnet",
  "selectors": [
    {
      "spec": "erc20",
      "events": ["Transfer"],
      "addresses": ["0xA0b86991c6218b36c1d19D4a2e9Eb0cE3606eB48"]
    }
  ],
  "predicate": "args.value > 1000000",
  "gate": {
    "module": "threshold",
    "config": { "count": 3, "window_ms": 3600000 }
  },
  "actions": ["alerts"]
}

Fields

Monitor rejects an unrecognized top-level field (#[serde(deny_unknown_fields)]); a selector entry’s keys are the one exception, checked later by the chain’s decoder rather than by serde, per the selector table below.

The whole monitor (selectors, predicate, and gate) must compile against the live decoder, matcher, and gate catalog before the write is persisted, so every refusal quoted below surfaces as a 422 on the write, per Resources § Write-time validation.

Selector keys

Each selector entry carries a core-readable spec reference plus decoder-owned keys. For the evm decoder those are events, functions, and addresses, all optional; any other key refuses by name at compile time: unknown selector key '<key>' (compile, crates/blockwatcher-evm/src/decoder/selector.rs).

The absent-vs-empty rule

events and functions share one presence matrix:

• Naming neither selects every event AND every function the spec declares, ABI-scoped rather than chain-wide, mirroring how an absent addresses means any address.
• Naming either selects only what it names; the omitted kind is left unselected.
• An explicit empty array is refused rather than treated as absence, because the two spellings mean opposite things and the empty one would compile an entry that can never fire: selector's 'events' is empty: omit the key to select every declaration of that kind in the spec, or name at least one (functions and addresses refuse with the same shape; the addresses message ends omit the key to match any address, or name at least one).

What else a selector refuses

• A non-string entry is refused by position, never silently dropped: selector's 'events' entry at position 1 is not a string; every entry must be a name.
• A name the spec does not declare is refused with a suggestion, the spec’s first declared event or function of that kind: unknown field 'Transfr' plus did you mean 'Approval'? (see Resources § The did-you-mean suggestion).
• An address must be a 0x-prefixed 40-hex-digit string: address '<raw>' must be a 0x-prefixed 40-hex-digit string.
• A mixed-case address claims an EIP-55 checksum and is held to it: address '<raw>' is mixed-case, which claims an EIP-55 checksum, but the checksum does not match; check for a mistyped character. An all-lowercase or all-uppercase spelling makes no checksum claim and is accepted as-is.

What a selected event or function actually decodes to, and which raw material reaches the selector per source, is documented on Selectors.

Predicate

predicate is one expression string, compiled at write time against the schemas the monitor’s selectors produce; a field or namespace it names that no selected schema declares refuses the write, with a bounded edit-distance suggestion. The language (syntax, operators, the three-valued evaluation, and the type families) is documented on Predicates and the expression language.

Gate

gate compiles at write against the same schemas as the predicate. Refusals (all 422):

• unknown field on the envelope or inside config (deny_unknown_fields)
• unknown module (message lists catalog gate names)
• window_ms 0 or > 86400000: same bound language as sink throttle.window_ms
• threshold.count < 2 or > 10000
• missing block.timestamp on the schema: gate requires 'block.timestamp'; this monitor's selectors do not expose it

Changing module or config drops persisted holds for that monitor; they are not migrated.

HTTP API reference

blockwatcher’s REST control plane is an axum router mounted on the [api].listen address from instance configuration. Every mutation goes through the same ControlHandle a --seed load and the engine’s own boot path use, and every read goes through core’s typed storage facade without taking any lock the engine holds: a read stays answerable while a write is in flight. There is no path prefix: routes are exactly as shown below, against the listener’s own address, assembled in router (crates/blockwatcher-api/src/serve.rs, lib.rs).

flowchart LR
    rpc["RPC endpoints<br/>(external chains)"]
    sources["Sources<br/>evm-rpc · evm-mempool"]
    sinks["Sinks<br/>webhook · script · log"]
    storage["Storage<br/>checkpoints · dead letters · resources"]
    api["REST API"]
    metrics["Metrics"]
    engine["engine<br/>bounded channels · checkpoints"]

    subgraph pipeline["Engine pipeline"]
        direction LR
        decoder["Decoder"]
        matcher["Matcher<br/>predicates"]
        gate["Gate<br/>threshold · max_once"]
        decoder --> matcher
        matcher --> gate
    end

    rpc --> sources
    sources -->|"decode and match"| decoder
    gate --> sinks
    api -->|"manages resources"| storage
    storage <--> engine
    engine -->|"drives"| pipeline
    engine -.->|"reports"| metrics

    classDef module fill:none,stroke:#a9a3e3
    classDef core fill:none,stroke:#8a8d86,stroke-dasharray: 5 5

    class rpc,sources,sinks,decoder,matcher,gate module
    class engine,api,storage,metrics core
classDef dim fill:none,stroke:#999999,color:#999999,opacity:0.35
classDef focus fill:#ffd43b,stroke:#d9480f,stroke-width:3px,color:#1a1a1a
class rpc,sources,sinks,storage,metrics,engine,decoder,matcher,gate dim
class api focus
click sources "../concepts/selectors.html"
click decoder "../concepts/chain-agnosticism.html"
click matcher "../concepts/predicates.html"
click gate "../concepts/gates.html"
click sinks "../concepts/delivery.html"
click storage "../concepts/resources.html"
click api "http-api.html"
click metrics "observability.html"
click engine "../concepts/pipeline.html"

Key takeaways

• Every route except GET /health requires a bearer token from the labelled [auth] table, checked in constant time; a missing or unknown token is 401, a valid token with too weak a scope is 403. Every failure returns the same ErrorBody shape.
• Writes go through the same ControlHandle a --seed load and the boot path use; reads go through core’s typed storage facade without taking any lock a write holds.
• Every resource kind shares one CRUD shape with ETag/If-Match optimistic concurrency: PUT with no If-Match creates, with If-Match updates, and DELETE requires If-Match.
• A monitor test is a dry run: it reports matches without delivering to any sink, registering a cursor, or moving a checkpoint.
• Network operations (pause, resume, skip, checkpoint delete, dead letters) act on persisted state and the running pipeline directly; none of them replay through the normal ingestion path.

Every request travels the same shape, from the door to the response body:

flowchart LR
    req["HTTP request"] --> health{"path is /health?"}
    health -->|"yes"| ok["200 ok<br/>no token required"]
    health -->|"no"| auth{"require_bearer"}
    auth -->|"missing/wrong"| e401["401 unauthorized"]
    auth -->|"valid"| scope{"require(min scope)"}
    scope -->|"too weak"| e403["403 forbidden"]
    scope -->|"allowed"| route{"read or write?"}
    route -->|"write"| handle["ControlHandle<br/>same path --seed and boot use"]
    route -->|"read"| facade["core's typed<br/>storage facade"]
    handle --> resp["JSON response"]
    facade --> resp

Bodies are JSON in both directions unless noted. The route table is grouped below the same way the route modules split it: health, status, schema, resources, monitor operations, network operations (crates/blockwatcher-api/src/routes/mod.rs).

Authentication

Every route except GET /health requires:

Authorization: Bearer <token>

checked by require_bearer (crates/blockwatcher-api/src/auth/mod.rs) against the labelled [auth] table. Each row’s secret is an env:NAME reference resolved at request time; the process never holds a copy of the value between requests. The comparison is constant-time on length-equal inputs; scheme matching is case-insensitive per RFC 7235. /health is exempted by an explicit path check ahead of the router, not by a separate mount, so a request to a path that genuinely doesn’t exist is refused for lacking a token before the router gets a chance to reveal that it wouldn’t have matched anyway. A valid token is then authorized per MethodRouter by require!(Scope::…): read lists and reads, operate pauses/resumes/ replays, admin writes resources (including script sinks).

A missing or unknown token gets:

{ "error": { "code": "unauthorized", "message": "a bearer token is required" } }

with status 401 and a WWW-Authenticate: Bearer response header.

curl -H "Authorization: Bearer $BLOCKWATCHER_API_TOKEN" http://127.0.0.1:8080/status

Errors

Every failure (an engine refusal or one the API layer decides on its own) leaves through the same body shape, ErrorBody/ErrorDetail (crates/blockwatcher-api/src/error.rs):

{
  "error": {
    "code": "version_conflict",
    "message": "version conflict: expected 3, actual 5",
    "actual_version": 5
  }
}

actual_version appears only on a 412 version conflict; every other error omits it. A 500 always answers with a fixed generic body, {"error":{"code":"internal","message":"internal error"}}, regardless of what actually failed; the real detail goes to the server log only, never the wire, in body and into_response (error.rs).

Health

GET /health

The one route served without a token: a liveness probe has to answer even when secrets aren’t available yet. Asserts nothing about any pipeline.

Response 200:

{ "status": "ok" }

curl http://127.0.0.1:8080/health

Status

GET /status

A point-in-time snapshot of every currently running, paused, or abandoned pipeline, sorted by network id (blockwatcher_core::EngineStatus, crates/blockwatcher-core/src/status.rs).

Response 200, one entry per network in pipelines:

{
  "instance": "b3f1a2c4-9e21-4a6a-8c3e-1f2d3a4b5c6d",
  "paused_monitors": ["usdc-sepolia-alerts"],
  "pipelines": [
    {
      "network": "sepolia",
      "source": { "status": "catching_up", "behind": 42 },
      "checkpoint": { "cursor": { "primary": 11424039, "secondary": 54 }, "module": "evm-rpc" },
      "head": { "primary": 11424081, "secondary": 0 },
      "lag": 42,
      "in_flight_events": 0,
      "event_queue": { "len": 0, "capacity": 1000 },
      "sink_queues": { "log-sink": { "len": 0, "capacity": 100 } },
      "counters": {
        "decoded": 118, "undecodable": 0, "matched": 12, "match_errors": 0,
        "delivered": 12, "dead_lettered": 0, "dispatch_failed": 0,
        "checkpoint_write_failed": 0, "dead_letter_write_failed": 0,
        "misrouted": 0, "checkpoint_regressions_refused": 0,
        "untimestamped": 0, "gate_persist_failed": 0, "gate_hits_dropped": 0,
        "gated": 0, "gate_emitted": 0
      },
      "dead_letter_count": 0,
      "gate_outbox_depth": 0,
      "paused_monitors": []
    }
  ]
}

instance is an opaque per-process identity; two reads returning different values mean the engine restarted in between. It is null when the composition root embedding blockwatcher supplies none.

Top-level paused_monitors is every monitor an operator has paused, as persisted — pipeline membership does not enter into it, so a monitor on a paused network still appears here. It is null when storage would not answer, deliberately never an empty list: a caller cannot tell “nothing is paused” from “the store did not answer” from an empty list, and this field is what a consumer reads as the truth about pause.

source.status is one of starting, live, degraded (carries reason), catching_up (carries behind), paused (the control-plane pause view, never something a source itself reports), or abandoned (no running pipeline and no pause — a failed restart left the network unattended). checkpoint is null until the first event on that network fully completes; lag is null whenever either side it subtracts is unknown.

dead_letter_count is read fresh from storage on every call — unlike counters.dead_lettered, an in-memory count that resets to 0 whenever this pipeline restarts (a sink or monitor edit restarts every network that references it). A caller that wants “how many dead letters exist right now” should read dead_letter_count, not the counter. It is null, never 0, when that storage read fails.

gate_outbox_depth is how many committed gate emissions are still awaiting delivery: the durable outbox rows written when a gate emitted that no sink worker has settled yet. Also read fresh from storage on every call, so a paused network’s pending emissions stay visible for as long as they sit undelivered; a non-zero depth on a paused pipeline means a resume still owes deliveries. Like dead_letter_count, it is null, never 0, when the storage read fails.

A pipeline entry’s own paused_monitors is the monitors that running pipeline is actually suppressing, read from its published set. Compare with the top-level field: that is what an operator asked for, this is what is in force. They diverge only when a republish failed after a persisted pause, which is logged and repaired by the next rebuild.

curl -H "Authorization: Bearer $TOKEN" http://127.0.0.1:8080/status

Schema

GET /specs/{id}/schema

The whole vocabulary a predicate written against this spec may address: compiled through the spec’s chain decoder, not read off the raw stored payload, so it’s the decoder’s own translation rather than the ABI or IDL text itself.

Response 200 (blockwatcher_types::SchemaSet):

{
  "events": [
    { "name": "Transfer", "kind": "event", "fields": [
      { "name": "from", "ty": "address" },
      { "name": "value", "ty": "uint" }
    ] }
  ],
  "namespaces": {
    "tx": [{ "name": "hash", "ty": "bytes" }]
  }
}

Errors: 404 not_found if the spec doesn’t exist.

curl -H "Authorization: Bearer $TOKEN" http://127.0.0.1:8080/specs/usdc-erc20/schema

Resources: networks, specs, sinks, monitors

Every resource kind answers on the identical CRUD shape, generated once by the resource_routes! macro so a client written against any kind works against all of them (crates/blockwatcher-api/src/routes/resources.rs):

The collection GET returns the bare list, unpaginated: resource counts are operator-scale, not event-scale. Every item route carries optimistic concurrency through ETag / If-Match:

• GET {id} returns the resource with its current version in ETag ("3").
• PUT {id} with no If-Match is a create: 201 with the new version in ETag, or 409 already_exists if the id is already there. PUT {id} with If-Match: "<version>" is a conditional update: 200 with the new version on success, 412 version_conflict (body carries actual_version) if the version has moved on, 404 not_found if the record is gone. The path’s id and the body’s id field must agree, or the write is refused as 422 invalid_resource.
• DELETE {id} requires If-Match: "<version>": omitting it is 428 precondition_required, since a delete has no later read that would reveal it clobbered a change it never saw. A version that doesn’t match is 412 version_conflict; success is 204.

If-Match takes a strong, quoted, numeric ETag exactly as GET returned it ("3"); anything else is 400 malformed_if_match.

Resource bodies, from crates/blockwatcher-types/src/resource.rs (every shape rejects an unrecognized field except Monitor.selectors[], whose extra keys are decoder-owned and checked later, at compile time):

Network: source.config’s shape is the named module’s own; for evm-rpc (EvmRpcConfig, crates/blockwatcher-evm/src/source/rpc/config.rs) that’s a pool of endpoints (each an EndpointDef with name, a url_secret reference, optional priority/rate_limit/weight, crates/blockwatcher-evm/src/source/endpoint.rs), a required start_block, and the module’s own tunables:

{
  "id": "sepolia",
  "chain": "evm",
  "source": {
    "module": "evm-rpc",
    "config": {
      "start_block": 11424310,
      "endpoints": [
        { "name": "primary", "url_secret": "env:SEPOLIA_RPC_URL", "priority": "high", "rate_limit": { "rps": 10 } }
      ],
      "confirmations": 12,
      "max_lag_blocks": 100,
      "poll_interval_ms": 3000,
      "logs_window": { "initial": 1000, "max": 5000 },
      "probe_interval_ms": 30000
    }
  }
}

Spec: payload is the chain’s own decode artifact, opaque to core (a Solidity ABI array for evm):

{ "id": "usdc-erc20", "chain": "evm", "payload": [ { "type": "event", "name": "Transfer", "inputs": [] } ] }

SinkDef: retry is optional; a sink with none falls back to [engine].default_retry. throttle is optional too, and caps how often the sink admits deliveries; a sink with none is not throttled at all, and a present throttle object fills in any omitted field with 60 deliveries per 60,000ms. aggregate is optional as well, and batches several matches into one digest delivery; a sink with none delivers every match on its own, unbatched. See Delivery policies for what each field means and how a rejection is enforced:

{
  "id": "ops-slack",
  "module": "webhook",
  "config": { "url_secret": "env:OPS_SLACK_WEBHOOK_URL" },
  "retry": { "max_attempts": 5, "initial_backoff_ms": 200, "max_backoff_ms": 30000 },
  "throttle": { "max_deliveries": 60, "window_ms": 60000 },
  "aggregate": { "window_ms": 30000, "max_batch": 100 }
}

Monitor: predicate is optional (no predicate matches everything the selectors decode); actions names one or more SinkDef ids:

{
  "id": "usdc-sepolia-transfers",
  "network": "sepolia",
  "selectors": [{ "spec": "usdc-erc20", "addresses": ["0x1c7D..."], "events": ["Transfer"] }],
  "predicate": "args.value > 1_000e6",
  "actions": ["ops-slack"]
}

An optional gate is a sibling of predicate. Absent is passthrough:

{
  "id": "usdc-burst",
  "network": "sepolia",
  "selectors": [{ "spec": "usdc-erc20", "addresses": ["0x1c7D..."], "events": ["Transfer"] }],
  "predicate": "args.value > 1_000e6",
  "gate": { "module": "threshold", "config": { "count": 3, "window_ms": 3600000 } },
  "actions": ["ops-slack"]
}

Errors common to every write: 422 codes covering unresolvable references, an uncompilable predicate or gate, an unconstructible module config, and so on: see Resources: write-time validation for exactly what’s checked per kind, and 404 not_found / 409 already_exists / 412 version_conflict for the concurrency cases above. DELETE on a Network, Spec, or SinkDef additionally refuses as 422 still_referenced while any stored monitor still names that id.

A gate write is 422 when:

• the module name is unknown (the message lists catalog gate names, the same shape as an unknown sink module)
• this monitor’s selectors do not expose block.timestamp: gate requires 'block.timestamp'; this monitor's selectors do not expose it
• count or window_ms is out of range (threshold.count is 2..=10000; window_ms is 1..=86400000)

curl -H "Authorization: Bearer $TOKEN" http://127.0.0.1:8080/monitors

etag=$(curl -s -o /dev/null -w '%header{etag}' \
  -H "Authorization: Bearer $TOKEN" \
  http://127.0.0.1:8080/monitors/usdc-sepolia-transfers)

curl -X PUT http://127.0.0.1:8080/monitors/usdc-sepolia-transfers \
  -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
  -H "If-Match: $etag" \
  -d '{"id":"usdc-sepolia-transfers","network":"sepolia","selectors":[{"spec":"usdc-erc20","addresses":["0x1c7D..."],"events":["Transfer"]}],"predicate":"args.value > 2_000e6","actions":["ops-slack"]}'

curl -X DELETE http://127.0.0.1:8080/monitors/usdc-sepolia-transfers \
  -H "Authorization: Bearer $TOKEN" -H "If-Match: \"4\""

Monitor operations

POST /monitors/{id}/pause
POST /monitors/{id}/resume
POST /monitors/{id}/test

pause and resume persist the monitor’s pause: a process restart does not clear it, and every path that rebuilds the monitor set — a network, sink, or spec write that restarts the pipeline, or a fresh boot — reads the persisted set back, per the pause handler’s own doc comment, which covers both operations (crates/blockwatcher-api/src/routes/monitors_ops.rs). The persist is what succeeds; publishing the change to a pipeline that happens to be running is best-effort, logged rather than reported on failure. Both answer 204 on success.

test is a dry run: it reports every match the monitor’s current selectors, predicate, and gate (fresh empty journal, no persist of gate_hits) would produce against supplied input, delivering nothing: no sink is called, no cursor is registered, and no checkpoint moves, which is what makes it safe to point at a monitor serving live traffic. The request body is exactly one of payloads (caller- supplied raw payloads) or fetch (a bounded history read through the network’s own source module):

{ "payloads": [ { "json": { "from": "0x...", "value": "1000000" } } ] }

{ "fetch": { "from": { "primary": 11424000, "secondary": 0 }, "to": { "primary": 11424100, "secondary": 0 }, "limit": 50 } }

Response 200:

{
  "results": [
    { "matched": [ { "id": "...", "monitor": "usdc-sepolia-transfers", "network": "sepolia", "event": { "kind": "event", "name": "Transfer", "fields": {}, "cursor": { "primary": 0, "secondary": 0 } } } ], "undecodable": 0, "no_match": false, "eval_errors": 0 }
  ],
  "explanations_supported": true,
  "held": 0,
  "dropped": 0
}

held and dropped live on the report (not each result): they count this request’s gate Retain and Discard decisions. A monitor with no gate leaves both at 0. One payload against threshold with count: 3 returns matched: [] and held: 1. Three in-window payloads fire one digest whose matches are flattened into matched.

Each result’s explanation field is present only when a decoded event didn’t match and the matcher module was consulted for why: omitted when nothing needed explaining, JSON null when the matcher has no explain support, and the explanation tree otherwise.

no_match is true when no selector wanted this input at all — a log at the wrong address/topic0, a function no monitor named — as opposed to a decoded event the predicate rejected, or an unremarkable input with nothing to report. All three otherwise leave matched empty and undecodable/eval_errors at 0; no_match is what tells them apart.

Errors: 422 invalid_resource when both or neither of payloads / fetch is present, or when fetch targets a source that can’t fetch history (message "this source cannot fetch history"); 413 too_many_payloads past the shared cap of 100 inputs (blockwatcher_core::MAX_TEST_INPUTS).

curl -X POST http://127.0.0.1:8080/monitors/usdc-sepolia-transfers/pause \
  -H "Authorization: Bearer $TOKEN"

curl -X POST http://127.0.0.1:8080/monitors/usdc-sepolia-transfers/test \
  -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
  -d '{"payloads":[{"json":{"from":"0x1","to":"0x2","value":"5000000"}}]}'

Network operations

POST   /networks/{id}/pause
POST   /networks/{id}/resume
POST   /networks/{id}/skip
DELETE /networks/{id}/checkpoint
GET    /networks/{id}/dead-letters
DELETE /networks/{id}/dead-letters
POST   /networks/{id}/dead-letters/{match_id}/replay
DELETE /networks/{id}/dead-letters/{match_id}

pause and resume persist the network’s pause in exactly the same sense as the monitor operations above: a restart does not clear it, and a paused network is never spawned at boot, per the pause handler’s own doc comment (crates/blockwatcher-api/src/routes/networks_ops.rs). pause cancels and drains the running pipeline as part of the same request; resume restarts it. Both answer 204 on success; a restart failure inside resume, after the persisted pause has already been cleared, is logged rather than returned as an error.

Skip

POST /networks/{id}/skip

Moves a paused network’s checkpoint and source start_block forward, without replaying the gap in between: catch-up while the operator has decided it isn’t worth reading.

{ "to": "tip" }

{ "to": { "block": 11424100 } }

Response 200 (blockwatcher_core::SkipReport):

{ "network": "sepolia", "cursor": { "primary": 11424100, "secondary": 0 }, "start_block": 11424100, "previous_checkpoint": { "cursor": { "primary": 11423000, "secondary": 12 }, "module": "evm-rpc" } }

Refusal cases, all named explicitly rather than left to fall through a generic status:

• Not paused: 409 conflict: "cannot skip network '{id}' while it is not paused; pause it first".
• Rewind: 422 invalid_resource when the target is behind the current checkpoint: "cannot skip to block {N}: checkpoint is already at {M}".
• Unsupported tip: 422 invalid_resource, "this source cannot report a confirmed tip", when "to": "tip" targets a source module that has no way to answer it (e.g. evm-mempool).
• A tip lookup that is merely unreachable right now (a transient RPC failure) is 503 unavailable instead, distinct from the source genuinely not supporting the concept at all.

Checkpoint delete

DELETE /networks/{id}/checkpoint

An explicit reset for an orphaned checkpoint (one whose network resource has already been deleted). Deleting a network deliberately leaves its checkpoint behind so a network re-created under the same id resumes where the old one stopped; this route is how an operator clears that row for good once they know it won’t be reused. The checkpoint is the only row a network delete leaves waiting: pending gate emissions are dead-lettered by the delete itself (see the gate delivery guarantee), so they show up in this network’s dead letters rather than sitting in an outbox nothing reads.

Response: 204 on success.

Errors: 409 conflict ("cannot reset checkpoint for network '{id}' while the network resource still exists") while the network resource is still there; 404 not_found if there’s no checkpoint for the id at all.

Dead letters

GET    /networks/{id}/dead-letters
DELETE /networks/{id}/dead-letters
POST   /networks/{id}/dead-letters/{match_id}/replay
DELETE /networks/{id}/dead-letters/{match_id}

A dead letter can be listed, replayed, or discarded without an attempt at delivery, singly or in bulk. Discarding is permanent and does not consult the sink at all; replaying is the only route that resends a letter.

These routes answer for any id that still holds letters, whether or not its network resource exists: a deleted network’s letters (including the ones its own delete minted from pending gate emissions) stay listable, replayable (the sink resource outlives the network), and discardable. An id with neither a network resource nor letters is 404 not_found, so a typo’d id still refuses rather than answering an empty list.

GET takes offset and limit query parameters (limit defaults to 100) and pages the queue in insertion order:

curl -H "Authorization: Bearer $TOKEN" \
  "http://127.0.0.1:8080/networks/sepolia/dead-letters?offset=0&limit=50"

Response 200:

{
  "entries": [
    {
      "match_id": "5f1136ec114001c7f726218b59527365",
      "monitor": "usdc-sepolia-transfers",
      "sink": "ops-slack",
      "cursor": { "primary": 11424039, "secondary": 54 },
      "attempts": 5,
      "reason": "permanent: webhook host returned 500 five times",
      "payload": { "type": "match", "id": "5f1136ec...", "monitor": "usdc-sepolia-transfers", "network": "sepolia", "event": {} }
    }
  ]
}

payload is omitted for a letter recorded before replay support existed, which is exactly the case replay below refuses outright, since there is nothing to resend. A stored payload is a tagged SinkEvent (type: match or type: retracted). Legacy rows that stored a bare Match object still load as type: match.

POST /networks/{id}/dead-letters/{match_id}/replay

Rebuilds the letter’s sink from its stored config and re-enters the same retry path a live delivery uses. Success deletes the letter: 204. Exhaustion leaves it queued with attempts and reason updated in place and answers 502 replay_failed. A letter with no stored payload refuses as 422 invalid_resource with the message "dead letter has no replay payload (recorded before replay support)" rather than attempting anything. A letter whose payload is a retraction refuses as 422 invalid_resource with "dead letter payload is a retraction; only match events can be replayed".

Identity is (match_id, sink): MatchId is not unique across sinks. ?sink= may be omitted when that match id is unique on the network; otherwise the call answers 409 conflict with "match '{id}' is dead-lettered for more than one sink; pass sink to name the row". The same query applies to the single-letter DELETE below.

curl -X POST \
  "http://127.0.0.1:8080/networks/sepolia/dead-letters/5f1136ec114001c7f726218b59527365/replay?sink=ops-slack" \
  -H "Authorization: Bearer $TOKEN"

DELETE /networks/{id}/dead-letters

Discards every letter matching the optional sink and monitor query filters, without attempting delivery on any of them. Absent filters means “any”; both present combine as AND. Nothing matching discards zero, which is not an error.

Response 200:

{ "discarded": 3 }

curl -X DELETE \
  "http://127.0.0.1:8080/networks/sepolia/dead-letters?sink=ops-slack" \
  -H "Authorization: Bearer $TOKEN"

DELETE /networks/{id}/dead-letters/{match_id}

Discards one letter by its match id, without attempting delivery. The optional ?sink= query is the same disambiguation as replay: omit it only when the match id is unique.

Response: 204 on success.

Errors: 404 not_found when no letter with that match id (and sink, if given) exists — which is what lets a caller tell a race against a concurrent replay from a success. 409 conflict when the match id is queued for more than one sink and sink was omitted.

curl -X DELETE \
  "http://127.0.0.1:8080/networks/sepolia/dead-letters/5f1136ec114001c7f726218b59527365?sink=ops-slack" \
  -H "Authorization: Bearer $TOKEN"

Observability

blockwatcher exposes two independent views of a running instance: the HTTP API’s GET /status (a point-in-time JSON snapshot of every pipeline, covered on that page) and a Prometheus scrape endpoint carrying the same events as cumulative counters and gauges since process start. This page covers the second one: how to turn it on, exactly what it exports, and what to alert on.

flowchart LR
    rpc["RPC endpoints<br/>(external chains)"]
    sources["Sources<br/>evm-rpc · evm-mempool"]
    sinks["Sinks<br/>webhook · script · log"]
    storage["Storage<br/>checkpoints · dead letters · resources"]
    api["REST API"]
    metrics["Metrics"]
    engine["engine<br/>bounded channels · checkpoints"]

    subgraph pipeline["Engine pipeline"]
        direction LR
        decoder["Decoder"]
        matcher["Matcher<br/>predicates"]
        gate["Gate<br/>threshold · max_once"]
        decoder --> matcher
        matcher --> gate
    end

    rpc --> sources
    sources -->|"decode and match"| decoder
    gate --> sinks
    api -->|"manages resources"| storage
    storage <--> engine
    engine -->|"drives"| pipeline
    engine -.->|"reports"| metrics

    classDef module fill:none,stroke:#a9a3e3
    classDef core fill:none,stroke:#8a8d86,stroke-dasharray: 5 5

    class rpc,sources,sinks,decoder,matcher,gate module
    class engine,api,storage,metrics core
classDef dim fill:none,stroke:#999999,color:#999999,opacity:0.35
classDef focus fill:#ffd43b,stroke:#d9480f,stroke-width:3px,color:#1a1a1a
class rpc,sources,sinks,storage,api,engine,decoder,matcher,gate dim
class metrics focus
click sources "../concepts/selectors.html"
click decoder "../concepts/chain-agnosticism.html"
click matcher "../concepts/predicates.html"
click gate "../concepts/gates.html"
click sinks "../concepts/delivery.html"
click storage "../concepts/resources.html"
click api "http-api.html"
click metrics "observability.html"
click engine "../concepts/pipeline.html"

Key takeaways

• blockwatcher exposes two independent views of a running instance: /status (point-in-time JSON) and /metrics (cumulative Prometheus counters and gauges), forked from the same events but never derived from each other.
• The metrics listener is disabled by default and, once enabled, runs on its own socket with no authentication and no relation to the API’s bearer token.
• Every pipeline metric is defined once as a named constant and carries a pipeline label; sink-scoped emissions also carry a sink label except the counters marked “pipeline only”.
• blockwatcher-rpc’s connection pool and evm-mempool sources publish their own metrics under separate prefixes, outside the per-network pipeline contract.
• A rising dead-letter count, a growing in_flight_events, and a restart failure are the alerting signals this page calls out explicitly.

Turning it on: [metrics]

Disabled by default, for the same reason [api] is: a listener nobody asked for is a listener nobody remembered to firewall.

[metrics]
enabled = true
listen = "127.0.0.1:9090"

Once enabled, GET /metrics on listen serves Prometheus exposition text (crates/blockwatcher-metrics/src/lib.rs): no path prefix, no authentication, and no relation to the API’s labelled [auth] table. The metrics listener is a separate axum::Router on its own socket, bound independently of the control plane. A deployment that wants the scrape endpoint restricted to its own network should bind it to a private address or firewall the port; blockwatcher itself puts nothing in front of it. Recorder upkeep (flushing accumulated Prometheus internals) runs on a 5-second interval for as long as the process is up, so a scrape always sees fresh state rather than data queued since the last request. See Configuration reference for the full [metrics] section and its BLOCKWATCHER_METRICS__LISTEN override.

The pipeline metric contract

Every metric below is defined once, as a named constant, in crates/blockwatcher-core/src/metrics.rs: the module’s own doc comment calls these names “a contract: an exporter and its dashboards are built against them,” fixed once shipped. Every emission carries a pipeline label naming the network id; an emission scoped to one sink also carries a sink label, except the rows marked “pipeline only” below, which stay pipeline-scoped even though the event they count is sink-adjacent. An emission scoped to one gate also carries monitor and gate (module name).

That is every metric blockwatcher-core exports through the metrics facade. A test in metrics.rs pins each constant’s string literal against the contract, so a typo in the definition itself cannot silently drift from what a dashboard expects.

The same events, twice

Each pipeline event forks into two independent destinations, neither derived from the other:

flowchart LR
    event["pipeline event<br/>decode, match, deliver..."] --> atomics["PipelineCounters<br/>plain atomics"]
    event --> facade["metrics facade<br/>named constants"]
    atomics --> status["GET /status<br/>point-in-time JSON"]
    facade --> recorder["prometheus recorder<br/>5s flush"]
    recorder --> scrape["GET /metrics<br/>exposition text"]

Every row above has a twin: crates/blockwatcher-core/src/counters.rs’s PipelineCounters holds one plain atomic per status field (decoded, undecodable, matched, match_errors, delivered, dead_lettered, dispatch_failed, checkpoint_write_failed, dead_letter_write_failed, misrouted, checkpoint_regressions_refused, untimestamped, gate_persist_failed, gate_hits_dropped, gated, gate_emitted); the Prometheus contract additionally exposes in_flight_events, pipelines_aborted, the restart counters, the invalidation-failure counter, the invalidate / retract / journal-gap counters, dead_letters_pruned, and deliveries_throttled and digest_deliveries, which have no per-network JSON counterpart), and both are incremented from the same call site. The atomics feed GET /status’s counters object directly: a point-in-time read, no labels, no cardinality cost, nothing to scrape, while the Prometheus constants feed a time series an alerting rule can take a rate() of. Neither is derived from the other; they are two independent destinations for the same underlying event, which is why the numbers agree at any instant but serve different jobs: reach for /status when debugging one network right now, and for /metrics when watching every network’s trend over time.

Other metrics on this endpoint

blockwatcher-rpc’s connection pool (the load-balancing and circuit-breaking layer evm-rpc and evm-mempool both sit on) publishes its own operational metrics onto the same process-wide recorder, under a blockwatcher_rpc_ prefix (crates/blockwatcher-rpc/src/pool.rs). They are not part of the per-network contract above and carry an endpoint label (the RPC endpoint name from a network’s endpoints[] config) rather than pipeline:

evm-rpc sources publish metrics of their own under a blockwatcher_evm_ prefix (crates/blockwatcher-evm/src/source/rpc/run.rs); the reorg counter carries pipeline and depth, the skip counter carries only pipeline, and the contradiction and range-split counters carry pipeline and endpoint:

evm-mempool sources publish metrics of their own, under the same blockwatcher_evm_ prefix (crates/blockwatcher-evm/src/source/mempool/run.rs). The skip counter carries endpoint; the reconnect counter does not, since a lost subscription is not attributable to one endpoint:

Alerting hints

Dead-letter count rising. increase(blockwatcher_dead_letters_total[15m]) > 0 sustained, for one sink label, means that sink is failing permanently or exhausting retries, not a one-off blip. Pull the reason off GET /networks/{id}/dead-letters (each entry’s reason string leads with transient: or permanent:) before deciding whether to fix the sink target or just replay.

A sink’s throttle is swallowing everything. increase(blockwatcher_deliveries_throttled_total[15m]) sustained at roughly the same rate as increase(blockwatcher_matches_total[15m]) for one sink label means that sink’s throttle policy is dead-lettering nearly every match rather than occasionally shedding a burst. Widen max_deliveries or window_ms for that sink, or pause the monitors feeding it, rather than leaving it to accumulate dead letters that need replaying later. This match-for-match comparison only holds for a sink with no aggregate policy: blockwatcher_deliveries_throttled_total counts suppressed matches while blockwatcher_deliveries_total counts a digest as one delivery regardless of how many matches it carried, so for an aggregating sink the throttled-versus-delivered ratio is not match-over-match. Compare against blockwatcher_digest_deliveries_total instead to see how many of that sink’s deliveries were digests in the first place.

Checkpoint lag growing. Watch blockwatcher_in_flight_events for a value that climbs and never comes back down, alongside blockwatcher_checkpoint_write_failures_total ticking upward. The first says a delivery is stuck open (check which sink’s queue is backed up), and the second says storage itself is the problem; either one means the network’s checkpoint has stopped advancing, which is exactly the signal GET /status’s lag field surfaces from the other side.

Threshold never fires. Check blockwatcher_gate_untimestamped_total, whether count is higher than traffic, and whether invalidate drained holds (must not). Compare event timestamps, not wall clock. GET /status pipeline counters gated (Retain+Discard) and gate_emitted (one per Emit) are the point-in-time twins.

Checkpoint stuck and persist_failed ticking. Same class as a closed sink: fix storage; do not skip the event. Watch blockwatcher_gate_persist_failed_total.

Tip reorg “lost” a burst. If holds with cursor ≤ from were dropped, that is a bug (drain-all). Replay will not restore them.

Gate log lines carry network, monitor, and gate (module name).

A network’s source has stopped coming back. blockwatcher_pipeline_source_restart_failures_total > 0 for a pipeline label means the restart supervisor tried and failed to bring that network’s source back up, and, per the metric’s own contract, nothing further will retry it on its own; the pipeline is down until an operator intervenes. This is a stronger signal than an occasional blockwatcher_pipeline_source_restarts_total tick, which just means a source exited and was cleanly respawned. A failed invalidate is a different abandonment: watch blockwatcher_source_invalidation_failures_total for that.

A deep invalidate failed to finish. blockwatcher_source_invalidation_failures_total > 0 for a pipeline label means retract or rewind did not complete. The checkpoint is left unrewound and the source is not restarted; this is not a crash-restart failure. Inspect dead letters for type: retracted payloads, then intervene.

A deep invalidate left a journal gap. increase(blockwatcher_journal_gap_total[15m]) > 0 for a pipeline label means that network’s invalidate cursor plus journal_depth did not cover the checkpoint high-water mark, so some already-delivered match ids cannot be retracted. The matching error log names the network, from, the checkpoint, and the configured depth. Raise journal_depth (it must stay well above confirmation depth) and treat consumer-side undo for ids older than the window as an operator problem: the engine will still rewind and restart.

A source keeps invalidating. increase(blockwatcher_source_invalidations_total[15m]) climbing on one pipeline is unusual for a quiet chain; pair it with blockwatcher_retracts_total to see whether sinks are actually receiving undos. An unrecovered retract leaves the checkpoint unrewound: watch blockwatcher_source_invalidation_failures_total, blockwatcher_dead_letters_total, and in_flight_events on that network rather than expecting a restart.

A reorg past the tracked window. increase(blockwatcher_evm_reorgs_total[15m]) with depth="beyond_window" on one pipeline means no tracked ancestor matched the live chain. The source invalidates from just below the oldest tracked height, a rewind bounded by the tracker’s own depth rather than genesis: the engine retracts journaled deliveries above that bound and restarts the source, the same path as any other invalidation, so expect blockwatcher_source_invalidations_total to tick alongside this depth label.

A bloom-skip rate of zero is not itself a fault. blockwatcher_evm_bloom_skips_total staying flat on a network with a selective monitor (one whose filter names addresses or topic0s) and an active chain is informational, not a fault: it means bloom_screen is disabled for that network, or the merged filter is broad enough (no addresses and no topic0s) that there is nothing left for a header’s bloom to refute. Read it alongside the filter a network’s monitors actually produce before treating it as a signal of anything gone wrong.

A bloom-contradiction count that is not zero. increase(blockwatcher_evm_bloom_contradictions_total[15m]) > 0 for a pipeline and endpoint pair means that endpoint served a header bloom that failed to admit a log it returned for the same block, which can only happen when its blooms do not describe its own logs. Two things follow, and neither is automatic:

• The running source instance that observed the contradiction has already disabled bloom_screen for itself, so no further window on that particular instance is at risk. That protection belongs to the instance, not the endpoint or the process: a restart of that pipeline’s source (the supervisor recovering from an exit, a proven-reorg invalidation, or a monitor change that escalates to a restart) builds a fresh instance with screening enabled again, against the same endpoint that already proved its blooms unreliable. Set bloom_screen = false for that network to make the disabling survive a restart; nothing else does.
• The contradiction proves the endpoint’s blooms are wrong, but says nothing about when the endpoint started serving wrong ones. Every window this source screened earlier in its run rested on the same untrustworthy blooms; the contradiction is only the first one this source happened to catch because a later window’s own headers still forced a real fetch. Read blockwatcher_evm_bloom_skips_total for the same pipeline to see how many windows were screened before the trip, and re-scan that range through POST /monitors/{id}/test’s fetch mode (which never screens) or a fresh network with start_block covering it: the checkpoint has already advanced past those windows, and nothing rewinds them on its own.

Read the matching warning log for the network, endpoint, the contradicting window’s own block range, the first contradicting block, and the contradiction count, and treat that endpoint’s blooms as untrusted for any other purpose too.

An evm-mempool connection is flapping. increase(blockwatcher_evm_mempool_reconnects_total[15m]) > N for a pipeline label means that network’s WebSocket subscription keeps dropping and reconnecting; GET /status’s source.status only ever shows whichever state (live, degraded, catching_up) is current at scrape time, so a short-lived flap between two scrapes is invisible there even though it loses whatever was pending in the node’s mempool during the gap.

Running with Docker

docker/ ships a two-service Compose stack (blockwatcher itself, plus a companion web UI), meant as a runnable local demo, not a production manifest. It has its own instance config, its own Dockerfiles, and its own .env, and it seeds the sink-script-monitor example on first boot: a usdc-script-test monitor watching USDC Transfer events on Sepolia, delivering every match to the dashboard’s ui-ingest webhook and to a notify-script script sink that appends one line per match to /data/script-sink-events.log inside the blockwatcher container.

The stack: docker/compose.yaml

services:
  blockwatcher:
    build:
      context: ..
      dockerfile: docker/Dockerfile.blockwatcher
    command: ["--config", "/etc/blockwatcher/blockwatcher.toml", "--seed", "/opt/script-sink/resources"]
    env_file:
      - .env
    environment:
      UI_INGEST_URL: http://ui:8080/ingest
      UI_INGEST_SECRET: ${UI_INGEST_SECRET:?set it in docker/.env}
    volumes:
      - ./blockwatcher.toml:/etc/blockwatcher/blockwatcher.toml:ro
      - ../examples/sink-script-monitor:/opt/script-sink:ro
      - blockwatcher-data:/data
    healthcheck:
      test: ["CMD", "curl", "-fsS", "http://localhost:8080/health"]
      interval: 5s
      timeout: 3s
      retries: 30
    restart: unless-stopped

  ui:
    build:
      context: ..
      dockerfile: docker/Dockerfile.ui
    environment:
      BLOCKWATCHER_API_URL: http://blockwatcher:8080
      BLOCKWATCHER_API_TOKEN: ${BLOCKWATCHER_API_TOKEN:?set it in docker/.env}
      UI_OPERATOR_SECRET: ${UI_OPERATOR_SECRET:?set it in docker/.env}
      UI_INGEST_SECRET: ${UI_INGEST_SECRET:?set it in docker/.env}
      RUST_LOG: info
      UI_ALLOWED_HOSTS: "127.0.0.1,localhost,[::1],ui:8080"
    ports:
      - "127.0.0.1:8080:8080"
    volumes:
      - ui-data:/data
    depends_on:
      blockwatcher:
        condition: service_healthy
    restart: unless-stopped

volumes:
  blockwatcher-data:
  ui-data:

The container and volume topology (a different picture from The dashboard’s application-level request flow, which this stack’s ui service is the one running) looks like this:

flowchart LR
    subgraph blockwatcher_c["blockwatcher service, no published port"]
        blockwatcherbin["blockwatcher binary"]
        blockwatchervol[("blockwatcher-data<br/>/data/blockwatcher.db<br/>/data/script-sink-events.log")]
    end
    subgraph ui_c["ui service, 127.0.0.1:8080"]
        uibin["ui server"]
        uivol[("ui-data<br/>/data/ui.db")]
    end

    blockwatcher_c -->|"depends_on:<br/>service_healthy"| ui_c
    ui_c -->|"REST proxy<br/>to blockwatcher:8080"| blockwatcher_c
    blockwatcher_c -->|"webhook POST<br/>to ui:8080/ingest"| ui_c

Both services build from the repository root (context: ..) so each Dockerfile can COPY whichever part of the workspace it needs: the blockwatcher service’s build only touches crates/; the ui service’s build touches ui/.

blockwatcher boots with --config /etc/blockwatcher/blockwatcher.toml (the file bind-mounted read-only from ./blockwatcher.toml) and --seed /opt/script-sink/resources, the example’s resource directory bind-mounted read-only from ../examples/sink-script-monitor (what that seed loads, and when, is covered below). env_file: .env is where BLOCKWATCHER_API_TOKEN comes from; UI_INGEST_URL is a value blockwatcher itself never reads at boot: it’s resolved later, at delivery time, once the ui service asks blockwatcher to create a sink. Its healthcheck curls its own /health (the one route the HTTP API serves without a bearer token), five seconds apart, up to 30 times, which is what ui’s depends_on: condition: service_healthy waits on before starting. blockwatcher-data is the one persistent volume: it holds /data/blockwatcher.db, the SQLite store named in blockwatcher.toml below, so pipeline state, resources, and checkpoints survive a docker compose restart or a rebuild. Note there is no ports: entry for this service at all: its API is reachable from ui over the compose network at http://blockwatcher:8080, but not published to the host; reaching it directly from outside the stack means adding a port mapping yourself or running docker compose exec blockwatcher ….

ui is the only service exposed to the host, and only on the loopback interface (127.0.0.1:8080:8080, not 0.0.0.0). BLOCKWATCHER_API_TOKEN, UI_OPERATOR_SECRET, and UI_INGEST_SECRET are required at compose-parse time (:?set it in docker/.env); the last two must not be the same value.

UI_ALLOWED_HOSTS and UI_INGEST_URL work together, and neither makes sense read alone. On its own first boot (ui/server/src/main.rs), the ui service calls blockwatcher’s API to create a webhook sink named ui-ingest, whose url_secret names UI_INGEST_URL (a reference blockwatcher resolves in its own environment, not the UI’s), which is exactly why the blockwatcher service (not ui) is the one carrying that variable above. From then on, every match any monitor produces that names ui-ingest among its actions gets delivered as a webhook request from the blockwatcher container straight into ui, addressed by compose service name and arriving with Host: ui:8080 (a header this companion checks against an allowlist that defaults to loopback names only) (ui/server/src/guard.rs, ui/server/src/config.rs). Ingest also presents x-blockwatcher-ingest from UI_INGEST_SECRET; operators reach /api with a session from UI_OPERATOR_SECRET. The Host/Origin guard is what keeps a DNS-rebinding page from looking like this host. Without UI_ALLOWED_HOSTS naming ui:8080 explicitly, that legitimate ingest call would be indistinguishable from an attack and rejected identically. ui-data is this service’s own persistent volume, separate from blockwatcher’s: the two services never share a data directory.

The two Dockerfiles

docker/Dockerfile.blockwatcher: a two-stage build. The first stage (rust:1.88-bookworm) copies just Cargo.toml, Cargo.lock, and crates/, then cargo build --release -p blockwatcher: no ui/ in this stage’s build context, since the binary crate doesn’t need it. The second stage (debian:bookworm-slim) installs only ca-certificates (for outbound TLS to an RPC provider) and curl (for the healthcheck above), copies the release binary in, and sets it as ENTRYPOINT. Two stages, not one, is what keeps the shipped image free of the Rust toolchain and the crate source it was built from.

docker/Dockerfile.ui: three stages, because the UI is two components. node:22-bookworm builds the web assets (npm ci, npm run build) from ui/web; a second, independent rust:1.88-bookworm stage builds ui/server; the final debian:bookworm-slim stage copies the server binary and the web build’s dist/ output into one image, EXPOSEs 8080, and sets three environment defaults (UI_STATIC_DIR=/app/static, UI_BIND=0.0.0.0:8080, UI_DB_PATH=/data/ui.db) that the compose file above never overrides, so they’re exactly what the running container uses.

The same two Dockerfiles are what a vX.Y.Z tag pushes to GHCR as ghcr.io/thethirdorigin/blockwatcher and ghcr.io/thethirdorigin/blockwatcher-ui, both tagged with the workspace version. The UI image tag is the release it shipped with, not ui/server’s own crate version.

docker/blockwatcher.toml: the containerized instance config

[api]
enabled = true
listen = "0.0.0.0:8080"

[[auth.tokens]]
label = "compose"
scope = "admin"
secret = "env:BLOCKWATCHER_API_TOKEN"

[storage]
module = "sqlite"
config = { path = "/data/blockwatcher.db" }

Two differences from the annotated example on the Configuration reference page are worth calling out. First, [api].listen binds 0.0.0.0, not 127.0.0.1: correct inside a container, where “loopback” means only the container’s own network namespace and would make the ui service’s cross-container call unreachable; the compose file is what keeps this from being exposed carelessly, since blockwatcher publishes no host port at all. Second, there is no [metrics] section here, so the Prometheus scrape endpoint described on the Observability page stays at its default (disabled) in this stack; enabling it means adding the section to this file and, if it should be reachable from outside the container, a ports: mapping in compose.yaml to go with it. There is also no [engine] section, so every engine tunable (channel capacities, drain deadline, retry policy, the matcher module) runs at whatever blockwatcher-core’s own defaults are. docker/ itself carries no resources/ tree: the network, spec, sink, and monitor JSON the stack seeds is examples/sink-script-monitor’s, bind-mounted in, and once the first boot has loaded it the ui service is how those four resource kinds are created and edited against the running blockwatcher API.

Seeding under compose

The command: in compose.yaml passes --seed /opt/script-sink/resources, the read-only mount of examples/sink-script-monitor/resources: one Sepolia network, the USDC ERC-20 spec, the ui-ingest and notify-script sinks, and the usdc-script-test monitor wiring them together. --seed is a one-time, first-boot-only load (it never touches a store that already holds anything, see Seed), so the seed lands exactly once: every later docker compose up boots with whatever blockwatcher-data’s SQLite file already holds, and the running instance is authoritative from then on. A fresh seed therefore takes a docker compose down -v first, which discards both data volumes. To seed a different resource directory instead, point the bind mount (and the --seed path) at it before the first boot, or POST resources through the ui service’s API-proxying UI once it is healthy.

Running it

cp docker/.env.example docker/.env

Edit docker/.env and set BLOCKWATCHER_API_TOKEN to any string, UI_OPERATOR_SECRET and UI_INGEST_SECRET to two different strings (compose interpolates all three with :?set it in docker/.env, and the companion refuses to boot if the last two are equal), and SEPOLIA_RPC_URL to an RPC endpoint, the variable the seeded network’s url_secret names. Any other url_secret a resource names later has to resolve from this same .env too, since env_file puts every variable in it into the blockwatcher container’s environment whether blockwatcher itself defines it or not.

Then stamp a recent start block into the seeded network resource (the seed’s start_block is absolute, and a stale one replays history):

./examples/sink-script-monitor/setup.sh

docker compose -f docker/compose.yaml up --build -d

The dashboard becomes reachable at http://127.0.0.1:8080 once its healthcheck dependency on blockwatcher passes, and matches appear there as Sepolia produces USDC transfers. The script sink’s copy of the same matches is a log inside the blockwatcher container:

docker compose -f docker/compose.yaml exec blockwatcher \
  tail -f /data/script-sink-events.log

Each line carries the full sink event as raw={...} JSON; the README’s Docker walkthrough carries a jq filter that unpacks it into the transfer’s interesting fields. docker compose down -v removes both named volumes along with the containers (and re-arms the first-boot seed); drop -v to keep blockwatcher-data/ui-data across a teardown.

Monitoring ERC-20 transfers [RPC]

Your first monitor already walked examples/source-rpc-monitor/ end to end: running it, reading the output, narrowing a predicate through the API. This page takes a different pass at the same directory: every file, reproduced in full, annotated field by field, as a reference for what each line actually does rather than a “do this next” narrative. Read that page first if you haven’t run the example yet; read this one when you want to know exactly what you’re looking at.

source-rpc-monitor/
├── .env.example
├── blockwatcher.toml
├── setup.sh
└── resources/
    ├── networks/sepolia.json
    ├── specs/usdc-erc20.json
    ├── sinks/log-sink.json
    └── monitors/usdc-sepolia-transfers.json

.env.example

SEPOLIA_RPC_URL=https://sepolia.infura.io/v3/YOUR_API_KEY
BLOCKWATCHER_API_TOKEN=local-test-token

Two variables, both required. Nothing under resources/ ever holds SEPOLIA_RPC_URL’s actual value: resources/networks/sepolia.json below names the variable itself, and blockwatcher resolves it from its own process environment the moment it needs it. That indirection exists because the URL usually is the credential: most providers put the API key somewhere in the path or query string, so the file that would otherwise carry it stays out of the resource entirely and out of anything that might get committed. BLOCKWATCHER_API_TOKEN can be any string here: the example’s blockwatcher.toml binds the API to 127.0.0.1 only, so there is no network exposure to defend against, just a token the example’s own curl commands need to match.

blockwatcher.toml

[api]
enabled = true
listen = "127.0.0.1:8080"

[[auth.tokens]]
label = "operator"
scope = "admin"
secret = "env:BLOCKWATCHER_API_TOKEN"

[metrics]
enabled = true
listen = "127.0.0.1:9090"

[storage]
module = "sqlite"
config = { path = "blockwatcher.db" }

[engine]
event_channel_capacity = 1000
sink_channel_capacity = 100
drain_deadline_ms = 5000
matcher = { module = "expr", config = {} }

Every section here is covered generally on the Configuration reference; what’s worth noting about this instance of it: [metrics] is turned on (unlike the Docker stack’s config, which leaves it off), so curl localhost:9090/metrics works alongside the API without any extra setup (see Observability for what’s on that endpoint). [storage].config.path is a bare relative filename, blockwatcher.db, so the database lands next to wherever the process’s current directory is when it starts: inside examples/source-rpc-monitor/ if you cd there first, per the running instructions. [engine]’s three explicit values (event_channel_capacity, sink_channel_capacity, drain_deadline_ms) each differ from EngineConfig’s own defaults (256/64/10000 respectively, see Configuration reference); an empty [engine] section would boot with those defaults instead.

setup.sh

#!/usr/bin/env bash
set -euo pipefail
cd "$(dirname "$0")"

[ -f .env ] || { echo "error: no .env — run: cp .env.example .env, then edit it" >&2; exit 1; }
set -a; . ./.env; set +a

: "${SEPOLIA_RPC_URL:?error: SEPOLIA_RPC_URL is unset in .env}"
case "$SEPOLIA_RPC_URL" in
  *YOUR_API_KEY*) echo "error: SEPOLIA_RPC_URL still holds the placeholder YOUR_API_KEY" >&2; exit 1 ;;
  http://*|https://*) ;;
  *) echo "error: SEPOLIA_RPC_URL must be an http(s) URL, not a ${SEPOLIA_RPC_URL%%:*}: one" >&2; exit 1 ;;
esac

head_hex=$(curl -fsS "$SEPOLIA_RPC_URL" \
  -H 'Content-Type: application/json' \
  -d '{"jsonrpc":"2.0","method":"eth_blockNumber","params":[],"id":1}' \
  | jq -re '.result') || { echo "error: the endpoint did not answer eth_blockNumber" >&2; exit 1; }

start=$(( head_hex - 20 ))

net=resources/networks/sepolia.json
tmp=$(mktemp)
jq --argjson b "$start" '.source.config.start_block = $b' "$net" > "$tmp" && mv "$tmp" "$net"

(Trimmed of its own comments and echo lines above, the repository copy explains its own reasoning inline.) The script is three checks and one mutation, in order: a .env file exists at all; the RPC URL is set, isn’t still the literal placeholder, and has a scheme that’s actually http(s) rather than, say, a stray ws:// pasted from the wrong example; and a live eth_blockNumber call succeeds against it, subtracted by 20 blocks and written into sepolia.json’s source.config.start_block in place, using jq and a temp file so a crash mid-write can never leave the resource file truncated. Nothing about this is required for blockwatcher itself to run: it’s a convenience so a fresh clone doesn’t need a human to look up a current Sepolia block number by hand. Running the network file with its original start_block: 11424310 still works if that block hasn’t rolled out of the RPC provider’s retained history; it just costs a longer catching_up wait if 11424310 is very old by the time you try it.

resources/networks/sepolia.json

{
  "id": "sepolia",
  "chain": "evm",
  "source": {
    "module": "evm-rpc",
    "config": {
      "start_block": 11424310,
      "endpoints": [
        {
          "name": "primary",
          "url_secret": "env:SEPOLIA_RPC_URL",
          "priority": "high",
          "rate_limit": { "rps": 10 }
        }
      ],
      "confirmations": 12,
      "max_lag_blocks": 100,
      "poll_interval_ms": 3000,
      "logs_window": { "initial": 1000, "max": 5000 },
      "probe_interval_ms": 30000
    }
  }
}

chain: "evm" selects the evm decoder family; source.module: "evm-rpc" selects confirmed-block scanning over evm-mempool‘s pending stream (see Selectors § The source for what that choice determines about what a selector here can ever match). endpoints is a pool of one: name is this instance’s own label for it (shows up in the blockwatcher_rpc_* metrics’ endpoint label, per Observability); priority: "high" matters once a second endpoint with a lower priority is added: the pool prefers higher-priority candidates and only falls back when they’re rate-limited or breaker-open; rate_limit.rps: 10 caps outbound calls to a level a free-tier provider key tolerates. confirmations: 12 is how many blocks must sit on top of one before its logs are trusted (deeper than the testnet’s own instant-final default, a realistic value for demonstrating a real reorg-safety margin rather than the bare minimum). max_lag_blocks: 100 is the threshold past which source.status reports catching_up instead of live. poll_interval_ms: 3000 is how often the source checks for a new head. logs_window governs how many blocks one eth_getLogs call spans while catching up: starting at 1000 and growing to 5000 as the gap narrows. None of these seven tuning keys are required: every one has its own default, and the quickstart’s network file carries none of them, but each is set here to a number an operator running against a real testnet would actually pick, rather than a value chosen only to demonstrate that the field exists.

resources/specs/usdc-erc20.json

{
  "id": "usdc-erc20",
  "chain": "evm",
  "payload": [
    {
      "type": "event",
      "name": "Transfer",
      "anonymous": false,
      "inputs": [
        { "name": "from",  "type": "address", "indexed": true },
        { "name": "to",    "type": "address", "indexed": true },
        { "name": "value", "type": "uint256", "indexed": false }
      ]
    },
    {
      "type": "event",
      "name": "Approval",
      "anonymous": false,
      "inputs": [
        { "name": "owner",   "type": "address", "indexed": true },
        { "name": "spender", "type": "address", "indexed": true },
        { "name": "value",   "type": "uint256", "indexed": false }
      ]
    }
  ]
}

This is a Solidity ABI fragment list: the raw artifact evm’s decoder compiles once, at write time, into the schema a selector and predicate actually work against. indexed: true on from/to/owner/spender means those parameters live in the log’s topics, not its data; value is indexed: false because ERC-20’s Transfer/Approval standard puts the transferred or approved amount in the log’s data word instead (a deliberate ABI design choice this fragment merely records, not something blockwatcher infers). anonymous: false on both means each event keeps its normal topic0 event signature hash, which is what the decoder actually dispatches on. Only Transfer is ever selected by this example’s monitor; Approval sits in the spec unused, because nothing requires a spec to carry only what one particular monitor asks for: a second monitor on the same network watching approvals could reference this same spec file without it changing at all.

resources/sinks/log-sink.json

{
  "id": "log-sink",
  "module": "log",
  "config": {},
  "retry": {
    "max_attempts": 1,
    "initial_backoff_ms": 100,
    "max_backoff_ms": 1000
  }
}

module: "log" puts each match on the process’s own stdout as a single line of JSON, the shipped sink with the least that can go wrong at delivery time, since there’s no network call and no credential to resolve. config: {} because the log sink takes no configuration at all. retry.max_attempts: 1 means exactly one delivery attempt: on failure (realistically, only a broken stdout pipe) the match dead-letters immediately rather than retrying a failure mode backoff can’t fix. initial_backoff_ms/max_backoff_ms are present but never exercised at max_attempts: 1: they’d matter only if this sink retried, which it doesn’t.

resources/monitors/usdc-sepolia-transfers.json

{
  "id": "usdc-sepolia-transfers",
  "network": "sepolia",
  "selectors": [
    {
      "addresses": ["0x1c7D4B196Cb0C7B01d743Fbc6116a902379C7238"],
      "spec": "usdc-erc20",
      "events": ["Transfer"]
    }
  ],
  "predicate": "args.value > 0",
  "actions": ["log-sink"]
}

network: "sepolia" ties this monitor to the network resource above: one network, always, per monitor. The one selector entry restricts to a single contract address (Sepolia’s USDC), names usdc-erc20 as the spec to decode against, and lists events: ["Transfer"]: with events present and functions absent, selector compilation fills only the events table with that one name and leaves the functions table empty; Approval, though declared in the spec, is never decoded by this selector because it was never named. In English, the predicate args.value > 0 reads “keep every transfer that moved a nonzero amount” , which in practice is nearly every one, since a zero-value Transfer is legal ABI-wise but rare in the wild; it exists mainly to show that a predicate is present and doing something, not to meaningfully filter this particular feed. actions: ["log-sink"] sends every match to the one sink above; a monitor with more sinks in its actions list would fan the same match out to each of them independently.

Variations

Watch a different contract. The spec is generic ERC-20 (nothing in usdc-erc20.json is USDC-specific), so pointing at a different token on Sepolia only means a different addresses entry on the monitor (or a second monitor entirely, if you want both watched at once). PUT the existing monitor with a new address and a fresh If-Match:

export TOKEN=$BLOCKWATCHER_API_TOKEN

etag=$(curl -s -o /dev/null -w '%header{etag}' \
  localhost:8080/monitors/usdc-sepolia-transfers -H "Authorization: Bearer $TOKEN")

curl -s -X PUT localhost:8080/monitors/usdc-sepolia-transfers \
  -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' -H "If-Match: $etag" \
  -d '{
    "id": "usdc-sepolia-transfers",
    "network": "sepolia",
    "selectors": [{
      "addresses": ["0xfFf9976782d46CC05630D1f6eBAb18b2324d6B14"],
      "spec": "usdc-erc20",
      "events": ["Transfer"]
    }],
    "predicate": "args.value > 0",
    "actions": ["log-sink"]
  }'

(0xfFf9… is Sepolia’s canonical WETH9: any ERC-20 deployment works the same way.)

A predicate with more than one clause. Your first monitor already showed a single threshold; the predicate language supports combining conditions with &&. To keep only large transfers that don’t originate from a specific address (say, a known exchange hot wallet you want to exclude from the feed):

args.value > 500e6 && args.from != 0xF977814e90dA44bFA03b6295A0616a897441aceC

Both operands type-check against the schema usdc-erc20 compiles: args.value and args.from are both Transfer’s own declared parameters (value a uint256, from an address). Excluding by sender has to go through args.from here, not tx.from: a log-decoded occurrence never carries tx.from/tx.to/tx.value at all (see the selectors comparison table), since nothing in a log’s own envelope holds them; only a functions selector, decoding calldata rather than a log, has a tx.from to read.

Deliver to a webhook instead of the log sink. Add a second sink resource using the webhook module (its url_secret follows the same env:NAME indirection as the network’s RPC URL above), then point the monitor’s actions at it instead of (or alongside) log-sink:

{
  "id": "ops-webhook",
  "module": "webhook",
  "config": {
    "url_secret": "env:OPS_WEBHOOK_URL",
    "headers": { "X-Source": "blockwatcher-source-rpc-monitor" },
    "timeout_ms": 10000
  },
  "retry": { "max_attempts": 5, "initial_backoff_ms": 200, "max_backoff_ms": 30000 }
}

PUT it to /sinks/ops-webhook, export OPS_WEBHOOK_URL in the same shell blockwatcher runs in, then PUT the monitor again with "actions": ["ops-webhook"]. A higher max_attempts than the log sink’s 1 makes sense here: an HTTP endpoint has real transient failure modes (a momentary 503, a timeout) that a retry can actually recover from, unlike a broken stdout pipe.

Monitoring ERC-20 transfers [Mempool]

examples/source-mempool-monitor/ is the evm-mempool twin of examples/source-rpc-monitor: same contract, same usdc-erc20 spec file, same sink shape: watching pending calls to USDC’s transfer function instead of confirmed Transfer events. This page gives it the same file-by-file tour, but leads with what’s different, because what’s different here isn’t cosmetic.

What’s different from watching confirmed events

evm-mempool trades completeness for latency: every fact below is already established in more depth on Selectors § The position problem and Delivery guarantees § The evm-mempool exception; this is the short version an operator needs before running this example:

• The cursor is a per-run arrival counter, not a chain position. There is no start_block to stamp (a pending transaction has no place in the chain yet), and a restart’s fresh checkpoint starts the counter over, so it can never be replayed across a process boundary.
• Dedupe on the transaction’s own hash, never on the match id. The same pending call seen again after a restart, or even twice within one run if it gets mined mid-hydration, is minted a different arrival number and therefore a different id.
• tx.status and every block.* field are always absent. Nothing here has a receipt or a mined block to read them from.
• tx.index is usually absent, but not always. A call can get mined between the pending-tx notification and the hydration lookup that fetches its full data: the copy that comes back then carries a real transactionIndex it didn’t have a moment earlier.
• This is why this source falls outside at-least-once delivery. A crash can lose whatever was pending during the gap; nothing resumes it.
• Resuming replays nothing: there’s no history behind a paused mempool feed to catch up on, unlike a paused evm-rpc network.

.env.example

SEPOLIA_WS_URL=wss://sepolia.infura.io/ws/v3/YOUR_API_KEY
SEPOLIA_RPC_URL=https://sepolia.infura.io/v3/YOUR_API_KEY
BLOCKWATCHER_API_TOKEN=local-test-token

Three variables where the confirmed-events example needed two. The new one, SEPOLIA_WS_URL, is what newPendingTransactions is subscribed over: it must be ws:///wss://, never the http(s) scheme SEPOLIA_RPC_URL uses. The two are usually the same provider project’s websocket and HTTP endpoints; SEPOLIA_RPC_URL here does a different job than in the confirmed-events example: not scanning blocks, but hydrating each pending hash the subscription reports into a full transaction via eth_getTransactionByHash. That lookup happens for every hash the subscription hands over, ahead of anything the monitor’s selector or predicate would otherwise rule out. So spend tracks the chain’s overall mempool arrival rate, not how many of those pending calls end up matching this example’s monitor.

blockwatcher.toml

[api]
enabled = true
listen = "127.0.0.1:8080"

[[auth.tokens]]
label = "operator"
scope = "admin"
secret = "env:BLOCKWATCHER_API_TOKEN"

[metrics]
enabled = true
listen = "127.0.0.1:9090"

[storage]
module = "sqlite"
config = { path = "blockwatcher.db" }

[engine]
event_channel_capacity = 1000
sink_channel_capacity = 100
drain_deadline_ms = 5000
matcher = { module = "expr", config = {} }

Identical, key for key, to source-rpc-monitor’s instance config: the [api]/[metrics]/[storage]/[engine] sections have nothing evm-mempool-specific about them; every difference between the two examples lives in the network resource below, not in the process’s own plumbing.

setup.sh

Unlike the confirmed-events example’s setup.sh, there is no start_block to compute or stamp here: nothing pending has a chain position for a script to look up. What it does instead: confirm .env exists and both URLs are set, confirm SEPOLIA_WS_URL actually uses a ws/wss scheme and SEPOLIA_RPC_URL an http/https one (catching the two swapped, a URL still holding the YOUR_API_KEY placeholder, or either one obviously mistyped), and make one real eth_blockNumber call against the HTTP endpoint to prove it answers. It cannot exercise the WebSocket endpoint itself, since only blockwatcher’s own boot ever dials that one, so the script’s own final message says as much rather than implying a check it didn’t perform.

resources/networks/sepolia-mempool.json

{
  "id": "sepolia-mempool",
  "chain": "evm",
  "source": {
    "module": "evm-mempool",
    "config": {
      "ws_url_secret": "env:SEPOLIA_WS_URL",
      "endpoints": [
        {
          "name": "primary",
          "url_secret": "env:SEPOLIA_RPC_URL",
          "priority": "high",
          "rate_limit": { "rps": 10 }
        }
      ],
      "reconnect_ms": 1000,
      "idle_policy": {
        "ping_after_ms": 30000,
        "pong_deadline_ms": 10000
      }
    }
  }
}

source.module: "evm-mempool" is the one field that changes which raw material a selector on this network can ever see. See Selectors § The source. ws_url_secret names the subscription endpoint, resolved the same env:NAME way as every other secret reference in this config; endpoints is the same EndpointDef pool shape evm-rpc uses, here doing hydration calls rather than log/block scans. Three keys have no counterpart in the confirmed-events example’s network config at all: reconnect_ms is the delay before redialing after the WebSocket connection drops; idle_policy.ping_after_ms is how long the connection can go without a frame before blockwatcher sends its own ping to check it’s still alive; idle_policy.pong_deadline_ms is how long it then waits for the pong before declaring the connection dead and reconnecting. When any of the three losses above happens, GET /status reports source.status: degraded with a reason string that’s the full, human-readable error (e.g. no frame arrived within 10s of an idle ping; the connection is presumed half-open for a pong that never came, per crates/blockwatcher-evm/src/ws.rs), not a short code; SourceStatusView passes that string through verbatim (crates/blockwatcher-core/src/status.rs). The short codes dial_failed/stream_closed/transport_error/idle_timeout exist on a different surface entirely: they’re the reason label on the blockwatcher_evm_mempool_reconnects_total counter (and the matching field in this source’s own tracing logs), one increment per subscription lost. That’s the surface to alert on for a flapping connection, since a status snapshot only ever shows whichever state is current, and a reconnect that lands before the next scrape reads as healthy either way. Absent from this file entirely, and refused if added: start_block. There is no chain position for a pending transaction to resume from, so the config has nothing to name.

resources/specs/usdc-erc20.json

Same id, same two events as the confirmed-events example’s spec, plus three function fragments this example actually uses:

{
  "type": "function",
  "name": "transfer",
  "stateMutability": "nonpayable",
  "inputs": [
    { "name": "to", "type": "address" },
    { "name": "amount", "type": "uint256" }
  ],
  "outputs": [ { "name": "", "type": "bool" } ]
}

(approve and transferFrom follow the same shape in the actual file, with their own parameter lists.) A function fragment’s inputs are its call arguments, decoded from the transaction’s calldata rather than a log’s topics or data, which is why functions selectors have nothing to do with indexed, a concept that only means something for an event’s log encoding. stateMutability and outputs are recorded from the ABI but don’t affect decoding or matching; only inputs, alongside the function’s own name (hashed into its first-four-bytes selector), matter to what a functions selector can dispatch on. This spec is not the same file as the confirmed-events example’s: it’s a separate copy under this example’s own resources/, extended with the three function fragments above that copy doesn’t carry, but it keeps the same id and the same two event fragments unchanged. That’s the pattern worth reaching for whenever you want both a confirmed audit trail and an early pending-call signal off the same contract: one evm-rpc network and one evm-mempool network, each with its own copy of a spec that started identical and only grows the fragments the mempool side actually needs, rather than each deployment maintaining two divergent ABI descriptions of the same contract from scratch.

resources/monitors/usdc-sepolia-pending-transfers.json

{
  "id": "usdc-sepolia-pending-transfers",
  "network": "sepolia-mempool",
  "selectors": [
    {
      "addresses": ["0x1c7D4B196Cb0C7B01d743Fbc6116a902379C7238"],
      "spec": "usdc-erc20",
      "functions": ["transfer"]
    }
  ],
  "predicate": "args.amount > 0",
  "actions": ["log-sink"]
}

The one selector-body difference from the confirmed-events monitor: functions: ["transfer"] where that one had events: ["Transfer"]. This isn’t a style choice: an events selector would still compile cleanly against this network, since compilation only ever validates a selector against its spec, never against which source module the network happens to run, but it could never produce a single match here: evm-mempool never emits a log at any point in its lifetime for such a selector to catch. args.amount here is transfer’s own second parameter (named amount, not value, because that’s what this ABI fragment calls it); it is unrelated to args.value, the Transfer event’s third parameter the other example’s predicate reads. Both predicates read as “keep every nonzero transfer,” just against two different decoded shapes.

resources/sinks/log-sink.json

Identical to the confirmed-events example’s, with the same module, same single-attempt retry policy, and the same reasoning: nothing about the log sink’s own behavior changes based on which source fed it a match.

When mempool watching is worth it

This source earns its place when the thing you care about is someone tried to call this function, and minutes or even seconds of latency change the value of knowing: flagging a large pending transfer before it’s mined, watching for a specific address’s activity the moment it hits a node’s mempool, anything where “usually right, occasionally wrong, but fast” beats “always right, but a block or twelve confirmations later.” It earns its place a lot less (arguably not at all) anywhere the record has to be complete or auditable: billing off matched events, anything feeding a ledger, anything where a transaction that gets dropped or replaced after this source already delivered it would leave a consumer holding a phantom event with no retraction ever coming. The evm-rpc twin of this example is the one to reach for whenever that completeness matters more than the head start.

Variations

Widen the functions watched. The spec already declares approve and transferFrom alongside transfer; add them to the selector’s functions list to catch pending approvals and delegated transfers too:

"functions": ["transfer", "approve", "transferFrom"]

Exclude a known address instead of just thresholding the amount. args.amount > 0 only rules out the zero-amount edge case; combine it with tx.from (always present on a functions selector, mined or not) to drop calls from an address you already know about (say, a market maker’s hot wallet that transfers constantly and would otherwise dominate the feed):

args.amount > 1_000e6 && tx.from != 0xF977814e90dA44bFA03b6295A0616a897441aceC

Split the two endpoints across different provider projects. ws_url_secret and endpoints[].url_secret are independent secret references: nothing requires them to name variables from the same provider account. Pointing the subscription at one provider and the hydration pool at another spreads load and removes a single provider outage as a way to lose the feed entirely; it costs nothing but a second env:NAME variable and a second free-tier key.

Troubleshooting

A symptom-first index into failure modes that are either expected behavior worth recognizing, or genuine problems with a specific fix. Every quoted string below is copied from the code that produces it, not paraphrased, so it’s safe to grep the same text against your own logs.