blockwatcher (binary)

blockwatcher is the composition root: it wires the engine, the module catalog, storage, the REST control plane, and the metrics exporter together, and implements none of them itself. Catalog construction lives in blockwatcher-embed; this crate calls build_catalog there so registration has one home. Every other rule it appears to enforce (what a resource must look like, what makes a monitor compilable) actually lives in blockwatcher-core or in a module crate; this crate only decides when each of those pieces gets constructed, in what order, and what the process does when asked to stop (lib.rs). It ships as a library target plus a thin binary, so the crate’s own integration tests can drive a boot in-process rather than only through a spawned process (lib.rs), and it is Unix-only in the sense that its one stop path is SIGTERM/SIGINT: a build for a target without those signals would have no way to ask it to stop, so it does not compile instead of compiling into a monitor nothing can shut down (lib.rs).

Its production dependencies are exactly the ALLOW_BLOCKWATCHER entry scripts/check-dep-graph.sh lists (crates/blockwatcher/Cargo.toml):

• blockwatcher-types, blockwatcher-ports, blockwatcher-core: the vocabulary and the engine this crate boots
• blockwatcher-embed: build_catalog and the same engine re-exports a host uses; this crate’s evm / expr / sinks features also enable the matching blockwatcher-embed features
• blockwatcher-api: the REST control plane, served once the engine exists
• blockwatcher-metrics: the Prometheus recorder and scrape endpoint, installed and served independently of the API
• blockwatcher-storage: the only storage modules (memory, sqlite); a plain, non-optional dependency, so both are always present regardless of feature flags. Gate modules are registered the same way sinks are, through embed’s catalog fold (blockwatcher-gates is always-on, not a binary feature). Module families (expr, evm, sinks) are not direct dependencies: this crate’s feature flags forward onto blockwatcher-embed (see Feature-flag wiring below)
• serde, serde_json: InstanceConfig and every resource shape a seed file or check deserializes
• thiserror: the derive behind BootError, ConfigError, SeedError
• toml: parsing an instance configuration file
• figment: layering the TOML file with BLOCKWATCHER_* environment overrides
• tokio (rt-multi-thread/macros/signal/net features): the async runtime main.rs builds, and the stop-signal plumbing run.rs installs
• tokio-util: CancellationToken, the listener shutdown signal Running holds
• tracing, tracing-subscriber (env-filter feature): init_tracing, which sends every diagnostic to stderr so stdout stays reserved for a running monitor’s own match output

Embed is the crate that links module families; this crate’s evm / expr / sinks features forward onto embed rather than declaring those crates. The dependency gate still exempts the binary for the families embed pulls in, because they appear in its transitive tree. Workspace map § Glue and test is where this crate’s role in the three rings is described, and Architecture decisions § Dependencies always point toward the vocabulary, never toward the engine is the rule that keeps every module crate usable from some other binary entirely: none of them may depend back on this one, or on blockwatcher-core.

Installation and building and Configuration reference already cover this crate from an operator’s side: how to build it, what each feature flag removes, what check’s exit codes mean, and every InstanceConfig key. This page does not restate any of that; it covers the boot order verified against run.rs’s actual source, the drain mechanics behind the exit codes, and which blockwatcher-embed catalog registrations a feature flag removes.

Key takeaways

• blockwatcher is the composition root: it wires the engine, the module catalog, storage, the REST control plane, and the metrics exporter together, and implements none of them itself. Catalog construction is blockwatcher-embed::build_catalog; this crate’s feature flags forward onto embed rather than declaring the module crates. Nothing in the workspace depends on it back.
• Every rule it appears to enforce actually lives in blockwatcher-core or a module crate; this crate only decides when each piece gets constructed, in what order, and what happens when asked to stop.
• It ships as a library plus a thin binary, and is Unix-only in the sense that its one stop path is SIGTERM/SIGINT.

Responsibilities

• Parses the command line by hand (cli.rs) into one of Command’s Run, Check, PruneCheckpoints, Help, or Version variants, without reading a file or constructing a module until a command actually runs (cli.rs).
• Loads instance configuration once at boot, TOML layered with BLOCKWATCHER_* environment overrides, and never re-reads it for the life of the process (config.rs).
• Boots in the order Startup sequence below verifies: resolves everything that can refuse without a side effect first, constructs storage, claims both listen ports, seeds only an empty store, and starts the engine last (run.rs::boot, run.rs).
• Registers no modules of its own: blockwatcher_embed::build_catalog folds every compiled-in family, feature-gated so what a build can select is exactly what it linked (run.rs, crates/blockwatcher-embed/src/catalog.rs).
• Validates a seed directory offline, against the exact modules this binary compiled in, without starting a pipeline or opening a listener (check.rs).
• Owns the process’s stop signal and the shutdown drain that follows it, mapping the result onto the process exit code (run.rs::run, run.rs).
• Sweeps orphaned checkpoint rows whose network resource no longer exists, offline, outside the running process (prune.rs).

Not this crate’s job: deciding whether a resource is valid, compiling a predicate, or running a pipeline (blockwatcher-core, called through Engine::start/Engine::validate and ControlHandle); implementing any module a config can select by name (each lives in its own module crate, gated behind this crate’s own feature flags); serving HTTP or Prometheus (blockwatcher-api::serve and blockwatcher_metrics::serve, both handed an already-bound listener by this crate and otherwise left alone, run.rs).

Key types and functions

Startup sequence

boot (run.rs) is the single function every Run command goes through, and its actual order, read straight from the source rather than assumed, is:

1. Resolve everything that can refuse without a side effect. If [api].enabled, parse [api].listen and resolve [auth] (config.api_listen()/config.auth()); if [metrics].enabled, parse [metrics].listen; resolve the engine config, including the compiled-in matcher fallback (run.rs).
2. Build the module catalog (blockwatcher_embed::build_catalog), folding every compiled-in family’s factories (run.rs).
3. Construct storage, via the catalog’s factory for [storage].module, inline in boot; warn if the resolved module is memory, since nothing survives that backend across a restart (run.rs).
4. Bind both listen ports via bind_listener, metrics before API, installing the Prometheus recorder via install_recorder in the same step as the metrics bind (run.rs).
5. Apply the seed, only if --seed <dir> was given and only into a store still holding zero resources of every kind, checked inline in boot (run.rs; the emptiness check and the load/validate/persist steps are apply_seed, run.rs).
6. Start the engine, Engine::start(EngineDeps { storage, catalog, config }), then build the one ControlHandle onto it (run.rs).
7. Spawn the listener tasks, serve_metrics/serve_api, now that the engine and control handle exist for them to serve (run.rs).

flowchart TD
    a["1: resolve api/metrics addr + token,<br/>resolve engine config,<br/>inline in boot()<br/>run.rs"] --> b["2: blockwatcher_embed::build_catalog()<br/>blockwatcher-embed/src/catalog.rs"]
    b --> c["3: construct storage<br/>via catalog's factory,<br/>inline in boot()<br/>run.rs"]
    c --> d["4: bind metrics listener<br/>+ install_recorder<br/>run.rs"]
    d --> e["4: bind api listener<br/>via bind_listener<br/>run.rs"]
    e --> f{"seed dir given?"}
    f -->|"yes, and store<br/>holds zero resources"| g["5: apply_seed:<br/>load, validate, persist<br/>run.rs"]
    f -->|"no, or store<br/>already non-empty"| h["6: Engine::start(EngineDeps)<br/>run.rs"]
    g --> h
    h --> i["ControlHandle::new(engine)<br/>run.rs"]
    i --> j["7: spawn listener tasks<br/>serve_metrics / serve_api<br/>run.rs"]

Two orderings here are easy to get backwards from memory, and both are deliberate, per boot’s own doc comment (run.rs):

• The catalog is built, and storage constructed, before either listener is bound, but both listeners are bound before the seed is applied. Claiming a port is one of the last things that can refuse without having written anything an operator has to undo; seeding is the first thing that writes. A typoed listen address must be discovered before a store holding any resource stops being eligible for seeding, not after.
• The engine starts last. Starting it runs real sources and delivers real events to real sinks. Everything above it, including a metrics recorder that refuses to install on a cold process, must have already refused if it was going to, so a boot that reports failure has sent nothing anywhere.

The check command

blockwatcher check <dir> (check.rs) is a wholly separate path from boot: it never reads an instance config file, never opens a real listener, and never touches real storage. check (check.rs) calls validate (check.rs), which:

1. Loads the seed directory (seed::load), the same all-or-nothing read apply_seed uses.
2. Resolves the engine config from InstanceConfig::default() rather than any real file, so the matcher this check validates against is the one this build compiled in, and the refusal a build with none produces is identical to what a real boot would say.
3. Runs seed::validate, the exact function apply_seed calls: it builds a real ModuleCatalog, constructs a scratch in-memory store, persists the bundle into it, and runs the whole bundle through Engine::validate (the same construction and compilation path Engine::start runs, minus actually spawning a pipeline).

A pass prints Summary’s Display to stdout (check.rs:55-63) and returns 0:

ok: 3 networks, 5 specs, 2 sinks, 8 monitors

A failure prints the refusal to stderr and returns 1 (check.rs). Because validation constructs every module, a sink config that resolves a secret through env:NAME needs that variable present in check’s own environment, exactly as a real boot would need it. check never returns 2 or 64: those two codes belong respectively to a drain that aborted at the deadline and to a command line this binary could not parse, and check goes through neither the drain path nor the general argument parser once its own subcommand name has matched inside parse (cli.rs).

Feature-flag wiring

Cargo.toml’s optional features each forward onto the matching blockwatcher-embed feature, all on by default. This crate does not declare the module crates itself; embed is the one composition façade that links them:

[features]
default = ["evm", "expr", "sinks"]
evm = ["blockwatcher-embed/evm"]
expr = ["blockwatcher-embed/expr"]
sinks = ["blockwatcher-embed/sinks"]

blockwatcher_embed::build_catalog (crates/blockwatcher-embed/src/catalog.rs) is where the effect of each flag is entirely mechanical: a #[cfg(feature = "...")] block around one fold, feature by feature, over the source, decoder, matcher, and sink families. This crate’s flags also enable the matching blockwatcher-embed features so the catalog this binary links is the catalog embed would build with the same flags:

Storage’s own registration, blockwatcher_storage::registry::storages::get_all() (memory, sqlite), is not behind any feature at all: blockwatcher-storage is a plain, non-optional dependency of embed (catalog.rs in blockwatcher-embed), so no flag ever removes it.

A build missing a feature does not merely fail to offer the module by name; the factory function itself is absent from the binary, since the whole fold is compiled out. A config naming a module a build didn’t link refuses at catalog lookup time with the alternatives that build actually carries, never with the module it lacks silently treated as unavailable without saying so (blockwatcher-embed’s own tests, an_unknown_storage_module_is_refused_naming_what_this_build_carries, assert exactly this against the real catalog).

Shutdown and drain

run (run.rs) installs StopSignals (Unix SIGTERM/SIGINT, or the three Windows equivalents) before calling boot, specifically so a signal arriving during a slow first boot (seeding, module construction) is answered here rather than escalated to a kill by whatever supervisor sent it (run.rs). A tokio::select! races the signal against boot itself, biased so a boot that has already finished always wins over a signal that happened to arrive in the same instant: a completed boot always gets a real drain rather than being torn down mid-construction (run.rs). A signal that wins the race (boot still in flight) exits 0 immediately, with a warning that a store may hold part of an interrupted seed and that a store holding any resource is never seeded again (run.rs).

Once boot succeeds, run waits for the next stop signal, then calls Running::shutdown (run.rs), which:

1. Calls Engine::shutdown(). This is blockwatcher-core’s own drain: every pipeline’s drain_pipeline (crates/blockwatcher-core/src/engine/drain.rs) awaits its tasks up to EngineConfig::drain_deadline_ms (default 10_000), escalating to hard_cancel plus a 500ms grace on timeout, then aborting any straggler; see blockwatcher-core § Boot, restart, and shutdown for that ladder in full. The result is a ShutdownReport { drained, aborted }.
2. Cancels listener_shutdown (a CancellationToken), which both serve_metrics and serve_api are watching, telling each axum::serve call to stop accepting new connections and let whatever it is already answering finish (run.rs).
3. Waits for each listener task, for at most listener_grace. This is Duration::from_millis(engine_config.drain_deadline_ms), computed once at boot (run.rs) and reused rather than given its own separate number: a request left half-answered must not be able to extend the shutdown any further than a wedged sink already could. A listener that does not finish inside its grace is abandoned (the handle is dropped, not aborted, so an in-flight response gets whatever time the process exit leaves it) with a tracing::warn! naming which listener (run.rs).

exit_code (run.rs) maps the ShutdownReport alone onto the process exit code: 0 if aborted is empty, 2 otherwise. Config, boot, and seed failures short-circuit earlier and exit 1; a command line parse could not make sense of exits 64 (cli.rs). The full table:

prune-checkpoints (prune.rs) is offline and outside this lifecycle entirely: it loads instance config, opens storage the same way boot does, sweeps or lists (--dry-run) orphaned checkpoint rows, and returns only 0 or 1 (prune.rs); it never starts an engine, never binds a listener, and never goes through the drain above.

Neighbours

blockwatcher depends on, in production (the same list named in full above):

• blockwatcher-types
• blockwatcher-ports
• blockwatcher-core
• blockwatcher-embed
• blockwatcher-api
• blockwatcher-metrics
• blockwatcher-storage
• serde, serde_json
• thiserror
• toml
• figment
• tokio (rt-multi-thread/macros/signal/net features)
• tokio-util
• tracing, tracing-subscriber (env-filter feature)

and, in [dev-dependencies] only:

• blockwatcher-testkit: shared test scaffolding
• blockwatcher-ports (fakes feature): the in-memory fakes a boot test constructs against instead of a real module
• blockwatcher-evm-testkit: a real Pool<EvmEndpoint> for tests that need the evm feature’s real construction path rather than a fake
• tempfile: scratch directories for seed and config tests
• tokio (macros/rt-multi-thread/time/test-util features): paused-time and multi-threaded async tests
• figment (test feature): the env/fs jail config.rs’s tests run inside

Nothing in the workspace depends on blockwatcher: it is the composition root, the end of every dependency chain rather than the start of one (per the dependency table, where its row is the longest but has no crate pointing back at it).

Reading the source

1. lib.rs: the module doc comment (why the binary is Unix-only, why a library target sits beside it) and init_tracing.
2. cli.rs: Command, parse, and USAGE: the whole surface a user types against, before any file is read or any module constructed.
3. config.rs: InstanceConfig and its sections, load, and engine_config’s matcher-fallback logic (including journal_depth).
4. seed.rs: SeedBundle, load, validate, persist_or_clear: read this before run.rs::apply_seed, which is a thin caller over exactly these functions. Catalog construction is blockwatcher_embed::build_catalog.
5. run.rs: boot, Running, and run, in that order; read boot’s own doc comment (run.rs) beside Startup sequence above.
6. check.rs: check and validate, both short, and both callers of seed.rs functions already covered above.
7. prune.rs: prune_checkpoints and sweep, the one command that never touches the engine at all.
8. main.rs: the actual entry point, and how little of the crate’s real decision-making happens here.