Delivery guarantees

The pipeline named the Progress tracker as the thing that turns “every dispatched match done” into “safe to persist this cursor.” This page opens that mechanism up: what “done” means precisely, what happens when it never becomes true, and what a crash actually costs a consumer versus what it never costs them. Delivery scenarios is this page’s companion for a different question: given a monitor, a sink, and whatever policies are configured on it, what actually happens in the situation in front of you right now, walked as flow diagrams rather than field-by-field prose.

flowchart LR
    rpc["RPC endpoints<br/>(external chains)"]
    sources["Sources<br/>evm-rpc · evm-mempool"]
    sinks["Sinks<br/>webhook · script · log"]
    storage["Storage<br/>checkpoints · dead letters · resources"]
    api["REST API"]
    metrics["Metrics"]
    engine["engine<br/>bounded channels · checkpoints"]

    subgraph pipeline["Engine pipeline"]
        direction LR
        decoder["Decoder"]
        matcher["Matcher<br/>predicates"]
        gate["Gate<br/>threshold · max_once"]
        decoder --> matcher
        matcher --> gate
    end

    rpc --> sources
    sources -->|"decode and match"| decoder
    gate --> sinks
    api -->|"manages resources"| storage
    storage <--> engine
    engine -->|"drives"| pipeline
    engine -.->|"reports"| metrics

    classDef module fill:none,stroke:#a9a3e3
    classDef core fill:none,stroke:#8a8d86,stroke-dasharray: 5 5

    class rpc,sources,sinks,decoder,matcher,gate module
    class engine,api,storage,metrics core
classDef dim fill:none,stroke:#999999,color:#999999,opacity:0.35
classDef focus fill:#ffd43b,stroke:#d9480f,stroke-width:3px,color:#1a1a1a
class rpc,sources,api,metrics,engine,decoder,matcher,gate dim
class sinks,storage focus
click sources "selectors.html"
click decoder "chain-agnosticism.html"
click matcher "predicates.html"
click gate "gates.html"
click sinks "delivery.html"
click storage "resources.html"
click api "../reference/http-api.html"
click metrics "../reference/observability.html"
click engine "pipeline.html"

Key takeaways

• A checkpoint only ever advances past the longest contiguous run of fully-done events from the front; one stuck event holds back everything behind it.
• Overload is backpressure, not data loss: bounded channels stall the pipeline rather than dropping a message.
• A crash can only repeat work, never skip it, because a cursor only advances once every match tied to it is finished.
• Deterministic match ids let a consumer treat a redelivered duplicate as the same occurrence rather than a new one.
• A delivery that exhausts its retry budget becomes a durably recorded dead letter, and the guard only completes once that write lands.
• Sinks consume a closed SinkEvent set: Match to apply, Retracted to undo by match_id, or Digest to apply several matches bundled by aggregation at once. All three are at-least-once; retracts and digests share the same retry and dead-letter path as a plain match.
• A deep invalidate is a typed source outcome, not a failure: the engine drains, retracts journaled ids, rewinds, then restarts. Retracts for a sink finish before any replacement Match from that network.
• A monitor gate decides whether a predicate-true hit becomes a match at all. Quiet gate hits are not deliveries, not dead letters, and do not hold the checkpoint. Sink throttle / aggregate still apply only after a match exists.

The contract, precisely

A network’s checkpoint is a cursor plus whatever the source needs to verify a resume. Progress (crates/blockwatcher-core/src/progress.rs) is the structure that decides when a cursor is safe to hand the checkpoint writer: it tracks every registered event as a Slot in arrival order, each carrying a remaining count of outstanding matches, and only ever publishes the cursor of the longest contiguous run of fully-done slots starting from the front, in advance_locked (progress.rs). “Fully done” means every match dispatched from that event has either reached its sink or been recorded as a dead letter. There is no third outcome. One still-open slot anywhere in that prefix holds every slot behind it back, even ones that finished long ago (CompletionGuard::complete, progress.rs, exercised directly in middle_event_finishing_last_gates_the_prefix, progress.rs).

Three consequences follow directly from that rule:

• Checkpoints only ever move forward past finished work. A CompletionGuard dropped without calling complete() (a worker task dying mid-delivery, say) permanently stalls the prefix at that slot rather than letting it expire or time out (progress.rs; see a_dropped_guard_stalls_the_checkpoint, progress.rs). A stall is visible in the status API as in_flight_events; nothing here manufactures a false “done.”
• Overload is backpressure, not data loss. The channels between source, processor, and each sink worker are bounded and never drop a message on the full side: a slow sink fills its own channel, which stalls the processor’s dispatch loop, which stalls the source’s own send. Nothing downstream of a full queue is asked to discard anything; the entire pipeline just waits.
• A crash can only repeat work, never skip it. Resuming means reading the last persisted checkpoint and continuing from there. Anything that cursor already covers is gone for good from the source’s perspective, but by the rule above that only happens once every match tied to it is finished, so the only way a crash surfaces to a consumer is by re-delivering an event whose checkpoint write hadn’t landed yet, never by silently dropping one.

Deterministic match IDs are what make that repetition harmless. MatchId::derive hashes the network, monitor, cursor, an index within the event, and a structural walk of the decoded event’s own field tree (crates/blockwatcher-types/src/id.rs): nothing about it depends on wall-clock time, retry count, or which run produced it. Re-processing the same on-chain event after a restart decodes the same bytes at the same slot and derives the exact same id (match_id_re_derivation_from_the_same_content_is_stable, id.rs), which is what lets a consumer treat two deliveries of the same id as one occurrence rather than two.

Happy path

One raw event, one match, one sink, nothing goes wrong:

sequenceDiagram
    participant Source as Source task
    participant Processor as Processor task
    participant Progress as Progress tracker
    participant Sink as Sink worker task
    participant CPW as Checkpoint writer task
    participant Store as checkpoint storage

    Source->>Processor: event at cursor C
    Processor->>Processor: decode + match (1 match)
    Processor->>Progress: begin(C, outstanding: 1)
    Processor->>Sink: dispatch match M
    Sink->>Sink: deliver Match(M) (success)
    Sink->>Progress: guard.complete()
    Progress->>Progress: prefix fully done, watermark = C
    Progress->>CPW: publish checkpoint(C)
    CPW->>Store: persist checkpoint(C)

An event with zero matches is registered too, with zero outstanding: it completes the moment begin runs, which is why an event that matched nothing still lets the cursor advance immediately rather than wait on deliveries that were never dispatched. An event whose monitors all retained or discarded at the gate is registered with zero outstanding the same way.

Crash after partial delivery, then resume

Two matches from one event, going to two different sinks. One delivery lands before the crash; the other is still retrying when the process dies. Because the checkpoint requires both guards, cursor C never got persisted. The last durable checkpoint is still whatever preceded it:

sequenceDiagram
    participant Source as Source task
    participant Processor as Processor task
    participant Progress as Progress tracker
    participant SinkA as Sink worker task A
    participant SinkB as Sink worker task B
    participant Store as checkpoint storage

    Note over Store: persisted checkpoint = C-1
    Source->>Processor: event at cursor C
    Processor->>Progress: begin(C, outstanding: 2)
    Processor->>SinkA: dispatch match M1
    Processor->>SinkB: dispatch match M2
    SinkA->>SinkA: deliver M1 (success)
    SinkA->>Progress: guard.complete() (M1)
    Note over Progress: M2 still outstanding, watermark holds at C-1
    Note over SinkB: process crashes mid-retry of M2
    Note over Store: restart resumes from C-1 (never advanced)

    Source->>Processor: re-emit event at cursor C
    Processor->>Progress: begin(C, outstanding: 2)
    Processor->>SinkA: dispatch match M1 (same MatchId as before)
    Processor->>SinkB: dispatch match M2 (same MatchId as before)
    SinkA->>SinkA: deliver M1 again (duplicate)
    Note over SinkA: consumer recognizes the duplicate by M1's id
    SinkA->>Progress: guard.complete() (M1)
    SinkB->>SinkB: deliver M2 (success this time)
    SinkB->>Progress: guard.complete() (M2)
    Progress->>Progress: prefix fully done, watermark = C
    Progress->>Store: checkpoint(C) persisted

Nothing about which sink had already succeeded survives the crash: the in-memory Progress state is gone, and the only source of truth on restart is the last persisted checkpoint. Re-reading from C-1 means the whole event at C is decoded and matched again, M1 included, which is exactly the duplicate a consumer’s dedup-by-id exists for.

Dead letters

When deliver_with_retry (crates/blockwatcher-core/src/pipeline/delivery.rs) exhausts a sink’s retry budget (every attempt spent on a transient or rate-limited failure, or a permanent/retry-narrower error that skips retry entirely), the sink worker does not drop the event. It builds a DeadLetter carrying the event’s match id, which monitor and sink produced it, the cursor it traces back to, the attempt count, a reason string, and the original SinkEvent payload, and writes it through the network’s storage backend, in dead_letter (crates/blockwatcher-core/src/pipeline/sink_worker.rs). Only after that write durably lands, and, for a Match, only after the delivery journal row is written, does the worker complete its guard, in dead_letter’s success branch (sink_worker.rs); if the write itself fails, the worker retries the write forever on its own doubling backoff rather than completing early, in dead_letter’s retry loop (sink_worker.rs): the same “never advance past an unrecorded loss” rule as everywhere else in this contract, just applied one layer further out. A dead-lettered Match stays in the journal so a later invalidate can still retract it; a dead-lettered Retracted does not.

An operator interacts with a network’s dead-letter queue through these routes, in router (crates/blockwatcher-api/src/routes/networks_ops.rs):

• GET /networks/{id}/dead-letters pages the queue in insertion order: a pure storage read, in list_dead_letters (crates/blockwatcher-core/src/control/dead_letters.rs).
• POST /networks/{id}/dead-letters/{match_id}/replay re-runs the same deliver_with_retry path against the sink the letter names, rebuilt fresh from its stored config. Success deletes the letter (204); exhaustion bumps its attempts and reason in place and leaves it queued, in replay_dead_letter (dead_letters.rs). A letter recorded before payload storage was added has no payload to replay and is refused outright rather than silently no-op’d (dead_letters.rs, message NO_REPLAY_PAYLOAD). A letter whose payload is SinkEvent::Retracted is likewise refused (NO_REPLAY_RETRACTED): only match events can be replayed.
• DELETE /networks/{id}/dead-letters/{match_id} and DELETE /networks/{id}/dead-letters (the latter with optional sink/monitor filters, returning the count discarded) discard letters outright, without attempting delivery, in discard_dead_letter/discard_dead_letters (dead_letters.rs). Replaying is the only route that resends a letter; discarding is permanent.

The sqlite storage backend is what makes a dead-letter queue survive a restart at all; the memory backend does not, matching its own no-persistence contract.

Delivery policies

Everything on this page is configured per sink, on the SinkDef resource (a JSON document written through PUT /sinks/{id} or a seed directory, hot-reloaded like every resource): nothing here is hardcoded or instance-config. One complete sink showing every policy field beside the module’s own config:

{
  "id": "alerts",
  "module": "webhook",
  "config": {
    "url_secret": "env:ALERTS_WEBHOOK_URL",
    "headers": { "x-source": "blockwatcher" },
    "timeout_ms": 10000,
    "body_template": "{% if type == \"digest\" %}{{ matches | length }} matches{% else %}{{ type }}: {{ monitor }}{% endif %}"
  },
  "retry": { "max_attempts": 8, "initial_backoff_ms": 200, "max_backoff_ms": 30000 },
  "throttle": { "max_deliveries": 60, "window_ms": 60000 },
  "aggregate": { "window_ms": 30000, "max_batch": 100 }
}

config belongs to the named module (here webhook; its fields, including the optional body_template renderer, are that module’s own documentation). retry, throttle, and aggregate are the engine’s, identical for every module, each optional: omit retry for the engine-wide default, omit throttle for no throttling, omit aggregate for unbatched delivery. The values shown are each field’s defaults.

A retry whose initial_backoff_ms sits above its own max_backoff_ms is refused by the same validate call throttle and aggregate go through, at the same two points described below for aggregation: put_sink refuses the write, validate_and_build refuses to boot a stored row that fails the same check. DeliveryRetry::backoff_bounds (crates/blockwatcher-types/src/resource.rs) is what every retry loop reads its starting backoff and ceiling from, deliver_with_retry and the sink worker’s journal, dead-letter, and forget retries alike, so the floor against an out-of-range pair lives in one place rather than at each of those call sites.

A sink’s throttle field (SinkDef.throttle: Option<Throttle>, crates/blockwatcher-types/src/resource.rs) caps how often that sink admits successful deliveries, independent of the retry and dead-letter policy above. The fields that define it:

• max_deliveries: the number of successful Match deliveries the window admits before it starts refusing more, default 60.
• window_ms: the window’s length in milliseconds, default 60_000.

A zero in either field is refused at write time, the same “validate before constructing anything” discipline every resource write goes through: a sink’s throttle is either absent or fully valid, never partially so.

At runtime, SinkWorker (crates/blockwatcher-core/src/pipeline/sink_worker.rs) holds a fixed admission window per sink, anchored at the first admitted delivery and reset once window_ms has elapsed. The window belongs to one network’s pipeline, not to the sink id globally: a sink named by monitors on several networks gets one independent window per network, each with its own budget, rather than one shared budget across all of them. A Match that arrives inside the current window’s budget is delivered normally; a Match that would exceed it never reaches the sink at all. It is dead-lettered instead, through the same dead_letter path an exhausted retry budget uses, with a reason naming the policy that suppressed it:

permanent: throttled: policy allows 60 deliveries per 60000ms; replay this dead letter to deliver it

The permanent: class prefix is the same one every dead-letter reason carries; nothing about the throttle path is a distinct class of dead letter. A throttled match is exactly as listable and replayable as any other dead letter, through the same GET /networks/{id}/dead-letters and POST /networks/{id}/dead-letters/{match_id}/replay routes described above: replaying one delivers it directly, without re-checking the window it was suppressed under.

Only a delivered Match spends budget. A failed attempt already pays for itself through the ordinary retry-then-dead-letter path and must not also consume quota meant for the next real delivery. Retracted events skip the throttle check entirely, at the same call site that would otherwise apply it: a correctness signal outranks a fatigue policy, so an undo is never the thing a noisy sink drops on the floor.

The window is runtime-only state, held in memory beside the sink worker rather than persisted: a pipeline restart opens a fresh window with zero deliveries counted. An operator relying on the throttle to bound a sustained rate should not expect the count to survive a restart.

Aggregation

A sink’s aggregate field (SinkDef.aggregate: Option<Aggregate>, crates/blockwatcher-types/src/resource.rs) batches several matches into one digest delivery instead of sending each on its own. The fields that define it:

• window_ms: how long the batch stays open once the first match joins it, default 30_000, capped at 86_400_000 (one day).
• max_batch: the number of matches that closes the batch immediately, default 100, capped at 10_000.

A zero in either field is refused the same way a zero throttle field is, and so is a value above either cap: the caps bound what one open window can cost, since every match in it is held in memory and the checkpoint stays behind all of them until the window closes. max_batch prices that worst case in matches, window_ms prices it in time. Each route runs a SinkDef’s policy through the same validate call, but they enforce it at different points. put_sink (crates/blockwatcher-core/src/control/writes.rs) refuses to persist a SinkDef whose policy fails validation. validate_and_build (crates/blockwatcher-core/src/engine/boot.rs) runs the identical check against rows already in storage, a restored backup or a hand-edited row among them, and refuses to start the engine rather than refusing a write. A sink’s aggregation policy is either absent or fully valid, never partially so.

At runtime, AggregateBuffer (crates/blockwatcher-core/src/pipeline/aggregate.rs) holds one batch per sink worker, and only a Match ever joins it; the buffer flushes, delivering whatever it holds as a single event, on any of:

• the batch reaches max_batch;
• window_ms elapses since the first match joined the batch, not the most recent one, so a steady trickle still reaches the sink once per window instead of waiting forever for a batch that never fills;
• the sink worker’s channel closes, which is what a cooperative drain looks like from this worker’s side: a partial batch still in progress is flushed rather than dropped;
• a Retracted event arrives. The buffer flushes first and the retract is delivered after, so a retract can never overtake the match it retracts: per-sink delivery order is the only evidence a sink has of which of the two came last. Retracted events are never buffered themselves and never count toward max_batch, the same correctness-outranks-fatigue rule that exempts them from the throttle above.

A flush of exactly one match delivers a plain SinkEvent::Match: a quiet sink whose matches never overlap inside one window sees the identical wire format it would see with no aggregate field set at all. A flush of two or more delivers a single SinkEvent::Digest { matches }, rendered by canonical_body as:

{
  "type": "digest",
  "matches": [
    { "id": "…", "monitor": "usdc-sepolia-transfers", "network": "sepolia", "event": { } },
    { "id": "…", "monitor": "usdc-sepolia-transfers", "network": "sepolia", "event": { } }
  ]
}

Every buffered match holds its CompletionGuard uncompleted for as long as it sits in the batch, exactly as an in-flight delivery does. A buffered window stalls the checkpoint by design: the batch’s oldest match cannot pass the checkpoint until the whole batch it ends up part of is delivered or dead-lettered, even though nothing has been sent to the sink yet. A hard-cancel or crash drops those guards uncompleted rather than flushing them, the same safe-by-default direction a mid-delivery crash already takes elsewhere in this contract: on replay, the source re-emits the events those matches came from, they are re-matched, and re-buffered, and eventually re-delivered, deduplicated by MatchId exactly as any other redelivered match is. There is no persisted window state to resume from; the window itself is runtime-only, the same stance the throttle above takes.

On a successful digest delivery, every contained match is journaled at its own cursor, in arrival order, before any guard completes, the same journal-then-complete ordering a lone match’s delivery follows. On retry exhaustion, the digest does not dead-letter as one row: each contained match is dead-lettered individually, carrying the digest’s own failure reason, one match at a time and in arrival order. Per match, the record order is the same as a lone dead-lettered match’s: the dead-letter row is written first, then that match is journaled, then the next match in the digest follows the same two steps. This is the replay caveat to carry forward: a dead-lettered digest member replays as a single match, never as the digest it originally arrived in, because POST /networks/{id}/dead-letters/{match_id}/replay only ever knows how to resend one match at a time.

The delivery policies compose rather than interact specially: a digest counts as exactly one delivery against the sink’s throttle window, the same one unit a lone match would have spent, so aggregating a burst into fewer deliveries is one way an operator keeps a bursty sink under a tight throttle budget. A digest the throttle refuses is suppressed exactly like a suppressed lone match would be, except it dead-letters every contained match individually rather than one row, matching how a delivered digest is journaled.

Monitor gates versus sink policies

throttle and aggregate are fields on SinkDef. A gate is a field on Monitor. They are not interchangeable:

• threshold gate: N predicate-true hits whose block.timestamp values span ≤ window_ms mint one digest of those hits, then reset the session. Held hits persist in gate_hits and complete with outstanding 0, so a 60-minute threshold does not stall the checkpoint for 60 minutes.
• aggregate sink: already-minted Matches sit in memory and the checkpoint stays behind all of them until the window flushes. That is why a long aggregate window is operationally expensive.
• max_once gate: at most one alert per event-time window; later hits are discarded with no dead letter.
• throttle sink: later Matches are dead-lettered and remain replayable. Only successful deliveries spend budget. Restarts reset the in-memory window.

Full gate contract: Gates.

The evm-mempool exception

Everything above assumes a source whose cursor is a real position in a feed it can resume from. evm-mempool is the one shipped source where that assumption fails: its cursor is a per-run arrival counter, not a chain position, so its checkpoint is only a dedupe watermark for the current process’s lifetime: a restart cannot replay whatever was pending in the node’s mempool while the process was down, and that work is simply gone. The same instability reaches into identity: because the counter restarts from zero-relative-to-the-fresh-checkpoint on every process start, the identical pending call seen again after a gap is minted a different arrival number and therefore a different MatchId. Consumers of this source deduplicate on the transaction’s own hash, not on the match id. Selectors § The position problem covers the mechanics in full.

What a sink receives: SinkEvent

Sink::deliver takes a SinkEvent, not a bare match (crates/blockwatcher-ports/src/sink.rs):

#![allow(unused)]
fn main() {
pub enum SinkEvent {
    Match(Match),
    Retracted { match_id: MatchId },
    Digest { matches: Vec<Match> },
}
}

(crates/blockwatcher-types/src/event.rs)

That set is closed: core reasons about every variant exhaustively, the same way it reasons about the canonical value model. Digest is what aggregation delivers in place of several separate Match events; it only ever reaches a sink whose SinkDef sets aggregate. A sink module that only handles Match will silently keep work a later Retracted asked it to undo, or drop every match a later Digest bundled together.

Every variant travels the same deliver_with_retry path (crates/blockwatcher-core/src/pipeline/delivery.rs): the same retry budget, the same backoff, the same dead-letter write when the budget runs out, and the same CompletionGuard (one per contained match, for a Digest). A failed undo can stall the checkpoint or land in the dead-letter queue; it is never dropped on the floor. Consumers must treat a redelivered Retracted as idempotent on match_id, just as they already treat a redelivered Match as idempotent on Match.id, and a redelivered Digest’s contents as idempotent per contained Match.id for the same reason.

Built-in sinks (webhook, script, log) render the event as tagged JSON through one helper, canonical_body (crates/blockwatcher-sinks/src/lib.rs):

{ "type": "match", "id": "…", "monitor": "…", "network": "…", "event": { } }

{ "type": "retracted", "match_id": "…" }

{
  "type": "digest",
  "matches": [
    { "id": "…", "monitor": "…", "network": "…", "event": { } },
    { "id": "…", "monitor": "…", "network": "…", "event": { } }
  ]
}

That tagging is a wire break against a bare Match object. A DeadLetter’s payload is Option<SinkEvent>: a row written before this shape existed, or written as a legacy Match object, still loads as SinkEvent::Match. Replay through the API only accepts a Match payload; a retraction payload is refused rather than re-sent (NO_REPLAY_RETRACTED, crates/blockwatcher-core/src/control/dead_letters.rs).

Shallow vs deep invalidation

A source whose cursor is a real chain position can see its feed fork. Whether sinks hear about that fork depends on which side of the confirmation barrier moved:

──── already emitted / checkpointed ────|──── unconfirmed window ────| head
                                        ^ confirmations barrier

• Shallow. Only the unconfirmed side moves. Those cursors were not durably offered to sinks (or a mid-scan race is retried in place, inside the source). No Retracted leaves the module.
• Deep. The fork cuts into work already emitted. Restarting the source alone would offer the new fork as new Matches (often with new MatchIds) and leave the consumer holding the orphan with no undo signal.

Core never names a chain, a reorg, or a confirmation depth. A source that has positively detected a deep invalidate returns SourceOutcome::Invalidated { from } from Source::run (crates/blockwatcher-ports/src/source.rs): a typed non-success, not a SourceError. The from cursor is the proven point; everything with cursor > from previously implied by this source is on a dead fork. A source whose entire tracked history is refuted (no tracked ancestor matches the live chain at all) proves divergence only at or below the oldest height it can still vouch for, bounded by the tracker’s own depth rather than any named fork block; when that depth reaches back to block 0, the bound is genesis, but that is the tracker’s own limit speaking, not a claim the fork itself sits there. The EVM RPC source is what translates a break against already-emitted work into Invalidated at the module boundary: a walk that finds a still-matching ancestor invalidates from that proven fork (beyond_confirmations); a walk that finds none invalidates from just below the oldest tracked height (beyond_window); journaled deliveries above it are retracted the same as any other invalidate (the journal-depth caveat below still applies), and the source restarts from the rewound checkpoint. Both are the same on a mid-run break as on resume. Mempool and other sources that have no finality never produce it.

The engine supervisor handles Invalidated as a control path, not a crash (crates/blockwatcher-core/src/engine/invalidate.rs):

1. Cooperative cancel and drain of that network’s pipeline (the same deadline drain any restart uses; no abort on this path unless the deadline is missed).
2. Prune gate_hits for this pipeline where cursor > from. Do not delete rows with cursor ≤ from.
3. list_deliveries_after(network, from) against the bounded delivery journal.
4. One-shot retract pass: a SinkEvent::Retracted { match_id } for each retained id, through the same deliver_with_retry live traffic uses.
5. Wait until every retract is delivered or dead-lettered. An unrecovered retract leaves the checkpoint unrewound and does not restart.
6. Rewind the checkpoint to from, unless no write could move the cursor strictly backward: from is not behind the stored checkpoint, no checkpoint is stored at all (writing one would fabricate delivery progress), or the post-drain checkpoint read failed. A rewind must move the cursor backward or not at all, so in each of those cases the write is refused instead and counted as blockwatcher_rewinds_refused_total.
7. Re-read the persisted checkpoint, then restart the source so replacement Matches can flow.

In-flight work on the old pipeline is dropped with it: it was never checkpointed, so it is not a silent skip. For a given sink, every Retracted from an invalidate finishes before any post-restart Match from that network is offered. Retracts go to the sinks that network’s pipeline was spawned with. After the network resource is deleted, those ids live on a runtime-only tombstone so a late Invalidated still notifies only those sinks, not every other network’s consumers. Sibling networks are not touched. One network’s invalidate does not abort the others.

sequenceDiagram
    participant Src as Source
    participant Sup as Supervisor
    participant Drain as Pipeline drain
    participant Journal as Delivery journal
    participant Sink as Sink worker
    participant Store as Checkpoint

    Src->>Sup: Invalidated { from }
    Sup->>Drain: cooperative cancel + drain
    Drain-->>Sup: outgoing work finished
    Sup->>Journal: list_deliveries_after(from)
    Journal-->>Sup: retained (cursor, match_id) rows
    loop each retained id, per sink
        Sup->>Sink: Retracted { match_id }
        Sink->>Sink: deliver_with_retry
        Note over Sink: success forgets the row,<br/>dead-letter keeps it
    end
    Note over Sink: all retracts delivered or dead-lettered
    Sup->>Store: rewind checkpoint to from
    Sup->>Src: restart (replacement Matches)

The delivery journal

A Match that is delivered or dead-lettered is recorded in a per-network journal before its completion guard completes (journal_delivery, crates/blockwatcher-core/src/pipeline/sink_worker.rs). That ordering is load-bearing: completing first would let the checkpoint pass a match the journal never durably saw, so a later invalidate could not retract it. Retracted is not journaled as a positive fact; a successful retract forgets the row. A retract that exhausts to a dead letter keeps the journal row so a later invalidate can re-offer it.

The journal is bounded by instance config journal_depth (default 1024, measured in cursor primary units: block number on EVM). On each record_delivery, rows whose cursor.primary is strictly behind recorded.primary.saturating_sub(journal_depth) are dropped as part of the same write. There is no operator prune API and no “disable journal” switch in this version; the depth is meant to sit well above typical confirmation windows (12–64).

When an invalidate’s from plus journal_depth does not cover the checkpoint high-water mark, rewind and restart still happen, but only ids still retained are retracted. The gap is counted (blockwatcher_journal_gap_total) and logged at error, naming the network, the invalidate cursor, the checkpoint, and the configured depth, never silent. See Observability and Troubleshooting. Deliveries at or below the invalidate’s own from bound are never retracted even when the true fork lies deeper than the source could prove: block from.primary itself is re-emitted in full on restart, so any already-delivered match inside it arrives again as a byte-identical duplicate under the at-least-once contract, not a gap.

A gate’s Emit carries the same durability shape one layer earlier: it rides an engine-owned outbox row per sink, written in the same storage transaction that consumes the hits its journal is built from, and held until that sink’s delivery outcome is itself durable. A crash between those two points loses nothing: the row survives to be drained and redelivered, at the cost of a possible duplicate, never a gap, the same trade this page’s dead-letter and journal contracts already make. Downstream consumers dedupe on Match.id, deterministic for the same reason a redelivered plain match already is. See Gates for the emission-to-outcome path in full.