Delivery scenarios

Delivery guarantees is the field-by-field reference: what each policy’s config keys do, what a Progress slot tracks, exactly which route replays a dead letter. This page is its companion for a different question: given a monitor, a sink, and whatever policies are configured on it, what actually happens, end to end, in the situation an operator is looking at right now? Every diagram here traces a real code path already described on that page; nothing here is a new mechanism, only a new view of the ones that exist.

Picking a scenario

The full path, one picture

Every match takes the same route from decode to checkpoint, whether or not any policy below is configured. Policies insert themselves as named branch points on this one path; they never create a second path:

flowchart TD
    A[Event decoded, matched] --> B["Progress::begin: guard created,<br/>cursor stalls until it completes"]
    B --> C{Sink has<br/>throttle?}
    C -->|admitted| D{Sink has<br/>aggregate?}
    C -->|refused| DL1["Dead letter<br/>(reason: throttled)"]
    D -->|no| E[Deliver attempt]
    D -->|yes| F["Buffer holds the match<br/>(guard stays uncompleted)"]
    F -->|flush trigger fires| G{Flush held<br/>one match or many?}
    G -->|one| E
    G -->|many| H[Deliver as one Digest]
    E --> I{Attempt<br/>result?}
    H --> I
    I -->|success| J[Journal the delivery]
    I -->|exhausted retries| K["Dead letter<br/>(per contained match, if a digest)"]
    J --> L["guard.complete()"]
    K --> L
    DL1 --> L
    L --> M["Progress advances the checkpoint<br/>once every guard in the prefix is done"]

Two things this picture makes visible that are easy to miss reading the policies one at a time: every exit except the throttle refusal passes through a delivery attempt first (aggregation defers when an attempt happens; it never skips one), and every exit reaches guard.complete(). There is no branch that drops a match without either delivering it, dead-lettering it, or leaving its guard open (which stalls the checkpoint rather than silently passing it). See The contract, precisely for why that third option doesn’t exist.

Retry and backoff

One delivery attempt, from dispatch to either success or exhaustion. This is the branch under “Deliver attempt” in the picture above:

flowchart TD
    A[Attempt delivery] --> B{Result?}
    B -->|success| Z[Journal + complete guard]
    B -->|classified error| C{Class?}
    C -->|Permanent| D[Skip remaining attempts]
    C -->|Transient / RateLimited| E{Attempts<br/>remaining?}
    C -->|RetryNarrower| E
    E -->|yes| F[Wait backoff, doubling<br/>from initial_backoff_ms<br/>up to max_backoff_ms]
    F --> A
    E -->|no| D
    D --> G["Dead letter<br/>(reason names the class)"]

A Permanent error (a config the sink will never accept, a 4xx a retry can’t fix) skips straight to dead-lettering rather than spending the rest of the attempt budget on a result that cannot change; see Dead letters for the exact field a dead letter carries and the API routes (GET .../dead-letters, POST .../dead-letters/{id}/replay, and the DELETE routes that discard one or many) that read, replay, or discard it. retry’s fields (max_attempts, initial_backoff_ms, max_backoff_ms) are what this diagram’s boxes read from; omitting retry on a SinkDef uses the engine-wide default from blockwatcher.toml, not “no retry at all.”

Throttle: admission and suppression

The branch under “Sink has throttle?” above. A window opens on the first admitted delivery and holds for window_ms:

flowchart TD
    A[Match arrives at the sink worker] --> B{Window still open,<br/>and budget left?}
    B -->|no window open yet, or window_ms elapsed| C[Open a fresh window]
    C --> D[Admit, spend one unit]
    B -->|open and budget remains| D
    D --> E[Deliver normally]
    B -->|open and budget exhausted| F["Dead letter<br/>(reason: throttled, attempts: 0)"]

Only a delivered match spends budget: a failed attempt already paid for itself through the retry path above and must not also cost throttle quota meant for the next real delivery. A throttled match is not silently lost: it’s dead-lettered through the exact same path an exhausted retry uses, so it’s listed and replayable identically (replaying delivers it directly, without re-checking the window it was suppressed under). See Delivery policies for the two config fields and their defaults.

Aggregation: hold and flush

The branch under “Sink has aggregate?” above. Unlike throttle, a held match is not resolved yet: it sits with its guard open until one of four things flushes the buffer:

flowchart TD
    A[Match arrives, sink has aggregate] --> B[Add to the open buffer]
    B --> C{Which flush trigger<br/>fires first?}
    C -->|buffer reaches max_batch| D[Flush now]
    C -->|window_ms since the FIRST<br/>held match elapses| D
    C -->|sink worker's channel closes<br/>= a cooperative drain| D
    C -->|a Retracted event arrives<br/>for this sink| E[Flush the buffer,<br/>THEN deliver the Retracted]
    D --> F{One match held,<br/>or several?}
    F -->|one| G[Deliver as a plain Match]
    F -->|several| H[Deliver as one Digest]

The window is anchored to the first match that joined it, not the most recent: a steady trickle of matches still reaches the sink once per window rather than waiting forever for a batch that never fills. A Retracted is never buffered and never counts toward max_batch: it flushes whatever is already held first, so a retract can never overtake the match it retracts (per-sink delivery order is a consumer’s only evidence of which came last). See Aggregation for the digest wire shape, the exhaustion-dead-letters-each-match-individually behavior, and why a dead-lettered digest member replays as a single match, never as the digest it arrived in.

The one behavior this diagram can’t show, because it’s an absence, not a branch: a hard-cancel or crash while matches are held drops those guards uncompleted rather than flushing them. Nothing here reads that as “lost”: the checkpoint simply doesn’t advance past them, and on restart the source re-emits the events those matches came from, which re-match, re-buffer, and eventually re-deliver, deduplicated by MatchId like any other redelivery. There’s no persisted window to resume from; the window is runtime-only, same as throttle’s.

Threshold gate, not aggregate

The branch sits before Progress::begin for a match. Three hits spanning ≤ window_ms produce one digest and complete outstanding for that emit; two hits complete with outstanding 0 and persist holds. Contrast the aggregate diagram on this page: every buffered match there holds a guard open. Full contract: Gates.

Composing policies

The two flow diagrams above never run in isolation on a sink that configures both. Four compositions are worth naming because each answers a question an operator will actually ask:

• A digest costs one throttle unit, not one per contained match. Aggregating a bursty sink into fewer, larger deliveries is a direct way to keep it under a tight throttle budget: ten matches held into one digest spend the same one unit of window budget a single match would.
• A throttled digest still dead-letters every contained match individually. The throttle refusal happens at the delivery-attempt boundary, after aggregation has already flushed a Digest; suppressing it doesn’t collapse it back into one dead-letter row, since replay still needs to resend each match on its own.
• Retract bypasses both policies, at the same call site each time. A Retracted skips the throttle check entirely and flushes (rather than joins) an aggregate buffer. This is deliberate in both cases: a correctness signal (undoing a match a consumer may have already acted on) outranks a fatigue policy meant for noisy matches, so an undo is never the thing a loud sink drops on the floor.
• A reorg’s retract pass ignores both policies too, for the same reason: see the sequence diagram in the next section. The retracts an invalidate sends ride the same deliver_with_retry path live traffic does, but they are never throttled or buffered; only the ordinary retry budget applies to them.

Reorg and invalidation

Whether a fork produces a Retracted at all depends on which side of the confirmation barrier it cuts into. This is a three-way branch, not two:

flowchart TD
    A[Source detects its tracked chain<br/>no longer matches the live head] --> B{Where does the<br/>fork land?}
    B -->|only in the unconfirmed window,<br/>never durably offered| C["Shallow: retried in place<br/>inside the source. No Retracted."]
    B -->|cuts into already-emitted work,<br/>a tracked ancestor still matches| D["Deep, proven fork:<br/>Invalidated { from: fork point }"]
    B -->|cuts deeper than the tracker<br/>can prove: no tracked ancestor matches| E["Deep, unproven depth:<br/>Invalidated { from: just below<br/>oldest tracked height }"]
    D --> F[Engine invalidate sequence]
    E --> F

The tracker’s own memory bounds how deep a fork can be proven: it keeps clamp(2 × confirmations, 8, 64) entries, so a reorg deeper than 64 blocks is a real limit of the design, not an oversight; the source still invalidates, just from the deepest point it can still vouch for rather than the true fork block. Both deep branches converge on the same sequence once Invalidated { from } is raised: this is the existing diagram from Delivery guarantees, repeated here because it’s the scenario this page’s readers are most often looking for:

sequenceDiagram
    participant Src as Source
    participant Sup as Supervisor
    participant Drain as Pipeline drain
    participant Journal as Delivery journal
    participant Sink as Sink worker
    participant Store as Checkpoint

    Src->>Sup: Invalidated { from }
    Sup->>Drain: cooperative cancel + drain
    Drain-->>Sup: outgoing work finished
    Sup->>Journal: list_deliveries_after(from)
    Journal-->>Sup: retained (cursor, match_id) rows
    loop each retained id, per sink
        Sup->>Sink: Retracted { match_id }
        Sink->>Sink: deliver_with_retry
        Note over Sink: success forgets the row,<br/>dead-letter keeps it
    end
    Note over Sink: all retracts delivered or dead-lettered
    Sup->>Store: rewind checkpoint to from
    Sup->>Src: restart (replacement Matches)

One gap worth knowing about explicitly: if from plus journal_depth doesn’t reach back far enough to cover the checkpoint’s high-water mark, the rewind and restart still happen, but only the ids the journal actually retained get a Retracted. That gap is never silent: it’s counted (blockwatcher_journal_gap_total) and logged naming the network, the invalidate cursor, the checkpoint, and the configured depth. See The delivery journal.

evm-mempool never produces Invalidated at all: its cursor has no chain position to fork against. See The evm-mempool exception.

What the guarantee actually is, by scenario

“At-least-once” is the contract everywhere, but what that means in practice (duplicates, gaps, and whether a consumer hears about it) differs by scenario:

“Gaps possible” only ever appears where the journal’s bounded depth couldn’t reach far enough, or on the one source that never promised a resumable position in the first place. Every other row’s answer is “no” because of the same one rule: a checkpoint only advances once every match tied to it has either reached its sink or become a durably recorded dead letter, so anything a crash or a restart repeats was, by construction, never durably finished.

Configuration cookbook

Named starting points, not the only correct answer for every case: each maps a goal to the fields that get you there, with the reasoning so you can move off the suggested numbers with the right things in mind.

“I don’t want duplicate delivery noise from one burst of activity.” Set aggregate with a window_ms wide enough to catch a typical burst and a max_batch above your usual burst size, so bursts land as one digest instead of many separate matches — HTTP batching of matches that already fired. Widening window_ms trades delivery latency for fewer messages; a buffered match holds the checkpoint for the whole window, so an unreasonably wide window (the cap is one day, 86_400_000 ms) is a real cost, not a free knob. If the burst should not fire at all until N predicate-true hits, that is the threshold-gate recipe below, not sink aggregate.

“Notify only if Transfer occurred more than 3 times in the last 60 minutes of block time, then start counting again.” Set the monitor’s gate to threshold with count: 3 and window_ms: 3600000. Do not use sink aggregate for this: aggregate would mint a Match per Transfer and stall the checkpoint for the whole window.

“At most one webhook per hour of block time; I do not want the rest queued.” Set gate to max_once. Do not use throttle if you intend those hits to vanish; throttle dead-letters them for replay.

“I want to bound how often a flaky or rate-limited downstream gets hit, without losing any match.” Set throttle to the rate your downstream can actually sustain. Nothing is lost: everything the window refuses becomes a replayable dead letter, so you’re trading “deliver immediately” for “deliver on demand, in bulk, via POST /dead-letters/{id}/replay” rather than trading away completeness.

“I need to know the alert reached the sink even through a real chain reorg, not just that it was attempted once.” This is a confirmations and journal_depth question, not a sink-policy one. confirmations (default 12) is a latency-for-safety trade: a deeper value makes an already-emitted match’s retraction by a reorg less likely, at the cost of how quickly it reaches the sink at all. Because the tracker that proves a fork’s exact depth caps at 64 blocks regardless of confirmations, a journal_depth (default 1024, measured in cursor-primary units: block number on EVM) that comfortably exceeds your confirmations value is what makes a proven-fork retract actually reach every affected match; a shallower journal risks the “unproven depth” row above.

“This monitor watches pending transactions and I need to understand what I’m giving up.” evm-mempool has no resumable cursor and no reorg signal at all: a restart loses whatever was pending, and identity is only stable within one process’s lifetime, so consumers must deduplicate on the transaction hash rather than MatchId. If your use case needs delivery guarantees, watch the confirmed chain with evm-rpc instead; evm-mempool is for latency-sensitive, best-effort visibility into what hasn’t landed yet, not for anything that must not be missed. See The evm-mempool exception.

“I want every dropped or suppressed event to show up somewhere I can alert on.” Nothing here is ever silently dropped, and that’s true without any config, but the observable surface differs by scenario: throttle and retry-exhaustion both produce dead letters (poll GET /networks/{id}/dead-letters or alert on its growth); a deep reorg’s retracts are visible in the Retracted events your sinks receive; a journal gap is a named metric (blockwatcher_journal_gap_total). See Observability for the full metric list.