Gates

A predicate answers a per-event yes-or-no. A gate answers a question that depends on other hits of the same monitor: did Transfer occur three times in the last hour of event time? Has this monitor already alerted in this window? The engine asks that question after selector and predicate succeed, and before a match exists.

Omit gate and the engine behaves as it does today: every predicate-true decode is a match. Name a gate module the same way a sink names its module ({ "module": "threshold", "config": { … } }), and the engine offers each hit to that compiled module together with an engine-owned journal of earlier holds.

Delivery guarantees still describe what happens after a match (or digest) is minted. This page is the seam before that mint. Sink throttle and aggregate are the wrong layer for “count three Transfers”: they run after a Match already exists, and aggregate stalls the checkpoint for the whole window.

flowchart LR
    rpc["RPC endpoints<br/>(external chains)"]
    sources["Sources<br/>evm-rpc · evm-mempool"]
    sinks["Sinks<br/>webhook · script · log"]
    storage["Storage<br/>checkpoints · dead letters · resources"]
    api["REST API"]
    metrics["Metrics"]
    engine["engine<br/>bounded channels · checkpoints"]

    subgraph pipeline["Engine pipeline"]
        direction LR
        decoder["Decoder"]
        matcher["Matcher<br/>predicates"]
        gate["Gate<br/>threshold · max_once"]
        decoder --> matcher
        matcher --> gate
    end

    rpc --> sources
    sources -->|"decode and match"| decoder
    gate --> sinks
    api -->|"manages resources"| storage
    storage <--> engine
    engine -->|"drives"| pipeline
    engine -.->|"reports"| metrics

    classDef module fill:none,stroke:#a9a3e3
    classDef core fill:none,stroke:#8a8d86,stroke-dasharray: 5 5

    class rpc,sources,sinks,decoder,matcher,gate module
    class engine,api,storage,metrics core
classDef dim fill:none,stroke:#999999,color:#999999,opacity:0.35
classDef focus fill:#ffd43b,stroke:#d9480f,stroke-width:3px,color:#1a1a1a
class rpc,sources,sinks,storage,api,metrics,engine,decoder,matcher dim
class gate focus
click sources "selectors.html"
click decoder "chain-agnosticism.html"
click matcher "predicates.html"
click gate "gates.html"
click sinks "delivery.html"
click storage "resources.html"
click api "../reference/http-api.html"
click metrics "../reference/observability.html"
click engine "pipeline.html"

Key takeaways

• A gate is a sixth port: one compiled decision per monitor over an engine-owned hit journal. The module never opens sqlite.
• Retain / Discard complete with zero outstanding: holding a hit does not stall the checkpoint. Only an Emit journals a delivery.
• Window membership uses decoded block.timestamp (unix seconds), never wall-clock time. Catch-up does not dump history into “now.”
• On invalidate the engine prunes journal rows with cursor strictly after from. It does not drain the whole bag. A tip reorg must not wipe holds from before the fork.
• One gate per monitor. Compose behaviours with two monitors. Per-event filters stay in the predicate.

Assumptions (what the operator is agreeing to)

Guarantees (what the engine actually promises)

Contrast operators will get wrong

Where it sits

flowchart LR
    src[Source] --> dec[Decode]
    dec --> pred[Predicate]
    pred --> gate[Gate]
    gate --> sink[Sink]

Inside Processor::process_one (crates/blockwatcher-core/src/pipeline/processor.rs), after a monitor accepts a decoded event:

1. If the monitor has no gate, mint a match as today.
2. If it has a gate, load that monitor’s persisted hits, append this hit, call Gate::on_hit, persist the resulting journal, then either stay quiet or mint SinkEvent::Match / Digest from the indices the module returned.
3. Quiet (Retain or Discard) registers outstanding 0 for this monitor’s sinks. Other monitors on the same raw event still add theirs.
4. The raw event still registers with Progress exactly once, after every monitor has been considered, for the total outstanding across all of them, including zero.

A hit the gate kept is not a match. Sinks never see it. Dead-letter replay cannot resurrect it. The only durable trace is the gate_hits row (and the metrics on Observability).

Resource shape

On Monitor, sibling of predicate (crates/blockwatcher-types/src/resource.rs):

{
  "id": "usdc-burst",
  "network": "eth-mainnet",
  "selectors": [{ "spec": "erc20", "events": ["Transfer"] }],
  "predicate": "args.value > 0",
  "gate": {
    "module": "threshold",
    "config": { "count": 3, "window_ms": 3600000 }
  },
  "actions": ["alerts"]
}

"gate": { "module": "max_once", "config": { "window_ms": 3600000 } }

gate is this monitor’s decision rule (like predicate), not a shared resource id (like actions: SinkId[]). Journals are per (pipeline, monitor). The envelope is ModuleSel-shaped: module plus config object, deny_unknown_fields on the envelope. The named module validates config at write (Modules, P9). An unknown module name is 422, listing catalog names, the same as an unknown sink module.

gate omitted: no module is constructed; the engine emits the hit.

There is no gates array. Order of stacked gates would be a footgun (max_once before threshold never reaches count 3). Two monitors with the same selectors and different gates are how an operator composes behaviours.

Decisions: Retain, Discard, Emit

Gate::on_hit (crates/blockwatcher-ports/src/gate.rs) is total after a successful compile, the same way Matcher::matches is total after compile. It sees the journal including this hit as the last element and returns exactly one of:

The module returns indices, never a payload it invented. Core calls existing Match::new on each chosen DecodedEvent.

A buggy on_hit after compile is counted (blockwatcher_gate_decision_invalid_total) and treated as Discard of this hit only. It does not stall the pipeline.

Event time

Time-window gates require block.timestamp in the compiled schema (unsigned or non-negative int). Otherwise the write refuses:

gate requires 'block.timestamp'; this monitor's selectors do not expose it

Window membership:

(newer_ts.saturating_sub(older_ts) as u128) * 1000 <= window_ms

window_ms is 1..=86_400_000 (one day, inclusive). threshold.count is 2..=10_000.

A predicate-true hit that still has no usable timestamp at runtime is not inserted, does not stall, and increments blockwatcher_gate_untimestamped_total. That counter is a decoder hole the write-time check is supposed to have closed; it is not a third way to configure a gate.

Shipped modules

Both crates depend on types + ports only. Catalog family gate, same register path as sinks (build_catalog in crates/blockwatcher-embed/src/catalog.rs).

threshold: session digest

Config: { "count": u32, "window_ms": u64 }.

On a hit, drop a prefix until the remaining span fits window_ms (those drops are internal, not operator-visible, not dead letters). If len >= count, Emit the oldest count indices. Else Retain.

Worked example, count: 3, one-hour window:

• T1@5m, T2@20m, T3@30m → one digest [1, 2, 3]. Journal empty of them.
• T4@45m, T5@50m, T6@55m → second digest [4, 5, 6].

That is reset-on-fire, not “wait out the original hour,” and not a cooldown that would drop T4–T6. Leftover hits that were not part of the emitted N stay in the journal.

max_once: first hit per window

Config: { "window_ms": u64 }.

If last_emit_ts is none, or this event_ts is outside the window from last_emit_ts, Emit([this]) and record last_emit_ts. Else Discard([this]). The bag does not grow.

The payload is the first hit as a Match, not a digest of everything seen in the hour. Later in-window hits are gone: not dead-lettered, not replayable. If the operator needs those hits later, that is a different product (follow-up), not throttle.

Passthrough

Omit gate. Tests also ship an in-memory fake that always Emit([this]); it is not an operator-facing module in a default binary the way threshold is.

Persistence and crash

Storage (sqlite schema v4 and every Storage impl, including memory and the ports fake) keeps:

• gate_hits: (pipeline, monitor, cursor_primary, cursor_secondary, event_index) plus event_ts and the decoded event JSON.
• gate_meta: (pipeline, monitor) → last_emit_ts, last_emit_cursor, optional opaque aux blob.

Identity is those columns, never a string-joined key.

After insert, if that monitor has more than 10_000 rows, the oldest are dropped and blockwatcher_gate_hits_dropped_total ticks.

Persist the new journal before completing the event. An Emit writes the journal and meta in one storage transaction so a crash cannot drop the cooldown after the emitted rows are already gone. A persist failure begins Progress with outstanding 1 and does not complete the guard: the checkpoint stays on this cursor, same as a closed sink channel. Skipping begin would let a later cursor complete and strand this one behind it.

Restart: sqlite still has the holds; the next hit sees them. Memory storage does not survive a process exit, matching its own contract.

Delete or rewrite a monitor (including changing gate.module or config): delete that monitor’s gate_hits and gate_meta. Holds are not migrated across compiled artifacts.

Invalidate

After drain, before rewind/restart, for that pipeline:

1. Delete gate_hits rows whose cursor is strictly after from.
2. Clear last_emit_ts / last_emit_cursor only when last_emit_cursor is strictly after from. An empty journal is not stale: after Emit those rows are gone on purpose, and a tip reorg must not reset max_once. A reorg of the emitted hit itself (cursor after from) does clear the cooldown so the replacement on the new fork can fire.
3. Existing delivery-journal retract: Retracted for already-emitted Matches after from, as Delivery guarantees already describes.

Replay from from feeds on_hit again. Duplicate Emit is at-least-once; Match ids are deterministic.

Forbidden: DELETE FROM gate_hits WHERE pipeline = ? with no cursor predicate. A two-block tip reorg must not wipe holds with cursor ≤ from. Replay only re-emits after from; those earlier holds would never come back, and a later threshold would under-fire.

Gate::on_invalidate exists for follow-up aux (e.g. forever-distinct keys). threshold and max_once do not need it. The engine has already pruned holds before that hook runs; a module must not use the hook to drain the journal.

Dry-run

POST /monitors/{id}/test stays inert: no persist, no sink, no checkpoint. It constructs a fresh empty journal for the request and feeds payloads or fetch events in order. It does not read stored gate_hits. A three-payload body can fire a threshold digest. A single payload against threshold with count: 3 reports a retained hit and an empty matched list.

TestReport reports flattened emitted Matches (existing clients still see the events that would have been delivered) plus held (Retain decisions) and dropped (Discard decisions) without emitting.

What a gate is not

• Not a predicate. “Odd block number” / “before noon UTC” belong in predicate (block.number % 2, block.timestamp % 86400).
• Not sink throttle. Throttle caps deliveries after a Match exists and dead-letters the rest (replayable). max_once drops later hits.
• Not sink aggregate. Aggregate batches Matches and stalls the checkpoint until the window closes. A 60-minute aggregate window is a 60-minute checkpoint stall. A 60-minute threshold window is not.
• Not on_tick. A gate cannot fire because “an hour passed with no hit.” That needs a timer-driven path and a checkpoint-without-cursor spec, which this version does not have.
• Not module-owned storage. A gate crate that opens sqlite, or that clears all RAM on any reorg, is not this port.

Chain-agnosticism

The gate sees Cursor and canonical Value only. event_ts is the block.timestamp path the decoder already put in the tree. Chain-agnosticism is unchanged: core does not learn a chain to “do windows.” A decoder that does not expose block.timestamp cannot use a time-window gate; the write fails closed.

Sources must provide event timestamps

A time-window gate windows on decoded block.timestamp, which presumes a block: a feed of not-yet-mined data has none to stamp. Each source module declares SourceCaps { event_timestamps: bool } (crates/blockwatcher-ports/src/source.rs). evm-rpc declares true; evm-mempool declares false, since a pending transaction has not been included in a block yet.

PUT /monitors/{id} checks the target network’s source caps against what the compiled gate needs, the same 422 compile_failed class every other compile refusal in this document uses. A gated monitor is invalid configuration on a network whose source does not declare event_timestamps, not a monitor that silently never fires. The same check runs the other way: PUT /networks/{id} refuses to swap a network’s source module out from under an existing gated monitor, because doing so would leave that monitor pointed at a feed its gate can no longer window against. Neither route lets the pairing exist for even one write.

Delivery guarantee

An emission’s durability starts inside the same storage transaction that consumes the journal members it was built from: Emit does not exist as a decision without also existing as a row a sink can still be delivered from. The sink worker deletes that row only once the delivery outcome itself is durable: landed in the delivery journal, or recorded as a dead letter, never before. A crash between those two points redelivers on the next start rather than losing the emission: duplicates are possible, gaps are not. A pending emission outlives the monitor that minted it: deleting the monitor leaves its outbox row in place, and the row is still delivered, or dead-lettered when its sink no longer has a worker, on the next start. Deleting the emission’s network dead-letters the row right away instead, because a deleted id gets no next start for the drain to run at: each contained match becomes a dead letter for the pipeline id, and the dead-letter surface keeps answering for any id that still holds letters, so they stay listable, replayable, and discardable while the network stays deleted. Whatever that delete-time sweep could not finish (a storage fence that did not resolve, a lapsed budget, a failed write) is dead-lettered by the next boot’s orphan sweep, which also covers rows stranded before the sweep existed. Every pipeline’s pending rows are counted by /status as gate_outbox_depth, read fresh from storage, so an emission parked behind a paused network stays visible until a resume delivers it. See Delivery guarantees for the outbox’s shape and the startup drain that sweeps whatever a crash left behind. Invalidate reaches into a pending emission’s row the same way it reaches into the hit journal: reorged-out members of the row are pruned at invalidation, and their canonical replacements re-accumulate through the gate on replay.