ADR-0023: Fetch Progress Uses Durable Named Consumers

Context

BSFG fetch semantics are unary paged pull. That means progress over a retained stream must be tracked somewhere durable and replayable. The architecture needs a clear answer to:

• how a fetch client identifies its read position
• whether progress survives restarts and failover
• whether multiple instances can share one logical consumer identity
• how ConfirmReceipt relates to stored fetch progress

JetStream consumers are the server-side mechanism that track delivery and acknowledgement state. Durable consumers persist progress on the server, while ephemeral consumers do not persist long-term state and are intended for short-lived use. :contentReference[oaicite:0]{index=0}

Options Considered

• minimal broker-side setup
• implementation freedom for clients

|

• progress semantics become fragmented
• harder failover and recovery
• weak alignment with JetStream model

| | Ephemeral consumers | Create short-lived consumers on demand for each fetch session. |

• simple for ad hoc inspection
• little persistent state to manage

|

• progress does not persist reliably
• poor fit for durable cross-zone replication
• restart semantics are weak

| | Stateless stream reads only | Read raw stream positions directly with no consumer abstraction. |

• appears simple conceptually
• close to a bare log model

|

• acknowledgement floor is externalized
• recovery and shared-consumer behavior become custom protocol work
• gives up useful JetStream primitives

| | Durable named consumers (Selected) | Each logical fetcher uses a stable consumer identity whose progress is persisted server-side. |

• progress survives restarts
• clean recovery semantics
• good fit for pull-based batch processing
• aligns with server-side acknowledgement tracking

|

• consumer naming and lifecycle must be governed
• administrative sprawl is possible if names are unmanaged

|

Decision

BSFG fetch progress uses durable named consumers.

FetchFacts(consumer_name, ...)
ConfirmReceipt(consumer_name, ...)

A consumer_name identifies the logical reader whose progress is tracked against the stream. That identity is durable and survives client restarts.

Consequences of this choice:

• fetch position is server-side state, not only client memory
• multiple client instances may share one durable consumer identity when appropriate
• consumer progress and acknowledgement floor remain observable at the log substrate
• ad hoc inspection may still use ephemeral tooling, but it is not the canonical BSFG path

Consequences

Benefits:

• restarts and failover do not lose fetch progress
• boundary replication remains operationally stable
• consumer identity is explicit in the API contract
• the design aligns with pull-based batch consumption in JetStream :contentReference[oaicite:1]{index=1}

Tradeoffs:

• consumer names become part of architecture governance
• unused durable consumers must be cleaned up administratively
• client implementers must understand shared-vs-dedicated consumer identity semantics