ADR-0029: Cross-Zone Synchronization Uses BSFG Peer Protocol, Not Native Stream Mirroring

Context

Each zone owns its own JetStream domain and object store. Facts must still move across zones. NATS JetStream already provides native source and mirror replication, which asynchronously copies messages across streams and can recover after disconnects. Those agreements are configured on the receiving side, and mirrored messages retain origin sequence numbers and timestamps. :contentReference[oaicite:0]{index=0}

However, BSFG has already committed to an explicit boundary protocol:

AppendFact
FetchFacts
ConfirmReceipt
PutObject

The architecture therefore needs to decide whether cross-zone movement should be implemented primarily by native broker-to-broker mirroring, or by BSFG peers speaking the boundary protocol.

Options Considered

• built-in asynchronous replication
• recovers after disconnects
• preserves source sequence and timestamp metadata in mirrors

|

• boundary semantics become broker-feature semantics
• confirmation and authorization are less aligned with BSFG RPC contract
• artifact/object lifecycle remains outside the same explicit peer protocol
• deletes in origin are not replicated through source/mirror agreements

| | Shared broker federation across zones | Expose zones through a broader NATS topology such as leaf-node style transparent routing. |

• strong built-in message routing
• convenient broker-level connectivity

|

• boundary becomes infrastructure-transparent rather than explicit
• trust domains blur operationally
• weak fit for appliance-style audited crossings

| | BSFG peer protocol as the canonical cross-zone path (Selected) | Zones synchronize by having BSFG peers call each other through the explicit boundary RPC contract. |

• keeps boundary crossing explicit and auditable
• aligns transport, security, idempotency, and confirmation semantics in one model
• works uniformly for facts and artifact references
• does not depend on broker features becoming the architectural contract

|

• more application logic than pure broker mirroring
• BSFG peers must manage polling, confirmation, and retry loops explicitly

| | Hybrid with broker mirroring hidden underneath | Expose BSFG RPC at the surface but secretly rely on native source/mirror for most cross-zone movement. |

• can reuse built-in broker replication
• less peer-side replication logic

|

• two competing mental models
• harder debugging when broker and BSFG semantics diverge
• weakens architectural clarity

|

Decision

Cross-zone synchronization uses the BSFG peer protocol as the canonical boundary mechanism. Native JetStream source and mirror are not the primary architectural primitive for zone crossing. :contentReference[oaicite:1]{index=1}

BSFG(zone A) --Connect RPC/mTLS--> BSFG(zone B)
     |                                   |
 JetStream A                         JetStream B

This means:

• facts move through FetchFacts, AppendFact, and ConfirmReceipt
• large artifacts move through PutObject and referenced facts
• zone authorization stays aligned with mTLS-authenticated peer identity
• confirmation semantics remain “durable commit in target zone log,” not broker-specific replication state

Native JetStream replication features remain useful implementation tools in other contexts, but they are not the canonical expression of the BSFG boundary architecture. Connect remains the chosen boundary RPC protocol and is explicitly designed to work over ordinary HTTP rather than a transport bound to one specific HTTP framing model. :contentReference[oaicite:2]{index=2}

Consequences

Benefits:

• boundary crossing stays visible as an explicit protocol act
• security, idempotency, and confirmation are defined once, in one place
• broker internals do not become the public architectural contract
• fact transport and artifact reference handling stay in the same conceptual model

Tradeoffs:

• BSFG peers must implement synchronization logic that native mirroring could have provided
• some built-in replication convenience is traded for architectural clarity
• operators must monitor peer sync behavior in addition to broker health