Observability & Operations Guide

Audience: Operators, SREs, platform engineers. Use: Understand the monitoring model, alerting signals, and operational visibility expectations.

Overview

BSFG deployments require operational discipline to maintain reliability in industrial environments. Operators must be able to detect issues, understand system state, and perform safe upgrades without data loss.

This guide provides practical procedures for:

• Monitoring replication health and consumer lag
• Detecting and diagnosing operational failures
• Responding to incidents with confidence
• Performing safe node restarts and upgrades
• Auditing cross-zone synchronization

Monitoring Architecture

Typical BSFG monitoring architecture involves:

BSFG Node
├── Metrics (Prometheus format)
│   └→ Prometheus Server
│       └→ Grafana Dashboards
└── Logs (structured JSON or syslog)
  └→ Log Aggregation (ELK, OpenSearch, Splunk, Loki)

Components

• BSFG Node: Emits metrics and logs during operation
• Metrics Exporter: Exposes metrics in Prometheus format on `/metrics` endpoint
• Prometheus: Scrapes metrics at regular intervals (e.g., every 15 seconds)
• Grafana: Visualizes metrics in dashboards, defines alerts
• Log Aggregation: Centralizes logs from all zones for analysis and audit

Core Metrics

BSFG should expose metrics in Prometheus format. Key metrics to monitor:

Replication Metrics

Ingestion Metrics

Consumer Metrics

Artifact Metrics

System Metrics

Alerting Rules

Configure alerts based on these thresholds. Adjust thresholds per deployment based on expected throughput and latency.

Log Structure

BSFG nodes should emit structured logs for observability. Logs should include:

Log Fields

• timestamp: RFC3339 format (e.g., 2026-03-06T14:30:00Z)
• level: INFO, WARN, ERROR, FATAL
• message_id: Unique identifier for the fact (if applicable)
• from_zone: Source zone (producer or peer)
• to_zone: Destination zone (consumer or peer)
• operation: AppendFact, FetchFacts, ConfirmReceipt, PutObject
• predicate: Fact predicate (if applicable)
• result: success, retryable_error, permanent_error
• error_message: Human-readable error (if error)
• duration_ms: Operation latency

Example Log Entry

{
  "timestamp": "2026-03-06T14:30:45.123Z",
  "level": "INFO",
  "zone": "enterprise-bsfg",
  "message_id": "msg_abc123def456",
  "operation": "AppendFact",
  "predicate": "order_created",
  "result": "success",
  "duration_ms": 12,
  "buffer_fill_percent": 45
}

Troubleshooting Scenarios

Replication Has Stopped (Lag Growing)

Symptom: Replication lag continuously increases, facts are not flowing between zones.

Diagnosis:

1. Check replication lag metric: is it > 30 seconds?
2. Check consumer backlog: is it growing?
3. Check network connectivity: ping between BSFG nodes
4. Check firewall rules: is traffic allowed on RPC port (9443)?
5. Check certificates: are they valid and trusted?
6. Check logs for TLS errors or connection timeouts

Resolution:

• If network unreachable: restore connectivity, replication resumes automatically
• If certificate expired: rotate certificate, restart node
• If consumer backlog growing: restart consumer, monitor confirmation rate
• If no obvious cause: restart BSFG node (no data loss due to durability)

Consumer Backlog Growing (Not Draining)

Symptom: Consumer backlog continuously increases; facts are fetched but not confirmed.

Diagnosis:

1. Check confirm rate: is it zero or very low?
2. Check consumer process: is it running or hung?
3. Check logs for consumer errors or exceptions
4. Check if consumer is idempotently processing (not getting stuck on duplicates)

Resolution:

• Restart consumer process
• Review consumer code for blocking operations or deadlocks
• Verify consumer can write to its destination (database, file system)
• Monitor confirm rate recovery after restart

Artifact Retrieval Failures

Symptom: GetObject calls failing with "not found" or access denied errors.

Diagnosis:

1. Check artifact reference in fact: bucket, key, digest
2. Check object store: does the bucket exist? Is the key present?
3. Check storage access: can BSFG node read from the bucket?
4. Check retention policy: has the artifact been garbage collected?

Resolution:

• If artifact missing: verify producer uploaded it correctly
• If access denied: check IAM permissions for BSFG node
• If storage offline: restore storage, consumer retries automatically
• If artifact expired: implement better retention policy or upload frequency

TLS Handshake Failures

Symptom: TLS handshake errors in logs, replication unable to establish connections.

Diagnosis:

1. Check certificate validity: is it expired?
2. Check certificate CN: does it match the peer's zone identity?
3. Check CA trust: is the certificate CA trusted by peers?
4. Check certificate chain: is the full chain available?

Resolution:

• Renew certificate if expired
• Verify CN matches expected zone identity
• Ensure all peers have updated CA root certificate
• Restart BSFG node to reload certificates

Store Buffer Exhaustion

Symptom: Store buffer fill > 95%, risk of backpressure or data loss.

Diagnosis:

1. Check replication lag: facts being produced faster than replicated?
2. Check consumer backlog: facts being replicated faster than consumed?
3. Check retention policy: TTL too long, facts not being truncated?

Resolution:

• Increase buffer capacity if throughput permanently increased
• Reduce TTL if facts are being retained longer than needed
• Tune replication and consumer concurrency
• Verify backpressure policy (reject vs. drop oldest) is appropriate for SIL level

Operational Playbooks

Playbook: Safe Node Restart

Restarting a BSFG node is safe due to durability guarantees:

1. Check replication lag to baseline (record pre-restart state)
2. Stop BSFG service gracefully (signal SIGTERM, wait for shutdown)
3. Verify service stopped (check process list)
4. Start BSFG service (service restart or systemctl restart)
5. Wait 30 seconds for startup and peer reconnection
6. Check replication lag metric: should return to baseline within 1 minute
7. Check consumer backlog: should drain as normal
8. Check logs for any errors during startup

Playbook: Certificate Rotation (Planned)

Rotate certificates before expiration without service interruption:

1. Generate new certificate and key for zone (e.g., enterprise-bsfg)
2. Sign certificate with enterprise PKI CA
3. Copy certificate and key to BSFG node(s)
4. Reload BSFG configuration (SIGHUP or restart)
5. Verify certificate loaded: check certificate_expiry_seconds metric
6. Test peer connectivity: RPC calls succeed
7. Archive old certificate for audit trail
8. Document rotation in change log

Playbook: Emergency Certificate Rotation (Key Compromise)

If a node's private key is compromised, rotate immediately:

1. Generate new certificate and key immediately
2. Copy new cert/key to node
3. Restart BSFG service (forces reconnection with new certificate)
4. Verify peers accept new certificate (check TLS handshake success)
5. Revoke old certificate in PKI system
6. Destroy old private key securely
7. Alert security team and document incident

Playbook: Rolling Upgrade (Multi-Node HA)

Upgrade BSFG binary or dependencies with zero downtime:

1. Plan upgrade during low-traffic period (if possible)
2. Drain traffic from node 1 (stop accepting new RPC calls)
3. Wait 30 seconds for in-flight RPC to complete
4. Stop node 1 gracefully
5. Upgrade binary and dependencies on node 1
6. Start node 1, wait for peer reconnection (30 seconds)
7. Verify replication resumes: check lag metric
8. Re-enable traffic to node 1
9. Repeat for nodes 2, 3, etc. (one at a time)

Monitoring Dashboard Recommendations

A comprehensive Grafana dashboard should include:

• Replication Health: Lag trend, frontier progress, fetch/confirm rates
• Ingestion Health: Append rate, append failures, buffer fill
• Consumer Health: Backlog trend, lag trend, confirm rate
• Artifact Status: Upload/retrieval counts and failures, storage usage
• System Health: Node up/down, certificate expiry, TLS errors, RPC latency
• Logs: Recent errors, warnings, and notable events

Pre-Deployment Checklist

• ☐ Monitoring stack deployed (Prometheus, Grafana, or equivalent)
• ☐ Metrics endpoint configured on each BSFG node
• ☐ Prometheus scrape jobs defined for all zones
• ☐ Grafana dashboards created for key metrics
• ☐ Alert rules configured with appropriate thresholds
• ☐ Alert destinations configured (PagerDuty, Slack, email, etc.)
• ☐ Log aggregation pipeline deployed
• ☐ Structured logging enabled in BSFG configuration
• ☐ Log retention policy defined (at least 30 days)
• ☐ Audit logging enabled (all RPC operations logged)
• ☐ Runbooks documented for common scenarios
• ☐ On-call rotation and escalation procedures defined
• ☐ Incident response procedures documented
• ☐ Monitoring tools access and authentication configured

Cross-Links to Related Documentation

• Operations Runbook — Detailed failure modes and recovery
• Enterprise + IDMZ + 2 Plants — Deployment reference
• Peer Replication — Understanding replication protocol
• Identity Model — Certificate management
• Network Policy — Connectivity and firewall