Multi-Replica Reference Architecture

0 How to read this document

This reference names the mechanism behind each claim, not a vague adjective. It tells you which state lives on a single replica, which is shared, which background job is safe to run at N copies and which is not, and which config key controls each one. Most CrossConnect deployments run a single application node, and the Performance & Capacity Planning Guide sizes that one node well into the thousands of devices. You move to more than one replica for two reasons: to survive a node failure or a rolling upgrade with no downtime, and to serve more requests or operators than one node should carry. The rest of this document is the reference shape for that move.

1 Posture at a glance

CrossConnect ships as a set of containers: one application image, PostgreSQL, and an optional Batfish analysis sidecar. You can run them under Docker Compose, Kubernetes, or a managed-database setup such as Cloud Run plus Cloud SQL. The application tier scales out (add more replicas), with one boundary you have to engineer on purpose. Four facts frame every decision below.

2 Reference topology

Here is the overall shape. A health-checked load balancer spreads traffic across identical application replicas. Every replica shares one PostgreSQL primary and one rate-limit / session store. The primary streams to an optional read replica that absorbs reporting and AI-retrieval reads, keeping that load off the primary. Configuration analysis runs against a Batfish sidecar reached on a separate path (out of band). Every edge below is labeled with its transport.

flowchart TB
  LB["Load balancer
L7 · TLS term · health checks
session affinity"]
  subgraph APP["APPLICATION TIER · stateless, scale 2..N"]
    direction LR
    A1["App replica 1
:8080 · UI · REST"]
    A2["App replica 2
:8080 · UI · REST"]
    AN["App replica N
:8080 · UI · REST"]
  end
  subgraph STORE["SHARED STATE · one of each"]
    direction LR
    PG[("PostgreSQL primary
:5432 · system of record")]
    RO[("Read replica
reporting · AI reads")]
    RS[("Shared counter / session store
rate limit · sessions")]
  end
  BF["Batfish sidecar
:8888 · single session
config analysis"]
  LB -- "HTTPS · affinity by cookie" --> A1
  LB --> A2
  LB --> AN
  A1 -- "JDBC / TLS" --> PG
  A2 --> PG
  AN --> PG
  PG -- "streaming replication" --> RO
  A1 -. "shared counters / sessions" .-> RS
  A2 -.-> RS
  AN -.-> RS
  A1 -- "RPC · out of band" --> BF
  A2 --> BF
  AN --> BF
  classDef app fill:#173a6b,stroke:#0f2a4f,color:#ffffff;
  classDef store fill:#1797b3,stroke:#0d7d90,color:#ffffff;
  classDef ext fill:#ffffff,stroke:#9aa8c0,color:#173a6b;
  class A1,A2,AN app;
  class PG,RO,RS store;
  class LB,BF ext;
  

The default ports come from the shipped configuration: the application listens on server.port (8080), PostgreSQL on 5432, and the Batfish sidecar on 8888 via CROSSCONNECT_BATFISH_URL. The managed reference (Cloud Run) sets containerConcurrency: 50 and autoscales from minScale: 1 to maxScale: 5. It keeps one warm replica so requests do not pay the roughly five-second JVM cold start.

3 The request path is stateless; what is not

CrossConnect serves dashboards from in-memory rollups, and reads and writes PostgreSQL as its system of record. The request path itself keeps no lasting per-user state between calls, so a request can land on any replica. But three pieces of state are not local to a single replica, and most of this architecture exists to handle them correctly.

The configuration-analysis engine (Batfish) is shared too, but it is reached on a separate path and holds no per-request state, so it scales as its own pool rather than as part of the request path (§8). The background sweeps are a fourth concern, covered in §7.

4 How a request flows

flowchart LR
  REQ["Operator / API request"] --> LB{"Load balancer
route by session cookie"}
  LB --> R["Any app replica
stateless"]
  R --> RL{"Rate-limit check
per-tenant · per-IP"}
  RL -- "within budget" --> SES["Resolve session
affinity or shared store"]
  RL -. "429 + Retry-After" .-> REJ["Rejected"]
  SES --> SRV["Serve from in-memory rollups
or read / write PostgreSQL"]
  classDef app fill:#173a6b,stroke:#0f2a4f,color:#ffffff;
  classDef gate fill:#fdf0dd,stroke:#e0892a,color:#173a6b;
  classDef store fill:#e3f3f6,stroke:#1797b3,color:#173a6b;
  classDef ext fill:#ffffff,stroke:#9aa8c0,color:#173a6b;
  class R,SRV app;
  class LB,RL gate;
  class SES store;
  class REQ,REJ ext;
  

When the rate-limit budget and the session both resolve against shared state, the operator sees one consistent limit and one continuous session no matter how the load balancer spreads their requests. Add or remove a replica and none of that changes. When they resolve against replica-local memory instead (the shipped default), the budget and session belong only to whichever replica answered. That is exactly why §5 and §6 exist.

5 Shared state, component by component

6 The rate limiter is per-replica today

This is the shared-state trap most teams hit first. The API rate limiter (RateLimitFilter, applied to /api/v1/*) is a fixed-window counter kept per (tenant, IP) in an in-memory map on each replica. The default is crossconnect.ratelimit.requests-per-window: 100 over crossconnect.ratelimit.window-seconds: 60, which is 100 requests per minute per key. The source code itself is clear that this is a single-instance design: “In-memory for v1, single-instance only. When the first deploy needs horizontal scale, the bucket store moves behind a port and a Redis impl plugs in.” The shared-counter implementation is a defined extension point, not yet shipped.

flowchart LR
  subgraph NOW["DEFAULT · per-replica counters"]
    direction TB
    C1["Replica 1
budget 100/min"]
    C2["Replica 2
budget 100/min"]
    C3["Replica 3
budget 100/min"]
    EFF["Effective fleet limit
100 × N per min"]
    C1 --> EFF
    C2 --> EFF
    C3 --> EFF
  end
  subgraph FIX["SHARED · one counter store"]
    direction TB
    S1["Replica 1"]
    S2["Replica 2"]
    S3["Replica 3"]
    SS[("Shared counter
one 100/min budget")]
    S1 --> SS
    S2 --> SS
    S3 --> SS
  end
  classDef app fill:#173a6b,stroke:#0f2a4f,color:#ffffff;
  classDef gate fill:#fdf0dd,stroke:#e0892a,color:#173a6b;
  classDef store fill:#1797b3,stroke:#0d7d90,color:#ffffff;
  class C1,C2,C3,S1,S2,S3 app;
  class EFF gate;
  class SS store;
  

7 Scheduled sweeps: the once-per-node hazard

CrossConnect runs roughly twenty @Scheduled background jobs: discovery, reachability probing, golden-config drift, report delivery, retention purges, and AI-intent expiry among them. Each one is a Spring scheduled method that fires on a fixedRate or fixedDelay timer. The shipped build has no leader election, ShedLock, or PostgreSQL advisory lock. On a single node that is correct and simple. At N replicas, every timer fires on every replica, so each job runs N times per interval. This is the single most important thing to engineer before you go multi-replica.

flowchart TB
  T(["Timer fires
fixedRate / fixedDelay"])
  T --> R1["Replica 1
runs the sweep"]
  T --> R2["Replica 2
runs the sweep"]
  T --> R3["Replica 3
runs the sweep"]
  R1 --> DUP{"No leader gate
in the build"}
  R2 --> DUP
  R3 --> DUP
  DUP --> X["N× discovery probes · N× report emails
racing deletes on purge sweeps
N× drift analysis"]
  classDef gate fill:#fdf0dd,stroke:#e0892a,color:#173a6b;
  classDef app fill:#173a6b,stroke:#0f2a4f,color:#ffffff;
  classDef ext fill:#ffffff,stroke:#9aa8c0,color:#173a6b;
  class R1,R2,R3 app;
  class DUP gate;
  class T,X ext;
  

Most timers are already held back by configuration that is off by default: discovery (crossconnect.discovery.enabled: false) and automation (crossconnect.automation.enabled: false) do not run until you enable them. So a worker-role split is often just a matter of turning them on for the worker replica and leaving them off everywhere else.

8 The Batfish single-session constraint

The Batfish sidecar runs a single shared analysis session. If several /analyze calls hit one session at once, they line up and thrash a single snapshot, so each call can run past its timeout. CrossConnect handles this inside each replica with two mechanisms in BatfishConfigBackend. A fair ReentrantLock queues analyze calls so each one finishes quickly and returns real results, rather than falling back to a rough estimate after a timeout. A SingleFlight coalescer merges identical-config requests that arrive at the same time into one computation. Results are cached and addressed by the config's SHA-256 hash in a persistent finding cache, so repeat questions skip the engine entirely.

flowchart LR
  subgraph R["Each replica"]
    direction TB
    REQ["Analyze requests
warmer + UI · per device"]
    SF{"SingleFlight
collapse identical configs"}
    LK{"Fair ReentrantLock
serialize one at a time"}
    REQ --> SF --> LK
  end
  CACHE[("Finding cache
keyed by config SHA-256")]
  BF["Batfish sidecar
single shared session"]
  LK -- "cache hit" --> CACHE
  LK -- "cache miss · /analyze" --> BF
  BF --> CACHE
  classDef app fill:#173a6b,stroke:#0f2a4f,color:#ffffff;
  classDef gate fill:#fdf0dd,stroke:#e0892a,color:#173a6b;
  classDef store fill:#1797b3,stroke:#0d7d90,color:#ffffff;
  classDef ext fill:#ffffff,stroke:#9aa8c0,color:#173a6b;
  class REQ app;
  class SF,LK gate;
  class CACHE store;
  class BF ext;
  

9 Caches and single-flight coalescing

Every in-memory cache in CrossConnect is per-replica; the build has no distributed cache. That is on purpose and almost always fine, because each cache is either time-bounded (it expires on a TTL) or hash-addressed: two replicas may briefly hold different values, but neither holds a wrong one for long.

Because the finding cache keys on a plain hash of the configuration text, it stays stable across replicas and restarts: the same config always produces the same key. So a shared on-disk cache directory lets replicas reuse each other's analysis results without any coordination. The time-bounded dashboard rollups are the only caches where replicas can visibly disagree, and only for the length of the TTL.

10 Database connection math

This is the one number that bites at scale. Each replica opens its own HikariCP connection pool (SPRING_DATASOURCE_HIKARI_MAXIMUM_POOL_SIZE), and the primary has to hold all of those pools at once, plus a reserve for replication, maintenance, and admin sessions. Keep each replica's pool to what one node needs (10 to 25, per the Capacity Planning tiers), not what the whole fleet needs.

11 Rolling upgrades and autoscaling

flowchart LR
  V1["Replica @ v1
serving"] --> DR["Drain
readiness → false
LB stops routing"]
  DR --> RP["Replace @ v2
warm caches in background"]
  RP --> RD{"Readiness probe
warm complete?"}
  RD -- "ready" --> V2["Replica @ v2
serving"]
  RD -. "not yet" .-> RP
  classDef app fill:#173a6b,stroke:#0f2a4f,color:#ffffff;
  classDef gate fill:#fdf0dd,stroke:#e0892a,color:#173a6b;
  classDef ext fill:#ffffff,stroke:#9aa8c0,color:#173a6b;
  class V1,V2 app;
  class RD gate;
  class DR,RP ext;
  

• Readiness gating. A replica reports ready only after it has started up and warmed its cache. The shipped /actuator/health/readiness probe includes a check that the database is reachable; /actuator/health/liveness reports process health. The load balancer and orchestrator both watch readiness, so traffic never reaches a cold or draining replica.
• Surge policy. maxUnavailable: 0, maxSurge: 1 brings a new replica up before retiring an old one, so capacity stays flat through the rollout.
• Autoscaling. Scale on CPU, since interactive load is CPU-bound once the cache is warm. A target near 60% CPU leaves room for the warm-up cost of a newly added replica. The Cloud Run reference caps at maxScale: 5 with containerConcurrency: 50.
• Warm-up cost. A new replica builds its in-memory rollups on its first traffic, using the startup warmers (DashboardCacheWarmer and, where Batfish is reachable, BatfishWarmer). Readiness gating hides that from users; time to readiness on restart is roughly nine seconds at the reference workload.

12 Failure modes and recovery

flowchart TB
  subgraph F["Each shared component fails into a bounded state"]
    direction TB
    AR["App replica down
health check fails"] --> ARX["LB reroutes to healthy replicas
affinity sessions re-auth"]
    PGF["PostgreSQL primary down"] --> PGX["Writes pause · cached reads continue
promote hot standby"]
    BFF["Batfish down"] --> BFX["available = false · flagged in UI
all other reads / writes unaffected"]
    RSF["Counter / session store down"] --> RSX["Rate limit falls open per-replica
shared sessions re-authenticate"]
  end
  classDef app fill:#173a6b,stroke:#0f2a4f,color:#ffffff;
  classDef store fill:#1797b3,stroke:#0d7d90,color:#ffffff;
  classDef gate fill:#fdf0dd,stroke:#e0892a,color:#173a6b;
  classDef ext fill:#ffffff,stroke:#9aa8c0,color:#173a6b;
  class AR,PGF app;
  class BFF,RSF gate;
  class ARX,PGX,BFX,RSX ext;
  

13 Observability across replicas

Every replica emits the same metrics it does on a single node, exported at /actuator/prometheus, with OpenTelemetry tracing always wired and span export gated by a runtime toggle (CROSSCONNECT_TRACING_EXPORT_ENABLED, off by default). The extra work at scale is aggregation: tag each series by replica so you can see both the fleet total and any single replica drifting away from the pack.

• Per replica: CPU, heap (jvm.memory.used vs max), request latency (http.server.requests p50/p95/p99), cache hit rate, DB pool in-use (hikaricp.connections.active / pending), and time to readiness on restart.
• Fleet aggregate: total request rate, rate-limit rejections, error rate, and the spread of latency across replicas. A widening spread points to one unhealthy replica before users feel it.
• Shared dependencies: PostgreSQL connection count against max_connections, replication lag to the read replica, counter/session store availability, and Batfish contention (crossconnect.batfish.analyze, …analyze.lockwait, …fallback).
• Sweep duplication: once you are multi-replica, watch for the once-per-node signature: discovery and report volume that scales with the replica count is the tell that a sweep is running ungated.

14 Pre-go-live checklist