Performance & Capacity Planning

0 How to read this document

This is a sizing reference, not a marketing sheet. Each section explains what is behind a number: the cache and how long it holds (its TTL), the lock in front of the config engine, the pool default, the meter you watch. The sizing tiers are conservative starting points, anchored to the measured reference workload at the back of this guide. Test at your own scale before going live.

1 Where the cost lives

CrossConnect is a single application process backed by a PostgreSQL database, plus an optional helper service for config analysis (a Batfish sidecar). Sizing it comes down to finding the few places where work is genuinely expensive and confirming the cheap paths stay cheap. Four facts frame every number here:

2 The request path & sizing model

The figure below is both the architecture and the sizing model. Each arrow is labelled with what makes that path cheap or expensive, because that is what you are sizing for. Cached summaries served from memory keep many operators cheap to support; writes and uncached lists go to PostgreSQL; analysis across the whole fleet is the one path that runs a single request at a time on a single engine session.

flowchart LR
  subgraph CLIENTS["OPERATORS & INTEGRATIONS"]
    direction TB
    OP["Operators
dashboards · lists · UI"]
    API["Integrations
/api/v1/* · rate-limited"]
  end
  subgraph APPNODE["CROSSCONNECT APPLICATION NODE"]
    direction TB
    APP["App process
Java 21 · Spring Boot 3.4 · Vaadin"]
    CACHE[("In-memory caches
memoized · 15-60s TTL")]
    POOL["HikariCP pool
sized per tier"]
  end
  PG[("PostgreSQL
system of record")]
  BF["Batfish sidecar
single session · fair lock"]
  OP -- "cached read (cheap)" --> APP
  API -- "fixed-window limit" --> APP
  APP -- "serve hot rollup" --> CACHE
  APP -- "write / uncached list" --> POOL
  POOL -- "JDBC / TLS" --> PG
  CACHE -. "miss: fleet-read 60s TTL" .-> POOL
  APP -- "analyze (queues on lock)" --> BF
  BF -. "JDBC: per-device config read on miss" .-> PG
  classDef app fill:#173a6b,stroke:#0f2a4f,color:#ffffff;
  classDef store fill:#e3f3f6,stroke:#1797b3,color:#173a6b;
  classDef gate fill:#fdf0dd,stroke:#e0892a,color:#173a6b;
  classDef ext fill:#ffffff,stroke:#9aa8c0,color:#173a6b;
  class APP app;
  class CACHE,PG store;
  class BF gate;
  class OP,API,POOL ext;
  

The sizing model follows directly from that path. You size four things, in order: the application node (CPU for handling many operators at once, memory for the data it keeps in flight), the database (connections matched to the pool, RAM for the active working set, disk for how long you keep history), the config engine (memory that grows with the size of the fleet, moved off the app node once it gets heavy), and the storage volume (the one number that keeps growing over time). The later sections cover each one in turn.

3 Caches: why concurrency is cheap

A Standard node handled 150 dashboard loads at once with memory near half a gigabyte because the expensive summaries are computed once and then reused. Several caches do this work. The dashboard report memos and both Batfish answer caches use a per-key single-flight lock, so even when many requests for the same cold entry arrive together, the result is computed exactly once, not once per request. The fleet-read cache is a plain short-TTL cache: it holds the rebuilt fleet config map for 60 seconds, which collapses the repeated per-question reads inside a single page load.

All report memos are per-tenant. A multi-tenant deployment multiplies cache memory by the number of active tenants, but each entry is a small summary, so the total stays modest next to overall heap.

4 The config engine: the one serialization point

Formal analysis across the whole fleet (reachability, ACL, and IP-address-conflict questions) runs against a single Batfish session. This is the only place in the request path where simultaneous requests wait in line rather than run side by side, so it pays to know how it behaves under load and why it is the first thing to move off the app node.

flowchart LR
  R1["request A
config X"] --> SF{"single-flight
same config?"}
  R2["request B
config X"] --> SF
  R3["request C
config Y"] --> LK
  SF -- "collapse to one" --> LK["fair ReentrantLock
one Batfish session"]
  LK -- "analyze
12s request timeout" --> BF["Batfish sidecar
5s connect timeout"]
  BF -- "result" --> CACHE[("result + finding cache
warm = microseconds")]
  BF -. "error / timeout" .-> FB["heuristic fallback
crossconnect.batfish.fallback++"]
  classDef app fill:#173a6b,stroke:#0f2a4f,color:#ffffff;
  classDef gate fill:#fdf0dd,stroke:#e0892a,color:#173a6b;
  classDef store fill:#e3f3f6,stroke:#1797b3,color:#173a6b;
  classDef ext fill:#ffffff,stroke:#9aa8c0,color:#173a6b;
  class BF app;
  class SF,LK,FB gate;
  class CACHE store;
  class R1,R2,R3 ext;
  

• Single-flight first. Several first-time requests for the same config are merged into one analyze call. Different configs still take turns on the lock.
• Fair lock second. A fair ReentrantLock lines up distinct analyses so each finishes quickly and the cache fills cleanly. Watch crossconnect.batfish.analyze.lockwait to see how deep the queue gets on the shared session.
• Capped timeouts. The sidecar HTTP client waits at most 5 seconds to connect and 12 seconds for an analyze request. If the engine is slow or unreachable, the request falls back to a rule-of-thumb answer instead of hanging, and that fallback is counted.
• The slow first run is paid only once. The on-disk finding cache (kept 30 days, keyed by analyzer version) means a given config is analysed once across the life of the process; later reads, even after a restart, are served from cache.

5 The database & the connection pool

PostgreSQL is the single system of record. The application reaches it through a HikariCP connection pool, and the pool size is the main dial between how much the app does at once and how much load lands on the database. The build does not override Hikari, so the framework defaults apply until you set the pool size yourself for each tier.

Flyway (version 10.20.1) owns the database schema; Hibernate runs in validate mode and never changes the schema on its own. Primary keys use UUIDv7 (time-ordered IDs), which keeps related rows close together in the index on insert. That holds down index bloat and extra write work as history piles up.

6 Discovery, webhooks & background work

Background work runs on small, fixed-size worker pools on a fixed schedule. None of it grows with how many operators are online, and the heaviest job (discovery) ships turned off, so it never inflates a sizing estimate unless you turn it on.

7 The rate limiter

Integration (API) traffic on /api/v1/* is rate-limited: each tenant-and-IP pair gets a fixed number of requests per time window. The operator UI is not rate-limited. The default is 100 requests per 60 seconds; raise it to your measured integration rate plus some headroom.

A rejected request returns HTTP 429 with an RFC-7807 problem-detail body and a Retry-After header. Today the counter lives in memory on each replica, so the first time you run more than one replica it has to become shared state behind a common store (see §13).

8 Gather your sizing inputs

Five numbers drive the model. Fill these in first; everything after derives from them.

9 Pick your tier

Find input A (devices) in the first column. That row is your tier for the rest of the model. If input B (operators) points you to a higher tier than A does, use the higher one: more operators at once add CPU, not memory.

10 Provision the three components

For your tier, build each piece to these specs. On Pilot, Small, and Standard they all run together on one node; from Large up they split apart. The figure shows how the layout changes as the config engine, the one place work runs one at a time, is pulled off the shared node.

flowchart TB
  subgraph T1["PILOT / SMALL / STANDARD · single node"]
    direction LR
    A1["App"] --- D1[("PostgreSQL")]
    A1 --- B1["Batfish
co-located"]
  end
  subgraph T2["LARGE · two nodes"]
    direction LR
    A2["App + PostgreSQL"] --- B2["Batfish
dedicated host"]
  end
  subgraph T3["X-LARGE+ · distributed by role"]
    direction LR
    LB["Load balancer"] --> A3["App replicas"]
    A3 --> D3[("PostgreSQL
primary + replica")]
    A3 --> B3["Batfish
cluster"]
  end
  T1 -.->|"fleet grows"| T2
  T2 -.->|"fleet + operators grow"| T3
  classDef app fill:#173a6b,stroke:#0f2a4f,color:#ffffff;
  classDef store fill:#e3f3f6,stroke:#1797b3,color:#173a6b;
  classDef gate fill:#fdf0dd,stroke:#e0892a,color:#173a6b;
  classDef ext fill:#ffffff,stroke:#9aa8c0,color:#173a6b;
  class A1,A2,A3 app;
  class D1,D3 store;
  class B1,B2,B3 gate;
  class LB ext;
  

10a · Application node

Leave roughly 30% of node RAM above the heap for off-heap memory, metaspace, and the operating system. The reference node used about 0.5 GB of heap at 155 devices, so these figures are comfortable, not tight.

10b · PostgreSQL (the system of record)

Connection math from §5: max_connections ≥ (pool × replicas) + 20. Use JDBC over TLS (an encrypted connection) for any database link that is not on the same machine.

10c · Batfish config-analysis engine

The engine is optional. If no sidecar is reachable, config-analysis answers fall back to a built-in rule-of-thumb estimate (counted on crossconnect.batfish.fallback) instead of failing.

11 Apply the settings

Set these on the application container. The first three are sized from §10; the rest are the same for every production deployment. The full tunables list is in Appendix A.

Distributed tracing is always built in, but exporting the data is a runtime switch that is off by default, so you do not need a collector to run. When you want traces, turn it on and point OTEL_EXPORTER_OTLP_ENDPOINT at your collector (OpenTelemetry exporter 1.43.0).

12 Plan storage & retention

The database storage in §10b is a starting point. It grows mainly with three things you control, and over time storage is the biggest cost driver.

13 Probes, orchestration & the meters to alert on

The endpoints and settings to wire into Kubernetes, Cloud Run, or your scheduler, plus the meters that warn you when a component is running out of headroom.

Health & metrics endpoints

Orchestration (Kubernetes / Cloud Run)

• Set CPU and memory requests = limits to the §10a application spec; the app runs at a steady level, not in bursts.
• Probes: a startupProbe on /actuator/health/readiness with about a 30 s budget (boot takes about 9 s), then separate readinessProbe and livenessProbe. Keeping them separate stops the orchestrator from killing a healthy pod that is still warming up.
• Rolling upgrades: maxUnavailable: 0, maxSurge: 1. A new replica serves right away; its first dashboard read fills the cache under the single-flight memo (§3), the background warmer fills it ahead of time, and the disk-backed finding cache means config analysis does not have to be redone.
• Autoscaling (HPA): scale the app tier on CPU at about 70% for workloads heavy on uncached reads or reports. Cached dashboard traffic rarely triggers it. Before running more than one replica, plan for the two shared-state items in the HA note below.

What to alert on

14 Validate, then go live

• Stand up the tier, load your real inventory (or a representative sample of it), and run a short load test at your peak operator count (B). CybrIQ provides the test harness.
• Watch four signals: application heap usage (stay under about 70% of -Xmx), how long requests wait for a database connection (hikaricp.connections.pending near zero), p95 response time on the dashboards and your busiest lists, and crossconnect.batfish.fallback (should stay at 0).
• Set up liveness and readiness probes; the app becomes ready in about 9 seconds and warms its caches in the background, so roll upgrades one replica at a time for zero downtime.
• If any signal is tight, move up one tier on the constrained component only (usually CPU for operators, RAM and storage for fleet size and retention, and a dedicated engine host if lock-wait is rising).

15 Worked example

16 Reference: measured performance

The sizing above is anchored to a real, measured reference workload (a Standard-class node, 4 vCPU, 155 devices):

The full method and complete result set are in the CrossConnect Performance Report.

A Appendix · tunables & defaults reference

The settings that matter for sizing, with the values the build ships with. The defaults are chosen to work out of the box; a production deployment sets the heap, pool, secrets, and rate limit explicitly.

B Appendix · meters reference

The custom meters most useful for capacity work. Read them at GET /actuator/metrics/{name} with no extra infrastructure, or scrape them from /actuator/prometheus. The standard framework meters (jvm.*, hikaricp.*, http.server.requests, system.cpu.usage) are exposed too.