AI API Cost Analysis

0 How to read this document

There is exactly one metered, per-token cost path in CrossConnect: the conversational AI. (Metered means you pay per use; per-token means the model charges by the small chunks of text it reads and writes.) Everything else runs on infrastructure you already own and have already paid for: CPU, memory, and the Postgres database. This document leads with the AI cost model because it is the only line item with an outside vendor invoice attached, then sets it against the wider API surface so you can see how a single AI question fans out into the database underneath it.

Every model parameter, default, and property name below is read straight from the shipping code and the application.yml config file, so the facts are real. The cost math (the latency bands, endpoint tiers, and how cost grows with fleet size) is a directional estimate from reading the code paths, not a measurement taken under load. Treat the relative ranking (which thing costs more than which) as solid, and the absolute numbers as a starting budget to confirm once you run it under load. We do not quote model list prices here, since those change often. Your bill is the model id and the token counts described below, multiplied by your provider's current price list.

1 The AI cost model in one page

The only spend that grows as you use the product more is the model API (the call out to the AI service). Each call to POST /api/v1/assistant/ask sends a prompt (the question plus its supporting data) to your model endpoint and is billed per token on both sides: the text going in and the text coming back. The cost of a single turn (one question and answer) is the tokens in the request times the input rate, plus the tokens in the reply times the output rate. Nothing else on the platform carries a per-call charge.

Two facts frame the whole bill. First, the AI ships off by default. Out of the box the provider is the built-in stub, a fixed renderer that formats data mechanically and never calls a paid model, so a fresh install has zero AI spend (§6). You turn the paid AI on deliberately. Second, once it is on, the input side drives the cost, not the output. The system prompt (the assistant's standing instructions), the tool-result data, and any multi-step agentic loop are all input tokens, and together they usually dwarf the size-limited reply.

flowchart LR
  Q["Operator question
(one turn)"] --> PROV{"Provider?
stub vs model"}
  PROV -- "stub (default)" --> FREE["Deterministic render
0 tokens, 0 spend"]
  PROV -- "model endpoint" --> IN["INPUT tokens
system prompt · tool-result JSON · history"]
  IN --> MODEL["Model endpoint
your key, per-token billed"]
  MODEL --> OUT["OUTPUT tokens
capped by max-tokens"]
  OUT --> BILL(["Turn cost =
in·rate_in + out·rate_out"])
  classDef app fill:#173a6b,stroke:#0f2a4f,color:#ffffff;
  classDef gate fill:#fdf0dd,stroke:#e0892a,color:#173a6b;
  classDef store fill:#e3f3f6,stroke:#1797b3,color:#173a6b;
  classDef ext fill:#ffffff,stroke:#9aa8c0,color:#173a6b;
  class MODEL app;
  class PROV gate;
  class IN,OUT,BILL store;
  class Q,FREE ext;
  

2 Anatomy of one AI request

A turn is not always a single model call. It is a prompt assembled from several parts, sometimes a short loop where the model fetches data step by step, and a size-limited reply. The cost is simply the total tokens that assembly adds up to. The request lifecycle is implemented in ai/AssistantService.java and the two client adapters (AnthropicLlmClient, ChatModelLlmClient).

flowchart LR
  ASK["POST /api/v1/assistant/ask"] --> WI{"Write intent
detected?"}
  WI -- "yes" --> PROP["Queue proposal
confirm-before-commit
(no model call)"]
  WI -- "no" --> RES{"Resolve client
per tenant"}
  RES -- "stub" --> KW["Keyword dispatch
~85 read tools
1 query per match"]
  RES -- "model · keyword" --> KW
  RES -- "model · agentic" --> LOOP["Agentic loop
model picks tools
up to 4 iterations"]
  KW --> ASM["Assemble prompt
system + tool-result JSON"]
  LOOP --> ASM
  ASM --> GEN["model.generate()
billed input + output"]
  GEN --> CITE{"Citations
validate?"}
  CITE --> AUD["Write ai_audit_entry
prompt · tools · answer · latency"]
  classDef app fill:#173a6b,stroke:#0f2a4f,color:#ffffff;
  classDef gate fill:#fdf0dd,stroke:#e0892a,color:#173a6b;
  classDef store fill:#e3f3f6,stroke:#1797b3,color:#173a6b;
  classDef ext fill:#ffffff,stroke:#9aa8c0,color:#173a6b;
  class GEN app;
  class WI,RES,CITE gate;
  class KW,LOOP,ASM,AUD store;
  class ASK,PROP ext;
  

One design rule keeps cost bounded: the AI only advises, it never makes a change itself. Anything it proposes is queued for a human to approve, which means the expensive open-ended "go fix it" agent loop simply does not exist here. The most a single turn can spend is fixed in advance by the four-step tool cap and the output-token ceiling.

3 What drives the token count

Tokens are the unit you pay for, so controlling the bill means controlling the tokens. Five things set the size of a turn. The system decides the first three; the user shapes the last two.

The takeaway for a budget: a narrow question about one thing ("status of device X") is a small, predictable turn. A broad question across the whole fleet ("summarize every compliance gap") is the expensive one, because it drags a large set of results into the prompt. Output is capped, so input is where a question gets expensive.

4 Defaults, model, and provider switching

The AI layer can swap between providers through LangChain4j (0.36.2), so you are not locked to one vendor. The deployment-wide default, plus any per-customer override, decides which model id applies, and therefore which price list. All values below are read from the shipping configuration.

Provider switching. A customer (tenant) can bring their own account and endpoint. The resolver (ai/LlmClientResolver.java) reads that customer's ai_settings (provider, model, base URL, temperature, max-tokens, encrypted key) and builds a matching client against an Anthropic, OpenAI, or any OpenAI-compatible endpoint. The deployment-wide default applies when a customer has no settings of their own. This matters for cost ownership: spend lands on, and can be capped by, the customer whose key served the call.

5 The cost levers

Every setting that moves the bill, roughly in order of how much it matters. All are configuration; none require a code change.

6 The stub path: AI off, zero spend

When the paid AI is off, the platform does not go dark. The default stub provider (ai/StubLlmClient.java) is a fixed renderer: it runs the same read tools, then formats the results into plain markdown without ever calling a model. No model call means no tokens and no spend.

7 Audit, attribution & chargeback

Every turn that runs is recorded, which is what lets you manage AI spend rather than treat it as an opaque line item. The AiAuditEntry row (table ai_audit_entry) is written for every turn in AssistantService and is scoped to one customer (tenant).

8 The wider REST surface that feeds the model

The AI does not run on its own. Every tool it uses is a real call into the same web service the operators use, and those calls carry their own cost. That cost is infrastructure (servers you already own), not a per-use vendor charge. This surface is what makes one AI question cheap and another heavy: the difference is which endpoints the tools hit underneath. The surface is roughly 640 endpoints across 115 controllers. By method, that is about 242 GET (reads), 165 POST, 68 DELETE, 34 PATCH, and 18 PUT. Reads make up most real traffic.

Four things set an endpoint's infrastructure cost: how many trips it makes to the database, how fast that work grows as the fleet grows, whether it reaches a heavy subsystem (Batfish, the model, live SNMP/SSH, or flow processing), and whether the result is cached (saved and reused). The first three set the raw cost; the cache divides it down. The very-heavy tier reaches out to an outside process, and the AI model is one of those.

flowchart LR
  REQ["API request"] --> F["Auth + tenant filter"]
  F --> SVC["Controller → service"]
  SVC --> DB[("PostgreSQL
tables × rows")]
  SVC --> HVY{"Heavy
subsystem?"}
  HVY -- "Batfish RPC" --> BF["formal model
multi-second"]
  HVY -- "AI model" --> LLM["metered tokens
1–5 s/turn"]
  HVY -- "live device" --> SNMP["SNMP / SSH
2–5 s/device"]
  SVC --> CACHE{"Cache hit?"}
  CACHE -- "yes (90s TTL)" --> FAST["served from memory
cost ÷ many"]
  classDef app fill:#173a6b,stroke:#0f2a4f,color:#ffffff;
  classDef gate fill:#fdf0dd,stroke:#e0892a,color:#173a6b;
  classDef store fill:#e3f3f6,stroke:#1797b3,color:#173a6b;
  classDef ext fill:#ffffff,stroke:#9aa8c0,color:#173a6b;
  class LLM,BF app;
  class HVY,CACHE gate;
  class DB,FAST store;
  class REQ,F,SVC,SNMP ext;
  

9 The heaviest endpoints behind a turn

When the AI answers a fleet-wide question, the tools it runs can land on the endpoints below. The top of the list is the AI call itself; the rest are the heavy reads it may pull data from. These are the ones to measure and protect first.

10 Caching, the non-AI cost multiplier

The fleet-wide reads all work the same way: a short-lived result cache that computes once and shares the result. The first caller does the work; any other callers that arrive during the window wait on that one computation instead of each launching their own. When the cache already has the answer, a heavy endpoint serves it from memory in single-digit milliseconds.

flowchart LR
  C1["caller 1"] --> SF{"in flight?"}
  C2["caller 2"] --> SF
  C3["caller 3"] --> SF
  SF -- "no" --> COMP["compute once
fleet-wide read"]
  SF -- "yes" --> WAIT["wait on the
one computation"]
  COMP --> CACHE[("90s TTL cache")]
  WAIT --> CACHE
  CACHE --> SERVE["serve all callers
for the window"]
  classDef app fill:#173a6b,stroke:#0f2a4f,color:#ffffff;
  classDef gate fill:#fdf0dd,stroke:#e0892a,color:#173a6b;
  classDef store fill:#e3f3f6,stroke:#1797b3,color:#173a6b;
  classDef ext fill:#ffffff,stroke:#9aa8c0,color:#173a6b;
  class COMP app;
  class SF gate;
  class CACHE,SERVE store;
  class C1,C2,C3,WAIT ext;
  

Where caching is missing matters too: the topology graph/diagram, imports, and committing observations to the source of truth are not cached, because each result is specific to one request and cannot be reused. Those are protected instead by per-endpoint rate limits and a separate pool for heavy work, so they cannot swamp the rest of the system. None of this is metered spend; it is the infrastructure cost behind a free (stub) or paid (model) AI answer alike.

A Appendix, AI cost configuration reference

Environment variables drive the AI cost settings. The defaults favor zero spend, and turning AI on and choosing a model are both explicit, deliberate steps. Every key and default below is read from application.yml and the client code.

B Appendix, endpoint cost tiers

A few example endpoints in each tier, to show the non-billed cost that sits behind any AI turn. The rest of the surface follows these same patterns.