Data Collection Reference

0 How to read this document

Every collector below is a real protocol implementation, not a mock-up. Each section names the mechanism rather than an adjective: the OID or column walked, the port bound, the library and pinned version, the staging entity written, and the canonical record it becomes. Collectors that are opt-in or off by default are labelled, so you can tell which sockets a stock deployment opens.

1 How collection works

CrossConnect reads, it does not intercept. Every input is either a control-plane query the network already answers, an announcement the gear already broadcasts, configuration text, or a record an operator or another system hands in. Nothing is sniffed off the wire: there is no packet capture, no SPAN or mirror feed, and no payload inspection anywhere in the platform.

Four families of collector feed one pipeline. Each writes to staging, an append-only set of discovered_* observations. Validation then confidence-scores each staged observation and commits it into the source of truth, the canonical store every read, view, and AI answer resolves against. The derived layer sits on top: it computes scores and rollups from that source of truth and writes no new facts of its own.

flowchart LR
  subgraph COL["FOUR COLLECTOR FAMILIES"]
    direction TB
    A["Active discovery
SNMP · LLDP · ICMP"]
    P["Passive listeners
flow · mDNS · DHCP · traps"]
    C["Config & formal model
SSH · Batfish"]
    X["Application input
REST · CSV · inbound API"]
  end
  STG[("Staging
append-only discovered_*")]
  G{"Validation
confidence-score · commit"}
  SOT["Source of truth
canonical records"]
  AUD[("Audit chain
hash-linked · HMAC")]
  DER["Derived layer
pure functions"]
  A --> STG
  P --> STG
  C --> STG
  X --> STG
  STG --> G
  G --> SOT
  SOT --> AUD
  SOT --> DER
  classDef app fill:#173a6b,stroke:#0f2a4f,color:#ffffff;
  classDef store fill:#e3f3f6,stroke:#1797b3,color:#173a6b;
  classDef gate fill:#fdf0dd,stroke:#e0892a,color:#173a6b;
  classDef ext fill:#ffffff,stroke:#9aa8c0,color:#173a6b;
  class A,P,C,X,SOT,DER app;
  class STG,AUD store;
  class G gate;
  

2 The SNMP sweep: session & credentials

SNMP is the Simple Network Management Protocol, the standard way to read facts off network gear. The discovery worker (crossconnect.discovery.enabled, default false; set source=snmp to drive real SNMP rather than the seed source) queries each managed device over UDP/161 on a schedule. The session is built by SnmpDiscoverySource on snmp4j 3.8.2, and the per-device probe methods live in SnmpProbe. The session details a security reviewer or a network engineer wiring up an account will need:

sequenceDiagram
  participant SCH as Scheduler
(interval-ms)
  participant SRC as SnmpDiscoverySource
  participant DEV as Managed device
SNMP agent
  participant PRB as SnmpProbe
  participant STG as DiscoveryStagingService
  SCH->>SRC: open tenant sweep
  SRC->>SRC: build read-only target
(pinned credential first)
  SRC->>DEV: GET system scalars (161/UDP)
  DEV-->>SRC: sysDescr · sysName · sysUpTime
  SRC->>PRB: probeAll(session, device)
  loop each MIB family (best-effort)
    PRB->>DEV: GETBULK walk (maxRows 20)
    DEV-->>PRB: varbinds → typed Fact records
  end
  PRB-->>SRC: SnmpSweepResult
  SRC->>STG: stage() inserts discovered_* rows
  Note over STG: append-only · stamped observedAt
nothing in source of truth changes yet
  

3 SNMP inventory & topology MIBs

A MIB (Management Information Base) is the catalog of values a device exposes over SNMP. The inventory probes answer the basics: what is this device, what are its ports, and how is it wired. Each writes a single staging entity per natural key per sweep. The literal OID roots are in Appendix A.

4 SNMP routing, multicast, timing, environment

On the same session, discovery walks the more specialized MIBs below: routing, multicast, clock timing, and physical health. All are best-effort, so a device that does not support a given MIB is skipped for that one. Reachability is handled separately by an ICMP (ping) probe.

5 Configuration over SSH & the formal model

Collecting a device's running configuration unlocks the formal-analysis layer, where CrossConnect reasons about the config itself. This collection is opt-in (crossconnect.discovery.collect-config, default false); the bean only loads when the flag is true, so a default deployment never opens an SSH session. The collector is SshConfigCollector on sshj 0.38.0: it opens a read-only interactive shell, turns off paging, issues one read-only show command, captures the text, and exits. It never issues a configure or any other command that would change device state.

VendorCliProfile.forVendor() matches the device vendor slug without caring about case. The SSH connect and read timeouts are both 20 seconds. Host-key verification accepts the device key for a read-only collection session, and passwords are decrypted in memory only for the duration of that session.

6 Cloud-managed sources (vendor REST)

Some networks are run from a vendor's cloud dashboard rather than managed device by device. For those, CrossConnect pulls inventory from the vendor's dashboard API over HTTPS instead of walking SNMP. CloudVendorSourceService turns on per tenant once a base URL and bearer token are configured. It reads only the documented dashboard endpoints below, and the token is stored with AES-256-GCM and decrypted in memory only for the duration of the pull.

7 Passive listeners

Passive listeners query nothing. They listen for traffic summaries and the announcements gear already broadcasts on its own, and none of them looks inside a packet payload. Each listener is off by default (its enabled flag is false) and binds no socket until it is enabled and pinned to a tenant UUID, so the platform adds no network surface unless an operator turns one on.

8 Application & integration input

Not every input comes off the wire. Operators and other systems hand records in directly. Manual entry is the one path that writes the source of truth without validation, because a person is treated as the authority. Everything else arrives as an observation and is validated like anything from a collector.

Outbound paths (signed webhooks and SIEM / chat sinks) send data out rather than take it in, and they are covered in the Security & Architecture reference.

9 The staging entities

Every collector lands in a discovered_* staging table (plus the health and inbound tables). Each row carries id, tenantId, observedAt, and, where it applies to a device, a deviceId. Rows are only ever inserted, never updated, and the newest observedAt per natural key is the one that counts. The set written by the collectors in this document, with the distinctive fields each one carries:

10 From signal to truth: validation

No matter where it came from, every collector above feeds the same pipeline under the same rules. Validation is the step that decides which observations become trusted records.

flowchart LR
  O["1 · Observe
discovered_* · untrusted"] --> G{"2 · Validate
confidence-score · commit"}
  G --> S["3 · Documented
source of truth"]
  S --> A[("4 · Audit
hash-linked entry")]
  S --> D["5 · Derived
pure functions"]
  classDef gate fill:#fdf0dd,stroke:#e0892a,color:#173a6b;
  classDef truth fill:#173a6b,stroke:#0f2a4f,color:#ffffff;
  classDef store fill:#e3f3f6,stroke:#1797b3,color:#173a6b;
  class G gate;
  class S,D truth;
  class O,A store;
  

• Staging is append-only. Each run writes a fresh row per natural key (for example device plus ifIndex), and the newest observedAt is the one that counts, so re-running a sweep is safe and never duplicates. Old staging rows are cleared automatically once they pass the retention window (discovery.staging.retention-days, default 14) by a scheduled sweep.
• Validation is the trust gate. ValidationService confidence-scores staged observations based on how many sources agree: two independent sources agreeing within the 24-hour window score Confirmed, a single source scores Inferred, and one that resolves to nothing scores Unconfirmed. Once an observation earns enough confidence it is committed, and that is what creates the canonical record. An observation never overwrites a record silently.
• The source of truth is hash-linked. Every commit and edit publishes a RecordChangeEvent on the EventBus and is captured into a tamper-evident audit chain: contentHash = SHA-256(tenantId · kind · occurredAt · actor · payload · previousHash), then signed with HMAC-SHA256. Because each entry chains to the one before it, the history of how a record reached its current state can be proven. A tenant's whole chain can be checked end to end (EventAuditService.verifyChain(tenantId)), and it is kept on its own retention policy (audit.retention-days, default 90).
• The derived layer reads, it does not collect. Data-quality, compliance, readiness, reachability, and the AV and presence views are all computed purely from the source of truth, and they store no new facts of their own.

11 Worked example: one link, wire to map

Take one fact and follow it the whole way through. Switch acc-sw-3 port Gi1/0/14 is patched to core-sw-1 port Gi1/0/1. Every signal in this document goes through the same five moves, read, stage, validate, commit, prove. This traces one of them in full, with the exact OIDs, staging tables, services, and records.

1. Trigger. The discovery scheduler (interval-ms, default about 5 minutes) opens a tenant sweep and, for each documented device, has SnmpDiscoverySource build a read-only snmp4j session to the management IP on UDP/161 with the pinned credential. No agent is installed and no port is opened on the device.
2. Read the wire. On acc-sw-3 the sweep walks IF-MIB and LLDP-MIB. From ifTable / ifXTable it reads ifName (Gi1/0/14), speed, MAC, and admin and oper status. From lldpRemTable it reads, on local port Gi1/0/14, the remote lldpRemSysName (core-sw-1) and lldpRemPortId (Gi1/0/1), with chassis and port subtypes. When the sweep later reaches core-sw-1 it reads the mirror-image neighbour, so the one link is observed independently from both ends.
3. Parse. SnmpProbe decodes the raw varbinds into a SnmpSweepResult: a list of SnmpInterfaceFact and SnmpLldpNeighbor records, each stamped with tenantId, the resolved deviceId, and observedAt.
4. Stage, append-only. DiscoveryStagingService inserts a DiscoveredInterface row for acc-sw-3 / Gi1/0/14 and a DiscoveredNeighbor row (local Gi1/0/14 to remote core-sw-1 / Gi1/0/1). Rows are inserted, never updated; the newest observedAt per natural key is operative, so the next sweep is idempotent. Nothing in the source of truth has changed yet: the link is observed, not documented.
5. Validate, the trust gate. ValidationService confidence-scores the staged neighbour. It resolves both endpoint names, and since both are managed devices this is a candidate cable between two known ports. It checks whether a Cable already records the pair; if not, the link is queued to commit. Because both switches reported the same adjacency, the link scores Confirmed. (A one-sided LLDP sighting would score Inferred, and a neighbour whose name resolves to no device would score Unconfirmed and be flagged as a possible rogue.) The link then waits in the validation queue with its evidence and score, still outside the source of truth.
6. Commit. Once the link has earned enough confidence, commit() is the only write path from observation into truth. It find-or-creates the two Interface endpoints and writes one Cable joining acc-sw-3:Gi1/0/14 and core-sw-1:Gi1/0/1, stamping discoveredAt. The observation is now a record.
7. Record and prove. The write publishes a RecordChangeEvent on the EventBus. The audit plugin captures it into the tamper-evident chain, contentHash = SHA-256(tenantId · kind · occurredAt · actor · payload · previousHash), HMAC-signed and linked to the previous entry for the tenant. The link is provable: who added it, when, and that the record has not been altered since.
8. Downstream updates, at no extra cost. Because the derived layer is computed purely from the source of truth, the new Cable immediately shows up on the topology graph and the network map; clears the data-quality "undocumented link" finding the LLDP sighting had raised while it was still staged; joins the Batfish topology input, so reachability checks now traverse it; and becomes citable by the AI assistant, which can reference the Cable record and its audit entry by primary key.

One LLDP value, read read-only off two switches, became a Confirmed, operator-approved, cryptographically recorded cable the whole platform now reasons over.

A Appendix, OID reference by probe

The literal OID roots each SNMP probe walks (GET for single values, GETBULK for tables). No state-changing OID set is ever issued, so collection can only read.

B Appendix, collector configuration reference

What gets collected is driven by configuration. Representative properties from application.yml follow. The defaults are deliberately quiet, with almost everything off until an operator turns on what they need: