MiBeeSteward v0.2.0 Internals: Distributed Consistency, Anti-Entropy, and the Change-Detection Engine

The previous post covered what v0.2.0 ships. This one covers why it’s designed that way — distributed consistency is the heaviest part of this release, and several seemingly casual decisions have specific trade-offs behind them.

The problem is clear: the center can only see its own LAN, but users have several. Dropping a center into every subnet isn’t realistic, so enter mibee-agent — a binary that only “scans local + reports to center.” But the moment an agent exists, it drags along a whole string of protocol questions: how to report, how to reconcile, how to detect offline, how not to overwhelm the center’s writes. Let’s take them one at a time.

The Agent ↔ Center Protocol

The protocol picks a push-report + pull-command hybrid. The agent is the only initiator; the center is entirely passive — that’s how it crosses NAT: the agent makes outbound HTTPS from inside, and the center opens no inbound port.

sequenceDiagram
    participant A as mibee-agent
    participant C as center

    A->>C: POST /agents/report<br/>(alive hosts + X-Network-State-Hash)
    C-->>A: 200 OK

    loop every 60s poll
        A->>C: GET /agents/commands
        C-->>A: pending cmds (maybe empty)
        A->>C: POST /commands/{id}/ack
        Note over A: run scan<br/>15min hard deadline
        A->>C: POST /commands/{id}/complete
    end

    Note over A,C: center down →<br/>backfill queue (cap=100)<br/>replays in order

Two channels, two jobs: the report channel pushes alive hosts (every 30s, carrying a state hash — see the next section), and the command channel pulls ad-hoc scan commands (60s poll, with an ack → execute → complete loop). The center can trigger “have agent X scan now” from the Web UI; the command enqueues, the agent picks it up on the next poll.

Two engineering details are worth pulling out:

Hard scan deadline — agent-triggered scans cap at 15 minutes (or timeout×2+60s). This is scar tissue: without a deadline, one hung TCP read nails a command in “acknowledged” forever, and the agent looks alive but does nothing.

Disconnect backfill — while the center is down, the agent’s reports aren’t lost; they land in a bounded in-memory queue (capacity 100) and replay in order when the center recovers. Only past 100 do you drop the oldest. A short center restart won’t leave any network “missing.”

The agent config is minimal — three required fields:

1
2
3
4
5
6
7

# configs/agent.yaml
center:
  url: "http://center.example.com:8080"
  auth_token: "<agent-token-from-center>"   # 32 random bytes; center stores only SHA-256
  report_interval: "30s"
network:
  name: "lan-hq"        # logical network the agent covers; must match a network created on the center

The agent has no database, no Web UI, no user system — just scan + report. mibee-agent -version reports the tag; misconfiguration shows up in the logs.

New agent-related API endpoints (all under /api/v1):

The Anti-Entropy Fast Path

The agent pushes a report every 30s. If a network is quiet — no devices added, no IP churn — the alive set 30s ago and now are identical. If the center dutifully ran the full device bridge each time (fingerprint classify, infer type, write device rows), it would be re-solving the same problem on a loop.

v0.2.0’s answer is an anti-entropy hash fast path:

Every report carries an X-Network-State-Hash header — a SHA-256 over the alive set’s identity + classification fields. The center compares it to the last stored hash:

• Hash matches → skip the entire device bridge; only refresh each device’s lease timestamp (scan_snapshots.last_seen_at). This is the steady-state path for quiet networks, costing ≈ one SHA-256 compare + N timestamp writes.
• Hash differs → run the per-host device bridge: fingerprint classify, infer type/brand/model, emit device_added / device_changed events as needed.

Why hash “identity + classification fields” rather than the whole device record? Because some fields naturally vary (RTT, last_seen_at), and folding them into the hash would leave it permanently “different,” voiding the fast path. Only fields whose change means a real change get hashed — that’s what makes a quiet network actually quiet.

The design borrows from Dynamo-style anti-entropy but is heavily simplified — no Merkle trees, no vector clocks. That’s because here the “truth” lives entirely on the agent side (it scans its local LAN); the center isn’t merging multiple replicas, it’s just deciding “do I need to recompute?” One hash compare is enough.

Dual-Track Offline Detection

Offline detection is where distributed systems most often grow bugs. v0.2.0 has two independent offline paths, but both emit the same device_lost event — downstream consumers don’t care which one fired.

The center’s own network uses consecutive-scan detection: a device missing from a scan round gets miss_count +1; only after 2 consecutive misses is it marked offline. This logic existed in v0.1.0 and fits the “center scans on a schedule” model — the cadence is known, so miss_count is meaningful.

flowchart TB
    SCAN["each scan round"] --> MISS["miss_count +1"]
    MISS -->|"miss_count < 2"| ONLINE["still online"]
    MISS -->|"miss_count ≥ 2"| EV1["device_lost"]

    classDef proc fill:#FFF3E0,stroke:#E65100,color:#BF360C
    classDef ok fill:#E8F5E9,stroke:#2E7D32,color:#1B5E20
    classDef lost fill:#FFEBEE,stroke:#F44336,color:#C62828
    class SCAN,MISS proc
    class ONLINE ok
    class EV1 lost

Each missed scan round increments miss_count; below the threshold (< 2) the device stays online, at the threshold (≥ 2) a device_lost event fires.

Agent-managed networks use lease detection: each report refreshes every device’s lease, the TTL defaults to 5 minutes, and a background LeaseSweeper runs every 60s marking leases past TTL as offline. At a 30s report interval, that’s roughly 10 consecutive missed reports before offline.

flowchart TB
    RPT["report every 30s"] --> LEASE["refresh lease<br/>TTL 5min"]
    LEASE --> SWEEP["LeaseSweeper<br/>every 60s"]
    SWEEP -->|"lease valid"| ONLINE2["still online"]
    SWEEP -->|"lease expired"| EV2["device_lost"]

    classDef proc fill:#FFF3E0,stroke:#E65100,color:#BF360C
    classDef ok fill:#E8F5E9,stroke:#2E7D32,color:#1B5E20
    classDef lost fill:#FFEBEE,stroke:#F44336,color:#C62828
    class RPT,LEASE,SWEEP proc
    class ONLINE2 ok
    class EV2 lost

Each report refreshes the device lease; the LeaseSweeper checks every 60s — while the lease is within TTL the device stays online, and once it expires a device_lost event fires.

Why not unify on one? Because the two scenarios have different signal sources:

• Center self-scan: the signal is “was it scanned this round,” naturally discrete per-round, and miss_count maps directly to “how many rounds in a row missed.”
• Agent report: the signal is “how long since the last report.” An agent can go fully silent (it crashed, the network dropped), and there’s no concept of “a round” anymore — only a time window. Forcing miss_count here means defining what “a round” is, then having the center wait dumbly for “the next round” while the agent is dead — detection lags and the semantics get fuzzy.

The lease model is more honest: it admits “I only know when I last saw this device,” and once the TTL passes, it’s expired. LeaseSweeper is a simple background goroutine doing DELETE FROM scan_snapshots WHERE last_seen_at < now() - ttl every 60s, O(number of expired devices).

Both paths write to the same change_log table and emit the same device_lost event — the Changes page and SSE stream see no difference.

The Change-Detection Engine

v0.1.0 only recorded current device state; v0.2.0 adds change detection. The core is a Watcher running in-process on the center, landing three event types in a change_log table.

flowchart TB
    SRC["scan result<br/>/ passive discovery"] --> DIFF["Watcher<br/>diff before vs after"]
    DIFF --> EVENTS["device_added<br/>device_changed<br/>device_lost"]
    EVENTS --> LOG[("change_log table")]
    LOG --> OUT["GET /changes<br/>SSE live stream<br/>Prometheus counter"]

    classDef src fill:#E3F2FD,stroke:#1565C0,color:#1565C0
    classDef proc fill:#FFF3E0,stroke:#E65100,color:#BF360C
    classDef ev fill:#FFEBEE,stroke:#F44336,color:#C62828
    classDef store fill:#E8F5E9,stroke:#2E7D32,color:#1B5E20
    classDef query fill:#F3E5F5,stroke:#9C27B0,color:#9C27B0
    class SRC src
    class DIFF proc
    class EVENTS ev
    class LOG store
    class OUT query

device_changed tracks 12 fields: name, type, brand, model, MAC, IP, status, open ports, detected services, Prometheus URL, node_exporter URL, scan attributes. Each event stores a structured before→after diff, not a full snapshot — you read “IP changed from .143 to .144” directly, without diffing two full records yourself.

A device_changed event’s JSON looks like:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

{
  "device_id": 42,
  "type": "device_changed",
  "network_id": 3,
  "changes": {
    "ip": { "before": "10.0.0.143", "after": "10.0.0.144" },
    "status": { "before": "online", "after": "online" }
  },
  "captured_at": "2026-07-13T02:14:31Z"
}

Three consumption paths: GET /api/v1/changes for paginated queries (filter by network + type), GET /api/v1/changes/watch for a live SSE stream (with keepalive — the browser Changes page auto-updates once connected), and the mibee_changes_total{type=...} Prometheus counter at /metrics. change_log retains 30 days by default, tunable via retention.change_log_days.

The Watcher runs only on the center — the agent does no change detection; it only reports the raw alive set. Diffing is centralized on the center to avoid multi-source judgment conflicts.

The Data-Driven Fingerprint Engine

In v0.1.0, device-identification rules were if and switch statements in Go; adding a signature meant changing code, recompiling, cutting a release. v0.2.0 turns identification rules into data — YAML files loaded at startup by a RuleClassifier.

flowchart TB
    RULES["YAML rules<br/>builtin + Recog"] --> LOAD["RuleClassifier<br/>loads at startup"]
    EVID["scan evidence<br/>SNMP/HTTP/TLS/banner"] --> MATCH["rule match"]
    LOAD --> MATCH
    MATCH -->|"declarative rule hits"| OUT["brand/model/type"]
    MATCH -->|"needs composite logic"| GO["Go fallback classifier"]

    classDef data fill:#E3F2FD,stroke:#1565C0,color:#1565C0
    classDef proc fill:#FFF3E0,stroke:#E65100,color:#BF360C
    classDef out fill:#E8F5E9,stroke:#2E7D32,color:#1B5E20
    classDef fallback fill:#F3E5F5,stroke:#9C27B0,color:#9C27B0
    class RULES,EVID data
    class LOAD,MATCH proc
    class OUT out
    class GO fallback

Leave the rule path empty to use the rules embedded in the binary (zero-config), or point it at an external directory for hot-reload. Out of the box: ~2554 rules — 20 hand-authored builtins + ~2534 imported from Rapid7’s Recog corpus (Apache-2.0, license-clean). nmap’s NPSL is deliberately not imported.

A Recog-style YAML rule looks roughly like this (simplified):

1
2
3
4
5
6
7

- fingerprint:
    os: "Ubuntu"
    os_version: "<%= data.fetch('version') %>"
  match: "Ubuntu(?:-)?(\\d+(?:\\.\\d+)*)"
  params:
    - match_group: 1
      name: version

The key engineering constraint: not all logic can be made declarative. A few composite judgments stay in Go, explicitly documented — the SNMP bitmask device-type heuristic (some devices encode type in specific bit combinations of an OID, not expressible as a regex), camera cross-evidence fusion (merging RTSP + ONVIF + SNMP evidence by confidence), and the database/remote-access byte-offset classifiers (reading specific response bytes).

To guarantee that “data-ifying” didn’t silently change behavior, there’s a parity test: it asserts the data-driven RuleClassifier produces byte-identical output to the hand-written classifiers it replaces. Any behavioral drift from a rule migration gets caught in CI. The standalone engine lives in the mibee-fingerprints-go module, consumed as a normal Go dependency — meaning other projects can reuse the same rule library.

Third-party rule corpora convert via cmd/fpimport/; Rapid7 Recog and IEEE-OUI / IANA-PEN data tables are both supported.

Known Limitations

A few things deliberately not done, to set expectations:

• Single-instance SQLite — the center is single-instance; no multi-center clustering. Not planned until scale demands it.
• No alerting — MiBeeSteward produces “asset freshness”; alerting is left to Alertmanager / Uptime Kuma. /metrics and /sd already expose the data.
• The eBPF passive observer requires a special build (make build-with-ebpf) and runtime privileges (CAP_BPF / CAP_NET_ADMIN); the default binary ships a no-op stub.

Related Links

• MiBeeSteward GitHub
• MiBeeSteward v0.2.0 Release Notes
• Standalone fingerprint engine module (mibee-fingerprints-go)
• v0.2.0 feature overview (promo post)

The distributed part of v0.2.0 really rests on one idea: the agent is the source of truth, the center is the aggregator, and the hash fast path makes quiet networks nearly free. Everything else — leases, dual-track offline detection, the change engine — is engineering follow-through around that core idea. If you’re building a similar multi-site discovery system, these trade-offs are worth borrowing.