MiBeeSteward v0.2.0: Distributed Discovery + Change Detection + Topology + Fingerprint Library

When v0.1.0 shipped, I wrote that it answers three questions: what devices are on the network, what they are, and how they relate. Honestly, v0.1.0 only answered the first half well — and only for the one LAN the center sat on.

The real-world layout is messier: one subnet in the office, one in the server room, another at home for test machines. The center lives in the office and literally cannot see the cameras in the server room. Getting it to scan the server room means crossing subnets — either opening up SNMP routing or just moving the center over there. Neither is elegant.

The stingier problem: v0.1.0 only records state, it doesn’t tell you what changed. The NAS plugged in last night, the Raspberry Pi that swapped IPs during the day, the switch that went offline last week — all drowning in the device list, waiting for you to eyeball-diff them.

v0.2.0 exists for exactly those two pains. One line: turn MiBeeSteward from a single-network discovery tool into a distributed, multi-LAN device portrait system. Full release notes at GitHub Release v0.2.0.

It ships two binaries this time — mibee-steward (center + Web UI) and mibee-agent (remote LAN discovery). Both pure Go, CGO-free, runnable as a regular user.

Distributed Discovery: Center + Agent

The headline change is the new mibee-agent binary. It runs on the remote LAN, scans the local network with the same scannerv2 engine as the center, and reports results back.

The design picks a pull model — the agent initiates every connection, so the center never needs to open an inbound port. This is friendly to home networks and cross-NAT setups: install the agent on the inside, let it make outbound HTTPS to the center, and punch zero firewall holes.

sequenceDiagram
    participant A as mibee-agent<br/>(remote LAN)
    participant C as center<br/>(Web UI + API)

    A->>C: POST /agents/report<br/>(alive hosts + state hash)
    C-->>A: 200 OK

    loop every 60s
        A->>C: GET /agents/commands
        C-->>A: pending scan cmds (maybe empty)
        A->>C: POST .../commands/{id}/ack
        A->>C: POST .../commands/{id}/complete
    end

The agent pushes an alive-host report every 30s and polls the center for ad-hoc scan commands every 60s. On receipt, the center converts the report into local device portraits via the device bridge. Agent-managed networks are excluded from the center’s own cross-subnet probing — the agent’s report is itself the liveness signal, so re-scanning would be redundant.

Agents page — agent token create/revoke (plaintext shown once), command history, manual scan trigger. Each token is bound to a network_id + agent_id; the center derives the network from the token, so a misconfigured agent can’t stamp devices onto the wrong network.

Change Detection Engine

This is the feature I personally wanted most. v0.1.0 recorded device state but computed no diff; v0.2.0 closes that gap. Three event types land in a new change_log table — device_added (new host appeared), device_changed (a known device’s identity fields shifted), device_lost (a device went offline).

device_changed tracks 12 fields: name, type, brand, model, MAC, IP, status, open ports, detected services, Prometheus URL, node_exporter URL, scan attributes. Each event stores a structured before→after diff, not a full snapshot, so it reads cleanly.

Query via GET /api/v1/changes (paginated, filterable by network + type), or stream live over GET /api/v1/changes/watch SSE with keepalive. Prometheus sees a mibee_changes_total{type=...} counter too.

Changes page — change history list; click any row to see the structured before→after diff. Once SSE connects, new events stream in without a refresh.

Passive Discovery

Scheduled scans have a natural blind spot: a host that boots between two scans waits until the next round to be found. v0.2.0 adds a long-running passive discovery service that watches for newly-appeared hosts at near-zero traffic and feeds them through the same device bridge. A device spinning up at 02:00 still gets a device_added event without waiting for the next scan.

flowchart TB
    SRC["Router ARP / cache<br/>mDNS + SSDP multicast"] --> DEDUP["5-min dedup"]
    DEDUP --> PRE["MAC precheck<br/>known host?"]
    PRE -->|"unknown MAC"| BRIDGE["device bridge<br/>trigger identify scan"]
    PRE -->|"known MAC"| SKIP["refresh only<br/>no rescan"]

    classDef src fill:#E3F2FD,stroke:#1565C0,color:#1565C0
    classDef proc fill:#FFF3E0,stroke:#E65100,color:#BF360C
    classDef out fill:#E8F5E9,stroke:#2E7D32,color:#1B5E20
    class SRC src
    class DEDUP,PRE,BRIDGE proc
    class SKIP out

Three sources, each independently toggleable: router SNMP ARP tables (widest cross-subnet coverage), the local ARP cache (reads /proc/net/arp, zero network traffic), and multicast listening (mDNS :5353 + SSDP :1900, listen-only, no queries sent — picks up cameras, printers, IoT, Macs, UPnP devices).

A key optimization is the MAC-primary known-host precheck: a device that drifts from .143 to .144 during DHCP renewal is recognized by MAC, skipping an expensive identify-scan loop. A 5-minute dedup window suppresses bursts. Runtime counters and the last 20 discoveries are exposed at GET /api/v1/discovery/status, with a dedicated Discovery page on the frontend.

Discovery page — passive-discovery runtime funnel (per-source hits, dedup count, identify-scan triggers), with the last 20 discoveries below.

Bridge-MIB Topology & New Probes

v0.2.0 starts laying groundwork for L2 topology. The new Bridge-MIB neighbor probe walks a switch’s dot1dTpFdbAddress / dot1dTpFdbPort / dot1dBasePortIfIndex to learn which MAC sits behind which switch port, persisting results to a new device_neighbors table. This is the foundation for a topology graph; LLDP/CDP is reserved for later.

flowchart TB
    SW["Switch<br/>SNMP reachable"] --> FDB["Walk BRIDGE-MIB<br/>FDB table"]
    FDB --> ADJ["L2 adjacency<br/>MAC ↔ port"]
    ADJ --> DB[("device_neighbors<br/>protocol=Bridge-MIB")]

    classDef dev fill:#E3F2FD,stroke:#1565C0,color:#1565C0
    classDef proc fill:#FFF3E0,stroke:#E65100,color:#BF360C
    classDef store fill:#E8F5E9,stroke:#2E7D32,color:#1B5E20
    class SW dev
    class FDB,ADJ proc
    class DB store

The active probe set also grows from 13 in v0.1.0 to 15, and the new additions are pointed:

iStoreOS is the interesting one — it puts the product name in the cert CN and “OpenWrt” in the issuer org, so you need both fields to identify it accurately.

Fingerprint Rule Library: Data-Driven

In v0.1.0, device-identification rules were hardcoded in Go; adding a signature meant changing code and recompiling. v0.2.0 turns identification rules into data — YAML files loaded at startup by a RuleClassifier.

flowchart TB
    YAML["YAML rule files<br/>or embedded in binary"] --> LOAD["RuleClassifier<br/>loads at startup"]
    LOAD --> MATCH["Evidence match<br/>brand/model/type"]
    MATCH --> PORTRAIT["Device portrait"]

    classDef data fill:#E3F2FD,stroke:#1565C0,color:#1565C0
    classDef proc fill:#FFF3E0,stroke:#E65100,color:#BF360C
    classDef out fill:#E8F5E9,stroke:#2E7D32,color:#1B5E20
    class YAML data
    class LOAD,MATCH proc
    class PORTRAIT out

Adding a device signature = one YAML entry, no Go code touched. Leave the rule path empty to use the rules embedded in the binary (zero-config), or point it at an external directory for easy hot-reload.

The out-of-the-box rule count is substantial: ~2554 rules — 20 hand-authored builtins + ~2534 imported from Rapid7’s Recog corpus (Apache-2.0). nmap’s NPSL is deliberately not imported (license-clean only). The standalone engine now lives in the mibee-fingerprints-go module, consumed as a normal dependency.

A handful of logic that can’t be expressed as a single declarative rule stays in Go (documented as such): the SNMP bitmask device-type heuristic, camera cross-evidence fusion, and the database/remote-access byte-offset classifiers. A parity test asserts the data-driven classifier produces byte-identical output to the hand-written classifiers it replaces.

Management UI

The frontend gains 4 new pages and 2 reusable components:

• Networks (admin) — create/edit/delete logical networks that agents bind to. Previously required direct SQLite access.
• Discovery — passive-discovery runtime funnel + recent discoveries.
• Changes — change history with structured before/after diffs + live SSE updates.
• Agents (admin) — agent token CRUD (one-time plaintext reveal) + command history + scan trigger.
• ColumnPicker component — the Devices page lets each user toggle optional columns (vendor, MAC, hostname, RTT, inferred type, OS, …), persisted to localStorage.
• ChangeDiff component — renders added/lost/changed events in the right shape each.

Upgrade

The center is a drop-in upgrade. New tables (change_log, agent_tokens, scan_snapshots, agent_commands, device_neighbors, topology_edges, subnets) are all created via CREATE TABLE IF NOT EXISTS — no manual migration, no new mandatory center config key, existing device data untouched.

The agent is optional; standing up a remote LAN probe takes five steps:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

# 1. Download the agent binary to the remote host
wget https://github.com/Mi-Bee-Studio/MiBeeSteward/releases/download/v0.2.0/mibee-agent-linux-amd64
chmod +x mibee-agent-linux-amd64

# 2. Install the systemd user unit (user-level, needs loginctl enable-linger)
cp deploy/mibee-agent.service ~/.config/systemd/user/

# 3. Create a network + agent token on the center (Networks → Agents pages)

# 4. Write configs/agent.yaml (three required fields)
# center.url / center.auth_token / network.name

# 5. Start — reports begin within ~30s
systemctl --user start mibee-agent

Bug Fixes & Operational Improvements

• CSRF-safe exports: CSV/JSON downloads now route through the API client — previously bypassed it via raw fetch, dropping the CSRF header (a real auth bug).
• Server bind-retry: on systemd restart, if the previous socket is in TIME_WAIT, retries binding for up to 30s instead of crash-looping.
• Agent HTTP keep-alive deadlock fix: pooled keep-alive connections that went half-open after a center restart no longer block reads for minutes (IdleConnTimeout=10s, bounded dialer); also enforces the scan deadline.
• golangci-lint v2: v0.1.0’s release never ran lint at all; CI now enforces the full v2.12.2 rule set, codebase is clean.
• Retention config consolidated under retention.* (heartbeat 7d, scan results 30d, audit 90d, change_log 30d — per-table configurable).

Related Links

• MiBeeSteward GitHub
• MiBeeSteward v0.2.0 Release Notes
• v0.1.0 intro post
• Technical deep-dive (anti-entropy protocol, dual-track offline detection, change engine internals)

The v0.2.0 tagline I wrote is “Distributed Discovery · Topology · Change Detection · Fingerprint Library.” If you ran v0.1.0 across multiple subnets, I’d strongly suggest upgrading and trying the agent — the moment a remote LAN comes online, the “finally seeing the whole picture” feeling is very real.

The project is under the PolyForm Noncommercial License (free for non-commercial use). Ideas or issues are welcome in the Issue tracker.