MiBeeNvr v0.9.0 → v0.9.1: ONVIF Zero-Config Auto-Discovery + Storage Rewrite + Lock-Free Camera Manager + Optional FFmpeg + Full i18n Repair

Nine cameras had been running on a Banana Pi M5 for over six months, and the engineering debt accumulated since v0.6 had reached the point where it needed paying off. The /api/cameras endpoint would stall for 14.9 seconds while all 9 recorders started up — not because the traversal was slow, but because the cm.mu RWMutex was blocking, holding every read operation out. ONVIF camera addition still required typing in IP addresses manually, and with a household of CW500s, DS-2CD2047G2s, and others, every new camera meant opening a terminal session — not sustainable. FFmpeg as the only third-party binary dependency meant every deployment needed a check for whether it was installed, and on Raspberry Pi you also had to worry about hardware encoder compatibility. And the older Xiaomi devices — Dafang, Xiaofang, Aqara G2 — couldn’t be integrated at all because they speak the TUTK protocol. None of these are new technical challenges; they’re engineering debt, and v0.9 pays it off in one shot.

This release ships 54 commits across 9 merged PRs (#49–#61), covering ONVIF zero-config auto-discovery, a full storage-layer read/write split rewrite, a lock-free camera manager refactor, FFmpeg demoted to optional dependency, Xiaomi TUTK legacy protocol support, surveillance grid protocol auto-selection with ONVIF full automation, a data repair CLI, plus critical fixes and code normalization. Each theme is detailed below.

v0.9.0 was validated against 9 real cameras (CW500 dual-lens ×2, DS-2CD2047G2, Outdoor Cam 4, IMILAB A1, Dafang, Xiaofang, Aqara G2, Xiaomi Yunyi). One point worth emphasizing: this is the last stable/compatible baseline since v0.1.0. The next major release will introduce breaking changes — with precise migration paths, config deltas, and impact scope documented — no silent breakage. The Recordings Gallery Grid View will be removed (Timeline and List views will be strengthened). For long-term stable deployments, pinning the v0.9.x line is recommended.

A patch — v0.9.1 — followed the very next day (July 19). The trigger was a burst of community feedback arriving on release day, focused on i18n completeness, recording fragments, and a handful of UX bugs (see issues #64/#68/#70). These aren’t features that v0.9.0 left unfinished — they’re a few long-standing hardcoded strings and one regression that this regression cycle exposed. A dedicated section at the end covers the v0.9.1 fixes in detail. Full v0.9.0 changelog at v0.9.0 Release Notes; v0.9.1 at v0.9.1 Release Notes.

ONVIF Zero-Config Auto-Discovery

Before v0.9, adding an ONVIF device followed this workflow: find the IP → open Settings → manually enter the profile → Hail Mary. For 3 cameras that works fine. Past 5 it becomes real overhead. v0.9’s auto-discovery delivers Hikvision-level plug-and-play — power it on, no configuration needed, the NVR finds the device itself.

sequenceDiagram
    participant CAM as Camera (ONVIF)
    participant NVR as MiBeeNvr
    participant UI as Frontend
    participant DB as SQLite

    CAM->>NVR: WS-Discovery Hello<br/>UDP 3702 multicast
    NVR->>DB: Dedup (endpoint + stable_id)
    NVR->>NVR: Enrich (brand/model/serial)
    NVR->>UI: SSE camera.added
    UI->>UI: Toast + pending_activation badge
    UI->>NVR: POST activate (credentials)
    NVR->>CAM: GetStreamURI + RTSP DESCRIBE
    NVR->>DB: Update active status
    NVR->>UI: SSE camera.activated

Auto-discovery operates in two modes running in parallel. The passive mode listens for WS-Discovery Hello multicast messages on UDP 3702 (zero latency — the device is discovered the moment it comes online). The active mode runs a Probe sweep every 60 seconds (configurable, minimum 30s enforced by RPi-3B performance constraints). Both modes feed into a unified registration pipeline: dedup (by ONVIF endpoint + stable_id, covering all protocols including archived cameras) → information enrichment → classification (active / pending_activation) → persistence.

Devices discovered without credentials enter the pending_activation state. The frontend CameraCard shows a pending badge with an activate button, which opens a credential dialog that calls POST /api/cameras/{id}/activate. On successful activation, SSE emits a camera.activated event and the frontend auto-refreshes the list. Settings adds a new auto-discovery configuration sub-endpoint at /api/settings/auto-discover, with controls for scan interval, listen interface, default credentials (returns a has_default_password boolean, never the plaintext), and scope blacklist. A 10-second startup grace period prevents the NVR from missing devices that haven’t finished DHCP+ONVIF initialization.

Storage Layer Rewrite

PR #53 is the single largest PR in this release — 31 commits across 117 files and ~10,800 lines changed. The driving force was a set of storage bottlenecks that kept surfacing during v0.8 development: N+1 queries, full-table-scan COUNT(*), and read-side lock contention while recorders wrote.

flowchart TD
    W["Write Pool<br/>1 conn, serial"] --> DB[(SQLite<br/>WAL mode)]
    R["Read Pool<br/>3 conn, parallel"] --> DB
    R --> CACHE["COUNT cache<br/>2s TTL"]
    CACHE --> API["daily-summary API"]
    DB --> MERGE["RollingMerge<br/>post-merge verify"]

    classDef w fill:#FFF3E0,stroke:#E65100,color:#BF360C
    classDef r fill:#E3F2FD,stroke:#1565C0,color:#1565C0
    classDef s fill:#E8F5E9,stroke:#2E7D32,color:#1B5E20
    classDef m fill:#F3E5F5,stroke:#9C27B0,color:#6A1B9A
    class W w
    class R,CACHE r
    class DB,API s
    class MERGE m

The core change is a read/write split connection pool: the writer uses 1 serial connection, readers use 3 parallel connections (WAL mode supports concurrent reads), with query_only=1 on reader DSNs. The full-table-scan COUNT(*) OVER() on the write path had previously demonstrated severe performance regression, so it is replaced by a COUNT cache (2s TTL) plus batched operations. Redundant indexes get unconditional DROP, auto_vacuum incremental migration (v23), and query metrics instrumentation across all paths.

A new GET /api/recordings/daily-summary endpoint delivers calendar-aggregated data, decoupling the frontend calendar view from the Gallery data source so calendar navigation no longer blocks Gallery pagination. At startup, a merge integrity scan checks whether every entry marked as merged actually has a corresponding MP4 file on disk; entries with missing files have their merge_status reset so playback falls back to frames instead of returning a dead-end 404. The RollingMergeManager now performs post-merge verification — when a merge reports success but the output MP4 is missing, delete_original is blocked to prevent destroying the source frames. Live merge uses blocking lock + retry instead of the previous try-lock, reducing edge-case merge omissions.

A keyframe extractor fix that had been pending for a while falls out of the storage rewrite: certain Xiaomi H.265 cameras only send parameter sets in the SDP or moov box, so KeyframeExtractor couldn’t obtain SPS/PPS and permanent merge failure ensued. The fix uses CodecParamProvider to fetch parameter sets live from the running recorder, so H.265 merges no longer fail on these cameras.

Lock-Free Camera Manager Refactor

On a setup with 9 cameras, the v0.8 camera manager revealed a design flaw: the cm.mu RWMutex protected the entire manager state, and slow operations — recorder Start/Stop, ONVIF handshake, disk writes — all held the write lock. Every read operation, including GET /api/cameras and the grid’s latest-frame polling, had to wait. And this lock had been there since the original v0.1 design — it had never been questioned.

The refactor eliminates this lock entirely using Write-on-Copy (COW) atomic snapshots plus per-camera single-flight guards:

• snapshot atomic.Pointer[snapshot]: an immutable atomically-published view containing the complete snapshot of recorders, hubs, configs, and failedStarts.
• apply(fn): the single write path. configMu is held only for nanosecond-level map copying plus disk writes — never across I/O operations.
• withCameraLifecycle(id, fn): a per-camera sync.Mutex managed via sync.Map, serializing start/stop/restart within each camera.
• All read paths (GetRecorder/GetHub/Status/statusSnapshot/counts) are completely lock-free.

ONVIFRecorder.Start no longer holds r.mu during the handshake — the multi-second ONVIF+RTSP+HTTP triple handshake no longer blocks grid polling.

Production benchmarks (Banana Pi M5, 9 cameras):

Most of that 14.9s was not doing useful work — several recorder Starts ran serially, each doing an ONVIF handshake plus RTSP connection while the write lock held all other 8 recorders and every read path waiting. After switching to COW + per-camera guard, one camera’s handshake no longer drags down the entire system.

FFmpeg Demoted to Optional Dependency

FFmpeg was previously the NVR’s only third-party binary dependency, responsible for media probing, transcoding, and timelapse merging. v0.9 relegates it to an optional accelerator: all core NVR functionality (recording, playback, live streaming, relay, timelapse, merge) works without FFmpeg installed; only H.265↔H.264 pixel-level transcoding requires it.

A new internal/mediaprobe package provides pure-Go MP4 box metadata probing (codec/duration/resolution/frame-count), 10-100× faster than ffprobe -count_frames. transcoding.GetMediaInfo prefers mediaprobe with ffprobe fallback; cleanup.probeDuration follows the same pattern. The timelapse selectTier defaults to TierGo with TierFFmpeg as optional. Placeholder images for thumbnails use pure-Go generation when FFmpeg is absent.

Four real-camera-recording bugs were fixed along the way: G.711 ParseSegment failure (previously errored without graceful degradation), FFmpeg -c:a copy failing on G.711 (changed to -c:a aac), MJPEG backup directory residue causing duplicate processing, and MJPEG→H265 false rejection. The install.sh script now includes optional FFmpeg auto-install (apt/dnf/yum/apk), non-blocking.

Xiaomi TUTK Legacy Protocol

Older Xiaomi cameras (Dafang, Xiaofang, Aqara G2, etc.) use a proprietary TUTK (IOT) P2P protocol — they don’t speak standard RTSP or ONVIF. Previously these devices could only be viewed through the Mi Home app, with no NVR integration possible. PR #52 ports the TUTK transport layer from go2rtc and implements a complete standalone stack.

The internal/tutk/ package includes: TransCode/XXTEA crypto transforms, codec constants, GenSessionID/ICAM/HL utility functions, FrameHandler/Packet, Dial + Conn (TransCode + worker + idle timeout). It supports three session types (0/16/25), with session25 featuring a 5-deep ReorderBuffer for shard reassembly. A custom ChaCha20-Poly1305 DTLS cipher suite is implemented.

Supported legacy models (7): Dafang (isa.camera.df3), Xiaofang (isc5c1), Aqara G2 (g2h), IMILAB A1, Loock V1 doorbell, Xiaobai, Mijia Yunyi. The MISS protocol gains two-way audio, PTZ motor control, device info queries (firmware/hardware versions), wakeUpCamera (battery-powered cat-eye/doorbell devices need RPC wake), quality selection "auto"/"hd"/"sd" (fixes isa.camera.hlc8 rejecting HD, corresponding to go2rtc issue #2114), and model-specific timestamp quirks. CS2 TCP adds 1s PING keepalive (matching Mi Home’s official behavior).

Divergences from the go2rtc upstream are documented in AGENTS.md: Pop buffer overflow drops old frames instead of erroring; the hdr double-copy issue is preserved bug-for-bug.

The frontend adds two-way audio UI, a Xiaomi PTZ control panel, and a device info panel.

Protocol Auto-Selection + ONVIF Full Automation

Before v0.9, the Surveillance Grid’s per-camera protocol selection was manual — the default was HLS, and you’d switch if the camera was H.265 or the browser didn’t support it. PR #54 makes protocol selection fully automatic.

The grid now auto-selects the best stream protocol per camera based on codec type and browser decoding capability, with runtime failure cascading: webrtc → flv → hls → mjpeg, degrading to a static snapshot if all fail. The backend had GET /api/cameras/{id}/protocols with codec-aware ranking for a while but the grid never consumed it — now it does.

ONVIF full automation closes 4 historical issues (#36/#48/#50/#51): camera metadata (brand/model/serial) from the discovery phase is now persisted instead of displayed and discarded. A real stream connection test (GetStreamURI + RTSP DESCRIBE, not just HTTP HEAD) replaces the old HEAD check. Profile selection uses resolution-based matching instead of blind profiles[0]. The Hikvision digest time-skew auth issue is fixed (fork v1.1.6). Bulk “add all” is supported, and subnet_hints editor plus ProfileToken persistence are added.

Live-only mode (issue #36): with recording_enabled=false, the recorder maintains the connection for live/relay but doesn’t write segment files. All four recorder families (H264/H265/MJPEG/HTTP-JPEG) are covered.

Friendly error messages are also shipped: friendlyError with 15 error code translations, audio buttons hidden for video-only cameras, advanced recording options collapsed, and severity-aware toast durations. A new doc docs/en+zh/streaming-protocol-selection.md covers the grid’s 4-layer protocol selection architecture in detail.

Data Repair CLI + FastProbeDuration

Data consistency issues in recording files had been accumulating: network instability caused lost segments, incomplete merge outputs, and SPS/PPS changes producing fragments that could never be merged. Previously these required hand-written SQL scripts. v0.9 turns them into CLI commands.

mibee-nvr repair duration: re-probes recording files to fix stuck duration=0 entries. Large files use FastProbeDuration (reads only the stts box, 100× faster than full parsing); MJPEG frame directories use frame-count estimation. The --prune flag deletes unrepairable corrupted fragments (DB row + file).

repair merge-status: resets merge_status for entries with missing merge output files (separated from the startup flow to avoid full-table scan plus per-file os.Stat on every boot).

repair fragments: cleans up merge_status='incompatible' dead fragments (SPS/PPS changed, never mergeable). Dry-run mode lists per-camera counts/duration/disk usage. --retry resets to pending, --force-delete removes the DB row and file.

FastProbeDuration comes from the new merge.ParseSegmentDurationOnly + mediaprobe.FastProbeDuration, stopping at the stts box instead of building a full sample table. Production result: 4285 fragments repaired in ~30 minutes (originally estimated at 10+ hours).

Documentation adds a “timeline recording missing” section to docs/en+zh/troubleshooting.md.

Critical Fixes

Several cross-PR fixes deserve separate mention.

Camera startup lifecycle (PR #60): POST /api/cameras/{id}/start previously passed the HTTP request ctx to rec.Start. When the HTTP response returned, the ctx was canceled, and the recorder’s run goroutine would exit via ctx.Err() — the camera appeared to “start then immediately stop.” Fixed by passing context.Background(), with the recorder lifecycle managed entirely by its own Stop().

Timezone offset (PR #60): Recordings at 16:00 Beijing time appeared at the 08:00 position on the timeline. Backend ParseMergeDuration caps long merge windows (8h/24h/natural-day/7d/30d downgrade to 1h with a warning), and the frontend parseDayStart returns local midnight.

Timeline truncation (PR #60): Xiaomi cameras with frequent reconnects could produce 5000+ segments in a single day; the asc+2000 limit made afternoon recordings invisible. Cap raised from 2000 to 10000.

AddCamera dedup (PR #56): Previously deduplicated only by ID. Auto-discovery generated random IDs, so the same physical ONVIF device could be registered twice — accumulating 777MB of redundant video in production. Now deduplicates by endpoint + stable_id.

AVI recording (PR #52): MJPEG camera AVI video-only muxer (replacing JPEG directory); per-camera health isolation prevents a single camera failure from cascading globally; SQLITE_BUSY fix (DSN busy_timeout 15s + incremental reconciliation + batch DELETE).

Merge audio codec mismatch (PR #52): G.711/Opus codec guards prevent audio track corruption.

Code normalization — golangci-lint v2 (PR #52+): Full adoption of golangci-lint v2.12.2. New linters include perfsprint (fmt.Sprintf→strconv/hex/concat), nilerr, nilnesserr, wastedassign, fatcontext, intrange, usestdlibvars, predeclared (fixed min builtin shadowing). Full interface{}→any migration with compile-time assertions. manager.go (2109 LOC) split into 6 cohesive files. //nolint:<linter> // TODO(#issue): <reason> format enforced, with inline rationale on all exclusion rules.

Data race fixes: session0.go writeAndWait race condition, wsstream viewer-not-registered panic, xiaomi TestCS2ConnError test flake, TestHealth/CameraForm relay test flake — all fixed.

Dependency upgrades: Go and npm deps fully upgraded to latest. gortsplib 5.5.2 → 5.6.1, gortmplib 0.3.2 → 0.4.0.

Important Announcement

v0.9.0 is the last stable/compatible baseline since MiBeeNvr v0.1.0. The next major release will do a comprehensive code normalization and architecture refactor that introduces breaking changes. Every breaking change will come with precise migration paths, configuration deltas, and documented impact scope — no silent breakage. The Recordings Gallery Grid View will be removed from the frontend; the Timeline and List views will be strengthened to cover its functionality.

For long-term stable deployments, pinning the v0.9.x line (currently v0.9.1) is recommended; wait for the next major’s migration guide before upgrading.

v0.9.1 Patch: Full i18n Repair + Recording Fragment Governance + Community Fixes

On the day v0.9.0 shipped, a concentrated batch of community reports arrived (issues #64/#68/#70), surfacing a few regression bugs and long-standing hardcoded strings. v0.9.1 is the next-day patch, focused on four things: i18n completeness, recording-fragment governance, UX fixes, and a docs audit. No breaking changes — just swap the binary.

Full i18n Repair (#68)

This is the heaviest item in the patch. A script cross-referenced every t('...') call in the code against the translation dictionaries and turned up two classes of systematic gaps:

• 19 keys referenced in code but missing from BOTH en.json and zh.json. Users were seeing raw key names in the UI — cameras.darkFrameFilter, cameras.passwordSet, cameras.recordingSchedule, cameras.twoWayAudioEnabled, common.close, live.retry, timeline.frameDrop, xiaomi.twoWayAudio*, and so on. Most of these were missed when features landed in v0.6/v0.7, not newly introduced by v0.9.0. All filled in this release.
• The entire Push-Out config section was hardcoded English. The platform selector, transcode strategy dropdown, and the whole “preset override” panel (resolution / framerate / bitrate / GOP / profile / B-frames / reset) had no i18n wiring at all. 14 new keys added; every hardcoded string routed through t().

Net result: en.json / zh.json grew from 1341 → 1377 keys, parity verified with zero missing. The audit script is kept and will be wired into CI to prevent recurrence.

Recording Fragment Governance

After v0.9.0 cleared the merge backlog, the fragment count itself was still on the high side. This release does a root-cause pass:

• Platform-aware segment-duration cap. The MP4 muxer holds all samples in RAM until the segment closes (moov is written last), so longer segments mean higher memory pressure. There was previously no upper bound, leaving low-RAM devices exposed to OOM. The cap is now set automatically by available RAM: ≤2 GB available (e.g. Raspberry Pi 3B) caps at 30s (OOM safety margin); >2 GB (Banana Pi M5, x86, etc.) caps at 2 minutes — directly halving the fragment count and easing rolling-merge pressure. Config values that exceed the platform cap are silently clamped with a warning.
• Periodic backfill sweep. Two new config options: rolling_backfill_interval (default 10m) and rolling_backfill_batch (default 500). Backfill previously ran only once at startup, so historical pending segments would re-accumulate during long runs. The background sweep handles this now; set to "0" to disable the periodic sweep (startup-only backfill remains).

Community Fixes (#64 / #68 / #70)

• Xiaomi TUTK no longer blocked by the pre-add vendor gate (#64): v0.9.0 already supported TUTK, but the check-vendor pre-check used during camera add hadn’t been updated and still flagged TUTK as compatible: false, throwing up a “not compatible” dialog that blocked users at the door. After the fix, both CS2 and TUTK return compatible: true. A classic “feature wired but entrance missed” regression.
• Docker custom ports (#64): devices like Synology NAS often have port 9090 occupied. Added HOST:CONTAINER port-mapping guidance for docker-compose and server.listen instructions for host-network mode; troubleshooting gains a “Docker / NAS 9090 port conflict” section.
• Xiaomi credential fields hidden (#68): Xiaomi authenticates via the cloud-account token, not per-camera username/password, yet the add form still showed both fields — easy to mistake as required. Fields are now hidden by protocol type.
• Merge-config “use global default” actually clears (#68): a deeply buried bug. The DELETE endpoint called UpsertCameraMerge(nil...), but that function uses COALESCE to preserve existing values — passing nil is a no-op, so per-camera overrides could never actually be cleared. Clicking “use global default” did nothing. A new ClearCameraMerge truly NULLs every field; the GET endpoint returns a customized flag so the frontend can distinguish “has override” from “fully default”.
• Push-out URL copy-to-clipboard (#70): the “1/1” badge on the camera card is now clickable, opening a popover that lists each push-out target (name / protocol / enabled state / full URL) with a copy button next to each address. Previously, seeing a push-out URL meant digging through the settings page.

Full Docs Audit

Before release, a full accuracy audit was run across all public docs (README + docs/en + docs/zh): 9 CRITICAL + 3 MODERATE + 1 MINOR fixes, 18 files, +664/-809 lines. A few representative ones:

• xiaomi-setup.md two-way audio description was reversed: the doc said “CS2 only”, but the code is “TUTK only” (CS2 returns two-way audio requires TUTK transport). Doc and code were exact opposites.
• timelapse.md removed the “v2” framing and fixed merge_duration misrepresentation: the doc promoted merge windows of 8h/12h/24h/natural-day/7d/30d, but the code silently clamps those to 1h (legacy strings) or rejects them (other >1h values). The doc was describing a capability that doesn’t exist. Corrected to the actual 1h ceiling; the Chinese file had also been polluted by a translation tool (every line prefixed with #XX|) and was cleanly rewritten.
• transcoding.md Prometheus metric names were wrong: the doc listed mibee_nvr_transcoding_*, which doesn’t exist — the real prefix is nvr_transcoding_*. Aligned with metrics.md.
• api/xiaomi.md check-vendor example was stale: still showed TUTK returning compatible: false; updated to actual post-v0.9.0 behavior.
• troubleshooting.md completed the repair subcommands: previously only documented repair duration; added repair merge-status and repair fragments (with --reset-fake-merged / --max-duration flags).

Separately, the README header demotes Raspberry Pi from “flagship positioning” to “one of the supported low-power ARM platforms” — badge from Raspberry Pi to generic ARM, hero slogan from Turn any Raspberry Pi... to Turn any low-power ARM device.... The project had effectively evolved from “Raspberry Pi-only NVR” to a cross-platform lightweight NVR (x86 servers / Banana Pi / Synology NAS / Raspberry Pi are all target platforms) a while ago; the docs finally caught up. Technical constraints (RPi 3B as the low-end validation baseline) are all retained.

Known Issues (Deferred)

Three issues were not fixed this round — each needs deeper changes or a real-device validation environment:

• TUTK cameras reconnect-flapping (#68-7): root cause identified — our TUTK worker has a 30s idle timeout that go2rtc doesn’t, which is the source of the reconnects. But the fix touches the protocol layer and needs a real TUTK device validation environment, so it’s deferred. Workaround: relay through go2rtc push-out.
• Recording page progress-bar flicker on click (#68-6): involves player-architecture changes; tracked for follow-up.
• Push-out auth (#70-2): tracked for follow-up.

Full changelog at v0.9.1 Release Notes.

Upgrade

Config is backward compatible — just swap the binary:

1
2
3
4
5

# Docker
docker compose --project-directory . -f deploy/docker/docker-compose.yml up -d

# Or one-line install
curl -fsSL https://raw.githubusercontent.com/Mi-Bee-Studio/MiBeeNvr/main/install.sh | sudo bash

After upgrading, check config.example.yaml for new config options (ONVIF auto-discovery parameters, auto-discovery credentials, storage pool configuration, etc.) and add them to your config as needed. ONVIF auto-discovery is enabled by default; you can turn it off in Settings if you don’t need it.

Related Links

• MiBeeNvr GitHub
• MiBeeNvr v0.9.0 Release Notes
• MiBeeNvr v0.9.1 Release Notes
• MiBeeNvr v0.7.0 Release Post
• MiBeeNvr v0.6.0 Release Post
• Xiaomi TUTK Implementation internal/tutk/
• streaming-protocol-selection docs