security-collector-exporter v0.3.0: Real-Time Security Monitoring with eBPF

From Static to Real-Time

The previous article introduced security-collector-exporter v0.1.0 — turning Linux security configuration states into Prometheus metrics. But v0.1.0 is essentially “snapshot-based”: periodically reading /etc, /proc, capturing the static configuration at a single point in time.

There’s an area of security operations that snapshots can’t cover: real-time security events. Someone running a reverse shell, a process escalating privileges, an abnormal network connection, someone loading a kernel module — these events happen and pass; you’d never see them at your next scrape.

eBPF is the natural tool for this problem. It can attach to various kernel hook points (tracepoints, kprobes), aggregate data in kernel space, with zero copy and low overhead. Starting from v0.2.0, security-collector-exporter added full eBPF support. After v0.3.0’s bug fixes and stability improvements, it’s finally production-ready.

What eBPF Collects

Five independent BPF programs, each attached to different kernel hook points, covering five categories of security events:

Exposed Prometheus metrics:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22

# Process metrics (labels: type=system/user/container/suspicious)
security_ebpf_process_exec_total
security_ebpf_process_exit_total
security_ebpf_process_active_count

# Network metrics (labels: direction=in/out, protocol=tcp/udp)
security_ebpf_connect_total
security_ebpf_connect_active
security_ebpf_connect_error_total{type="timeout|refused|reset"}

# File access metrics (labels: severity=critical/warning/info, operation=read/write)
security_ebpf_file_access_total

# Privilege escalation metrics (labels: type=setuid/setgid/capset, result=success/failure)
security_ebpf_privilege_escalation_total

# Kernel module metrics (labels: action=load/load_file)
security_ebpf_kernel_module_total

# Meta information
security_ebpf_up{status="active|degraded|disabled"}
security_ebpf_sample_rate

The process classification is particularly interesting — not simply system/user, but four categories: system (processes under system paths), user (user processes), container (container processes detected via cgroup ID), suspicious (suspicious interpreter calls). This enables precise filtering in PromQL:

1
2
3
4
5

# Check processes executed in containers
security_ebpf_process_exec_total{type="container"}

# Check suspicious process execution
security_ebpf_process_exec_total{type="suspicious"}

Overall Architecture

The architecture is divided into two layers: kernel space for collection and pre-aggregation, user space for reading and exposing metrics.

The 5 BPF programs in kernel space attach to 14 tracepoints. When events occur, they increment counters directly in percpu_array maps, with zero lock contention:

flowchart LR

    classDef tp fill:#E3F2FD,stroke:#1565C0,color:#1565C0
    classDef bpf fill:#FFF3E0,stroke:#E65100,color:#BF360C
    classDef map fill:#E8F5E9,stroke:#2E7D32,color:#1B5E20

    subgraph tp["Tracepoints"]
        t1@{ shape: rounded, label: "execve / exit" }
        t2@{ shape: rounded, label: "tcp state change" }
        t3@{ shape: rounded, label: "openat" }
        t4@{ shape: rounded, label: "setuid / setgid / capset" }
        t5@{ shape: rounded, label: "init_module" }
    end

    subgraph bpf["BPF Programs"]
        p@{ shape: proc, label: "process.c\nProcess Tracing" }
        n@{ shape: proc, label: "network.c\nNetwork Connections" }
        f@{ shape: proc, label: "file.c\nFile Access" }
        v@{ shape: proc, label: "privilege.c\nPrivilege Detection" }
        k@{ shape: proc, label: "kernel.c\nKernel Modules" }
    end

    subgraph maps["percpu_array Maps"]
        m1@{ shape: cyl, label: "exec / exit / active" }
        m2@{ shape: cyl, label: "connect / error" }
        m3@{ shape: cyl, label: "access by severity" }
        m4@{ shape: cyl, label: "escalation by type" }
        m5@{ shape: cyl, label: "module by action" }
    end

    t1 --> p --> m1
    t2 --> n --> m2
    t3 --> f --> m3
    t4 --> v --> m4
    t5 --> k --> m5

    class t1,t2,t3,t4,t5 tp
    class p,n,f,v,k bpf
    class m1,m2,m3,m4,m5 map

The user-space Manager loads BPF programs and attaches tracepoints. The Aggregator reads aggregated data from percpu_array every 5 seconds (with delta tracking for RPS calculation), ultimately exposed as Prometheus metrics by EbpfCollector:

flowchart LR

    classDef datasource fill:#E8F5E9,stroke:#2E7D32,color:#1B5E20
    classDef component fill:#EDE7F6,stroke:#4527A0,color:#311B92
    classDef endpoint fill:#FCE4EC,stroke:#C62828,color:#B71C1C

    maps@{ shape: cyl, label: "BPF percpu_array Maps" }

    subgraph go["Go User Space"]
        mgr@{ shape: notch-rect, label: "Manager\nLoad & Attach" }
        agg@{ shape: proc, label: "Aggregator\nRead + Delta Tracking" }
        smp@{ shape: notch-rect, label: "AdaptiveSampler\nAdaptive Sampling Rate" }
        col@{ shape: proc, label: "EbpfCollector" }
    end

    prom@{ shape: stadium, label: "Prometheus :9102/metrics" }

    maps -->|"Read every 5s"| agg
    mgr -.->|"Lifecycle"| maps
    agg --> col
    smp -.->|"Sampling rate"| col
    col -->|"/metrics"| prom

    class maps datasource
    class mgr,agg,smp,col component
    class prom endpoint

The AdaptiveSampler evaluates the actual event volume every 10 seconds. If it exceeds 2x the target RPS, it doubles the sampling rate (samples more sparsely); if below half, it restores (samples more densely), preventing eBPF event volume from overwhelming user space under high load.

Pitfalls in v0.2.0

v0.2.0 was the first eBPF-enabled version, with significant new code (~4000 lines), but running it after release revealed many issues.

BPF Program Bugs

Incorrect struct field in network BPF. The inet_sock_set_state tracepoint’s kernel struct doesn’t have a skbaddr field, but v0.2.0’s network.c referenced it. The BPF verifier refused to load it, completely breaking network monitoring.

UDP kprobe doesn’t exist. v0.2.0 attempted to use kprobe for UDP connection tracing, but the target kernel didn’t have the corresponding kprobe tracer. Load failed without graceful handling.

Inaccurate process exit classification. When sched_process_exit triggers, you can’t retrieve the category assigned during the original execve. v0.2.0 didn’t implement PID-to-category mapping, causing exit event classification to differ from exec.

Wrong map type for privilege detection. privilege.c used PERCPU_ARRAY, which has race conditions when multiple CPUs concurrently write to the same key. Switching to PERCPU_HASH fixed the issue.

ARM Device Blocking

This was a subtle bug. On ARM devices (Raspberry Pi, etc.), port version detection (the Java version detection logic in process_info.go) could stall the Collect() cycle. Certain processes (like mediamtx) would hang during version command execution with no timeout mechanism, causing the entire Prometheus scrape to time out.

CI Build Issues

BPF programs need clang to compile C code into bytecode (*_bpf.o), then embedded into the Go binary via go:embed. v0.2.0 had .o files in .gitignore, and the CI environment (ubuntu-latest) didn’t have clang, causing the build to fail immediately.

How v0.3.0 Fixed Them

BPF Program Fixes

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

// process.c: Added PID hash map, store category on execve, retrieve on exit
struct {
    __uint(type, BPF_MAP_TYPE_HASH);
    __uint(max_entries, 65536);
    __type(key, __u64);    /* pid_tgid */
    __type(value, __u32);  /* category */
} pid_category SEC(".maps");

// Store on execve
__u64 pid_tgid = bpf_get_current_pid_tgid();
bpf_map_update_elem(&pid_category, &pid_tgid, &category, BPF_ANY);

// Retrieve on exit
__u32 *cat_ptr = bpf_map_lookup_elem(&pid_category, &pid_tgid);
if (cat_ptr) {
    category = *cat_ptr;
    bpf_map_delete_elem(&pid_category, &pid_tgid);
} else {
    // Process existed before BPF loaded, guess category by comm name
    category = classify_by_comm();
}

classify_by_comm() is a fallback — for processes running before eBPF was loaded, we can’t get the execve event, so it uses coarse-grained matching on the first four bytes of the process name (syst → systemd, sshd → sshd, etc.). The BPF verifier doesn’t allow loops, so byte-by-byte comparison is hardcoded.

Aggregator Rewrite

v0.2.0’s Aggregator used in-memory arrays simulating percpu data (simulation mode). v0.3.0 completely rewrote it to directly read real BPF percpu_array maps, with delta tracking:

1
2
3
4
5
6
7

// ReadAndUpdateFromMaps reads all COUNTER-type BPF maps and calculates deltas
func (a *Aggregator) ReadAndUpdateFromMaps() uint64 {
    // For each counter map, for each key:
    //   current_value - previous_value = delta
    //   Accumulate delta for total event count, feed to AdaptiveSampler
    // processActive and connectActive are GAUGE, not involved in delta
}

ARM Blocking Fix

Three changes together solved this:

• executeVersionCommand added a 3-second timeout (context.WithTimeout)
• Cached empty version detection results for 5 minutes (avoid re-probing known non-responsive processes)
• getVersionFromBinary uses io.LimitReader to read only the first 1MB (previously read the entire binary, slow for large files)

CI Fix

Committed the compiled *_bpf.o files to git. These are architecture-independent BPF ELF bytecode (not x86 or ARM machine code), so there’s no cross-platform issue. CI no longer needs a clang toolchain.

Two Interesting Design Decisions

Adaptive Sampling

Under high load, eBPF event volume can be very large (tens to hundreds of thousands of events/sec). The AdaptiveSampler’s approach: set a target RPS (default 5000), evaluate actual RPS every 10 seconds — if it exceeds 2x the target, double the sampling rate; if below half, halve the sampling rate. The sampling rate range is limited to 1 ~ 10000.

This sampling rate is exposed as the security_ebpf_sample_rate metric, enabling restoration of true values during queries:

1
2

# Approximate restoration of real process execution rate
security_ebpf_process_exec_total * security_ebpf_sample_rate

Space-Saving Top-N

The code also implements the Space-Saving algorithm for maintaining Top-K frequent items in high-cardinality data streams. Fixed number of counters (default 100), new keys take a slot if available, otherwise replace the one with the smallest count and inherit its count. O(1) update, O(K log K) Top-N query. This component lays groundwork for future queries like “who’s executing frequently / who’s connecting to the network frequently.”

Deployment

Enabling eBPF requires additional parameters:

1
2
3
4
5
6
7
8

docker run -d \
  --name security-exporter \
  --privileged \
  -p 9102:9102 \
  ghcr.io/mickeyzzc/security-collector-exporter:0.3.0 \
  --ebpf.enabled \
  --ebpf.sample-rate=1 \
  --ebpf.max-events-per-second=5000

Without --ebpf.enabled, it runs only the v0.1.0 static collector, with no impact.

Kernel requirement: Linux 5.4+ (needs BTF support). Environments that can’t run eBPF automatically degrade to degraded status, with security_ebpf_up{status="degraded"} set to 0, running only the traditional collector.

Some useful eBPF alert rules:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

# eBPF monitoring degraded
- alert: EbpfMonitoringDegraded
  expr: security_ebpf_up{status="degraded"} == 1
  labels:
    severity: warning

# Suspicious process execution
- alert: SuspiciousProcessExecuted
  expr: increase(security_ebpf_process_exec_total{type="suspicious"}[5m]) > 0
  labels:
    severity: critical

# Privilege escalation failures spike (possible brute force)
- alert: PrivilegeEscalationAttempts
  expr: increase(security_ebpf_privilege_escalation_total{result="failure"}[10m]) > 5
  labels:
    severity: critical

# Kernel module loaded
- alert: KernelModuleLoaded
  expr: increase(security_ebpf_kernel_module_total{action="load"}[5m]) > 0
  labels:
    severity: warning

Version History

1
2
3

v0.1.0 (2026-05-14)  Static security configuration collection, 15 metric categories
v0.2.0 (2026-05-18)  eBPF real-time monitoring, 5 BPF programs, ~4000 new lines of code
v0.3.0 (2026-05-19)  BPF bug fixes, ARM compatibility, CI fixes, documentation cleanup

Project repository: github.com/mickeyzzc/security-collector-exporter