The previous articles on OOM tracing all used C for eBPF kernel-space programs. This is natural — C is eBPF’s “native language,” with the verifier, CO-RE, and libbpf toolchain all designed around C. But if you’ve followed the eBPF ecosystem, you’ll notice a clear trend: more and more people are writing eBPF in languages other than C. Rust’s Aya framework is already used in production by the Solana validator and Kubernetes Gateway API; meanwhile, Zig is trying to bring a new development experience with comptime, explicit allocation, and first-class C interop.
The previous articles showed how to use eBPF to observe OOM events. But we could only watch, not intervene. The kernel’s OOM Killer decides who lives and dies based on the oom_badness() algorithm, with no user control.
In 2025, Google engineer Roman Gushchin proposed the BPF OOM kernel patch series, aiming to let eBPF programs fully take over OOM handling policy. This is the biggest change to Linux memory management’s OOM subsystem in nearly two decades.
The first two articles covered eBPF fundamentals and OOM Killer event tracing. This article goes deeper: container-level OOM pinpointing, real-time memory allocation rate tracking, and implementing the same functionality with the Rust Aya framework.
Container-Level OOM Pinpointing In Kubernetes, “a Pod OOM’d” is actually a vague statement. A Pod consists of multiple containers, each belonging to different cgroups. eBPF can drill through this layer and precisely identify which container and which process caused the OOM.
bpftrace is great for quick probing and ad-hoc debugging. For production-grade monitoring tools, you need full eBPF programs. The architecture splits into two layers:
Kernel side: eBPF program written in C, attached to hook points, collecting event data User side: loader written in Go (or Rust / libbpf C), loading the eBPF program and reading events Architecture mermaid flowchart LR classDef kern fill:#E3F2FD,stroke:#1565C0,color:#1565C0 classDef user fill:#FFF3E0,stroke:#E65100,color:#BF360C classDef data fill:#E8F5E9,stroke:#2E7D32,color:#1B5E20 subgraph kernel["Kernel Space"] hook@{ shape: rounded, label: "oom_kill_process (kprobe)" } ebpf@{ shape: proc, label: "eBPF Program\nEvent Collection" } ring@{ shape: cyl, label: "Ring Buffer" } end subgraph userspace["User Space (Go)"] loader@{ shape: notch-rect, label: "bpf2go Loader" } reader@{ shape: proc, label: "RingBuf Reader\nEvent Parsing" } end hook --> ebpf --> ring ring --> reader loader -.-> ebpf class hook,ebpf,ring kern class loader,reader user eBPF Kernel Program (C) Name the C file oom_kprobe.bpf.c — the bpf suffix is a cilium/ebpf convention for bpf2go code generation:
eBPF (Extended Berkeley Packet Filter) started as a network packet filtering tool, but over nearly a decade it has evolved into the most powerful observability framework in the Linux kernel. It allows you to safely inject and execute custom programs without modifying kernel source code or loading kernel modules.
This article kicks off the series, using OOM (Out-of-Memory) monitoring as a concrete entry point to learn the core eBPF concepts and toolchain.
The Origin: Compliance Check Hassles Anyone in operations knows there’s no escaping one hurdle for domestic servers: Cybersecurity Level Protection (GB/T 22239-2019, commonly known as “Level Protection 2.0”). Whether you’re Level 3 or Level 2, auditors come asking about these things:
Is SSH root login disabled? Are password policies compliant? Is the firewall on? Is SELinux enforcing? Are there expired accounts? What’s the password validity period? Which ports are open? Are there high-risk services running? Are audit logs enabled? How long are they retained? There are plenty of compliance check tools on the market—search GitHub and you’ll find a bunch: Golin, EvaluationTools, Linux-Security-Compliance-Check, etc. But they all share one limitation: Run once, get a report, done. You check compliance today, and someone changes sshd_config tomorrow, turns off the firewall, installs a backdoor service—you’d never know.
From Static to Real-Time The previous article introduced security-collector-exporter v0.1.0 — turning Linux security configuration states into Prometheus metrics. But v0.1.0 is essentially “snapshot-based”: periodically reading /etc, /proc, capturing the static configuration at a single point in time.
There’s an area of security operations that snapshots can’t cover: real-time security events. Someone running a reverse shell, a process escalating privileges, an abnormal network connection, someone loading a kernel module — these events happen and pass; you’d never see them at your next scrape.
Introduction In the previous article, we reviewed the three-year evolution of stream-metrics-route and mentioned that the “dual hashmod scheduling” is the core scheduling mechanism of the entire gateway. However, during continuous production operation, one fatal flaw of hashmod became increasingly obvious—every scaling operation triggers full data redistribution.
This article documents the complete decision process of migrating from hash % N (hashmod) to Jump Consistent Hash: which candidate algorithms were evaluated, why Jump Hash was ultimately chosen, and the specific impact before and after migration.
Why This Was Built Anyone managing servers has probably had this experience: compliance audit comes, SSH into machines one by one to check—SSH config correct, SELinux enabled, firewall running, any expired accounts, password policies compliant. A few machines are fine; dozens or hundreds becomes purely manual grunt work.
And the more painful part: none of this has continuous monitoring. You check compliance today, someone changes a config tomorrow, and you’d never know.
Introduction It’s been exactly three years since the previous article Applying VictoriaMetrics Stream Aggregation for Metrics was published in March 2023. In these three years, the VictoriaMetrics ecosystem has undergone tremendous changes—let’s revisit the issues raised in that blog post, see what the official project has resolved, and where our stream-metrics-route project stands today.
I. Problems We Encountered Three Years Ago Let’s quickly recap the core issue list from the 2023 blog post: