The previous 7 articles broke down the storage architectures of each observability domain—TSDB, logging, tracing, RUM, profiling, and eBPF. This one pulls them together: how to pick storage for your scenario. Teams differ in scale, tech stack, budget, and requirements—there is no “best” storage, only the one that fits you.
Picking storage is a bit like picking a car—a startup needs a cost-efficient compact (Loki + VictoriaMetrics), an enterprise with compliance needs wants a heavy-duty truck (Elasticsearch), and a platform team wants a versatile van (ClickHouse unified). It comes down to four dimensions: data volume, query patterns, budget, and team size. The animation below walks three typical scenarios through the decision matrix:
Observability architecture has shifted from “fragmented” to “unified” over the past five years. Signals (metrics, logs, traces, profiling) have moved from independent specialized backends toward shared storage and a unified data model, while eBPF, as a next-generation collection mechanism, provides finer data granularity and broader coverage for this transformation. This article focuses on four dimensions: eBPF collection storage design, unified storage architecture (LGTM fragmented vs ClickHouse unified), the OpenTelemetry unified data model, and the evolution of object storage with compute-storage separation.
The previous articles on OOM tracing all used C for eBPF kernel-space programs. This is natural — C is eBPF’s “native language,” with the verifier, CO-RE, and libbpf toolchain all designed around C. But if you’ve followed the eBPF ecosystem, you’ll notice a clear trend: more and more people are writing eBPF in languages other than C. Rust’s Aya framework is already used in production by the Solana validator and Kubernetes Gateway API; meanwhile, Zig is trying to bring a new development experience with comptime, explicit allocation, and first-class C interop.
Continuous Profiling is a vital pillar of observability. Unlike traditional sampling profilers, continuous profiling captures application and kernel stack traces at a fixed frequency around the clock, generating flame graphs that help teams identify performance bottlenecks, memory leaks, and resource hotspots. This article focuses on the storage architecture of profiling data: starting from the Google pprof data model and flame graph construction algorithm, we analyze Pyroscope’s architectural evolution from V1 (TSDB+Parquet) to V2 (Metastore+Segments), Parca and FrostDB’s columnar storage innovation, commercial solutions from Datadog and Splunk, and conclude with comparative tables that highlight the differentiating choices across solutions.
Real User Monitoring (RUM) differs fundamentally from backend monitoring in its data model: RUM collects event streams with massive high-cardinality attributes — every user visit generates session_id, user_id, page_url, and hundreds of dimensions. Traditional TSDB label-set models degrade severely under this pattern.
This article analyzes the time-series storage design of three mainstream RUM solutions: Sentry (open-source all-in-one), Datadog RUM (SaaS benchmark), and Grafana Faro (Grafana LGTM ecosystem extension). The common trend across all three is leveraging columnar OLAP instead of TSDB.
Distributed tracing storage is one of the most technically divergent areas in the observability landscape. Unlike metrics and logging, which have relatively converged storage patterns (TSDB and ClickHouse-like systems), tracing storage has split into three distinct paths due to fundamental disagreements over indexing strategy.
This chapter analyzes the storage architectures of Jaeger, Grafana Tempo, Apache SkyWalking, Zipkin, and the commercial solutions from Datadog and Splunk, concluding with a horizontal comparison.
Introduction Logs generate the largest data volume and highest storage cost among the three pillars of observability. Unlike metrics with their fixed numeric structure, logs are variable-length text with semi-structured characteristics, posing unique storage design challenges: supporting full-text search while controlling storage costs.
The core design dimension of log storage systems is indexing strategy — deeper indexing means faster search, but also higher storage costs. From Elasticsearch’s full inverted index to Loki’s label-only indexing, each system makes different technical choices along this spectrum.
Introduction Time-Series Databases (TSDB) are the foundation of observability storage. The previous post outlined the main threads of storage evolution; this post focuses on the TSDB category—comparing the storage engines of four mainstream TSDBs: InfluxDB, Prometheus, VictoriaMetrics, and TimescaleDB. Whether you use Prometheus or are evaluating VictoriaMetrics, understanding the design tradeoffs of the underlying storage engines helps with more effective selection and usage.
Introduction Observability is a core component of cloud-native infrastructure. The five signals—Metrics, Logs, Traces, RUM, and Profiling—each generate massive volumes of data, and their storage efficiency directly determines platform cost boundaries and query performance. On the collection side, eBPF is reshaping how data is acquired; on the storage side, storage architecture evolution is the underlying foundation that enables observability to scale in production.
mermaid flowchart TD A[Five Signals of<br/>Observability] --> B[Metrics] A --> C[Logs] A --> D[Traces] A --> E[RUM + Profiling] style A fill:#9C27B0,color:#fff style B fill:#2196F3,color:#fff style C fill:#2196F3,color:#fff style D fill:#2196F3,color:#fff style E fill:#2196F3,color:#fff mermaid flowchart TD G[Storage Evolution<br/>Convergent Paths] --> H[Proprietary Format<br/>→ Columnar Standard] G --> I[Index Bloat<br/>→ Weak Index + Object Store] style G fill:#FF9800,color:#fff style H fill:#4CAF50,color:#fff style I fill:#4CAF50,color:#fff Figure: The five observability signals each produce data with different characteristics, yet their storage evolution converges on two core themes across all signals: the shift from proprietary storage formats to Parquet/Arrow columnar standards, and the move from full-field inverted indexes to weak-index strategies with object storage.
The previous articles showed how to use eBPF to observe OOM events. But we could only watch, not intervene. The kernel’s OOM Killer decides who lives and dies based on the oom_badness() algorithm, with no user control.
In 2025, Google engineer Roman Gushchin proposed the BPF OOM kernel patch series, aiming to let eBPF programs fully take over OOM handling policy. This is the biggest change to Linux memory management’s OOM subsystem in nearly two decades.