Observability Technology Series

Introduction

Observability has become a core infrastructure pillar of the cloud-native era. The five signals—Metrics, Logs, Traces, RUM, and Profiling—each generate massive volumes of data, and their storage efficiency directly determines platform cost boundaries and query performance. On the collection side, eBPF technology is reshaping how data is acquired. Behind all of this, storage architecture evolution is the foundation that enables observability to scale in production.

flowchart TD
    A[Five Signals of<br/>Observability] --> B[Metrics]
    A --> C[Logs]
    A --> D[Traces]
    A --> E[RUM + Profiling]

    style A fill:#9C27B0,color:#fff
    style B fill:#2196F3,color:#fff
    style C fill:#2196F3,color:#fff
    style D fill:#2196F3,color:#fff
    style E fill:#2196F3,color:#fff

flowchart TD
    G[Storage Evolution<br/>Convergent Paths] --> H[Proprietary Format<br/>→ Columnar Standard]
    G --> I[Index Bloat<br/>→ Weak Index + Object Store]

    style G fill:#FF9800,color:#fff
    style H fill:#4CAF50,color:#fff
    style I fill:#4CAF50,color:#fff

Figure: The five observability signals each produce data with different characteristics, yet their storage evolution converges on two core themes across all signals: the shift from proprietary storage formats to Parquet/Arrow columnar standards, and the move from full-field inverted indexes to weak-index strategies with object storage.

Introduction

Time-Series Databases (TSDB) are the foundation of observability storage. If the previous post was a bird’s-eye view of the “storage map,” this one zooms into the “core districts” — the storage engines of four mainstream TSDBs. Whether you’re just starting with Prometheus or evaluating VictoriaMetrics, understanding the design philosophy behind these storage engines will help you use them more effectively.

Quick Reference — Four TSDBs at a Glance

Database	Positioning	One-Liner
InfluxDB	Dedicated TSDB	The earliest popular TSDB; custom TSM engine (1.x/2.x), pivoted to columnar storage in 3.0
Prometheus	Cloud-native monitoring standard	Built-in local TSDB, single-node design, widely adopted in the Kubernetes ecosystem
VictoriaMetrics	High-performance alternative	Prometheus-compatible but more memory-efficient, supports clustering, excellent cold data compression
TimescaleDB	PostgreSQL time-series extension	Don’t want to learn a new query language? Use Postgres + time-series optimizations

If terms like “LSM-Tree,” “Compaction,” or “inverted index” sound unfamiliar — don’t worry. This post starts from the ground up, explaining what each storage engine is designed to solve.

Introduction

Logs generate the largest data volume and highest storage cost among the three pillars of observability. Unlike metrics with their fixed numeric structure, logs are variable-length text with semi-structured characteristics, posing unique storage design challenges: supporting full-text search while controlling storage costs.

The core design dimension of log storage systems is indexing strategy — deeper indexing means faster search, but also higher storage costs. From Elasticsearch’s full inverted index to Loki’s label-only indexing, each system makes different technical choices along this spectrum.

Distributed tracing storage is one of the most technically divergent areas in the observability landscape. Unlike metrics and logging, which have relatively converged storage patterns (TSDB and ClickHouse-like systems), tracing storage has split into three distinct paths due to fundamental disagreements over indexing strategy.

This chapter analyzes the storage architectures of Jaeger, Grafana Tempo, Apache SkyWalking, Zipkin, and the commercial solutions from Datadog and Splunk, concluding with a horizontal comparison.

Real User Monitoring (RUM) differs fundamentally from backend monitoring in its data model: RUM collects event streams with massive high-cardinality attributes — every user visit generates session_id, user_id, page_url, and hundreds of dimensions. Traditional TSDB label-set models degrade severely under this pattern.

This article analyzes the time-series storage design of three mainstream RUM solutions: Sentry (open-source all-in-one), Datadog RUM (SaaS benchmark), and Grafana Faro (Grafana LGTM ecosystem extension). The common trend across all three is leveraging columnar OLAP instead of TSDB.

Continuous Profiling is a vital pillar of observability. Unlike traditional sampling profilers, continuous profiling captures application and kernel stack traces at a fixed frequency around the clock, generating flame graphs that help teams identify performance bottlenecks, memory leaks, and resource hotspots. This article focuses on the storage architecture of profiling data: starting from the Google pprof data model and flame graph construction algorithm, we analyze Pyroscope’s architectural evolution from V1 (TSDB+Parquet) to V2 (Metastore+Segments), Parca and FrostDB’s columnar storage innovation, commercial solutions from Datadog and Splunk, and conclude with comparative tables that highlight the differentiating choices across solutions.

Observability architecture has undergone a profound shift from “fragmented” to “unified” over the past five years. Signals (metrics, logs, traces, profiling) have moved from independent specialized backends toward shared storage and a unified data model, while eBPF, as a next-generation collection mechanism, provides unprecedented data depth and breadth for this transformation. This article focuses on four dimensions: eBPF collection storage design, unified storage architecture (LGTM fragmented vs ClickHouse unified), the OpenTelemetry unified data model, and the evolution of object storage with compute-storage separation.

The previous 7 articles in this series analyzed the storage architectures of each observability domain—TSDB, logging, tracing, RUM, profiling, and eBPF—in depth. This article synthesizes that knowledge into a scenario-driven storage selection decision framework. Teams differ in scale, tech stack, budget, and requirements—there is no “best” storage, only the one that best fits your context. Use this article as a practical reference to quickly narrow down your options.

Beginner Analogy: Choosing observability storage is like choosing a vehicle — there’s no “best” car, only the one that fits your journey. A startup needs a cost-efficient compact (Loki + VictoriaMetrics); an enterprise with compliance needs a heavy-duty truck (Elasticsearch); a platform team wants a versatile van (ClickHouse unified). The key dimensions: data volume, query patterns, budget, and team size.