eBPF Collection Storage and Observability Unified Architecture Trends

June 27, 2026 Observability Observability, EBPF, Pixie, Hubble, ClickHouse, OpenTelemetry, Object Storage, Compute-Storage Separation Observability Technology Series 1974 words 10 min read

Observability architecture has undergone a profound shift from “fragmented” to “unified” over the past five years. Signals (metrics, logs, traces, profiling) have moved from independent specialized backends toward shared storage and a unified data model, while eBPF, as a next-generation collection mechanism, provides unprecedented data depth and breadth for this transformation. This article focuses on four dimensions: eBPF collection storage design, unified storage architecture (LGTM fragmented vs ClickHouse unified), the OpenTelemetry unified data model, and the evolution of object storage with compute-storage separation.

This article examines eBPF’s role in observability from a storage architecture perspective — where collected data is stored, how it’s stored, and how compute-storage separation is evolving. For collection-side implementation details (CO-RE, libbpf, Aya, protocol parsing, OOM tracing), see the eBPF Series.

Beginner Analogy: eBPF is like installing smart cameras on a highway — without modifying any vehicles (kernel code), you can capture data from all vehicles at checkpoints. Traditional monitoring requires installing sensors on each vehicle (modifying application code), while eBPF captures everything at the road surface (kernel layer), transparent to the applications passing through.
The animation below shows how eBPF data flows from kernel space to user space:

Animation: The loop shows the eBPF data pipeline: a network packet or syscall triggers the eBPF program (attached via kprobe/tracepoint), data is written to the kernel ring buffer, the user-space agent reads it via mmap, and sends it to the backend. The key contrast: traditional instrumentation requires application-level SDK changes and cannot see kernel events, while eBPF captures everything transparently at the kernel layer.

内核事件 → eBPF Hook → eBPF Program → Map → Userspace

eBPF Collection Storage

eBPF pushes collection capabilities into the kernel, delivering unprecedented fine-grained data — but also an explosion in data volume. This section analyzes the storage design of three representative eBPF observability solutions.

Pixie — eBPF-Based Zero-Instrumentation Observability Platform

The Pixie platform consists of three main components^[E1]: Vizier (data plane, deployed in monitored clusters), Cloud (control plane), and Vizier Operator (Kubernetes Operator).

Key Storage Characteristics^[E1]^[E3]:

Data never leaves the cluster: Collected data is stored in cluster node memory, not sent to any centralized backend by default
PEM collects data via eBPF in kernel space, with local in-memory short-term storage (max 24h) as the primary storage location for all cluster data
Multi-stage pipeline: eBPF collection → PEM preprocessing and aggregation → Kelvin node-to-cluster aggregation → Query Broker result merging

mermaid
flowchart TD
    PEM["PEM<br/>eBPF Collection + Local Memory 24h"]
    K["Kelvin<br/>Cluster-Level Aggregation"]
    QB["Query Broker<br/>Result Merging"]
    Cloud["Cloud<br/>Control Plane"]

    PEM --> K
    K --> QB
    Cloud -.->|No Data Storage| PEM
    Cloud -.->|No Data Storage| K

    style PEM fill:#4CAF50,color:#fff
    style K fill:#2196F3,color:#fff
    style QB fill:#9C27B0,color:#fff

Figure 1: Pixie “Collect-and-Compute” Architecture. PEM (green) collects data via eBPF in kernel space and stores it in local memory for up to 24 hours. Kelvin (blue) performs cluster-level aggregation, and Query Broker (purple) merges query results. The Cloud control plane does not store data, embodying the “data never leaves the cluster” design principle — only query results, not raw data, are transmitted.

Architecture Trend Insight: Pixie embodies a “collect-and-compute” design — data aggregation is performed on the collection side (PEM/Kelvin), with only query results rather than raw data being transmitted^[E1]^[E2].

Cilium/Hubble — eBPF Network Observability

Hubble is built on top of Cilium and eBPF, with its server component embedded within the Cilium Agent^[E5].

Flow Log Storage: Userspace Ring Buffer^[E5]:

Hubble stores monitoring events from the Cilium event monitor in a userspace ring buffer structure
The ring buffer stores a configurable number of events in memory, overwriting the oldest events when full (FIFO overwrite)
Lock-free design: The internal buffer length is 2^n, using atomic operations instead of locks

eBPF Data Collection Characteristics

The volume of data collected by eBPF is enormous, making data reduction the core challenge^[E10]:

Mechanism	Characteristics	Use Case
eBPF Ring Buffer	Kernel 5.8+, single buffer shared across CPUs, memory efficient	High-throughput event streams (recommended)
Perf Event	Per-CPU independent buffer	Compatibility scenarios
Hash Map	For counters and other aggregation state	In-kernel aggregation

In-Kernel Aggestion Strategy^[E10]: Compute summaries in the kernel rather than streaming raw events; intelligent sampling. This strategy is consistently applied across Pixie, Hubble, and Inspektor Gadget — collection-side aggregation + short-term memory storage + on-demand query result transmission.

Unified Storage Architecture (LGTM Fragmented vs ClickHouse Unified)

Grafana Stack’s “Fragmented” Architecture

The Grafana Stack employs a per-signal-type dedicated backend architecture: Mimir (metrics), Loki (logs), Tempo (traces), and Pyroscope (continuous profiling)^[E11]^[E12]. Each backend shares object storage but has its own database engine.

mermaid
flowchart TD
    Mimir["Mimir<br/>Metrics"]
    Loki["Loki<br/>Logs"]
    Tempo["Tempo<br/>Traces"]
    Pyro["Pyroscope<br/>Profiling"]
    G["Grafana<br/>UI Correlation Layer"]

    Mimir --> G
    Loki --> G
    Tempo --> G
    Pyro --> G

    style Mimir fill:#2196F3,color:#fff
    style Loki fill:#FF9800,color:#fff
    style Tempo fill:#f44336,color:#fff
    style Pyro fill:#9C27B0,color:#fff

Figure 2a: Grafana LGTM Fragmented Architecture. Each signal type has its dedicated backend — Mimir (blue), Loki (orange), Tempo (red), Pyroscope (purple). Backends share object storage but maintain independent database engines. Cross-signal correlation occurs at the visualization layer (Grafana), not the storage layer^[E16].

ClickHouse as Unified Observability Storage

ClickHouse has become the de facto standard for observability storage engines, with the core argument being “observability is a data analytics problem and should be treated as such”^[E16].

mermaid
flowchart TD
    GCH["Grafana"]
    CH["ClickHouse<br/>Unified Column Store"]
    Signals["Metrics / Logs<br/>Traces / Profiling"]

    CH --> GCH
    Signals --> CH

    style CH fill:#4CAF50,color:#fff
    style GCH fill:#2196F3,color:#fff

Figure 2b: ClickHouse Unified Storage Architecture. All signals share a single columnar storage engine, with correlation performed at the storage layer via standard SQL JOIN^[E18], eliminating data duplication and ETL overhead across multiple databases.

Column Store Advantages^[E16]:

Feature	Value for Observability
Compression	Logs/traces achieve ~14x average compression
Fast Aggregation	Rapid GROUP BY aggregation response
Fast Linear Scan	Can scan tens of GB/s (compressed), replacing inverted indexes
High Cardinality	Column store fundamentally solves the high cardinality problem^[E17]
SQL	Lowers the learning curve

5-Step Cost Optimization Stack^[E16]:

Codec Chain Composition: Delta/DoubleDelta + ZSTD, ~50% storage reduction
Tiered Storage: Hot data on NVMe SSD, migrated to S3 after 7 days
Primary Key Optimization: ORDER BY (toStartOfMinute(Timestamp), Service)
Materialized View Pre-Aggregation: Pre-compute aggregations on data arrival
Async Inserts: Server-side buffering, eliminating Kafka

Large-Scale Production Validation: According to the ClickHouse official playbook, Anthropic, OpenAI, Tesla, Didi, Shopee and others use ClickHouse (Note: The following case data is sourced from ClickHouse marketing materials and has not been independently verified) — Tesla built a platform ingesting tens of millions of rows/second; OpenAI reduced query latency from minutes to milliseconds; Didi migrated from Elasticsearch achieving 30% cost reduction with 4x query speedup^[E16].

Core Conclusion (triple-source cross-validation): Unified columnar storage replacing fragmented multi-database is a clear trend. ClickHouse official docs^[E18], ClickStack playbook^[E16], and Parseable comparison^[E19] all confirm: a single engine eliminates data duplication, natively supports high cardinality, and reduces TCO.

OpenTelemetry Unified Data Model

Unified Signal Architecture

The OpenTelemetry client is organized by Signal — each signal provides a specific observability capability (tracing, metrics, logs, baggage), with signals sharing a common context propagation subsystem but remaining functionally independent^[E25].

The OTLP data structure follows a unified hierarchy of Resource + InstrumentationScope + Signal^[E26]^[E25]:

mermaid
flowchart TD
    Resource["Resource<br/>service.name, k8s.pod.name"]
    Scope["InstrumentationScope<br/>Library + Version"]
    Metrics["Metrics"]
    Logs["Logs"]
    Traces["Traces"]
    Profiles["Profiles"]

    Resource --> Scope
    Scope --> Metrics
    Scope --> Logs
    Scope --> Traces
    Scope --> Profiles

    style Resource fill:#9C27B0,color:#fff
    style Scope fill:#2196F3,color:#fff
    style Metrics fill:#4CAF50,color:#fff
    style Logs fill:#4CAF50,color:#fff
    style Traces fill:#4CAF50,color:#fff
    style Profiles fill:#4CAF50,color:#fff

Figure 3: OpenTelemetry Unified Data Model. Resource (purple) captures entity information (service.name, k8s.pod.name), InstrumentationScope (blue) identifies the instrumenting library, and the four signals — Metrics, Logs, Traces, Profiles (green) — all share the unified Resource and InstrumentationScope hierarchy, enabling cross-signal entity correlation.

Resource: Captures entity information for telemetry records, describing the full entity hierarchy (from cloud host to container to application)
InstrumentationScope: Identifies the instrumenting library that generated the telemetry
Signal: All four signals — Metrics, Logs, Traces, and Profiles — share the unified Resource and InstrumentationScope

Profiles as the Fourth Pillar: Profiles have become the fourth signal in OpenTelemetry. A profile can record which spans were active at sampling time, enabling correlation between profiling and tracing^[E27].

Exemplar Mechanism

Exemplars serve as the bridge connecting metrics to traces: associating metric data points with sample trace_ids, allowing direct navigation from latency spikes to representative traces^[E28]^[E29].

Cross-Signal Correlation Storage Design

mermaid
flowchart TD
    Exemplar["Metrics → Traces<br/>Exemplar(trace_id)"]
    TraceID["Logs → Traces<br/>trace_id Injection"]
    SpanProfile["Profiles → Traces<br/>Active Span Recording"]
    Unified["Unified Base<br/>Resource +<br/>InstrumentationScope"]

    Unified --> Exemplar
    Unified --> TraceID
    Unified --> SpanProfile

    style Exemplar fill:#2196F3,color:#fff
    style TraceID fill:#FF9800,color:#fff
    style SpanProfile fill:#4CAF50,color:#fff
    style Unified fill:#9C27B0,color:#fff

Figure 4: Cross-Signal Correlation Mechanisms. Three correlation chains all build on a unified foundation: Metrics correlating to Traces via Exemplar carrying trace_id (blue); Logs correlating to Traces through trace_id/span_id injection in structured logs (orange); Profiles correlating to Traces by recording active spans at sampling time (green). The unified base (purple) ensures all signals share the same Resource + InstrumentationScope model.

Correlation Link	Mechanism	Storage Design Requirements
Metrics → Traces	Exemplar (trace_id/span_id attached to metric points)	Metrics storage must support exemplar field storage and query
Logs → Traces	Structured logs inject trace_id/span_id correlation IDs	Log storage must support trace_id indexing
Profiles → Traces	Profiles record active spans at sampling time	Profile storage must correlate span context
Unified Base	Resource (entity hierarchy) + InstrumentationScope (library identifier)	All signals share the same Resource model

Storage Layer Design Pattern Comparison: The LGTM multi-database pattern achieves correlation at the visualization layer through complex joins; the ClickHouse unified storage pattern uses standard SQL JOIN for native correlation at the storage layer^[E16]^[E18].

Core Conclusion (triple-source cross-validation): Cross-signal correlation is evolving from “UI-layer joins” to “storage-layer native correlation.” The OTel specification^[E25]^[E29], ClickHouse documentation^[E18], and community practice^[E28] all confirm this direction.

Object Storage and Compute-Storage Separation

Object storage (S3/GCS/Azure Blob) has evolved from “archive cold storage” to serving as the primary storage backend in observability^[E12]^[E14]:

Tempo: Trace data is ultimately persisted to object storage^[E14]
Mimir: Based on Prometheus TSDB blocks persisted to object storage^[E13]
Loki: Uses the Thanos Object Storage Client for standardized storage configuration^[E12]
ClickHouse: Tiered storage strategy migrates cold data to S3^[E16]

Tempo 3.0 is a canonical example of compute-storage separation^[E14]: write-read decoupling, Kafka as a persistent WAL, and object storage bridging both write and read paths.

mermaid
sequenceDiagram
    participant WP as WritePath
    participant K as Kafka(WAL)
    participant OS as ObjectStorage
    participant RP as ReadPath

    WP->>K: Persistent WAL
    K->>OS: Block-builder Write
    RP->>OS: Cold Data Query
    RP->>K: Hot Data(Live-store) Query

Figure 5: Tempo 3.0 Compute-Storage Separation Write-Read Model. WritePath writes data to Kafka as a persistent WAL; Block-builder asynchronously consumes and builds blocks written to object storage. ReadPath queries cold data directly from object storage and reads hot data from Kafka (Live-store), achieving complete decoupling of the write and read paths.

References

[E1] [Official] Pixie Architecture: https://docs.px.dev/reference/architecture/

[E2] [Secondary] Pixie Component Overview: https://blog.csdn.net/gitblog_00394/article/details/150926117

[E3] [Secondary] Pixie vizier-pem pod running out of memory: https://knowledge.newrelic.com/s/article/4408761257623

[E5] [Official] Hubble internals (Cilium docs): https://docs.cilium.io/en/latest/internals/hubble/

[E10] [Secondary] eBPF Observability Architecture: https://calmops.com/software-engineering/ebpf-observability-architecture-next-generation-monitoring/

[E11] [Official] Grafana: The open and composable observability platform: https://grafana.com/

[E12] [Official] Grafana Loki 3.4: Standardized storage config: https://grafanacon.org/blog/2025/02/13/grafana-loki-3.4-standardized-storage-config-sizing-guidance-and-promtail-merging-into-alloy/

[E13] [Official] Grafana Mimir architecture: https://grafana.com/docs/mimir/v3.0.x/get-started/about-grafana-mimir-architecture/

[E14] [Official] About the Tempo architecture: https://grafana.com/docs/tempo/next/reference-tempo-architecture/about-tempo-architecture/

[E16] [Official] How to engineer cost-efficient open source observability with ClickHouse (ClickStack): https://clickhouse.com/resources/engineering/observability-cost-optimization-playbook

[E17] [Secondary] The high-cardinality trap: https://clickhouse.com/resources/engineering/high-cardinality-slow-observability-challenge

[E18] [Official] Использование ClickHouse для обеспечения наблюдаемости: https://clickhouse.com/docs/ru/use-cases/observability/introduction

[E19] [Secondary] ClickHouse vs Parseable: https://www.parseable.com/blog/clickhouse-vs-parseable

[E25] [Official] OpenTelemetry — Overview (specs): https://opentelemetry.io/docs/specs/otel/overview/

[E26] [Secondary] OpenTelemetry in Practice: Cloud Native Observability Three Pillars Unified Standard: https://blog.csdn.net/xyghehehehehe/article/details/159112652

[E27] [Secondary] What is OpenTelemetry? (Dash0): https://www.dash0.com/knowledge/what-is-opentelemetry

[E28] [Secondary] Semantics First: Correlating Signals with OpenTelemetry: https://www.thequietkernel.com/engineering/system-engineering/30-09-2025-obs-semantics/

[E29] [Official] Using exemplars (OpenTelemetry .NET metrics docs): https://opentelemetry.io/pt/docs/languages/dotnet/metrics/exemplars/

Part of series: Observability Technology Series

← Previous Continuous Profiling Storage: From pprof to Pyroscope V2 and Parca Next → Observability Storage Selection Guide: A Scenario-Driven Decision Framework