Observability Storage Architecture Overview: From Gorilla to Parquet

Introduction

Observability has become a core infrastructure pillar of the cloud-native era. The five signals—Metrics, Logs, Traces, RUM, and Profiling—each generate massive volumes of data, and their storage efficiency directly determines platform cost boundaries and query performance. On the collection side, eBPF technology is reshaping how data is acquired. Behind all of this, storage architecture evolution is the foundation that enables observability to scale in production.

flowchart TD
    A[Five Signals of<br/>Observability] --> B[Metrics]
    A --> C[Logs]
    A --> D[Traces]
    A --> E[RUM + Profiling]

    style A fill:#9C27B0,color:#fff
    style B fill:#2196F3,color:#fff
    style C fill:#2196F3,color:#fff
    style D fill:#2196F3,color:#fff
    style E fill:#2196F3,color:#fff

flowchart TD
    G[Storage Evolution<br/>Convergent Paths] --> H[Proprietary Format<br/>→ Columnar Standard]
    G --> I[Index Bloat<br/>→ Weak Index + Object Store]

    style G fill:#FF9800,color:#fff
    style H fill:#4CAF50,color:#fff
    style I fill:#4CAF50,color:#fff

Figure: The five observability signals each produce data with different characteristics, yet their storage evolution converges on two core themes across all signals: the shift from proprietary storage formats to Parquet/Arrow columnar standards, and the move from full-field inverted indexes to weak-index strategies with object storage.

This opening post starts with the two storage evolution threads, then delves into the Gorilla compression algorithm—the shared theoretical foundation of all modern TSDBs—surveys the key technical consensus, and outlines three major evolution trends, setting the stage for subsequent deep-dives into each signal.

Beginner’s Cheat Sheet — Five Signals in One Line

Signal	Analogy	Example Data
Metrics	System’s “heart monitor”	CPU=80%, QPS=12k, mem=6GB
Logs	System’s “work diary”	[ERROR] 10:03 connection timeout
Traces	Request’s “delivery tracking”	gateway → order → payment → inventory
RUM	User’s “experience report”	FCP=3.2s, CLS=0.05
Profiling	Code’s “X-ray scan”	function call stack flame graph

If you’re new to observability, don’t be intimidated by the jargon—essentially it’s about examining your system from five different angles, with one core goal: when something breaks, you can quickly find out why.

Two Storage Evolution Threads

Following an in-depth survey of over a dozen systems—InfluxDB, Prometheus, VictoriaMetrics, TimescaleDB, Loki, Elasticsearch, ClickHouse, Tempo, Pyroscope, and more—two clear threads emerge in observability storage technology:

These two threads are not independent—columnar formats naturally support efficient sequential scanning and column pruning, providing the underlying capability for weak-index strategies; while the low cost of object storage makes “scanning instead of seeking” economically viable.

Beginner Analogy: Imagine recording daily temperatures. Uncompressed, each entry stores “2026-06-14 10:00:00 → 25.3°C” as a long string. But if the temperature barely changes, you’d naturally say “same as yesterday” — that’s compression in essence. Gorilla pushes this intuition to the extreme: timestamps are replaced with “same interval as before,” values with “nearly identical to the previous one.” The animation below demonstrates this process.

Theoretical Foundation: Gorilla Compression Algorithm

Gorilla is an in-memory time series database published by Facebook at VLDB 2015. Its compression algorithm is the foundation of modern TSDB compression technology, widely adopted by Prometheus, InfluxDB, VictoriaMetrics, TimescaleDB, and others^[1][16]. Gorilla compresses time series to an average of 1.37 bytes per data point, achieving a 12x compression ratio^[1].

Timestamp Compression: Delta-of-Delta Encoding

A key property of time series data is that collection intervals are typically fixed. Gorilla exploits this by encoding timestamps using “delta of delta”^[1][2]:

1

Formula: D = (t_n - t_{n-1}) - (t_{n-1} - t_{n-2})

Variable-length encoding rules^[2][5]:

Real-world effect: For a 15-second fixed-interval collection, the first timestamp stores the full value (64 bits), the second stores the delta (14 bits), and every subsequent timestamp needs only 1 bit^[2]. A single collection delay incurs an extra 32 bits of overhead, making consistent collection intervals critical to compression efficiency.

Animation: The loop above shows 5 samples collected at fixed 15-second intervals being encoded with delta-of-delta. Note that from t₃ onward, each row requires only 1 bit (green highlight) — because delta-of-delta is 0, a single flag bit suffices. This is the core principle behind Gorilla’s 12x compression ratio.

Floating-Point Value Compression: XOR Encoding

Consecutive floating-point values are typically similar (e.g., CPU utilization 47.2%, 47.3%), and XOR operations produce many leading/trailing zeros^[1][2]:

Production statistics (Facebook measurements)^[2]:

Animation: The loop above shows XOR compression of consecutive float values like CPU utilization. Key insight: adjacent samples are typically nearly identical (47.2% → 47.3% → 47.3%), and XOR produces all zeros 51% of the time (just 1 flag bit), with only a few meaningful bits in 30% of cases. This is the core of Gorilla’s value compression — it doesn’t compress individual values, it compresses the differences between consecutive values.

Overall Compression Results and Limitations

Data is grouped into chunks, each with fixed overhead (base timestamp 64 bits + first value 64 bits). Short chunks yield poor compression^[2][3].

flowchart TD
    TS[Timestamp Sequence] --> DD{Compute delta-of-delta}
    DD -->|D=0| B1[1 bit encoding]
    DD -->|"D ∈ -63~64"| B2[9 bits encoding]
    DD -->|"D ∈ -255~256"| B3[12 bits encoding]
    DD -->|Other| B4[36 bits encoding]

    style TS fill:#FF9800,color:#fff
    style DD fill:#2196F3,color:#fff
    style B1 fill:#4CAF50,color:#fff
    style B2 fill:#4CAF50,color:#fff
    style B3 fill:#4CAF50,color:#fff
    style B4 fill:#4CAF50,color:#fff

Figure: Gorilla timestamp encoding decision flow. The core assumption of delta-of-delta encoding is stable collection intervals—when D=0 (fixed intervals), each timestamp requires only 1 bit, which is the key to achieving a 12x compression ratio. If collection intervals are irregular, D=0 cases drop significantly, and the compression ratio falls from 12x to 5-7x.

Algorithm limitations^[2]:

Key Technical Consensus

Through cross-validation across multiple sources, the following technical consensus has emerged in observability storage^[1][3][E25]:

Three Major Evolution Threads

Synthesizing trends across all signal domains, observability storage is evolving along three main threads^{[E25][E27][E11]}:

Series Reading Guide

This series consists of 8 posts, best read in the following order:

Posts 2-6 can be read selectively based on interest. Posts 7-8 depend on technical background from preceding posts.

References

[1] [Official] Gorilla: A Fast, Scalable, In-Memory Time Series Database (VLDB 2015): https://www.vldb.org/pvldb/vol8/p1816-teller.pdf

[2] [Secondary] Gorilla Compression (Delta-of-Delta + XOR) Detailed Analysis: https://crackingwalnuts.com/distributed-systems/gorilla-compression

[3] [Secondary] Time-Series Database Internals: InfluxDB 3, TimescaleDB, and QuestDB Under the Hood: https://iotdigitaltwinplm.com/time-series-database-internals-influxdb-timescale-questdb-2026/

[5] [Official] InfluxDB Time Series Index (TSI) Details: https://docs.influxdata.com/enterprise_influxdb/v1/concepts/tsi-details/

[16] [Secondary] CHIMP: Efficient Lossless Floating Point Compression for Time Series Databases (VLDB 2022): https://vldb.org/pvldb/vol15/p3058-liakos.pdf

[E11] [Official] Grafana: The open and composable observability platform: https://grafana.com/

[E25] [Official] OpenTelemetry — Overview (specs): https://opentelemetry.io/docs/specs/otel/overview/

[E27] [Secondary] What is OpenTelemetry? (Dash0): https://www.dash0.com/knowledge/what-is-opentelemetry