TSDB Storage Architecture Deep Dive: InfluxDB, Prometheus, VictoriaMetrics, TimescaleDB

Introduction

Time-Series Databases (TSDB) are the foundation of observability storage. If the previous post was a bird’s-eye view of the “storage map,” this one zooms into the “core districts” — the storage engines of four mainstream TSDBs. Whether you’re just starting with Prometheus or evaluating VictoriaMetrics, understanding the design philosophy behind these storage engines will help you use them more effectively.

Quick Reference — Four TSDBs at a Glance

Database	Positioning	One-Liner
InfluxDB	Dedicated TSDB	The earliest popular TSDB; custom TSM engine (1.x/2.x), pivoted to columnar storage in 3.0
Prometheus	Cloud-native monitoring standard	Built-in local TSDB, single-node design, widely adopted in the Kubernetes ecosystem
VictoriaMetrics	High-performance alternative	Prometheus-compatible but more memory-efficient, supports clustering, excellent cold data compression
TimescaleDB	PostgreSQL time-series extension	Don’t want to learn a new query language? Use Postgres + time-series optimizations

If terms like “LSM-Tree,” “Compaction,” or “inverted index” sound unfamiliar — don’t worry. This post starts from the ground up, explaining what each storage engine is designed to solve.

InfluxDB

Storage Engine Evolution: From LSM to TSM

The core storage engine of InfluxDB 1.x/2.x is TSM (Time-Structured Merge Tree), evolved from LSM-Tree and specifically optimized for time-series data^[4][9]. The storage engine consists of four core components^[4]:

The diagram below illustrates the four core components of the TSM storage engine and their data flow:

flowchart TD
    A[WAL<br/>Write-Ahead Log] --> B[Cache<br/>In-Memory Cache]
    B --> C[TSM Files<br/>Time-Structured Merge Tree]
    C --> D[TSI<br/>Inverted Index]

    style A fill:#FF9800,color:#fff
    style B fill:#2196F3,color:#fff
    style C fill:#2196F3,color:#fff
    style D fill:#9C27B0,color:#fff

Figure description: The diagram shows the four components of the InfluxDB 1.x/2.x TSM storage engine. The write path flows left to right: WAL → Cache → TSM Files, with TSI providing efficient multi-dimensional query support. Orange indicates the write entry point, blue represents data storage and processing, and purple represents the indexing component.

Write Path and WAL Mechanism

Write flow^[4]:

1. Write request appended to the WAL file
2. Persisted to disk via fsync() (ensuring durability)
3. In-memory Cache updated
4. Acknowledgment returned after disk write succeeds

sequenceDiagram
    participant Client as 写入请求
    participant WAL as WAL
    participant Cache as Cache
    participant TSM as TSM Files
    Client->>WAL: 1. 追加写入
    WAL->>WAL: 2. fsync 刷盘
    WAL->>Cache: 3. 更新内存
    Cache-->>Client: 4. 返回确认
    Note over Cache,TSM: 定期 snapshot
    Cache->>TSM: 5. flush 为 TSM 文件

Figure description: The diagram illustrates the complete InfluxDB write path. The write request is first appended to the WAL and fsynced for durability, then the in-memory Cache is updated, and acknowledgment is returned immediately (steps 1-4). A background process periodically snapshots the Cache and flushes it as TSM files (step 5), enabling asynchronous data movement from hot to cold storage.

Cache characteristics^[4]: Organized by key (measurement + tag set + field key), each field independently sorted by time, storing uncompressed data. Rebuilt from WAL on restart. On query, Cache data is merged with TSM file data.

TSM File Format and Compression

TSM files store compressed series data in a columnar format, recording only the deltas between values in a series^[4]. The compaction process organizes data into long sequences to optimize compression and scan queries:

TSM file level merging (similar to LSM leveled compaction): data is grouped by series key and sorted by time; multiple TSM files are merged into higher levels. After merging, the WAL is truncated and the Cache is cleared.

TSI (Time Series Index)

As cardinality grows, early in-memory indexes hit limits at 1-4 million series^[5][6]. TSI persists the index to disk and uses mmap to let the OS manage it as an LRU cache^[6].

TSI Architecture (LSM-Tree based)^[5]:

flowchart TD
    A[Index<br/>单 Shard 完整索引] --> B[Partition<br/>分片分区]
    B --> C[LogFile L0<br/>内存索引+WAL<br/>阈值 5MB]
    B --> D[IndexFile<br/>L1/L2 层级合并<br/>mmap 不可变]
    C -->|compaction| D
    D --> E[SeriesFile<br/>跨库 series key<br/>自增 ID]

    style A fill:#9C27B0,color:#fff
    style B fill:#2196F3,color:#fff
    style C fill:#FF9800,color:#fff
    style D fill:#4CAF50,color:#fff
    style E fill:#9C27B0,color:#fff

**Figure description**: The diagram shows the TSI inverted index LSM-Tree architecture. The Index is divided into Partitions (blue), each containing a LogFile (orange, in-memory index with 5MB WAL threshold) and IndexFile(s) (green, immutable mmap, leveled merge). The SeriesFile (purple) maintains auto-increment IDs for all series keys across databases. LogFiles exceeding the 5MB threshold are compacted to L1 IndexFiles, and consecutive L1 files merge into L2.

Write flow^[5]: Series added to SeriesFile or looked up returning auto-increment series ID → Index maintains Roaring Bitmap tracking existing series IDs → Series hashed and sent to corresponding Partition → Partition writes series to LogFile → LogFile persists to WAL on disk and adds to in-memory index.

Compaction^[5]: LogFile exceeding 5MB is compacted to L1 IndexFile; two consecutive L1 files merge into L2 IndexFile.

IndexFile structure^[5]: TagBlocks (maintaining tag value index for a single tag key), MeasurementBlock (maintaining measurement and its tag key index), Trailer (offset info + HyperLogLog cardinality estimation sketch).

Read and Query — TSI provides composable iterator APIs^[5]:

• MeasurementSeriesIDIterator() → all series IDs for a measurement
• TagValueSeriesIDIterator() → all series IDs for a tag value
• Merge operations: Merge (deduplicated union), Intersect (intersection), Difference (set difference)

For example, a query WHERE region != 'us-west' across two shards is constructed as:

1
2
3
4

DifferenceSeriesIDIterators(
  MergeSeriesIDIterators(Shard1.MeasurementIterator, Shard2.MeasurementIterator),
  MergeSeriesIDIterators(Shard1.TagValueIterator("region","us-west"), Shard2.TagValueIterator("region","us-west"))
)

InfluxDB 3.0: FDAP Architecture (Columnar Storage Revolution)

InfluxDB 3.0 (formerly InfluxDB IOx) completely abandoned the custom storage format in favor of the FDAP technology stack^[7][8][9]:

flowchart TD
    A[Flight<br/>gRPC+Arrow Transport] --> B[DataFusion<br/>Vectorized Query Engine]
    B --> C[Arrow<br/>In-Memory Columnar Format]
    C --> D[Parquet<br/>Disk Columnar Storage]

    style A fill:#2196F3,color:#fff
    style B fill:#9C27B0,color:#fff
    style C fill:#4CAF50,color:#fff
    style D fill:#4CAF50,color:#fff

Figure description: The diagram shows the four-layer FDAP technology stack of InfluxDB 3.0. Flight provides gRPC-based Arrow array network transport; DataFusion is a Rust vectorized SQL query engine supporting projection/filter pushdown and automatic parallelization; Arrow is an in-memory columnar format enabling zero-copy sharing across components; Parquet provides efficient disk columnar storage. Blue indicates the transport layer, purple the query engine, and green the data format layer.

Core design decisions and data^[7]:

• Parquet replaces custom TSDB format — measured ~10x overall compression ratio vs raw data; InfluxData has not published exact comparison figures relative to TSM
• Arrow: in-memory columnar format, cache-friendly, eliminates serialization overhead, zero-copy sharing across components
• DataFusion: Rust-based vectorized SQL query engine, supports projection/filter pushdown and automatic parallelization
• Flight: gRPC-based network transport protocol for Arrow arrays, serialization-free

Architecture components^[8][9]:

• Ingester: Receives writes, caches in Arrow in memory (immediately queryable), WAL ensures durability, periodically flushes Parquet files to S3/GCS/Azure Blob
• Querier: Builds query plans, queries Ingester for recent unpersisted data, queries Catalog for Parquet file locations
• Compactor: Background merges small Parquet files into larger ones (similar to LSM compaction)
• Storage-compute separation (Lakehouse model): Stateless compute nodes, infinitely scalable storage on object store

Prometheus

Local TSDB Storage Engine

Prometheus ships with a built-in local on-disk TSDB using a custom efficient format, averaging only 1-2 bytes per sample^[10].

On-disk file layout^[10]:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

./data
├── 01BKGV7JBM69T2G1BGBGM6KB12     ← Persistent Block (2 hours)
│   ├── chunks/                     ← chunk segment files (max 512MB each)
│   │   └── 000001
│   ├── tombstones                  ← deletion markers
│   ├── index                       ← inverted index (label → series mapping)
│   └── meta.json
├── chunks_head/                    ← Completed Head Block chunks (mmap)
│   └── 000001
└── wal/                            ← Write-Ahead Log
    ├── 000000002                   ← 128MB segments
    └── checkpoint.00000001

Head Block and Persistent Block

• Head Block: The block currently receiving writes, held in memory, with WAL ensuring crash recovery^[10]
• Persistent Block: Once the Head Block is full (2 hours), it is persisted as an immutable Block containing chunks, index, and tombstones^[10]

Chunk Encoding and Gorilla Compression

flowchart TD
    A["Raw Data<br/>120 samples / 2h"] --> B["Gorilla XOR Encoding"]
    B --> C["delta-of-delta<br/>Timestamps"]
    B --> D["XOR<br/>Values"]
    C --> E["Sealed Chunk<br/>150-250 bytes"]
    D --> E
    E --> F["Immutable<br/>Concurrency-safe"]

    style A fill:#FF9800,color:#fff
    style B fill:#9C27B0,color:#fff
    style C fill:#2196F3,color:#fff
    style D fill:#2196F3,color:#fff
    style E fill:#4CAF50,color:#fff
    style F fill:#4CAF50,color:#fff

Figure description: The diagram shows the Prometheus Chunk structure. Each chunk contains 120 samples or 2 hours of data (whichever comes first), encoded with Gorilla XOR (delta-of-delta timestamps, XOR values), approximately 150-250 bytes in size. Chunks are immutable once sealed, ensuring safe concurrent reads.

Prometheus chunks are sealed, immutable, Gorilla-XOR encoded data blocks, typically 150-250 bytes^[2][3].

mmap Mechanism

Prometheus TSDB aggressively uses mmap (memory-mapped files)^[10][3]: index files and chunk files of persistent Blocks are mmap’d, as are completed chunks in the Head Block (chunks_head/). Benefits: OS-managed page cache (LRU), zero-copy access, reduced process memory footprint^[3].

WAL Mechanism and Remote Storage Interfaces

• WAL stored in wal/ directory with 128MB segments, containing raw uncompressed data^[10]
• At least 3 WAL files retained, WAL compression enabled by default (halves WAL size since 2.20.0)
• Supports out-of-order ingestion (tsdb.out_of_order_time_window)

Prometheus integrates with remote storage through four mechanisms^[10]: Remote Write (write to remote URL, snappy-compressed protobuf over HTTP), Remote Write Receiver (receive samples from other clients), Remote Read (read samples back from remote URL), and Remote Read responses.

Key limitations: Remote read path only fetches raw series data; all PromQL evaluation happens locally, creating scalability constraints^[10]. Default retention is 15 days, single-node with no clustering^[10].

VictoriaMetrics

Cluster Component Architecture

For time-series databases, “clustering” solves a fundamental problem: what happens when a single node can’t keep up? VictoriaMetrics splits write, storage, and query into three independent services, each independently scalable and focused on its own role.

VictoriaMetrics cluster employs a Shared-Nothing architecture consisting of three independently scalable services^[11]:

Animation: The animation shows VM cluster data flow in phases. Phase 1 (15-35% cycle) shows a write request entering vminsert and being distributed to vmstorage nodes via consistent hashing; Phase 2 (35-55%) shows a query request entering vmselect, fanning out to all vmstorage nodes; Phase 3 (55-75%) shows vmstorage returning results; the final phase summarizes the Shared-Nothing architecture.

Two smaller diagrams below show the write path and read path separately:

flowchart TD
    A[vminsert<br/>Write Router] --> B[vmstorage<br/>Node A]
    A --> C[vmstorage<br/>Node B]
    A --> D[vmstorage<br/>Node C]
    A --> E[vmstorage<br/>Node D]

    style A fill:#2196F3,color:#fff
    style B fill:#4CAF50,color:#fff
    style C fill:#4CAF50,color:#fff
    style D fill:#4CAF50,color:#fff
    style E fill:#4CAF50,color:#fff

Figure description: Write path. vminsert (blue) distributes data to vmstorage nodes (green) via consistent hashing over metric name and all labels.

flowchart TD
    A[vmselect<br/>Query Handler] --> B[vmstorage<br/>Node A]
    A --> C[vmstorage<br/>Node B]
    A --> D[vmstorage<br/>Node C]
    A --> E[vmstorage<br/>Node D]

    style A fill:#2196F3,color:#fff
    style B fill:#4CAF50,color:#fff
    style C fill:#4CAF50,color:#fff
    style D fill:#4CAF50,color:#fff
    style E fill:#4CAF50,color:#fff

Figure description: Read path. vmselect (blue) fans out queries to all vmstorage nodes (green) in parallel.

• vminsert: Distributes data to vmstorage nodes via consistent hashing on metric name + all labels^[11]
• vmstorage: Stores raw data; nodes do not communicate or share data
• vmselect: Executes queries, fetching data from all vmstorage nodes

Storage Engine: Improved MergeTree / LSM

vmstorage’s storage engine is based on a LSM-Tree-like structure, with data going through multi-level transformations^[12]:

flowchart TD
    A[In-memory Part<br/>1-17 MB] --> B[Small Part<br/>Disk ≤10 MB]
    B --> C[Big Part<br/>Disk ≤1 TB]

    style A fill:#FF9800,color:#fff
    style B fill:#2196F3,color:#fff
    style C fill:#4CAF50,color:#fff

Figure description: The diagram shows the three-tier LSM Part lifecycle in the VictoriaMetrics storage engine. In-memory Parts (orange, 1-17 MB) are the first flush product and immediately queryable; Small Parts (blue, disk ≤10 MB) result from memory flushes or merges; Big Parts (green, disk ≤1 TB) are produced by subsequent merges. The warm-to-cool color transition represents the data lifecycle from hot to cold.

Three LSM Part types^[12]:

Flush triggers^[12]: Size-based (buffer reaches ~120MB) or time-based (more than 2 seconds since last flush). Maximum 60 in-memory parts, consuming approximately 10% of system memory.

Merge Mechanism and Compression Algorithms

Merging does not run on a fixed schedule but is based on “causality”^[12]:

Merge Multiplier^[12]: Defines the ratio of total output part size to maximum input part size; default value is 7.5, maximum 15 parts per merge. Deduplication runs during merge, configured via -dedup.minScrapeInterval.

Compression algorithms^[3][12]:

• Hot data path: Gorilla encoding (delta-of-delta + XOR)
• Cold data (exported to S3 as Parquet): Gorilla + ZSTD stacked, achieving 25-35x compression ratio
• Significantly reduced memory footprint compared to Prometheus local storage, capable of handling tens of millions of time series on a single node

Level Downsampling

Beginner analogy: You don’t need to remember every second of CPU utilization from a week ago — hourly averages are enough to spot trends. TSDB “level downsampling” works on the same principle: the older the data, the lower the retained resolution, saving storage space without sacrificing trend analysis.

VictoriaMetrics does not have built-in automatic downsampling (it requires external tools like vmctl), but understanding this concept is crucial for grasping TSDB hot/cold tiering:

Animation: The animation shows four stages of TSDB level downsampling. Raw data (orange, 15s interval) is progressively merged through multiple compaction passes into larger time window aggregates — 5-minute rollups (avg/min/max, 20 points → 1) → 1-hour rollups (12 × 5m → 1) → 1-day rollup (24 × 1h → 1). Storage density improves exponentially as resolution decreases — from one data point per 15 seconds to one point per day, a 5760x reduction.

Performance Characteristics

Stream Aggregation: VictoriaMetrics provides built-in stream aggregation capabilities for real-time downsampling and aggregation at the ingestion side, reducing storage volume and query load. For a deeper dive, see the Stream Metrics Series.

TimescaleDB

Core Positioning: PostgreSQL Extension

TimescaleDB is a PostgreSQL extension that layers time-series optimizations on top of Postgres’ robust transaction engine, replication, and ecosystem^[3][15].

Hypertable and Chunk Architecture

flowchart TD
    A[Hypertable<br/>Logical Table] --> B[Chunk 1<br/>T0-T1]
    A --> C[Chunk 2<br/>T1-T2]
    A --> D[Chunk 3<br/>T2-T3]
    A --> E[Chunk N<br/>Tn-1-Tn]

    style A fill:#9C27B0,color:#fff
    style B fill:#2196F3,color:#fff
    style C fill:#2196F3,color:#fff
    style D fill:#2196F3,color:#fff
    style E fill:#2196F3,color:#fff

Figure description: The diagram shows the TimescaleDB Hypertable logical architecture. The Hypertable (purple) is a user-transparent logical table automatically partitioned by time range into multiple Chunks (blue), each backed by a standard PostgreSQL heap table. INSERT operations route to the appropriate Chunk based on timestamp, while SELECT queries skip irrelevant time partitions through Chunk Exclusion.

• Hypertable: User-facing logical table, automatically partitioned by time^[15]
• Chunk: Backed by a standard Postgres heap table, subject to MVCC and locking constraints^[3][15]

Compression Strategy: Columnar Storage

Compression algorithms (per-column)^[15]: Dictionary Encoding, Delta Encoding, Gorilla Compression (floating-point XOR), Run-Length Encoding.

Compression effectiveness^[14][15]: Chunk compression reduces storage by up to 98%; the columnar format supports fast scans and aggregations, and columnar data can be queried transparently without decompression.

Hypercore (TimescaleDB 2.18, 2025)^[17]: Unified rowstore/columnstore engine, allowing a single hypertable to simultaneously hold hot rows plus compressed cold columns, with automatic transition based on chunk age.

Continuous Aggregates

Continuous aggregates are incrementally updated materialized views that avoid recomputing aggregations on every query^[14][15]:

flowchart TD
    A[Materialization Engine<br/>聚合引擎] --> B[Materialization Hypertable<br/>存储聚合结果]
    C[Invalidation Engine<br/>失效检测] --> A
    D[Query Engine<br/>查询引擎] --> B
    style A fill:#2196F3,color:#fff
    style B fill:#4CAF50,color:#fff
    style C fill:#FF9800,color:#fff
    style D fill:#9C27B0,color:#fff

Figure description: The diagram shows the four-component Continuous Aggregates architecture. The Invalidation Engine (orange) detects data changes in the raw hypertable, notifying the Materialization Engine (blue) to perform incremental aggregation. Results are stored in the Materialization Hypertable (green). The Query Engine (purple) transparently queries the pre-aggregated data.

Key mechanisms^[14]: Incremental updates (only processing new/changed data), materialization threshold (lag time point), two-phase transactions, invalidation engine tracking data changes. Supports hierarchical continuous aggregates (second→hour→day) and real-time aggregates (pre-aggregated data merged with latest raw data)^[14].

sequenceDiagram
    participant Raw as 原始 Hypertable
    participant IE as Invalidation Engine
    participant ME as Materialization Engine
    participant Mat as 物化 Hypertable
    Raw->>IE: 数据变更通知
    IE->>IE: 计算失效区间
    IE->>ME: 触发增量刷新
    ME->>Raw: 读取新增/变更数据
    ME->>ME: 执行聚合计算
    ME->>Mat: 写入物化结果

Figure description: The diagram shows the incremental refresh flow of TimescaleDB Continuous Aggregates. Data changes in the raw Hypertable notify the Invalidation Engine to compute invalidation intervals, triggering the Materialization Engine to read new/changed data, perform aggregation, and write results to the Materialization Hypertable. The entire process is transparent to users and executes incrementally.

Differences from Native TSDBs

TSDB Cross-Comparison

Storage Structure Comparison

Write Path and Compression Algorithm Comparison

Cluster/Distributed Architecture Comparison