Log Storage Architecture: From Inverted Indexes to Weak Indexing

Introduction

Logs generate the largest data volume and highest storage cost among the three pillars of observability. Unlike metrics with their fixed numeric structure, logs are variable-length text with semi-structured characteristics, posing unique storage design challenges: supporting full-text search while controlling storage costs.

The core design dimension of log storage systems is indexing strategy — deeper indexing means faster search, but also higher storage costs. From Elasticsearch’s full inverted index to Loki’s label-only indexing, each system makes different technical choices along this spectrum.

This article provides an in-depth analysis of five mainstream log storage systems — Grafana Loki, VictoriaLogs, Elasticsearch/OpenSearch, ClickHouse, and Quickwit — covering their architecture design, indexing strategies, and core technical tradeoffs.

Quick Reference — Log Storage Concepts in One Sentence

Concept	Analogy	Explanation
Inverted Index	Book index at the back	Records which words appear in which docs, fast search but index size is 3-5x data size
Weak Indexing	Table of contents only	Knows where logs roughly are, but needs scanning to find specific content
Chunk	Compressed archive	Groups log entries together and compresses them to save space
Object Storage	Cloud drive (S3/GCS)	Cheap but slower mass storage, ideal for cold data
Columnar Storage	Organizing data by column	Same column stored contiguously for better compression and read efficiency

If terms like inverted index or object storage are new to you, don’t worry — the diagrams and animations below will show them in the most intuitive way. The key takeaway: log storage design is a spectrum — deeper indexing means faster search but higher storage costs.

Grafana Loki

Design Philosophy: Index Labels, Not Log Content

Loki’s core philosophy is “do for logs what Prometheus does for metrics” — it indexes only streams (unique label sets), while log content itself is simply compressed and stored in chunks without building inverted indexes. This results in extremely small index sizes, reducing storage costs to approximately 1/10th of Elasticsearch or less^[L1][L3].

Underlying Data Structures: Index and Chunks

Index: Records where logs for a given label set are stored. Two formats — TSDB (recommended, derived from Prometheus TSDB index format, scalable) and BoltDB (deprecated)^[L1][L3].

Chunk on-disk format^[L1]:

Block internals (decompressed): Each log entry = ts(nanosecond timestamp) | len | log bytes | symbols reference^[L1]. The symbols mechanism references strings in structuredMetadata to reduce duplicate storage.

Write Path and Persistence

Loki’s write path employs a Cassandra-like consistent hashing architecture. The following diagram shows the complete flow from client to persistence:

sequenceDiagram
    participant Client as Client
    participant Dist as Distributor
    participant Ingest as Ingester
    participant Quorum as Quorum
    participant OSS as Object Storage

    Client->>Dist: HTTP POST (log stream)
    Dist->>Dist: Consistent hash → target ingester
    Dist->>Ingest: Forward stream
    Ingest->>Ingest: Create/append chunk
    Ingest-->>Dist: Ack
    Dist->>Quorum: Wait for majority
    Quorum-->>Dist: Majority confirmed
    Dist-->>Client: 200 OK
    Note over Ingest,OSS: Periodic flush
    Ingest->>OSS: Flush chunk
    Ingest->>OSS: Upload index

Figure 1: Loki Write Path. After the Distributor receives HTTP POST log streams, it hashes each stream to determine the target ingester via a consistent hash ring. The Ingester receives the stream, creates or appends a chunk for that (tenant, label set), and acknowledges the write. The Distributor waits for a majority (quorum) of acknowledgments before returning success. The Ingester accumulates chunks in memory and periodically flushes them to object storage; indexes are uploaded via the index shipper^[L1].

Persistence: Relies on object storage (S3/GCS/Azure) as the long-term storage backend, with ingester memory + WAL providing recent data and disaster recovery.

Beginner Analogy: Loki’s chunk is like packing scattered parcels into a large box. Without packing (ES’s approach), each parcel takes its own space and can be opened individually; packing saves space but finding a specific item means first opening the box. Loki chooses a strategy of “pack-and-store + label-only index” — it knows which box might contain what you need, but you have to open the box to know for sure.

The following animation shows the complete chunk lifecycle from creation to persistence:

Animation: The animation above shows the Loki chunk lifecycle — log streams continuously arrive at the Ingester, accumulate in memory buffer to ~256KB, are compressed via gzip/snappy into ~25KB chunks, and then flushed to object storage (S3/GCS) for persistence. Loki’s key optimization: it does not index log content at all, only records which stream’s logs are in which chunk, dramatically reducing storage costs. This stands in stark contrast to Elasticsearch, which builds inverted indexes for every single log entry.

Loki 3.0 TSDB Storage Engine Improvements

Loki 2.8 introduced the TSDB (Time Series Database) index format, based on the Prometheus TSDB sub-project, as the recommended index backend for schema v13+^[L1][L3]. The TSDB discussed here refers to Loki’"’"’s index file format (not a time-series database itself), which fundamentally replaced the BoltDB embedded KV store index used in earlier Loki versions and serves as the foundation for all new storage features.

From BoltDB to TSDB: Why the Switch?

Earlier Loki versions used BoltDB (an embedded B-tree KV store) as the index backend, uploading index files to object storage via the boltdb-shipper. BoltDB has three inherent limitations:

• Uncontrollable memory: BoltDB files cannot release memory on demand after mmap, causing querier memory to grow continuously under high cardinality
• High query overhead: Each query requires deserializing entire entry lists from BoltDB buckets, causing severe I/O amplification
• Mandatory index cache: BoltDB queries depend on external cache layers (memcached/redis), increasing operational complexity and cost

TSDB fundamentally solves these problems: it uses an inverted index structure (Postings + Series), supports zero-copy reads after mmap, eliminates the need for index caching, and enables dynamic query sharding^[L5][L6].

TSDB Write Path

The following diagram illustrates the complete write path with TSDB:

flowchart TD
    Agent["Log Collector<br/>(Promtail/Alloy)"] -->|push| Distributor
    Distributor -->|hash ring| Ingester
    Ingester -->|WAL + in-memory chunk| Chunk["Chunk (memory)"]
    Chunk -->|flush @ 256KB/30min| ObjStore["Object Storage<br/>S3/GCS"]
    Ingester -->|build TSDB block| TSDBBlock["TSDB Index Block"]
    TSDBBlock -->|shipper upload| ObjStore
    Compactor -->|merge & deduplicate| ObjStore

    style Agent fill:#FF9800,color:#fff
    style Distributor fill:#2196F3,color:#fff
    style Ingester fill:#2196F3,color:#fff
    style ObjStore fill:#9C27B0,color:#fff
    style Compactor fill:#E65100,color:#fff

Write path explanation: Logs flow through the Distributor (consistent hash routing) to the Ingester, which writes to WAL while building chunks in memory. When a chunk reaches ~256KB or the 30-minute age limit, it is compressed and flushed to object storage. Simultaneously, the Ingester builds a TSDB index block containing the inverted index for all streams in that period, which is asynchronously uploaded via the TSDB shipper. The Compactor merges small TSDB blocks and cleans up deleted series.

TSDB Index Structure and Query Flow

A TSDB index file contains the following core sections:

• Symbol Table: A deduplicated string dictionary – label names and values are referenced by integer symbols, significantly reducing index size
• Series: Entries sorted lexicographically by labels, each with the full label set and corresponding chunk references (including min/max time range)
• Postings: Label pair to matching series ID list mappings – the core of the inverted index
• Postings Offset Table: Label pair to postings section offset index, sorted by label name/value, partially loaded into memory during queries
• TOC (Table of Contents): File entry point recording offsets to each section

The animation below demonstrates how a {app="api"} query navigates the TSDB index structure to locate matching chunks:

Animation explanation: The animation above shows the complete TSDB index query path in four stages: (1) The query {app="api"} enters and looks up the matching label pair in the Postings Offset Table; (2) The offset directs to the postings section, returning matching series IDs [0, 2]; (3) These series entries are resolved to obtain full label sets and chunk references; (4) Overview showing the complete query chain.

Query Path and Dynamic Sharding

The TSDB query path is significantly more efficient than BoltDB’"’"’s^[L5][L6]:

1. Index Gateway caching: The Index Gateway acts as a front-end cache for TSDB indexes, downloading and caching recent TSDB blocks from object storage. Queriers retrieve index data via gRPC from the Gateway, avoiding direct S3 reads
2. Dynamic query sharding: TSDB stores per-chunk metadata (size in KB and line count) in each index entry. The Query Frontend uses this to estimate data volume for each time slice and dynamically determines shard granularity – targeting 300-600MB per shard. Compared to BoltDB’"’"’s static sharding (by fixed index row count), dynamic sharding provides more balanced distribution under fluctuating data volumes
3. No index cache needed: The compact and mmap-friendly TSDB format eliminates the need for memcached/redis as an index cache layer, reducing operational complexity

Structured Metadata (Loki 3.0)

Loki 3.0 enables Structured Metadata by default^[L4]: metadata fields that are neither labels (too large/high-cardinality) nor log content, but are commonly used in queries (e.g., trace_id, http.status, method), are stored in a dedicated metadata section. These structured metadata values are also indexed in TSDB, supporting queries like {app="api"} | status="500" – status is not a label, but can be quickly filtered through the structured metadata index.

Bloom Filters Evolution (Loki 3.0 -> 3.3)

Bloom Filters serve as Loki’"’"’s acceleration mechanism for compensating the lack of log content indexing^[L5][L6]:

• Loki 3.0: Builds Bloom filters from n-grams of log content, enabling quick checks of whether a chunk might contain a specific keyword, avoiding full scans
• Loki 3.3: Switches to building Bloom filters from structured metadata keys and key-value pairs (block schema V3). Since metadata value space is much smaller than content n-grams, Bloom filter sizes are reduced by several orders of magnitude

Limitations

Despite TSDB’"’"’s significant index performance improvements, Loki’"’"’s architectural limitations remain:

• High-cardinality labels are fatal: High label cardinality generates massive numbers of streams, inflating indexes and stressing ingester memory. Label cardinality must be strictly controlled
• Weak full-text search: Without indexing log content, large-scale text search relies on full scanning; Bloom filters only accelerate structured metadata scenarios
• Object storage dependency: Historical query latency is affected by object storage IOPS; Index Gateway caching alleviates but cannot completely eliminate this

VictoriaLogs (vmlogs)

Relationship with VictoriaMetrics

VictoriaLogs is developed by the same team as VictoriaMetrics. It is an independent log database that reuses VM’s storage engine design philosophy (per-day partitioning, merge compaction, columnar layout), and its cluster roles align with VM (vlinsert/vlselect/vlstorage)^[VL8][VL9]. Note that VictoriaLogs cluster mode does not automatically replicate between storage nodes like VM does; instead, it achieves HA by replicating to multiple independent targets at the ingestion layer (vlagent)^[VL8].

Storage Architecture: Columnar LSM-Tree

VictoriaLogs uses a columnar LSM-Tree architecture inspired by ClickHouse^[VL6][VL9]. The write pipeline:

1. Per-CPU sharded buffers: Incoming logs are distributed to per-CPU ring buffers to avoid lock contention
2. In-memory part (~1s): Buffer fills or timer triggers → flushed to in-memory part, immediately queryable
3. Small part: In-memory parts merge into small parts (written to disk, stay in page cache)
4. Big part: Small parts merge further into big parts (up to ~1TB, immutable, bypasses page cache on write)

Key property: Parts are immutable once written; compaction compresses fully with no wasted space. Snapshots are directory copies with no consistency coordination.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

victoria-logs-data/
└── partitions/
    ├── 20260109/               # Per-day partition (UTC)
    │   └── datadb/
    │       ├── 1882C35B4CE64498/   # Part (hex ID)
    │       │   ├── metadata.json            # Part metadata
    │       │   ├── index.bin + metaindex.bin # Block index
    │       │   ├── timestamps.bin            # Timestamp column (delta encoded)
    │       │   ├── values.bin0..127          # Field value columns (up to 128 shards)
    │       │   ├── bloom.bin0..127           # Per-column bloom filters
    │       │   ├── message_values.bin        # _msg field
    │       │   └── column_names.bin          # Column name index
    │       └── parts.json                    # Active part list
    └── 20260110/

Essence of columnar storage: Same-field values are stored contiguously. All status values sit together in values.bin — same-distribution values compress extremely well with ZSTD. A query | fields _time, status only reads those two columns, skipping everything else^[VL6].

Write Pipeline

The Mermaid diagram below shows the complete write path from log source to disk:

flowchart TD
    Logs["Log Sources"] --> Buffer["per-CPU Buffer"]
    Buffer --> InMem["In-Memory Part"]
    InMem --> Small["Small Part"]
    Small --> Big["Big Part"]
    Big --> Retention["Daily Delete"]

Index Design: Bloom Filters over Inverted Indexes

This is VictoriaLogs’ most distinctive design decision — using Bloom Filters instead of inverted indexes^[VL6][VL8]:

• Every column in every block has a bloom filter
• On ingestion, field values are auto-tokenized (words) and bloom filter fingerprints are built
• Bloom filter size: ~2 bytes per unique token (1,000 unique words ≈ 2KB bloom)^[VL6]
• On query: check bloom first → “definitely not” = skip block → “maybe” = read values

Two-level index filtering (index.bin + metaindex.bin):

• metaindex.bin stays in memory, mapping stream + time ranges to index blocks
• index.bin stores block headers (stream, time range, row count, byte offsets)
• Query path: filter metaindex (in-memory) → read matching index blocks → check bloom filters → read values

The animation below shows how bloom filters check each block, skipping non-matching blocks to read only potentially matching data:

Animation notes: Four stages of bloom filter query filtering: (1) Time-range filter narrows to the correct partition; (2) metaindex filters by stream, excluding non-matching blocks (web stream eliminated); (3) Each remaining block’s bloom filter is checked for “500” (✓ = maybe, ✗ = definitely not); (4) Only passing blocks read values — 50% reduction in blocks scanned.

Log Streams Concept

VictoriaLogs uses a stream concept similar to Loki^[VL8]:

• Stream fields (low cardinality): host, app, pod, container — identify the log source
• Non-stream fields (high cardinality): trace_id, user_id, ip — auto-indexed but not stream-scoped
• Auto-generates _stream_id and _stream labels (Prometheus format: {app="api",host="host-1"})

Compared to Loki: Loki only indexes stream labels. High-cardinality fields either become labels (index bloat) or go into JSON payloads (parsed at query time). VictoriaLogs auto-indexes all fields; high cardinality has no special overhead, and the stream selector {...} is optional.

LogsQL Query Language

LogsQL is VictoriaLogs’ custom query language with Unix-pipe style syntax^[VL8]:

Basic syntax: <filters> | <pipe1> | <pipe2> | ...

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

# Full-text search (defaults to _msg field)
error

# Time range (best practice — queries without time range may be slow)
_time:5m

# Exact field match
status:=500

# Keyword field search
log.level:error

# Phrase matching
"error: connection refused"

# Stream filtering (optional — unlike Loki, not required)
{app="nginx"}

# Pipe chain: sort → select fields → limit
_time:5m error | sort by (_time) desc | fields _time, host, _msg | limit 20

# Aggregation: group by host, compute error rate, filter >10%
_time:5m
  | stats by (host)
      count() logs,
      count() if (error) errors
  | math (errors / logs) as error_rate
  | filter error_rate:>0.1

Compared to LogQL (Loki)^[VL8]:

Performance Benchmarks

Third-party benchmarks (4 vCPU / 8GiB RAM, ~500GB logs, 7 days)^[VL11]:

Architecture Components

Single-node mode^[VL8]: One binary victoria-logs-prod, port 9428, zero config, auto-adapts to CPU/RAM.

Cluster mode (same binary, different flags)^[VL8]:

• vlinsert: Ingest gateway, distributes by stream hash across vlstorage nodes
• vlselect: Query gateway, fans out queries and merges results
• vlstorage: Storage node (each is a standalone instance)

Data durability: In-memory data flushes to disk every 5 seconds (configurable via -inmemoryDataFlushInterval)^[VL6], queryable within ~1 second of ingestion.

Compression and Cost

VictoriaLogs’ extreme compression comes from three layers:

1. Columnar layout: Same-type values stored contiguously, excellent ZSTD compression
2. Type-aware encoding: IP → 4 bytes, timestamps → delta encoding, repeated values → dictionary encoding
3. Immutable parts: Full compression on merge, no wasted space

Typical compression ratio 10x-100x+^[VL9]: structured logs (nginx access logs) can reach 100x+; random text logs ~10x. 100GiB disk stores over 1TB of uncompressed logs. Uses up to 30x less RAM and 15x less disk than ES/Loki^[VL9].

Elasticsearch / OpenSearch

Inverted Index Principles (Lucene Internals)

Each ES/OpenSearch shard is a Lucene index composed of multiple immutable segments^[ES13][ES17]. Core inverted index structures:

• Term Dictionary: Maps terms to their Posting Lists
• Posting List: Records which documents contain a given term (doc ID list), typically compressed using FOR (Frame of Reference) or Roaring bitmaps

Beginner Analogy: An inverted index is like the index section at the back of a book. Without an index, finding a keyword means flipping through every page — accurate but extremely slow; with an index, you look up the keyword and instantly find the page numbers. The cost is that the index itself takes up space — a book’s index may add dozens of pages. In log scenarios, this index bloat is far more severe: each log line contains dozens of terms, and the index size can reach 3-5x the original data size.

The following animation shows the complete inverted index construction process from raw log text:

Animation: The animation above shows the four stages of inverted index construction. First, a raw log line appears, then it is split into individual terms (orange blocks), and each term generates a dictionary entry pointing to a list of document IDs containing that term (green blocks). Note that a single log line with just 6 terms produces 6 index records — in production with tens of thousands of log lines per second, the index size quickly balloons to 3-5x the original data size. This is the fundamental reason why Elasticsearch is fast to search but expensive to store.

Segment Structure and File Formats

Lucene segments are immutable and contain multiple file types: .tim/.tip (term dictionary + term index, FST structure, prefix compression), .doc/.pos/.pay (posting lists), .fdt/.fdx/.fdm (stored fields), .dvd/.dim (doc values, columnar), .cfs/.cfe (compound files).

Write Path and NRT Mechanism

ES’s write path achieves near-real-time (NRT) searchability through the refresh mechanism. The following diagram shows the complete flow from document write to disk persistence:

sequenceDiagram
    participant Client as Client
    participant Buf as In-Memory Buffer
    participant TL as Translog
    participant FSCache as FileSystem Cache
    participant Disk as Disk

    Client->>Buf: Write document
    Client->>TL: Write translog
    Note over Buf,FSCache: refresh (default 1s)
    Buf->>FSCache: New segment (searchable)
    Note over TL,Disk: flush
    FSCache->>Disk: fsync segment to disk
    TL->>Disk: commit + clear translog

Figure 2: Elasticsearch NRT Write Mechanism. Documents are written to an in-memory buffer while simultaneously being written to the translog (transaction log). By default, a refresh occurs every second, which generates a new segment from the buffer into the filesystem cache — at this point, the documents become searchable, which is the essence of “near-real-time.” A flush operation executes a Lucene commit (fsync-ing segments to disk), clears the translog, and writes a commit point^[ES12][ES13].

Translog Persistence Mechanism

The translog ensures that acknowledged but not-yet-persisted operations between commits can be recovered after a crash^[ES11]:

• request (default): fsync + commit translog after every request
• async: Background fsync every sync_interval (default 5s)
• flush_threshold_size: Default 10GB, triggers a flush when translog reaches this size

Logging Scenario Optimization: ILM + Data Tiers

Data Tiers^[ES14]: Hot (NVMe/SSD, newest data) → Warm (cheaper SSD/HDD) → Cold (rarely queried) → Frozen (based on searchable snapshots, object storage).

ILM Five Phases^[ES15]: hot → warm → cold → frozen → delete, with actions per phase (rollover/shrink/forcemerge/allocate/searchable_snapshot/delete).

Pros and Cons for Logging

Pros: Strongest full-text search (inverted index + complex query DSL), mature ecosystem, powerful aggregation and analytics.

Cons: High storage costs (inverted index + doc values + stored fields cause index bloat), large JVM heap memory overhead, high-cardinality fields can lead to oversized shards.

ClickHouse as Log Storage

Columnar Storage for Logs

ClickHouse is an analytical columnar database that stores each column contiguously. When used for logs, log fields are mapped to columns, high-frequency filter columns (timestamps, tags) serve as ORDER BY/PARTITION BY keys, and log content is stored as a compressed String column.

MergeTree Engine

• Data is partitioned by PARTITION BY; within each part, data is sorted by ORDER BY keys and divided into granules (default 8192 rows) as the minimum read unit
• Sparse primary index: Each granule records primary key values (marks), allowing queries to skip irrelevant granules based on key ranges
• Columnar compression: Each column is compressed independently (LZ4 default, ZSTD configurable)^[CH19]

Data Skipping Indexes

ClickHouse’s columnar storage does not maintain row-level secondary indexes; instead, it uses data skipping indexes to record statistical information per index granularity^{[CH17][CH19][CH22]}:

• minmax: Records min/max values per index granularity. Altinity benchmark: 100 million row query from 0.167s → 0.017s^[CH19]
• bloom_filter: Bloom filter, suitable for equality/IN queries. 110 million row array has(x,42) from 0.505s → 0.063s, ~10x acceleration^[CH19]
• ngrambf_v1 / tokenbf_v1: n-gram/token Bloom filters for LIKE and phrase queries
• ClickHouse 25.9 introduced experimental full-text index (still experimental as of writing; assess stability before production use)^[CH20]

Quickwit

Positioning: Cloud-Native Search Engine for Logs/Traces

Quickwit is written in Rust, built on Tantivy (a Rust full-text search library comparable to Lucene), and designed specifically for observability workloads^[QW23][QW25]. Core architectural choice: all index data stored on object storage, compute-storage separation, with search costs approximately 1/10th of Elasticsearch^[QW25].

Architecture Overview

A Quickwit cluster consists of four core components^[QW23]:

• Indexer: Ingests data streams (Kafka, Kinesis, HTTP), buffers documents in memory, builds Tantivy splits upon threshold, uploads to object storage, and publishes metadata
• Searcher: Stateless search node that efficiently locates matching documents via hotcache, fetching only necessary inverted index data from object storage
• Metastore: Backed by PostgreSQL (cluster mode), manages split state, time ranges, file locations and other metadata
• Control Plane: Coordinator node responsible for split assignment, merge scheduling, and cluster topology management

Hotcache Deep Dive

Hotcache is the key innovation enabling Quickwit’s stateless search^[QW23][QW24]. In essence, the hotcache is a split’s inverted index term dictionary — containing all indexed terms with their posting list offsets and document frequencies.

1
2
3
4
5
6
7
8
9

// hotcache content structure (conceptual)
{
  "terms": [
    {"term": "error", "posting_offset": 0, "doc_freq": 1542},
    {"term": "timeout", "posting_offset": 124, "doc_freq": 233}
  ],
  "field_stats": { "doc_count": 50000, "avg_field_len": 128 },
  "block_index": { "block_size": 128, "blocks": [] }
}

When building an inverted index, Tantivy uses block-wise variable-length compression encoding — terms are grouped into blocks (128 terms per block), each block stores the first term’s value as a key, and subsequent terms store only the delta from the previous term. Combined with a skip list structure, looking up any term requires only O(log N) comparisons and a few random reads — this keeps the hotcache file typically a few KB to tens of KB.

Hotcache-driven search process:

1. Split pre-filtering: Searcher retrieves candidate splits from Metastore (already pruned by time range)
2. Fetch Hotcache: Downloads each candidate split’s hotcache from object storage (~60ms, micro-request)
3. Term Lookup: Binary searches the hotcache to check if the query term exists
4. On-demand Loading: Only for splits containing the matching term, fetch the full inverted index from object storage and decode the posting list
5. Skip Optimization: Splits without the term require zero additional I/O — this is Quickwit’s core advantage over traditional solutions

This design allows Searcher nodes to be fully stateless — no local index cache needed, elastic scaling at any time.

Index Configuration Modes

Quickwit offers three index modes per field, balancing search capability with storage cost^[QW23][QW25]:

Index configuration example (YAML):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22

version: 0.7
index_id: "nginx-logs"
doc_mapping:
  field_mappings:
    - name: message
      type: text
      tokenizer: raw
      record: basic
    - name: status
      type: u64
      fast: true
      indexed: false
    - name: latency_ms
      type: u64
      fast: true
    - name: body
      type: json
      stored: true
  timestamp_field: "@timestamp"
indexing_settings:
  commit_timeout_secs: 30
  split_num_docs_target: 1000000

Write Pipeline

Quickwit’s indexing write follows a “buffer → build → upload → publish” four-stage model:

flowchart TD
  A["Data Source<br/>Kafka / HTTP"] --> B["Indexer<br/>Buffer Accumulation"]
  B --> C["Build Split<br/>+ Hotcache"]
  C --> D["Upload to<br/>Object Storage"]
  C --> E["Publish Metadata<br/>to Metastore"]
  E --> F["Searcher<br/>Discovers Split"]

Stage details:

1. Buffer Accumulation: Indexer consumes data streams from Kafka / Kinesis / HTTP API, accumulating documents in memory. Default commit interval is 30 seconds or reaching split target size (~5GB uncompressed data)
2. Build Split: Triggers Tantivy index construction — tokenization, building inverted index, encoding fast fields, compressing doc store, and generating hotcache. Each split is UUID-named
3. Upload to Object Storage: Atomically uploads the split directory (all index files + hotcache) to S3/MinIO/GCS. Notifies Indexer upon success
4. Publish Metadata: Indexer writes split metadata to Metastore (PostgreSQL), including UUID, state (published), time range (min/max timestamp), file list and sizes
5. Split Discovery: Searchers poll the Metastore to discover newly published splits and begin serving them

Merge Policy: A background merge process periodically merges small splits into larger ones to reduce the number of splits scanned during queries. After merging, it updates split states in Metastore (staged → published → merged) and removes the merged small splits.

Query Flow

flowchart TD
  A["QQL Query Request"] --> B["Metastore<br/>Time Pruning"]
  B --> C["Load Hotcache<br/>from Object Storage"]
  C --> D{"Term<br/>Exists?"}
  D -->|"Yes"| E["Load Full<br/>Inverted Index"]
  D -->|"No"| F["Skip Split<br/>(Zero I/O)"]
  E --> G["BM25 Scoring<br/>& Merge"]
  F --> B
  G --> H["Read Doc Store<br/>Return Results"]

Query flow details:

1. Query Reception: User submits a QQL query via REST API (or gRPC), e.g. message:"connection refused" AND status:>=500
2. Time Pruning: Metastore uses each split’s timestamp min/max to quickly filter splits matching the time range — first critical pruning layer
3. Hotcache Pre-check: For each candidate split, downloads the tiny hotcache from object storage and checks if the query term exists — second pruning layer, core performance optimization
4. On-demand Loading: Only for splits containing the matching term, fetches the full inverted index from object storage and decodes the posting list
5. Scoring & Ranking: Uses Tantivy’s BM25 algorithm to score matching documents, merges and sorts across splits
6. Result Return: Reads full document rows from the doc store, returns up to top N results

Quickwit Query Language (QQL)

Quickwit provides a Lucene-like query language^[QW25]:

Performance & Benchmarks

Based on official benchmarks and community test data^[QW23][QW25]:

Kalvad case study: After migrating to Quickwit, TCO reduced by 10x, compression ratio 2x better than ES^[QW26].

Cluster Architecture Overview

flowchart TD
  CP["Control Plane<br/>Coordinator"] --> I["Indexer<br/>Writer Node"]
  CP --> S["Searcher<br/>Search Node"]
  I --> OBJ["Object Storage<br/>S3 / MinIO"]
  S --> OBJ
  I --> META["PostgreSQL<br/>Metastore"]
  S --> META

Key design decisions:

• Complete compute-storage separation: All index data persists in object storage; both Indexer and Searcher have no persistent state
• Metastore as source of truth: All split metadata (existence, state, time range) is managed by PostgreSQL
• Lightweight Control Plane orchestration: Handles Indexer/Searcher registration and discovery, split assignment strategy, merge task scheduling
• Elastic scaling: Searcher nodes can scale horizontally based on query load; new nodes simply read the split list from Metastore to start serving

Cross-System Comparison

Storage Cost Comparison

Core Technical Tradeoffs

The fundamental tradeoff in log storage is “index depth vs. storage cost”:

• ES/Quickwit index content → fast search but expensive storage (Quickwit mitigates with object storage + compression)
• Loki/VictoriaLogs/ClickHouse weakly index content → cheap storage but full-text search relies on scanning/skipping (VictoriaLogs compensates with auto field indexing, Loki with Bloom filters, ClickHouse with skipping indexes)

The following diagram illustrates where each system sits on the spectrum between index depth and storage cost:

flowchart TD
    ES["ES/OpenSearch<br/>Full Inverted Index"]
    VL["VictoriaLogs<br/>Auto-Index All Fields"]
    CH["ClickHouse<br/>Skipping Indexes + Columnar"]
    LK["Loki<br/>Labels Only"]

    ES --> VL
    VL --> CH
    CH --> LK

    style ES fill:#f44336,color:#fff
    style VL fill:#2196F3,color:#fff
    style CH fill:#9C27B0,color:#fff
    style LK fill:#4CAF50,color:#fff

Figure 3: Log Storage Indexing Strategy Spectrum. From ES’s full inverted index (red, fastest search but highest cost) to Loki’s label-only indexing (green, lowest cost but full-text search relies on scanning), each system makes different tradeoffs between index depth and storage cost. VictoriaLogs (blue) takes a middle-ground approach with auto-indexing of all fields, while ClickHouse (purple) achieves a unique balance for analytical workloads through skipping indexes and columnar compression.

References

[L1] [Official] Grafana Loki architecture: https://grafana.com/docs/loki/v3.2.x/get-started/architecture/

[L3] [Official] Loki storage (TSDB 2.8 recommended, object storage): https://archive.grafana.com/docs/loki/v3.0.x/configure/storage/

[L4] [Official] Loki 3.0 release notes: https://grafana.com/docs/loki/v3.0.x/release-notes/v3-0/

[L5] [Official] Loki 3.3 release notes: https://grafana.com/docs/loki/latest/release-notes/v3-3/

[L6] [Secondary] Grafana Loki 3.3 Blooms for structured metadata: https://portal.raintank.io/blog/2024/11/21/grafana-loki-3.3-release-faster-query-results-via-blooms-for-structured-metadata/

[VL8] [Official] VictoriaLogs architecture basics: https://victoriametrics.com/blog/victorialogs-architecture-basics/

[VL9] [Official] VictoriaLogs docs: https://docs.victoriametrics.com/victorialogs/

[VL6] [Secondary] VictoriaLogs per-day partitioning and retention: https://docs.victoriametrics.com/victorialogs/

[ES11] [Official] Elasticsearch translog: https://www.elastic.co/guide/en/elasticsearch/reference/8.17/index-modules-translog.html

[ES12] [Official] Elasticsearch translog flush: https://www.elastic.co/guide/en/elasticsearch/guide/master/translog.html

[ES13] [Official] OpenSearch force-merge: https://docs.opensearch.org/3.1/api-reference/index-apis/force-merge/

[ES14] [Official] Elasticsearch data tiers: https://www.elastic.co/guide/en/elasticsearch/reference/7.17/data-tiers.html

[ES15] [Official] Elasticsearch ILM phases and actions: https://docs-v3-preview.elastic.dev/elastic/docs-builder/docs/2927/manage-data/lifecycle/index-lifecycle-management/index-lifecycle

[ES17] [Official] Elasticsearch indices/documents/mappings: https://www.elastic.co/guide/en/elasticsearch/reference/current/documents-indices.html

[CH17] [Official] ClickHouse data skipping indices: https://clickhouse.com/docs/best-practices/use-data-skipping-indices-where-appropriate

[CH19] [Secondary] Altinity KB: ClickHouse skip indexes benchmarks: https://kb.altinity.com/altinity-kb-queries-and-syntax/skip-indexes/printview/

[CH20] [Official] ClickHouse 25.9 release (experimental full-text index): https://clickhouse.com/blog/clickhouse-release-25-09

[CH22] [Official] ClickHouse data skipping indices best practices: https://clickhouse.com/docs/best-practices/use-data-skipping-indices-where-appropriate

[QW23] [Official] Quickwit architecture: https://quickwit.io/docs/main-branch/overview/architecture

[QW24] [Official] Quickwit querying: https://quickwit.io/docs/overview/concepts/querying

[QW25] [Secondary] Quickwit overview: https://rustutils.com/tools/quickwit/

[QW26] [Official] Kalvad migration story: https://quickwit.io/stories/kalvad-migration