Distributed Tracing Storage Architecture: Jaeger, Tempo, SkyWalking and Commercial Solutions

Distributed tracing storage is one of the most technically divergent areas in the observability landscape. Unlike metrics and logging, which have relatively converged storage patterns (TSDB and ClickHouse-like systems), tracing storage has split into three distinct paths due to fundamental disagreements over indexing strategy.

This chapter analyzes the storage architectures of Jaeger, Grafana Tempo, Apache SkyWalking, Zipkin, and the commercial solutions from Datadog and Splunk, concluding with a horizontal comparison.

Beginner Analogy: Distributed tracing is like package delivery tracking — every parcel (request) generates a sign-off record (span) at each sorting center (service) it passes through, and the tracking number (trace_id) ties all records together. Look up the tracking number, and you see the parcel’s complete journey from sender to recipient.

The difference: tracing’s “tracking number” is a 128-bit trace_id, and each service also adds its own “internal work order number” (span_id). The animation below shows how trace context propagates across services:

Animation: The loop above shows a request flowing from Gateway through Service A, Service B, to Database. Each service preserves the same trace_id (the “tracking number”) but generates a unique span_id (its “sign-off record”). The traceparent HTTP header carries this context across service boundaries. When all spans are collected, they form a complete trace tree linked by parent_span_id — enabling end-to-end request visualization.

Jaeger

Jaeger is a CNCF graduated project originally open-sourced by Uber. As one of the reference implementations of OpenTelemetry, it has broad influence in the distributed tracing space.

Span/Trace Data Model

Jaeger’s TraceID at the data model layer is a 16-byte sequence with multiple representations^[J2]: a hex string at the UI layer (32 characters for 128 bits), a pair of unsigned 64-bit integers in the Go domain model (big-endian), and a byte sequence in the Protobuf model.

The Span model includes: traceId, spanId, parentSpanId, operationName, startTime, duration, tags, logs, process (service/tags), and references (follows-from / child-of)^[J4].

Storage Backend and Index Design

The core tradeoff in Jaeger’s backend is reflected in two paths^[J2]:

Cassandra path (KV storage + client-side indexing): Writing a single span is fast, but Jaeger must implement search capability on top of KV storage → severe write amplification: in addition to the span record itself, index entries must be written for service name, operation name, and each individual tag.

Elasticsearch/OpenSearch path (inverted index): Writing one span equals one write operation; all indexing happens inside the ES cluster. Search is powerful, results are consistent, and Kibana can directly query for aggregation analysis.

The official documentation explicitly recommends ES/OpenSearch over Cassandra for the search capability and write amplification reasons cited above^[J2].

Supported storage backends^[J1][J3]: Cassandra 4.x/5.x, Elasticsearch 7.x/8.x, OpenSearch 1.0+, ClickHouse (via Remote Storage API), Badger (embedded), memory (testing), Kafka (buffer only).

Sampling Mechanism (Core Feature)

Jaeger’s client implements consistent head-based (upfront) sampling: the root service makes the sampling decision and propagates it with the request, ensuring that spans in a trace are either all sampled or all not^[J6].

Client sampler types^[J6]: const (all/none), probabilistic (default 0.001 = 0.1%), ratelimiting (leaky bucket), remote (default, pulls strategy from agent/collector).

Adaptive sampling (v1.27+)^[J6]: The Collector observes received spans and recalculates sampling probabilities per service/endpoint combination to match --sampling.target-samples-per-second.

Note: Jaeger has no native tail-based sampling capability, which is a notable limitation compared to OTel Collector / Splunk.

Topology Analysis

The Dependencies page computes on-demand with in-memory storage; it is too expensive with distributed storage and requires offline jobs — the legacy spark-dependencies (periodic Spark) or the experimental jaeger-analytics (Flink streaming continuous build)^[J2].

Grafana Tempo

Tempo is Grafana Labs’ tracing storage system, sharing its design philosophy with Loki.

Design Philosophy: TraceID-Only Indexing (Loki-like)

Tempo’s core philosophy aligns with Loki: no inverted index on trace content, only index the TraceID, with trace content stored as compressed blocks in object storage^[T7][T24]. This collapses “indexing cost” from a multitude of attributes down to a single TraceID. Estimated ~100x cost reduction per designround analysis (no official figures from Grafana), enabling “infinite retention”^[T24].

Microservice Architecture and Components

All Tempo components compile into a single binary, with -target controlling which to start, supporting both monolithic and microservice modes^[T7].

Write Path Architecture:

flowchart TD
    A[Distributor] --> B[Kafka]
    B --> C[Block-builder]
    C --> D[ObjectStorage]
    B --> E[Metrics-generator]
    E --> F[Mimir/Prometheus]
    style A fill:#2196F3,color:#fff
    style B fill:#FF9800,color:#fff
    style C fill:#9C27B0,color:#fff
    style D fill:#4CAF50,color:#fff
    style E fill:#9C27B0,color:#fff
    style F fill:#4CAF50,color:#fff

The Tempo write path consists of 6 key components: Distributor receives multi-format spans and hashes by TraceID, writing to Kafka as a persistent WAL; Block-builder consumes Kafka and organizes spans into blocks by configurable time windows, sending them to object storage; Metrics-generator (optional) derives RED metrics from traces back to Prometheus/Mimir.

Read Path Architecture:

flowchart TD
    A[Querier] --> B[ObjectStorage]
    A --> C[Live-store]
    D[Kafka] --> C
    style A fill:#2196F3,color:#fff
    style B fill:#4CAF50,color:#fff
    style C fill:#FF9800,color:#fff
    style D fill:#FF9800,color:#fff

In the read path, Live-store consumes Kafka to provide near-real-time queries (last 30min1h); Querier searches both object storage blocks and live-store, using bloom filters and index for efficient trace location.

Key components in detail^[T7]:

• Distributor: Receives spans (Jaeger/OTLP/Zipkin formats), hashes by traceID to shard, writes to Kafka. Only ACKs to the client after Kafka confirmation^[T7]
• Kafka-compatible system: Acts as a persistent WAL between distributor and downstream, decoupling write and read paths^[T7]
• Block-builder: Consumes Kafka partitions, organizes spans into blocks by configurable time windows, sends to object storage^[T7]
• Live-store (read path): Consumes Kafka for recent data queries (last 30min1h)^[T7]
• Querier: Searches object storage blocks and live-store, using bloom filters and index for efficient trace location within blocks^[T7][T29]
• Metrics-generator (optional): Derives RED metrics and service graphs from traces, writes to Mimir/Prometheus^[T7][T12]

Block Structure and Parquet Storage Format

Since Tempo 2.0, Apache Parquet columnar format is the default^[T8]. A block is a directory in object storage^[T9]:

Schema design: Deeply nested Parquet schema oriented toward spans^[T9]:

• Intrinsic fields (top-level columns): trace ID, span ID, parent span ID, span name, span kind, duration, etc., always efficient to query
• Generic attributes: Remaining resource/span attributes stored in a generic Attrs column (key-value), requiring map scans (slower)
• Dedicated columns: Frequently queried attributes can be promoted to independent Parquet columns for acceleration^[T9][T30]

TraceQL Query Language (2.0+)

TraceQL is a PromQL/LogQL-like trace query language^[T13][T8]: {} always selects a set of spans from the current trace, e.g., { span.http.status_code = 500 }^[T15]. Parquet columnar storage allows TraceQL to read only the relevant columns, significantly reducing IO^[T9].

Apache SkyWalking

SkyWalking is an ASF top-level project, widely adopted in the Java ecosystem for its rich analysis capabilities and real-time topology computation.

OAP (Observability Analysis Platform) Architecture

flowchart TD
    A[Probes] --> B[OAP<br/>Receiver + Aggregator]
    B --> C[Stream Processing]
    B --> D[Batch Processing]
    C --> E[Storage]
    D --> E
    E --> F[UI]
    style A fill:#2196F3,color:#fff
    style B fill:#9C27B0,color:#fff
    style C fill:#2196F3,color:#fff
    style D fill:#2196F3,color:#fff
    style E fill:#4CAF50,color:#fff
    style F fill:#FF9800,color:#fff

SkyWalking logically comprises four parts: Probes, Platform backend (OAP), Storage, UI^[SW16]. OAP is the core brain, serving both Receiver and Aggregator roles — handling reception, cleansing, analysis, aggregation, storage, and querying^[SW17]. Probes (blue) report data to OAP (purple), which processes data through stream and batch pipelines before writing to storage (green), ultimately served to UI (orange) for visualization.

Two core OAP processing flows^[SW17]:

• Stream processing: Real-time reception of raw agent data, trace model parsing, generating service topology, real-time metrics, and call relationships
• Batch processing: Time-dimension historical data aggregation, producing hourly/daily metrics

Analysis rules are defined using three DSLs: OAL (trace/metrics aggregation), MAL (metrics), LAL (log)^[SW18]. OAP’s default architecture does not introduce MQ^[SW16].

Segment Data Model (Key Concept)

Segment is a SkyWalking-unique concept^[SW19]: a segment contains all spans from a single request within a single OS process. Across processes, each process generates an independent segment, linked via references to form a complete trace.

Three Span types^[SW19]:

• EntrySpan: Service provider/server side endpoint
• LocalSpan: Regular method invocation without remote service
• ExitSpan: Service client/MQ producer (e.g., JDBC database access, Redis read)

Topology Graph Calculation and Endpoint-Level Metrics

• Topology: OAP uses STAM (Streaming Topology Analysis Method), building real-time service/instance/endpoint dependency relationships from ExitSpan peer addresses and references^[SW18][SW19]. This is a significant advantage over Jaeger/Zipkin’s offline jobs (real-time topology)
• Endpoint-level metrics: OAL analyzes trace spans, aggregating RED metrics (QPS, P99, success rate) by endpoint^[SW17][SW18]

Storage Backend and BanyanDB

Pluggable storage interface supporting Elasticsearch, BanyanDB, MySQL, H2, TiDB^[SW16]. BanyanDB is SkyWalking team’s self-developed dedicated storage^[SW20].

BanyanDB data model hierarchy^[SW14]: Data organized as streams, measures, traces, and properties within groups.

• Measure: Time-series data point sequences, supporting GORILLA encoding + ZSTD compression. entity defines the unique time series
• Trace: Designed specifically for distributed tracing — groups spans by trace ID, one data point per span, requiring trace_id_tag_name, span_id_tag_name, and timestamp_tag_name
• TopNAggregation: Pre-computes Top/Bottom N entities at write time^[SW14]

Indexing mechanism^[SW14]: Three index structures:

• INVERTED: Suitable for measure tags, good query performance
• SKIPPING: Most stream tags, prioritizing space efficiency
• TREE: Designed specifically for traces, using user-controlled int64 sort keys (e.g., duration, timestamp) for tree indexing

Zipkin

Zipkin is one of the earliest tracing systems, open-sourced by Twitter. Its data model has been influential (the B3 propagation protocol is widely referenced by Jaeger/SkyWalking).

Architecture and Span Data Model

Zipkin has four components^[Z23]: collector, storage, search (query service), and web UI. Tracers record timing/metadata in-app, propagate IDs in-band only, and asynchronously report spans out-of-band.

The Span model is more streamlined than Jaeger/SkyWalking^[Z23][Z24]: core fields include traceId, id (spanId), parentId, name, timestamp, duration, annotations (timed events with endpoint), and tags. Propagation uses B3 headers (X-B3-TraceId / X-B3-SpanId).

Storage Backends

Originally built for Cassandra (Twitter internal), later plug-in based^[Z23]: Cassandra (scalable, flexible schema, high write throughput, native TTL), Elasticsearch (strong querying, complex analysis support), MySQL/Postgres (structured, easy maintenance — small/medium/testing only).

Dependency Graph Storage and Computation

Similar to Jaeger, Zipkin does not maintain dependency graphs in storage in real-time. Instead, it uses the zipkin-dependencies Spark job for offline aggregation: extracts parent→child service relationships from spans, aggregating into DependencyLinks^[Z23]. This is a key differentiator from SkyWalking’s real-time topology.

Commercial Solutions (Datadog/Splunk)

Datadog: Head Sampling + 15-Minute Full + 15-Day Intelligent Retention

Data volume insights^[D21]: Datadog’s customer survey shows that unsampled trace volume is approximately 5x log volume; even sampled, it remains ~2x log volume.

Sampling philosophy^[D21]: Adheres to head-based sampling — decisions are made at trace start and propagated via trace context. Per Datadog’s public architecture blog, default target is 10 trace/s (DD_APM_MAX_TPS; not explicitly documented in technical docs).

Storage/retention tiers^[D21][D22]:

• Live Search (15-minute rolling window): Query all ingested spans from the last 15 minutes (100% full, no indexing required)
• 15-day indexed retention: Queries beyond 15 minutes require spans indexed through retention filters
• Intelligent Retention Filter: Diversity sampling + 1% flat sampling

Trace Metrics (RED)^[D21]: Rate/error/duration metrics computed from 100% application traffic (not sampled) — a hybrid approach of “full metrics, sampled traces.”

Splunk APM: 100% Full-Volume No Sampling

Splunk claims 100% full-trace storage (product page wording; no independent verification in technical docs)^[D25][D26]: no sampling, stores all traces, enabling engineers to query any trace retroactively (even one-in-ten-thousand intermittent failures).

• Trace Analyzer: Based on 100% traces, search any trace by arbitrary tag value/error/latency combination^[D26]
• Tag Spotlight: Identify common characteristics of problem traces
• AlwaysOn Profiling: Continuous Java/.NET/Node.js CPU/memory monitoring, correlating code-level hotspots with specific traces^[D25]

VictoriaTraces

Positioning: Trace Storage on VictoriaLogs

VictoriaTraces is a trace database developed by the VictoriaMetrics team, built on top of VictoriaLogs^[VT1][VT2]. The core design concept: transform OTLP protobuf spans into structured logs, leveraging VictoriaLogs’ efficient columnar LSM-Tree engine. It provides Jaeger Query Service JSON API, compatible with Grafana’s Jaeger datasource^[VT3].

Architecture Modes

Single-node mode: A single binary victoria-traces-prod handles ingestion, storage, and querying; deployed as a StatefulSet in Kubernetes.

Cluster mode consists of three independent components^[VT4]:

Components communicate via HTTP: vtinsert → vtstorage via POST /internal/insert, vtselect → vtstorage via GET /internal/select/*.

Data Model: Span as Log

VictoriaTraces does not define a proprietary trace storage format. Instead, it flattens OTLP spans into structured log records^[VT3]:

Key advantage: all fields are automatically full-text indexed — no predefined index schema required. Unlike Tempo which needs manually configured dedicated columns for performant attribute queries, VictoriaTraces feels more like a log search engine.

Write Path

flowchart TD
  A["OTLP Spans<br/>gRPC / HTTP"] --> B["vtinsert<br/>Shard by Trace ID"]
  B --> C["vtstorage<br/>Transform to Log"]
  C --> D["VictoriaLogs<br/>Columnar LSM-Tree"]
  D --> E["Local Disk<br/>per-day partitions"]

Write flow details:

1. Data reception: Receives span streams via OTLP/gRPC or OTLP/HTTP endpoints
2. Trace ID sharding: vtinsert hashes trace_id to distribute spans across vtstorage nodes, ensuring all spans of the same trace land on the same node
3. Data transformation: vtstorage flattens OTLP spans — resource attributes become resource_attr:*, span attributes become span_attr:*, and duration is automatically computed
4. Columnar write: Written to local disk via VictoriaLogs’ LSM-Tree engine, partitioned by day. Unlike VictoriaMetrics, VictoriaTraces does not yet support object storage^[VT6]

Query Path

flowchart TD
  A["Grafana<br/>Jaeger Datasource"] --> B["vtselect<br/>Distribute Query"]
  B --> C["vtstorage<br/>LogsQL Search"]
  C --> D["VictoriaLogs<br/>Full-text Scan"]
  D --> E["Cross-node<br/>Result Merge"]
  E --> F["Return Trace<br/>Waterfall View"]

Two-layer query interface^[VT5]:

1. Jaeger-compatible API: /select/jaeger/api/traces/{trace_id} — exact trace ID lookup, plus Jaeger-native APIs for service list, operation list, and service dependency graph
2. LogsQL query interface: /select/logsql/query — full-text search across all span fields using LogsQL, supporting arbitrary field filtering including span attributes

LogsQL example:

1

resource_attr:service.name="payment" AND span_attr:http.status_code>=500

Performance Benchmarks

Official benchmark at 10,000 spans/s (4 CPU, 8 GiB RAM)^[VT6]:

At 30,000 spans/s high load, VictoriaTraces uses 3.7x less RAM and 2.6x less CPU than Tempo^[VT6]. Docker image is only 15.1 MB with zero external dependencies.

Comparison with Mainstream Solutions

Horizontal Comparison

Indexing Strategy Comparison (Core Divergence)

flowchart TD
    A[Tracing Index Strategy] --> B[Full Inverted Index<br/>Flexible, High Cost]
    A --> C[TraceID-Only Index<br/>Minimal Cost]
    A --> D[Hybrid<br/>Balanced]
    A --> E[Full-Text Index<br/>VictoriaTraces]
    style A fill:#FF9800,color:#fff
    style B fill:#f44336,color:#fff
    style C fill:#4CAF50,color:#fff
    style D fill:#9C27B0,color:#fff
    style E fill:#795548,color:#fff

Tracing storage indexing strategies can be divided into four schools: Full Inverted Index (red) — Jaeger/ES, Zipkin/ES, Datadog, Splunk — offers flexible search but high index cost; TraceID-Only Index (green) — Tempo — minimal cost but non-TraceID search relies on columnar scanning; Hybrid (purple) — SkyWalking BanyanDB’s multi-index structure and Datadog’s tiered retention; Full-Text Index (brown) — VictoriaTraces — built on VictoriaLogs full-text indexing, all fields auto-indexed with no predefined schema.

• Full Inverted Index camp: Jaeger (ES), Zipkin (ES), Datadog/Splunk — flexible search but high cost and write amplification
• TraceID-Only Index camp: Tempo — minimal cost, fast TraceID lookup, content search via columnar scan + TraceQL
• Hybrid camp: SkyWalking (BanyanDB INVERTED+SKIPPING+TREE multi-index, trace-specific TREE), Datadog (recent full + selective retention)
• Full-Text Index camp: VictoriaTraces — built on VictoriaLogs full-text indexing, all fields automatically indexed without predefined schema

Sampling Strategy Comparison

Jaeger Sampling Flow

sequenceDiagram
    participant RS as RootService
    participant C as Collector
    participant A as Agent
    participant DS as DownstreamService
    RS->>RS: Sampling decision
    RS->>DS: Propagate trace context
    DS->>DS: Inherit decision
    DS->>C: Report spans
    C->>C: Adaptive adjustment
    C->>A: Push strategy
    A->>RS: Update sampling config
    Note over RS,DS: Head sampling ensures trace consistency

Jaeger’s sampling flow illustrates the complete lifecycle of head-based sampling: the RootService makes the sampling decision at the request entry point, propagating it via trace context to downstream services (DownstreamService), which inherit the same decision to ensure whole-trace consistency. The Collector performs adaptive adjustments per service/endpoint combination and pushes updated strategies back through the Agent.

Capacity Estimation Reference^[T24]

5M spans/s pre-sampling → 500K spans/s post-sampling (10%) → 250 MB/s ingestion → 21.6 TB/day (raw JSON) → ~3 TB/day after ClickHouse columnar compression (5-10x) → 14 days ~300 TB (raw) / ~40 TB (compressed).

References

[J1] Jaeger Features. https://www.jaegertracing.io/docs/1.57/features/

[J2] Jaeger Architecture. https://www.jaegertracing.io/docs/1.57/architecture/

[J3] Jaeger CLI / Storage Backends. https://www.jaegertracing.io/docs/1.57/cli/#storage-backends

[J4] Jaeger Data Model. https://www.jaegertracing.io/docs/1.57/data-model/

[J6] Jaeger Sampling. https://www.jaegertracing.io/docs/1.57/sampling/

[T7] Tempo Architecture. https://grafana.com/docs/tempo/latest/operations/architecture/

[T8] Tempo 2.0 Release. https://grafana.com/blog/2023/03/07/grafana-tempo-2.0-is-here/

[T9] Tempo Parquet Design. https://grafana.com/docs/tempo/latest/operations/parquet/

[T12] Tempo Metrics-generator. https://grafana.com/docs/tempo/latest/metrics-generator/

[T13] TraceQL Documentation. https://grafana.com/docs/tempo/latest/traceql/

[T15] TraceQL Examples. https://grafana.com/docs/tempo/latest/traceql/examples/

[T24] Grafana Labs Keynote / Tempo Capacity Estimate. https://grafana.com/blog/2021/09/22/grafana-tempo-1.0/

[T29] Tempo Bloom Filters. https://grafana.com/docs/tempo/latest/operations/bloom-filters/

[T30] Tempo Dedicated Columns. https://grafana.com/docs/tempo/latest/operations/dedicated-columns/

[SW14] BanyanDB Data Model. https://skywalking.apache.org/docs/main/latest/en/setup/backend/banyandb/

[SW16] SkyWalking Architecture. https://skywalking.apache.org/docs/main/latest/en/concepts-and-designs/overview/

[SW17] SkyWalking OAP. https://skywalking.apache.org/docs/main/latest/en/setup/backend/oap/

[SW18] SkyWalking OAL. https://skywalking.apache.org/docs/main/latest/en/concepts-and-designs/oal/

[SW19] SkyWalking Segment Model. https://skywalking.apache.org/docs/main/latest/en/concepts-and-designs/segment/

[SW20] BanyanDB Overview. https://skywalking.apache.org/docs/main/latest/en/setup/backend/banyandb/

[Z23] Zipkin Architecture. https://zipkin.io/pages/architecture.html

[Z24] Zipkin Span Model. https://zipkin.io/zipkin-api/#/default/get_trace__traceId_

[D21] Datadog APM Trace Sampling. https://docs.datadoghq.com/tracing/guide/trace_sampling_and_storage/

[D22] Datadog Intelligent Retention. https://docs.datadoghq.com/tracing/trace_retention/

[D25] Splunk APM AlwaysOn Profiling. https://www.splunk.com/en_us/observability/apm.html

[D26] Splunk APM Trace Analyzer. https://docs.splunk.com/observability/en/apm/traces/trace-analyzer.html

[VT1] [Official] VictoriaTraces overview: https://docs.victoriametrics.com/victoriatraces/

[VT2] [Official] VictoriaTraces GitHub: https://github.com/VictoriaMetrics/VictoriaTraces

[VT3] [Official] VictoriaTraces key concepts: https://docs.victoriametrics.com/victoriatraces/keyconcepts/

[VT4] [Official] VictoriaTraces cluster: https://docs.victoriametrics.com/victoriatraces/cluster/

[VT5] [Official] VictoriaTraces querying: https://docs.victoriametrics.com/victoriatraces/querying/

[VT6] [Official] VictoriaTraces benchmark (distributed tracing with VictoriaLogs): https://victoriametrics.com/blog/dev-note-distributed-tracing-with-victorialogs/

[VT7] [Official] VictoriaTraces data ingestion: https://docs.victoriametrics.com/victoriatraces/data-ingestion/