Observability Storage Selection Guide: A Scenario-Driven Decision Framework

The previous 7 articles in this series analyzed the storage architectures of each observability domain—TSDB, logging, tracing, RUM, profiling, and eBPF—in depth. This article synthesizes that knowledge into a scenario-driven storage selection decision framework. Teams differ in scale, tech stack, budget, and requirements—there is no “best” storage, only the one that best fits your context. Use this article as a practical reference to quickly narrow down your options.

Beginner Analogy: Choosing observability storage is like choosing a vehicle — there’s no “best” car, only the one that fits your journey. A startup needs a cost-efficient compact (Loki + VictoriaMetrics); an enterprise with compliance needs a heavy-duty truck (Elasticsearch); a platform team wants a versatile van (ClickHouse unified). The key dimensions: data volume, query patterns, budget, and team size.

The animation below shows how different scenarios lead to different storage recommendations:

Animation: Three scenarios cycle through the decision matrix. Startup (green): cost-sensitive path → Prometheus + Loki. Enterprise (orange): compliance-driven → VictoriaMetrics + Elasticsearch. Platform Team (purple): unified storage → ClickHouse. The key takeaway: there is no “best” storage, only the best fit for your specific context.

This series is continuously updated. Future articles will cover collection architecture, query optimization, visualization, and AI-assisted observability.

Cross-Domain Storage Technology Comparison

Before diving into selection criteria, let’s first survey the representative projects in each domain and their core storage philosophies. The table below covers 14 representative solutions across six observability domains.

Observations: A clear evolutionary path emerges from this table—object storage is becoming the backend standard, underlying formats are trending toward Parquet/Arrow columnar storage, and the traditional combination of local disks + inverted indexes is gradually being marginalized in terms of cost and scalability. eBPF solutions take a different approach: shifting storage pressure forward to collection-side aggregation.

Selection Recommendation Matrix

The following table maps concrete scenarios to recommended solutions with core rationale. The assumptions for each scenario have been discussed in the preceding articles; we present the conclusions directly here.

Scenario-Driven Decision Trees

The following three diagrams translate the matrix above into actionable decision paths. Choose the entry branch based on your primary signal type.

Figure 1: Metrics Monitoring Selection

flowchart TD
    A[Metrics Monitoring] --> B[Small-scale/Single node]
    A --> C[Large-scale or need ACID]
    B --> D[Prometheus]
    C --> E[VictoriaMetrics]
    C --> F[TimescaleDB]
    style A fill:#FF9800,color:#fff
    style B fill:#2196F3,color:#fff
    style C fill:#2196F3,color:#fff
    style D fill:#4CAF50,color:#fff
    style E fill:#4CAF50,color:#fff
    style F fill:#4CAF50,color:#fff

Explanation: The key differentiator for metrics monitoring selection is scale. For small-scale single-node scenarios (a single machine or a few servers), Prometheus is sufficient. Once you involve multi-cluster, large-scale cloud environments, or need long-term storage, VictoriaMetrics’s Shared-Nothing architecture offers better scalability. If your tech stack is deeply integrated with PostgreSQL and you need full transactional semantics, TimescaleDB is the natural choice.

Figure 2: Logging Selection

flowchart TD
    A[Logging Scenario] --> B[Label query]
    A --> C[Search or aggregation]
    B --> D[Loki]
    C --> E[ES/OpenSearch]
    C --> F[ClickHouse]
    style A fill:#FF9800,color:#fff
    style B fill:#2196F3,color:#fff
    style C fill:#2196F3,color:#fff
    style D fill:#4CAF50,color:#fff
    style E fill:#4CAF50,color:#fff
    style F fill:#4CAF50,color:#fff

Explanation: Logging selection depends on your query pattern. If your primary access method is filtering by labels (namespace, pod, service, etc.), Loki’s minimal index and object storage backend deliver extremely low TCO. If you need full-text search (searching log body text) or complex text matching, ES/OpenSearch’s inverted index is the more mature choice. If your log volume is high and you frequently perform aggregation analytics (error rate trends, P99 latency distributions), ClickHouse’s columnar vectorized execution engine offers the best query performance.

Figure 3: Tracing Selection

flowchart TD
    A[Tracing Scenario] --> B[Low-cost long-term]
    A --> C[Need search or APM]
    B --> D[Tempo]
    C --> E["Jaeger(ES)"]
    C --> F[SkyWalking]
    style A fill:#FF9800,color:#fff
    style B fill:#2196F3,color:#fff
    style C fill:#2196F3,color:#fff
    style D fill:#4CAF50,color:#fff
    style E fill:#4CAF50,color:#fff
    style F fill:#4CAF50,color:#fff

Explanation: Tracing storage selection is a trade-off between query capability and cost. If your core requirement is massive low-cost long-term trace retention with TraceID-centric query patterns, Tempo’s object storage approach offers the best cost efficiency. If your team already uses the Elasticsearch ecosystem and needs powerful trace search and aggregation, Jaeger(ES) can leverage existing infrastructure. For all-in-one APM capabilities (service topology, metric correlation, trace analysis), SkyWalking’s OAP real-time analysis platform provides richer out-of-the-box functionality.

Observability Storage Cost vs. Query Capability Quadrant

The following table positions each solution along the dimensions of “cost” and “query capability”:

Cost estimates are based on self-managed open-source deployments. Actual costs depend on cloud vendor pricing, node specifications, replication factor, and retention strategy.

Cost Estimation Framework

The following table provides reference cost ranges for different data scales. All figures are approximate; validate with a PoC before committing.

Note: These are self-managed open-source estimates. Managed/SaaS solutions typically cost 2-5x more but eliminate operational overhead.

Cost Optimization Tips:

• Use tiered storage (hot/warm/cold) to reduce cold data costs
• Configure retention and sampling policies wisely to avoid unbounded storage growth
• Evaluate compression ratios—different solutions can vary 3-10x on the same data

Common Anti-Patterns and Pitfalls

Even experienced teams can fall into these traps:

High-Cardinality Label Explosion: Using high-cardinality values (user_id, request_id, IP addresses) as labels in Prometheus or Loki can bloat indexes by tens or hundreds of times. For high-cardinality scenarios, prioritize solutions like VictoriaMetrics or ClickHouse that handle cardinality better, or apply label cardinality reduction preprocessing.

Uncontrolled Full-Text Index Costs: ES/OpenSearch’s full-text inverted index is powerful, but indiscriminately indexing all log fields can cause 3-5x storage bloat and drastically reduce write throughput. Only create indexes on fields that need searching; disable indexing with index: false for non-searchable fields.

Improper Sampling Leading to Data Distortion: Aggressive sampling (e.g., “store only 1% of traces”) can significantly reduce costs, but also completely misses signals for low-frequency events like P99 latency spikes. Design tiered sampling strategies: fully sample errors and slow traces, downsample normal traces.

Neglecting Cross-Signal Correlation: Siloed selection (Prometheus for metrics, ES for logs, Jaeger for traces) isn’t inherently wrong, but failing to plan cross-signal correlation mechanisms (via exemplars and trace_id injection) means each signal operates in isolation during incident troubleshooting. Prioritize solutions that support OpenTelemetry semantics and unified correlation queries.

Series Summary and Outlook

This series has so far covered the following topics:

This is an open-ended series that will be continuously updated. Potential future topics include:

• Collection architecture design and optimization (Agent management, data pipelines, backpressure handling)
• Query performance optimization (Index design, caching strategies, query routing)
• Observability visualization (Dashboards, alerting, SRE SLO panels)
• AI-assisted observability (Anomaly detection, root cause analysis, intelligent alert noise reduction)
• Multi-cluster federation and cross-region replication

Thank you for reading—stay tuned.

Sources

For the complete reference list of this series, please see each article’s source section. The comparison data and selection recommendations in this article are synthesized from the following references:

• [E11] [Official] Grafana: The open and composable observability platform: https://grafana.com/
• [E12] [Official] Grafana Loki 3.4: Standardized storage config: https://grafanacon.org/blog/2025/02/13/grafana-loki-3.4-standardized-storage-config-sizing-guidance-and-promtail-merging-into-alloy/
• [E13] [Official] Grafana Mimir architecture: https://grafana.com/docs/mimir/v3.0.x/get-started/about-grafana-mimir-architecture/
• [E14] [Official] About the Tempo architecture: https://grafana.com/docs/tempo/next/reference-tempo-architecture/about-tempo-architecture/
• [E16] [Official] How to engineer cost-efficient open source observability with ClickHouse (ClickStack): https://clickhouse.com/resources/engineering/observability-cost-optimization-playbook
• [E17] [Secondary] The high-cardinality trap: https://clickhouse.com/resources/engineering/high-cardinality-slow-observability-challenge
• [E25] [Official] OpenTelemetry — Overview (specs): https://opentelemetry.io/docs/specs/otel/overview/
• [E28] [Secondary] Semantics First: Correlating Signals with OpenTelemetry: https://www.thequietkernel.com/engineering/system-engineering/30-09-2025-obs-semantics/