Continuous Profiling Storage: From pprof to Pyroscope V2 and Parca

Continuous Profiling is a vital pillar of observability. Unlike traditional sampling profilers, continuous profiling captures application and kernel stack traces at a fixed frequency around the clock, generating flame graphs that help teams identify performance bottlenecks, memory leaks, and resource hotspots. This article focuses on the storage architecture of profiling data: starting from the Google pprof data model and flame graph construction algorithm, we analyze Pyroscope’s architectural evolution from V1 (TSDB+Parquet) to V2 (Metastore+Segments), Parca and FrostDB’s columnar storage innovation, commercial solutions from Datadog and Splunk, and conclude with comparative tables that highlight the differentiating choices across solutions.

Beginner Analogy: A flame graph is like an “X-ray” of your code — the horizontal axis shows the proportion of function call counts, the vertical axis shows call depth, and the widest “flame” is the most time-consuming code path. Continuous profiling samples the CPU at fixed intervals (e.g., every 10ms), and after thousands of samples, the aggregated result forms a statistical picture of where your program spends its time.

The animation below shows how individual stack trace samples aggregate into a flame graph:

Animation: The loop shows how individual stack trace samples (Phase 1) merge into a flame graph (Phase 2). Each function frame’s width is proportional to the percentage of samples containing that function. The widest frame — dbQuery at 50% — is the performance optimization target. This is how continuous profiling turns thousands of 10ms-interval samples into an actionable performance picture.

pprof Data Model and Flame Graph Construction

pprof profile.proto Core Message Definitions

Almost all mainstream projects (Pyroscope, Parca, Datadog, Splunk, etc.) use Google’s pprof protobuf format as the exchange/serialization standard for profile data^[P1][P4][P7]. The pprof format consists of 5 core entities^[P1][P5]:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

message Profile {
  repeated ValueType sample_type = 1;   // Describes the semantics/unit of each Sample.value
  repeated Sample sample = 2;           // Collection of samples
  repeated Mapping mapping = 3;         // Address range → binary/library mapping
  repeated Location location = 4;       // Locations referenced by samples (program counters)
  repeated Function function = 5;       // Functions referenced by locations
  repeated string string_table = 6;     // Deduplicated string table (index 0 must be "")
  int64 time_nanos = 9;                 // Collection time (UTC nanoseconds)
  int64 duration_nanos = 10;            // Profile duration
}

message Sample {
  repeated uint64 location_id = 1;  // Stack trace, leaf at location_id[0] (leaf-first order)
  repeated int64 value = 2;         // Each value corresponds to one entry in sample_type
  repeated Label label = 3;         // Additional context labels
}

The five entities form a hierarchical relationship through reference deduplication:

flowchart TD
    P["Profile<br/>Top-level Container"]
    S["Sample<br/>Stack+Value+Labels"]
    L["Location<br/>Program Counter"]
    M["Mapping<br/>Binary Mapping"]
    F["Function<br/>Function Symbol"]

    P -->|"1 : N"| S
    S -->|"N : N"| L
    L -->|"N : 1"| M
    L -->|"N : 1"| F

    style P fill:#2196F3,color:#fff
    style S fill:#FF9800,color:#fff
    style L fill:#4CAF50,color:#fff
    style M fill:#9C27B0,color:#fff
    style F fill:#795548,color:#fff

Figure note: A Profile contains multiple Samples, each referencing a set of Locations (the call stack). Locations reference Mappings (linking to specific binaries/shared libraries) and Functions (linking to function symbols). The string_table stores all strings centrally, with fields referencing entries by index.

Key design points^[P1][P5]:

• Relationship structure: Sample → Location (N→1) → Mapping; Location → Function — a classic reference-deduplication model
• string_table deduplication: All strings are stored centrally and referenced by index, avoiding duplicate storage of function names and file names
• Leaf-first ordering: sample.location_id[0] is the top of the stack; must be reversed to root→leaf when constructing flame graphs
• Disk format: Serialized protobuf must be gzip compressed^[P1]

Flame Graph Construction Algorithm

A flame graph is essentially a call stack tree where each node is a callsite and leaf nodes accumulate the self value^[P5].

Construction Steps

1. Parse proto: Read the string_table, create index entries for each Mapping/Function
2. Location to frame: Convert each pprof Location to a frame (with function name + line number)
3. Stack reversal: pprof uses leaf-first ordering; must be reversed to root→leaf when building flame graphs
4. Tree construction: For each Sample, walk the reversed call chain to find/create child nodes level by level, accumulating value at the leaf
5. Width calculation: Each node’s width = sum(value) / total_value × canvas_width

Concrete Example

Suppose 3 samples are received (sampling every 10ms):

Samples A, B, and C share the prefix main and merge into the same root node; A and B further share the handleRequest subtree but diverge at the leaves (dbQuery vs parseJSON); C branches directly from main to validateToken. After aggregation, each node’s width proportion equals its value’s share of the total.

flowchart TD
    R["main<br/>100%"] --> H["handleRequest<br/>80%"]
    R --> V["validateToken<br/>20%"]
    H --> D["dbQuery<br/>50%"]
    H --> P2["parseJSON<br/>30%"]

    style R fill:#1565C0,color:#fff
    style H fill:#FF9800,color:#fff
    style V fill:#E65100,color:#fff
    style D fill:#4CAF50,color:#fff
    style P2 fill:#9C27B0,color:#fff

Figure note: A flame graph is essentially this call stack tree “laid horizontally” — showing call depth from bottom to top, with each node’s width proportional to its value share. Colors are typically assigned random warm tones to distinguish adjacent frames, color encodes no performance information.

Statistical Basis

The effectiveness of flame graphs rests on sampling theory: with enough samples (typically > 1000), a function’s width in the flame graph approximates its actual CPU time share. Sampling at 10ms intervals for 60 seconds produces 6000 samples, sufficient to cover most hotspot functions. With insufficient sample counts, low-frequency functions may be missed (known as “sampling blind spots”), mitigable by reducing the sampling interval or extending the collection duration.

Stacktrace Compression and Deduplication

Storage optimization for profile data focuses on call stacks and symbol information:

• Symbol deduplication: Function/file names are deduplicated via the string_table or a dedicated symbol library. Pyroscope 2.0 reports up to 95% reduction in symbol storage^[P6][P9]
• Columnar encoding: Stack traces are stored as columns, with repeated values compressed via run-length encoding^[P4][P8]
• Gorilla XOR encoding: Timestamps and numeric values are compressed using the Gorilla paper’s XOR encoding^[P8]
• Sample aggregation: Raw samples with identical call stacks and labels are merged, with values summed element-wise^[P7]

Pyroscope (Grafana Pyroscope)

Project Evolution and Phlare Merger

Grafana Pyroscope was formed by the merger of two open-source continuous profiling projects^[P2]: Pyroscope (founded in 2021) and Phlare (launched by Grafana Labs in 2022). In March 2023, Grafana Labs acquired Pyroscope Inc., merging Phlare and Pyroscope under the unified name Grafana Pyroscope^[P2][P6].

V1 Storage (TSDB + Parquet + SymDB)

V1 adopted a three-tier architecture^[P9]:

1

Ingestion → Head (in-memory) → Local Blocks (disk) → Object Storage (immutable blocks)

Core components^[P9]:

• Head: In-memory write buffer containing profileStore (profile slice buffer), symdb.SymDB (symbol database for deduplicating functions/locations), and profilesIndex (TSDB-based inverted index)
• Immutable Block structure:

1
2
3
4
5

<ULID>/
├── meta.json        # Block metadata
├── index.tsdb       # TSDB inverted index (profile series)
├── profiles.parquet # Merged Parquet profile data
└── symbols.symdb/   # Symbol database files

The following diagram shows the V1 write pipeline:

flowchart TD
    I[Ingestion] --> H[Head<br/>Memory Buffer]
    H --> L[Local Blocks<br/>On-Disk]
    L --> O[Object Storage<br/>Immutable Blocks]
    H -.-> S[SymDB<br/>Symbol Database]
    style I fill:#2196F3,color:#fff
    style H fill:#2196F3,color:#fff
    style L fill:#2196F3,color:#fff
    style O fill:#2196F3,color:#fff
    style S fill:#9C27B0,color:#fff

Figure note: The V1 write path follows a three-stage pipeline: data enters the Head memory buffer (with concurrent SymDB writes for symbol deduplication), then lands on local disk as Local Blocks, and is finally uploaded to object storage as immutable blocks. Each write involves 3 copies (profile data, index, and symbols), creating significant write amplification.

V2 Storage (Metastore + Segments) — Released April 2026

Pyroscope 2.0 is a ground-up rewrite with core goals: high write throughput, low storage cost, and scalable querying^[P10]. The biggest change: data is written directly to object storage, eliminating the need for local disks on ingesters; read and write paths are decoupled^[P10][P6].

Architecture components^[P10]:

• distributor: Stateless, diskless, horizontally scalable; routes profiles to segment-writers to co-locate profiles from the same application
• segment-writer: Stateless, diskless; accumulates profiles into small blocks (segments) and writes them to object storage
• metastore: The only stateful component, using Raft consensus replication and BoltDB persistence, maintaining block/shard metadata indices^[P10][P11]
• compaction-worker: Stateless; merges small segments into larger blocks with median first-compaction time under 15 seconds^[P10]
• query-frontend / query-backend: Stateless; query execution is represented as a DAG^[P10]

flowchart TD
    D[Distributor<br/>Stateless Router] --> S[Segment-Writer<br/>Writes Small Blocks]
    S --> O[Object Storage]
    M[Metastore<br/>Raft + BoltDB] -.-> O
    C[Compaction-Worker<br/>Segment Merging] --> O
    style D fill:#4CAF50,color:#fff
    style S fill:#4CAF50,color:#fff
    style O fill:#4CAF50,color:#fff
    style M fill:#9C27B0,color:#fff
    style C fill:#FF9800,color:#fff

Figure note: In the V2 architecture, both Distributor and Segment-Writer are stateless, diskless components that write data directly to object storage. The Metastore is the only stateful component (Raft replication + BoltDB persistence), responsible for maintaining block metadata indices. The Compaction-Worker merges small segments into larger blocks, reducing the number of files that need to be scanned during queries.

V1 vs V2 comparison^[P9][P11]:

Cost and scale results^[P6]:

• Write-path replication removed: V1 wrote 3 copies per profile, V2 writes only 1 copy
• Symbol co-location and deduplication: up to 95% reduction in symbol storage (~20x compression)
• 19.5 PB of profiling data processed (in production on Grafana Cloud Profiles since April 2025)

Parca and FrostDB

Parca Overall Architecture

Parca consists of two major components^[P3]: Parca (Server) for storing profiling data, and Parca Agent, an eBPF-based system-wide profiler. Series are identified by unique label combinations, queries use Prometheus-style label selectors, and the UI displays Icicle Graphs^[P3][P4].

Storage Architecture: Meta Store + Sample Store

Meta Store (based on the embedded KV store badger)^[P4]: Three entity types—Mappings (representing object files where stack traces originate), Functions (normalized functions), and Locations (single stack trace steps, referencing mapping + function + specific line number). On write, metadata is first normalized and deduplicated in the meta store.

Sample Store (FrostDB): Columnar storage for samples^[P4]. The logical table is labels.*, timestamp, stacktrace, value, taking advantage of column-level data repetition for compression.

FrostDB: A Columnar Database Built for Profiling

FrostDB (originally named ArcticDB, renamed in 2022 due to a trademark conflict^[P15]) is an embedded columnar database built by Polar Signals specifically for Parca^[P12][P15].

Design motivation^[P8][P15]: Parca initially used a Prometheus-style time series approach, but it could not support “attaching arbitrary labels (e.g., customer="X", distributed trace IDs/span IDs) to stack traces and filtering/aggregating by those labels” — this is the unbounded cardinality problem.

Core design choices^[P12][P15]:

• Columnar layout: Apache Parquet for storage (efficient encoding), Apache Arrow for queries (vectorized execution foundation)
• Dynamic Columns: Columns are created dynamically the first time a sub-column is encountered — this is FrostDB’s key differentiator from other columnar databases
• Immutable + Globally Sorted: Write-only, no updates; globally sorted by schema columns, enabling binary search without additional indexes
• Snapshot Isolation: Weak consistency, no read-after-write guarantee

Why build from scratch instead of using existing solutions^[P15]: The requirements were a Go-embeddable library (single static binary) with dynamic column capabilities — the only open-source option at the time was InfluxDB IOx, which was not yet ready.

The FrostDB table schema is illustrated below:

flowchart TD
    L[labels.*<br/>Dynamic Columns<br/>Created on First Sight] --> T[Immutable + Global Sort<br/>Binary Search]
    TS[timestamp<br/>Gorilla XOR Encoding] --> T
    ST[stacktrace<br/>Columnar Storage<br/>Run-Length Encoding] --> T
    V[value<br/>Numeric Column] --> T
    style T fill:#4CAF50,color:#fff
    style L fill:#FF9800,color:#fff
    style TS fill:#2196F3,color:#fff
    style ST fill:#2196F3,color:#fff
    style V fill:#2196F3,color:#fff

Figure note: FrostDB’s logical table contains four column types: labels.* are dynamic columns (automatically created when a new label is first encountered, solving the unbounded cardinality problem); timestamp uses Gorilla XOR encoding for time series compression; stacktrace is stored in columnar layout with Run-Length Encoding; value is a simple numeric column. The entire table is immutable and globally sorted across all columns, allowing binary search without additional indexes.

Parca Agent eBPF Collection Mechanism

Parca Agent is a sampling profiler that captures stack traces at 19 Hz per logical CPU^[P16]. The number 19 is prime to avoid collisions with other periodic events on the machine.

eBPF technical details^[P16]:

• Uses BPF CO-RE (Compile Once – Run Everywhere) + libbpf, pre-compiling all BPF programs and embedding them statically in the binary
• Sampling is performed by attaching BPF programs to perf_event, specifically the PERF_COUNT_SW_CPU_CLOCK event
• BPF maps communication: Stack traces map (stack trace ID → memory address array) + Counts map ((PID, user-space stack ID, kernel-space stack ID) triple → observed count)
• Every 10 seconds, all data is read and the maps are cleared/reset

Symbolization^[P16]:

• Kernel symbols: Symbolized immediately by the Agent, reading /proc/kallsyms
• Application symbols: For binaries with debug symbols, extracted symbols are uploaded to the server and symbolized at read time by the server
• Interpreted/JIT languages (Ruby/Node.js/Python/JVM): Must be parsed directly within the Agent

The complete eBPF collection flow is shown below:

sequenceDiagram
    participant K as Linux Kernel
    participant B as BPF Program
    participant M as BPF Maps
    participant A as Parca Agent
    participant S as Parca Server

    K->>B: perf_event trigger(19Hz)
    B->>M: Write stack traces
    B->>M: Write counts mapping
    Note over M: PID + stack IDs to count
    A->>M: Read & clear every 10s
    A->>A: Symbolize kernel symbols<br/>(/proc/kallsyms)
    A->>S: Upload .pprof
    S->>S: Symbolize app symbols<br/>(debug info)

Figure note: The Kernel triggers perf_event at 19Hz, the BPF program samples the current CPU’s call stack and writes to BPF Maps (including the Stack traces map and Counts map). The Agent reads from BPF Maps every 10 seconds and then clears them, immediately symbolizing kernel symbols (/proc/kallsyms), before uploading raw samples to the Server. The Server performs secondary symbolization for application symbols at query time (based on debug info). For interpreted/JIT languages, symbolization must be completed within the Agent.

Commercial Profiling Solutions

Datadog Continuous Profiler

Architecture^[P7]: Multiple independent profilers (CPU/wall time/exceptions/lock contention/allocations), each with a sampler and provider. An aggregator combines samples from all profilers, and an exporter serializes samples into .pprof files for upload via the Datadog Agent. The code for serializing into Google’s .pprof format is written in Rust^[P7].

Sample deduplication^[P7]: When labels and call stacks are identical, the exporter sums the values and merges them into a single sample — avoiding duplication of identical call stacks and labels.

Profile-to-Trace correlation^[P7]: The profiler attaches metadata (process ID, host name, runtime ID) to each .pprof file. The tracer informs the profiler which runtime ID maps to which AppDomain → service name, enabling the backend to find profiles related to specific traces/spans.

Splunk AlwaysOn Profiling

Based on the Splunk OpenTelemetry Collector (v0.44.0+) for collection^[P17]: Configure a profiling pipeline (OTLP gRPC receiver + Splunk HEC exporter)^[P17].

Key characteristics:

• Native OTel integration: Collects traces, metrics, and profiles simultaneously through a unified collector — all three signals share the same agent and pipeline
• Language support: Java (JFR), Python (cProfile), .NET (ETW), Node.js (CPU profiler), Go (pprof)
• Storage backend: Profile data is written to Splunk Indexer, stored separately from traces/metrics but queried uniformly
• Cross-signal correlation: SPL (Search Processing Language) enables cross-signal queries — “find CPU hotspot function → correlate with corresponding trace → investigate slow request root cause”

Key difference from Datadog: Datadog builds a customized pipeline with proprietary SDKs + a Rust serializer; Splunk chooses full OTel standard adoption, sacrificing some customization capability for ecosystem compatibility.

Profiling Storage Comparison

Storage Format Comparison

Collection Mechanism Comparison

References

Profiling References

[P1] [Official] Google pprof profile.proto: https://raw.githubusercontent.com/google/pprof/main/proto/profile.proto

[P2] [Official] Grafana Pyroscope: https://grafana.com/oss/pyroscope/

[P3] [Official] Parca Overview: https://www.parca.dev/docs/overview/

[P4] [Official] Parca Storage: https://www.parca.dev/docs/storage

[P5] [Official] Perfetto pprof Support: https://perfetto.dev/docs/design-docs/pprof-support

[P6] [Secondary] InfoQ: Grafana’s Pyroscope 2.0: https://www.infoq.com/news/2026/05/pyroscope-2-profiling/

[P7] [Official] Datadog .NET Continuous Profiler Under the hood: https://www.datadoghq.com/blog/engineering/dotnet-continuous-profiler/

[P8] [Official] Polar Signals: Introducing arcticDB (FrostDB): https://www.polarsignals.com/blog/posts/2022/05/04/introducing-arcticdb/

[P9] [Secondary] deepwiki: Pyroscope Storage System: https://deepwiki.com/grafana/pyroscope/2.4-storage-system

[P10] [Official] About the Pyroscope v2 architecture: https://grafana.com/docs/pyroscope/next/reference-pyroscope-v2-architecture/about-pyroscope-v2-architecture/

[P11] [Secondary] deepwiki: Storage v2 (Metastore + Segments): https://deepwiki.com/grafana/pyroscope/2.4.2-storage-v2-(metastore-+-segments)

[P12] [Official] FrostDB pkg.go.dev README: https://pkg.go.dev/github.com/polarsignals/frostdb

[P13] [Secondary] OneUptime: How to Build Continuous Profiling Setup: https://oneuptime.com/blog/post/2026-01-30-continuous-profiling-setup/view

[P15] [Official] Polar Signals: Introducing arcticDB (Design Motivation): https://www.polarsignals.com/blog/posts/2022/05/04/introducing-arcticdb/

[P16] [Official] Parca Agent Design: https://www.parca.dev/docs/parca-agent-design/

[P17] [Official] Splunk AlwaysOn Profiling Configuration: https://help.splunk.com/en/splunk-observability-cloud/monitor-application-performance/alwayson-profiling/troubleshoot-alwayson-profiling