Continuous Profiling: Observability Down to the Line of Code

The Question a Trace Cannot Answer

A latency alert fires. The trace is unambiguous: a request spent 800 milliseconds inside the pricing service, and the span has no children. Metrics, logs and traces have done their job and pointed at the right box, and then they go quiet. Nothing in that trace says which function was running, whether the time went to a regular expression, a lock, a garbage collector pause or a JSON serialiser that someone called in a loop.

That gap is where profiling lives. A profile is a statistical sample of what the code was actually executing, aggregated into a call tree and usually rendered as a flame graph. Developers have used profilers on laptops for decades. What changed is running them all the time, on every production host, and keeping the results.

From Occasional Tool to Always-On Signal

The idea is older than most of the tools implementing it. Google described its fleet-wide sampling profiler in the 2010 paper Google-Wide Profiling, showing that sampling a small rotating subset of machines produced accurate, stable profiles of the whole data centre with negligible overhead. Continuous profiling applies the same principle inside one organisation: sample stack traces a few dozen times per second on each CPU, ship them to a store, and index them by service, version and time so you can compare last Tuesday's deploy with today's.

Because the data is sampled rather than instrumented, overhead is typically a low single-digit percentage of CPU, which is what makes leaving it switched on acceptable.

Collecting Profiles Without Touching Code

Two collection styles exist. Language SDKs, such as those shipped with Grafana Pyroscope, run inside the process and can tag profiles with application context. The alternative is eBPF, where an agent on the host samples every process from the kernel without a restart or a code change. Parca, an open source project started by Polar Signals, pairs an eBPF agent that samples nineteen times per second per logical CPU with a store that queries profiles over time. The OpenTelemetry eBPF profiler, originally Elastic's Universal Profiling agent and donated to the project in 2024, unwinds stacks for C and C++, Go, Rust, Python, Java, Node.js, .NET, PHP, Ruby and Perl from a single agent. Both speak the widely supported pprof format, so the tooling around them is interchangeable.

An Open Standard Arrives

Profiling has been called the fourth pillar of observability for years, but it lacked a shared wire format. OpenTelemetry formed a profiling working group in 2022, merged a profiling data model in 2024, and in March 2026 announced that the Profiles signal had entered public alpha. Pyroscope 2.0 ingests OTLP profiles natively, and the eBPF profiler above exports them. The most useful consequence is correlation: the signal is designed to carry links to and from traces, metrics and logs, so a slow span can point at the profile captured while it ran.

What SRE Teams Use It For

Regression triage: diff the profile of the current release against the previous one and the function that doubled its share of CPU stands out immediately, without a reproduction.
Incident diagnosis: when a trace shows an unexplained gap, the span-linked profile shows the frames that filled it.
Cost: sorting the whole fleet's CPU by function often reveals that a logging library or a compression step is the single most expensive line of code you run, which is an easy win to hand to a FinOps review.
Capacity: knowing where cycles go turns capacity planning from extrapolation into engineering.

Getting Started Without a Big Project

Deploy an eBPF agent to one node pool and keep a week of retention. Managed platforms such as Datadog's Continuous Profiler or Grafana Cloud Profiles remove the storage question entirely. Then add one habit: after every deploy that moves a latency SLI, open the profile diff before opening the code. Profiling does not replace distributed tracing; it picks up where the trace stops. When the error budget is burning because of a slow code path, this is the signal that names the path.