culpert

Warning

Culpert is still very experimental and in development, use in production at your own risk.

Introduction

Per-span heap allocation profiler for Rust services.

Culpert is a sampled heap-allocation profiler for Rust libraries and services. It attributes allocations to existing spans and exports pprof-compatible profiles and CI-friendly diffs, helping catch allocation regressions before release.

It integrates with tracing, Cloudflare Foundations, or its own #[culpert::span_fn] macro. Culpert has already caught several real allocation regressions, including a memory leak, before they reached production at cloudflare!

What is it?

A #[global_allocator] wrapper that attributes every sampled allocation to the span it happened inside, exports pprof-format profiles so the existing tool ecosystem (stock pprof, Speedscope, Pyroscope, Polar Signals) keeps working, and ships a CLI with unbiased per-span reports plus a diff subcommand for CI/PR workflows. Geometric sampling with Bernstein correction provides unbiased allocation estimates while keeping profiling overhead low.

Three integration paths — pick whichever matches your service:

Your service uses…	Adapter	Cost
foundations	culpert's foundations feature	Zero instrumentation change. Existing #[span_fn] annotations become attribution keys.
The tracing crate	culpert's tracing feature	Compose a tracing_subscriber::Layer into your subscriber stack. Existing #[tracing::instrument] annotations become attribution keys.
Neither, or you want attribution independent of trace sampling	culpert::scope + #[culpert::span_fn]	One macro on the functions you want attributed. No external tracer.

Install

Library:

cargo add culpert                              # standalone
cargo add culpert --features foundations       # with foundations
cargo add culpert --features tracing           # with tracing

culpert-macros is re-exported by culpert (don't depend on it directly).

CLI:

cargo install --locked culpert-cli
# → installs the `culpert` binary into ~/.cargo/bin
culpert --help

# Optional feature for talking to a culpert-archive instance that
# sits behind Cloudflare Access — adds --cf-access-client-id /
# --cf-access-client-secret flags (with CF_ACCESS_CLIENT_ID /
# CF_ACCESS_CLIENT_SECRET env-var fallbacks) to `upload` / `pull`:
cargo install --locked --features cloudflare-access culpert-cli

Quickstart — standalone (#[culpert::span_fn])

The smallest viable setup. No external tracer:

[dependencies]
culpert = "0.2.1"

use culpert::{Config, LocalSpanContext, TrackingAllocator};
use std::alloc::System;

#[global_allocator]
static GLOBAL: TrackingAllocator<System> = TrackingAllocator::new(System);

fn main() {
    let _profiler = culpert::install(LocalSpanContext::new(), Config::default());

    handle_request();

    let profile = culpert::snapshot();
    let bytes = culpert::pprof::encode_gzipped(&profile).unwrap();
    std::fs::write("/tmp/prof.pb.gz", &bytes).unwrap();

}

#[culpert::span_fn("handle_request")]
fn handle_request() {
    // Every sampled allocation in here, transitively, is attributed to
    // span_name = "handle_request".
}

Every installation method returns a ProfilerGuard. Keep that guard alive while allocations should be recorded; dropping it stops profiling safely before application and thread-local teardown.

See examples/macros for a runnable version.

Quickstart — with foundations

[dependencies]
culpert = { version = "0.2.1", features = ["foundations"] }
foundations = { version = "5", default-features = false, features = ["tracing", "telemetry-server"] }

The default-features = false is required — foundations' default jemalloc feature declares its own #[global_allocator] which conflicts with culpert's TrackingAllocator and fails to link.

#[global_allocator]
static GLOBAL: TrackingAllocator<System> = TrackingAllocator::new(System);

#[tokio::main]
async fn main() {
    let driver = foundations::telemetry::init(TelemetryConfig {
        service_info: &service_info!(),
        settings: &settings.telemetry,
        custom_server_routes: vec![
            culpert::foundations::pprof_route("/debug/alloc/profile"),
        ],
    }).unwrap();

    let _profiler = culpert::foundations::install();

    // ... your existing app, with #[span_fn] annotations as usual.
}

Existing #[foundations::telemetry::tracing::span_fn] annotations become attribution keys for free. See examples/foundations (minimal) and examples/mock-axum (full HTTP service with pprof_route-served profile).

Quickstart — with the tracing crate

[dependencies]
culpert            = { version = "0.2.1", features = ["tracing"] }
tracing            = "0.1"
tracing-subscriber = "0.3"

use tracing_subscriber::prelude::*;

#[global_allocator]
static GLOBAL: TrackingAllocator<System> = TrackingAllocator::new(System);

fn main() {
    tracing_subscriber::registry()
        .with(culpert::tracing::layer())
        .with(/* your other layers — fmt, OTLP, etc. */)
        .init();

    let _profiler = culpert::tracing::install();

    // ... your existing app, with #[tracing::instrument] annotations as usual.
}

Existing #[tracing::instrument] annotations become attribution keys. See examples/tracing for a runnable version.

What you get

A profile (*.pb.gz) you can either feed to stock pprof or read with the shipped CLI. Output below is from the examples/mock-axum service under load; the same shape works for any of the three integration paths.

Tree report — culpert report <profile>

The default: hierarchical breakdown, with each sub-span nested under its parent. Built from span_parent_id labels emitted by whichever SpanContext was installed.

$ culpert report /tmp/mock-axum.pb.gz

Hierarchical span report (143179 samples, sample rate 4.00 KB/alloc):
  Tree shows span_name groupings under their parents. `bytes` is the
  Bernstein-corrected, unbiased estimate of allocated bytes (see
  CHANGELOG: geometric sampling). Use --flat for a simple sorted table.

vec                                            1.30 GB   70.48%  (self 1.30 GB)
(no span)                                    246.45 MB   13.07%  (self 246.45 MB)
json                                         180.92 MB    9.59%  (self 180.92 MB)
nested                                         3.55 MB    0.19%  (self 170.62 KB)
├─ parse_payload                              59.77 MB    3.17%  (self 59.77 MB)
├─ validate_payload                           59.57 MB    3.16%  (self 59.57 MB)
└─ build_response                            604.00 KB    0.03%  (self 604.00 KB)
strings                                        5.87 MB    0.31%  (self 5.87 MB)

--flat switches to a sorted-by-bytes table for users who prefer it. The bytes column is the unbiased estimate of total bytes allocated under each span — each underlying sample is weighted by 1 / (1 − exp(−bytes/rate)) (the Bernstein correction for geometric sampling). No raw column is shown: with geometric sampling the corrected value is the only one that means anything meaningful.

Drill into one span — --span <name>

$ culpert report /tmp/mock-axum.pb.gz --span json --top 4

Top callsites within span "json" (sample rate 4.00 KB/alloc):
  callsite                                                  samples         bytes  bytes %
  -------------------------------------------------------  --------  ------------  -------
  alloc::vec::Vec::from_iter::SpecFromIter                   23000      89.84 MB   50.00%
  alloc::fmt::format::{closure}                              23000      89.84 MB   50.00%

Drill into the unattributed bucket — --no-span

Useful for "is this my code's fault, or the runtime's?". Shows the top callsites of allocations that fired outside any span — i.e. tokio runtime work, framework internals, foundations' or tracing's own reporters, or code paths you haven't yet annotated.

$ culpert report /tmp/mock-axum.pb.gz --no-span --top 4

Top callsites in unattributed samples — outside any span:
  callsite                                                  samples         bytes  bytes %
  -------------------------------------------------------  --------  ------------  -------
  cf_rustracing_jaeger::Tag as Clone>::clone                  8720      34.06 MB   33.74%
  alloc::vec::Vec::append_elements                            6678      26.09 MB   25.84%
  alloc::boxed::Box::new_uninit                               6551      25.59 MB   25.34%
  bytes::bytes_mut::BytesMut::reserve_inner                   2307      16.00 MB   15.85%

Compare two profiles — culpert diff

For CI workflows: diff a "before" and "after" profile by span_name with both an absolute (--threshold-bytes) and a relative (--threshold-pct) gate. --format markdown produces output you can pipe straight into $GITHUB_STEP_SUMMARY:

$ culpert diff before.pb.gz after.pb.gz --format markdown

### culpert: allocation diff

- **before:** `before.pb.gz` — total 2.70 MB (estimated)
- **after:**  `after.pb.gz` — total 5.75 MB (estimated)
- **net Δ:** +3.05 MB (+113.21%)

#### Regressions

| Span | Before | After | Δ | Δ% |
|------|-------:|------:|--:|---:|
| `encode_response` | 1.53 MB | 4.58 MB | +3.05 MB | +200.00% |

To suppress spans that allocate less than 20 MiB in both profiles, add --min-span-bytes 20971520 (default: 0, disabled). A span reaching exactly 20 MiB on either side remains eligible, including new and disappeared spans. The existing delta-size and percentage gates still apply; whole-profile totals include all spans. JSON retains suppressed rows as quiet. In --tree output, the minimum applies to the displayed subtree totals, including children.

culpert diff before.pb.gz after.pb.gz --format markdown \
  --min-span-bytes 20971520 --threshold-bytes 1048576 --threshold-pct 10

CI can set CULPERT_MIN_SPAN_BYTES=20971520 instead of passing the flag. An explicit --min-span-bytes overrides the environment, including 0 to disable it. CI must install a CLI release containing this option.

Stock pprof works too

The on-disk format is canonical pprof, so everything in the ecosystem reads it:

pprof -tags        /tmp/mock-axum.pb.gz   # totals grouped by span_name + span_id
pprof -tagfocus="span_name:json" -text /tmp/mock-axum.pb.gz
pprof -http=:8090  /tmp/mock-axum.pb.gz   # interactive flame graph + source view

Persist profiles for CI — culpert upload / culpert pull

For CI workflows you usually want last week's profiles to compare against. The companion culpert-archive Cloudflare Worker stores .pb.gz files keyed by commit SHA; the upload / pull subcommands of culpert-cli are its first-party client. Endpoint and token are picked up from environment variables (CULPERT_ARCHIVE / CULPERT_TOKEN), commit SHA / branch from GITHUB_SHA / GITHUB_REF_NAME so the GitHub Actions step body is short:

# Push the just-captured profile under this commit
culpert upload /tmp/profile.pb.gz

# Pull last build's main-branch profile as a baseline. --allow-missing
# exits 0 (writing nothing) on 404 so the very first main run doesn't
# fail the build.
culpert pull --latest-of main -o /tmp/baseline.pb.gz --allow-missing

# Diff. Markdown to $GITHUB_STEP_SUMMARY, plus a fail-gated text run.
culpert diff /tmp/baseline.pb.gz /tmp/profile.pb.gz \
  --format markdown >> "$GITHUB_STEP_SUMMARY"
culpert diff /tmp/baseline.pb.gz /tmp/profile.pb.gz \
  --threshold-bytes 1048576 --threshold-pct 10   # exit 1 on regression

For drop-in CI use, this repo ships a reusable composite action that wraps the whole flow:

# In your workflow, after you've captured a profile:
- uses: rupert648/culpert/.github/actions/culpert-diff@main
  with:
    archive-url:   ${{ vars.CULPERT_ARCHIVE }}
    archive-token: ${{ secrets.CULPERT_TOKEN }}
    profile:       /tmp/my-service-profile.pb.gz
    culpert-cli:   ./target/release/culpert    # path to the binary

That one block does: culpert info (sanity check in the run log) → culpert pull --latest-of main --allow-missing → culpert diff --format markdown (posted to $GITHUB_STEP_SUMMARY and, on pull_request events, as a sticky PR comment) → culpert upload as the new baseline. Override defaults with the action's inputs — baseline-branch, threshold-bytes, threshold-pct, fail-on-regression, etc. See .github/actions/culpert-diff/action.yml for the full input schema.

Culpert's own rust.yml profiles example-macros and invokes the same action — that's the worked example. Currently warn-only (fail-on-regression: "false") until enough main-branch runs have accumulated to make gating meaningful.

The worker never parses the pprof bytes — it's dumb storage. Sample attribution, Bernstein correction, threshold logic all run in this CLI. See culpert-archive's README for the deploy recipe and HTTP surface.

Examples

Four standalone examples in examples/ showing each integration path:

Example	What it shows	Key wiring
mock-axum	foundations + axum service with /debug/alloc/profile HTTP endpoint, plus a load script	foundations::telemetry::init + culpert::foundations::pprof_route + #[foundations::span_fn]
foundations	minimal foundations wiring without an HTTP server — just init, work, snapshot, write	culpert::foundations::install + #[foundations::span_fn]
tracing	tracing crate integration via tracing_subscriber::Layer	culpert::tracing::layer() composed into a Registry + #[tracing::instrument]
macros	sampling-independent attribution with no external tracer; also doubles as a BT-vs-FP timing demo (set CULPERT_FP=1)	culpert::LocalSpanContext + #[culpert::span_fn]

Each prints a profile to /tmp/example-*.pb.gz. View with cargo run -p culpert-cli --bin culpert -- report <path> or pprof -tags <path>.

License

Dual-licensed under MIT or Apache-2.0.