feat(data-pipeline)!: add agentless stats export - #2309
Conversation
# Motivation Perform stats aggregation and export directly to the public intake when agentless tracing is enabled # Changes * Add paramateres on the TraceExporterBuilder * agentless_stats_endpoint - the fully formed URL. The API key is derived from the agentless export parameter * Refactor the stats exporter to add a path that wraps tje ClientStatsPayload into a top-level StatsPayloas * Hardcode the agent_version to libdd version + a libdatadog suffix to be able to differentiate metrics origin
|
✅ All CI checks and tests passed. 🎉 All green!🧪 All tests passed 🎯 Code Coverage (details) 🔗 Commit SHA: 79c8499 | Docs | View more details | Give us feedback! |
BenchmarksComparisonBenchmark execution time: 2026-08-31 20:16:55 Comparing candidate commit 79c8499 in PR branch Found 0 performance improvements and 2 performance regressions! Performance is the same for 133 metrics, 1 unstable metrics.
|
Artifact Size Benchmark Reportaarch64-alpine-linux-musl
aarch64-unknown-linux-gnu
libdatadog-x64-windows
libdatadog-x86-windows
x86_64-alpine-linux-musl
x86_64-unknown-linux-gnu
|
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: fc1ff2cd86
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 94541f2ccf
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
1a741a1 to
4554db3
Compare
ekump
left a comment
There was a problem hiding this comment.
This doesn't include fuzz and benchmark jobs. Please create tickets for those under this milestone to update those as well. You'll need to update the images in benchmarking-platform. Also please update the macOS AMI in ci-platform-machine-images before merging this PR. Please attach performance changes to the description of the PR
ekump
left a comment
There was a problem hiding this comment.
made a commit to add missing test coverage for additional_metric_tags. LGTM
|
/merge |
|
View all feedbacks in Devflow UI.
It will be processed automatically as soon as GitHub reports it as mergeable. View in MergeQueue UI.
The expected merge time in
|
|
/code blockers |
|
View all feedbacks in Devflow UI.
Checking merge blockers for #2309...
Detected 1 merge blocker(s) to address: 🟠 Pending
|
2e6061f to
f369f59
Compare
…ker, libdd-data-pipeline, li... (#2482) # Release proposal for libdd-capabilities-impl, libdd-crashtracker, libdd-data-pipeline, libdd-ddsketch, libdd-ffe, libdd-http-client, libdd-ipc, libdd-library-config, libdd-live-debugger, libdd-otel-thread-ctx, libdd-remote-config, libdd-shared-runtime, libdd-telemetry, libdd-trace-utils, libdd-tracer-flare and their dependencies This PR contains version bumps based on public API changes and commits since last release. ###⚠️ Crates left out of this proposal affected by its major bumps These publishable workspace crates are not part of this release but their dependency requirement was rewritten on this branch while their published version still requires the old major. If they are a dependency on your deployment not including them in the release could result in duplicate packages or symbol incompatibility. - `libdd-common` `5.2.0` → `6.0.0` affects: `libdd-profiling`, `libdd-sampling` - `libdd-trace-utils` `11.0.0` → `12.0.0` affects: `libdd-sampling` ## libdd-capabilities **Next version:** `3.0.1` **Semver bump:** `patch` **Tag:** `libdd-capabilities-v3.0.1` ### Commits - refactor: migrate HTTP & networking deps to workspace level (phase 4bis) (#2350) - feat: do not entirely disable connection pooling for periodic connections (#2440) ## libdd-common **Next version:** `6.0.0` **Semver bump:** `major` **Tag:** `libdd-common-v6.0.0` ### Commits - refactor: migrate HTTP & networking deps to workspace level (phase 4bis) (#2350) - feat: do not entirely disable connection pooling for periodic connections (#2440) - feat(data-pipeline)!: add agentless stats export (#2309) - feat(data-pipeline): add runtime-independent agentless sending (#2389) - feat(common)!: add HTTPS_PROXY support for hyper_backend (#2421) ## libdd-ipc-macros **Next version:** `1.0.1` **Semver bump:** `patch` **Tag:** `libdd-ipc-macros-v1.0.1` ### Commits - feat(sidecar)!: support appsec helper-rust integration with sidecar (#2310) ## libdd-otel-thread-ctx **Next version:** `1.1.0` **Semver bump:** `minor` **Tag:** `libdd-otel-thread-ctx-v1.1.0` ### Commits - feat(otel-thread-ctx): add update-and-attach operation (#2443) ## libdd-tinybytes **Next version:** `1.1.3` **Semver bump:** `patch` **Tag:** `libdd-tinybytes-v1.1.3` ### Commits - refactor: migrate HTTP & networking deps to workspace level (phase 4bis) (#2350) ## libdd-capabilities-impl **Next version:** `5.0.0` **Semver bump:** `major` **Tag:** `libdd-capabilities-impl-v5.0.0` ###⚠️ major bump forced due to: - `libdd-common`: ^5.1.1 → ^6.0.0 ### Commits - refactor: migrate HTTP & networking deps to workspace level (phase 4bis) (#2350) - feat: do not entirely disable connection pooling for periodic connections (#2440) - feat(data-pipeline): add runtime-independent agentless sending (#2389) ## libdd-http-client **Next version:** `2.0.0` **Semver bump:** `major` **Tag:** `libdd-http-client-v2.0.0` ###⚠️ major bump forced due to: - `libdd-common`: ^5.1.1 → ^6.0.0 ### Commits - refactor: migrate HTTP & networking deps to workspace level (phase 4bis) (#2350) - feat(data-pipeline): add runtime-independent agentless sending (#2389) - feat(common)!: add HTTPS_PROXY support for hyper_backend (#2421) ## libdd-remote-config **Next version:** `5.0.0` **Semver bump:** `major` **Tag:** `libdd-remote-config-v5.0.0` ###⚠️ major bump forced due to: - `libdd-common`: ^5.2.0 → ^6.0.0 ### Commits - fix(remote-config): refresh fetcher identity (#2469) - refactor: migrate HTTP & networking deps to workspace level (phase 4bis) (#2350) - fix(remote-config): reuse injected sleep capability (#2429) ## libdd-shared-runtime **Next version:** `4.0.0` **Semver bump:** `major` **Tag:** `libdd-shared-runtime-v4.0.0` ###⚠️ major bump forced due to: - `libdd-common`: ^5.2.0 → ^6.0.0 ### Commits - fix(shared-runtime): allow disabling worker fork restart (#2464) ## libdd-trace-utils **Next version:** `12.0.0` **Semver bump:** `major` **Tag:** `libdd-trace-utils-v12.0.0` ###⚠️ major bump forced due to: - `libdd-common`: ^5.2.0 → ^6.0.0 ### Commits - feat(trace-utils): add v1-native JSON log encoder brick (#2371) - feat(trace-utils): add v1-native agentless JSON encoder brick (#2370) - refactor: migrate HTTP & networking deps to workspace level (phase 4bis) (#2350) - fix(trace-stats): read OTel HTTP names for the status and method dimensions (#2323) - feat(data-pipeline): emit native trace export telemetry (#2338) - feat(data-pipeline)!: add agentless stats export (#2309) - fix(trace-utils): use vec map dedup when serializing (#2422) - feat(data-pipeline): add runtime-independent agentless sending (#2389) - fix(compression): align zstd behavior across targets (#2400) - feat(trace-utils)!: add from owned to SpanText (#2403) ## libdd-ffe **Next version:** `2.0.0` **Semver bump:** `major` **Tag:** `libdd-ffe-v2.0.0` ###⚠️ major bump forced due to: - `libdd-common`: ^5.1.1 → ^6.0.0 - `libdd-remote-config`: ^3.0.0 → ^4.1.0 - `libdd-trace-protobuf`: ^4.0.1 → ^5.0.0 ### Commits - refactor: migrate HTTP & networking deps to workspace level (phase 4bis) (#2350) - feat(ffe): support arbitrary semver core parts (#2413) - feat(ffe)!: send the split serial id on exposure events [EX-3425] (#2402) - feat(ffe): expose observeFullEvaluationData config-level FFI getter (#2373) - fix(ffe): report rejected flags as parse errors (#2339) - test: skip/shorten slow miri jobs (#2331) ## libdd-dogstatsd-client **Next version:** `6.0.0` **Semver bump:** `major` **Tag:** `libdd-dogstatsd-client-v6.0.0` ###⚠️ major bump forced due to: - `libdd-common`: ^5.2.0 → ^6.0.0 ### Commits - refactor: migrate HTTP & networking deps to workspace level (phase 4bis) (#2350) ## libdd-telemetry **Next version:** `8.0.0` **Semver bump:** `major` **Tag:** `libdd-telemetry-v8.0.0` ###⚠️ major bump forced due to: - `libdd-common`: ^5.2.0 → ^6.0.0 ### Commits - refactor: migrate HTTP & networking deps to workspace level (phase 4bis) (#2350) - refactor(telemetry): avoid doing two separate http requests in stop telemetry (#2435) ## libdd-trace-obfuscation **Next version:** `8.0.0` **Semver bump:** `major` **Tag:** `libdd-trace-obfuscation-v8.0.0` ###⚠️ major bump forced due to: - `libdd-common`: ^5.2.0 → ^6.0.0 - `libdd-trace-utils`: ^11.0.0 → ^12.0.0 ### Commits - fix(trace-obfuscation): scan all span meta for credit-card obfuscation (#2472) - refactor: migrate HTTP & networking deps to workspace level (phase 4bis) (#2350) - feat(data-pipeline)!: Obfuscate v04 spans in agentless context (#2418) ## libdd-tracer-flare **Next version:** `3.0.0` **Semver bump:** `major` **Tag:** `libdd-tracer-flare-v3.0.0` ###⚠️ major bump forced due to: - `libdd-common`: ^5.2.0 → ^6.0.0 - `libdd-trace-utils`: ^11.0.0 → ^12.0.0 ### Commits - refactor: migrate HTTP & networking deps to workspace level (phase 4bis) (#2350) ## libdd-crashtracker **Next version:** `3.0.0` **Semver bump:** `major` **Tag:** `libdd-crashtracker-v3.0.0` ###⚠️ major bump forced due to: - `libdd-common`: ^5.2.0 → ^6.0.0 ### Commits - refactor: migrate HTTP & networking deps to workspace level (phase 4bis) (#2350) - fix(crashtracking): filter out frames above faulting frame (#2428) - feat(sidecar)!: support appsec helper-rust integration with sidecar (#2310) - chore(crashtracking): use RAII remote ptrace API (#2416) - chore(crashtracking): bump libdd-libunwind-sys to v1.0.3 (#2414) ## libdd-data-pipeline-core **Next version:** `1.0.0` **Semver bump:** `major` **Tag:** `libdd-data-pipeline-core-v1.0.0` **Warning:** this is an initial release. Please verify that the version and commits included are correct. ## libdd-trace-stats **Next version:** `9.0.0` **Semver bump:** `major` **Tag:** `libdd-trace-stats-v9.0.0` ###⚠️ major bump forced due to: - `libdd-common`: ^5.2.0 → ^6.0.0 - `libdd-trace-obfuscation`: ^7.0.0 → ^8.0.0 - `libdd-trace-utils`: ^11.0.0 → ^12.0.0 ### Commits - refactor: migrate HTTP & networking deps to workspace level (phase 4bis) (#2350) - fix(trace-stats): read OTel HTTP names for the status and method dimensions (#2323) - feat(data-pipeline)!: add agentless stats export (#2309) - feat(trace-utils)!: add from owned to SpanText (#2403) ## libdd-data-pipeline **Next version:** `10.0.0` **Semver bump:** `major` **Tag:** `libdd-data-pipeline-v10.0.0` ###⚠️ major bump forced due to: - `libdd-common`: ^5.2.0 → ^6.0.0 - `libdd-trace-obfuscation`: ^7.0.0 → ^8.0.0 - `libdd-trace-stats`: ^8.0.0 → ^9.0.0 - `libdd-trace-utils`: ^11.0.0 → ^12.0.0 ### Commits - feat(trace-utils): add v1-native JSON log encoder brick (#2371) - chore: prepare crate for publishing (#2466) - refactor: migrate HTTP & networking deps to workspace level (phase 4bis) (#2350) - fix(trace-stats): read OTel HTTP names for the status and method dimensions (#2323) - fix(data-pipeline): pass obfuscation config to OTLP stats (#2444) - feat(data-pipeline): emit native trace export telemetry (#2338) - feat(data-pipeline): add fork-safe OTLP gRPC trace transport (#2273) - feat(data-pipeline)!: add agentless stats export (#2309) - feat(data-pipeline)!: Obfuscate v04 spans in agentless context (#2418) - feat(data-pipeline): add runtime-independent agentless sending (#2389) - feat(trace-utils)!: add from owned to SpanText (#2403) ## libdd-ipc **Next version:** `2.0.0` **Semver bump:** `major` **Tag:** `libdd-ipc-v2.0.0` ###⚠️ major bump forced due to: - `libdd-common`: ^5.2.0 → ^6.0.0 - `libdd-trace-stats`: ^8.0.0 → ^9.0.0 ### Commits - fix(ipc): drop the signal feature from libdd-ipc (#2431) ## libdd-live-debugger **Next version:** `1.0.0` **Semver bump:** `major` **Tag:** `libdd-live-debugger-v1.0.0` **Warning:** this is an initial release. Please verify that the version and commits included are correct. [EX-3425]: https://datadoghq.atlassian.net/browse/EX-3425?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: hoolioh <107922352+hoolioh@users.noreply.github.com>
The compact v0.4 exporter bypasses the full TraceExporter, so it cannot deliver dd-trace computed stats. The callback-owned flush path preserves the worker-free WASM lifecycle and its 210 KiB size budget. Refs: DataDog/libdatadog#2309
Motivation
Perform stats aggregation and export directly to the public intake when agentless tracing is enabled
Changes