feat(data-pipeline)!: add compression option for agentless export - #2205
Conversation
# Motivation Since we are sending data to the intake directly, we should be compressing requests as they can be multi MB. The compression uses the zstd crate, which was already used in trace-utils. It links to a C library, so it is not usable everyhwere (WASM for instance) and adds ~500KB of artifact size. For these reasons, it is hidden behind a feature flag and should be enabled as required (implementation using the agentless option mostly). Another option would be rustzstd, an full rust port. Tge library is smaller (300KB) but it is notably slower.. (2x to 5x slower, with worse compression ratios at the same level) # Changes * Add a compression strategy parameter to send_with_retry * implement zstd compression * since compressio is implemented lower down, remove compression option from SendData * Add a "compression" feature flag on data-pipeline to be used in languages where it is needed
📚 Documentation Check Results📦
|
🔒 Cargo Deny Results📦
|
🎉 All green!🧪 All tests passed 🎯 Code Coverage (details) 🔗 Commit SHA: bd3a767 | Docs | Datadog PR Page | Give us feedback! |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 70a0d4d48a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| payload: Vec<u8>, | ||
| headers: &HeaderMap, | ||
| retry_strategy: &RetryStrategy, | ||
| compression_strategy: CompressionStrategy, |
There was a problem hiding this comment.
Update the doctest call for the new argument
Adding compression_strategy here leaves the doctest immediately above with the old five-argument call, so cargo test --doc -p libdd-trace-utils --no-default-features now fails with E0061 for send_with_retry(&capabilities, &target, payload, &headers, &retry_strategy). Please pass CompressionStrategy::None (or another strategy) in the example so the required doc-test validation stays green.
Useful? React with 👍 / 👎.
| payload, | ||
| &headers, | ||
| &self.retry_strategy, | ||
| compression_strategy, |
There was a problem hiding this comment.
Report compressed bytes for compressed sends
When a caller uses SendDataBuilder::with_compression(CompressionStrategy::Zstd { .. }), this new argument moves compression inside send_with_retry, but send_payload has already computed payload_len from the uncompressed Vec. As a result successful compressed SendData requests record SendDataResult.bytes_sent/telemetry as the pre-compression size even though the field is documented as bytes in the payload sent and previously measured the compressed final_payload; return the final payload length or compress before recording the metric.
Useful? React with 👍 / 👎.
| test-utils = [] | ||
| regex-lite = ["libdd-common/regex-lite"] | ||
| # Enable zstd compression for the agentless trace intake sender. | ||
| compression = ["libdd-trace-utils/compression"] |
There was a problem hiding this comment.
Expose compression through FFI release features
Defining the feature only on libdd-data-pipeline leaves no way for the existing FFI release path to enable it: I checked the Cargo feature wiring and libdd-data-pipeline-ffi does not forward libdd-data-pipeline/compression, while libdd-profiling-ffi only exposes data-pipeline-ffi. In languages consuming the generated FFI artifacts, enabling data-pipeline support will therefore still build uncompressed agentless sends unless a pass-through feature is added.
Useful? React with 👍 / 👎.
Artifact Size Benchmark Reportaarch64-alpine-linux-musl
aarch64-unknown-linux-gnu
libdatadog-x64-windows
libdatadog-x86-windows
x86_64-alpine-linux-musl
x86_64-unknown-linux-gnu
|
yannham
left a comment
There was a problem hiding this comment.
Beside some Codex comments that look relevant, the rest LGTM
- Fix rustfmt on the send_with_retry doctest example - Keep test_agentless_export_body_shape compatible with the compression feature by asserting on content-encoding when compression is enabled - Report the compressed byte count in SendData telemetry by compressing in send_payload before measuring bytes_sent - Expose a 'compression' feature through the data-pipeline / profiling FFI crates and the builder so release artifacts can opt in
BenchmarksComparisonBenchmark execution time: 2026-07-29 16:34:23 Comparing candidate commit bd3a767 in PR branch Found 5 performance improvements and 19 performance regressions! Performance is the same for 153 metrics, 10 unstable metrics.
|
…er-flare (#2332) # Release proposal for libdd-ffe, libdd-http-client, libdd-tracer-flare and their dependencies This PR contains version bumps based on public API changes and commits since last release. ## libdd-capabilities **Next version:** `3.0.0` **Semver bump:** `major` **Tag:** `libdd-capabilities-v3.0.0` ### Commits - fix(telemetry)!: stop sending Connection: close to the Agent (#2286) - feat(capabilities)!: Add streaming to http capabilities (#2251) - feat: added file capability [APMSP-3780] (#2240) - feat: added environment capability [APMSP-3780] (#2239) ## libdd-common **Next version:** `5.1.1` **Semver bump:** `patch` **Tag:** `libdd-common-v5.1.1` ### Commits - feat(telemetry)!: Add Installation signature and AppProduct changes payloads (#2213) - refactor: migrate to workspace dependencies, phase 4 (#2296) - refactor: migrate to workspace dependencies, phase 3 (#2283) - refactor: consolidate core dependencies at workspace level (phase 1) (#2253) - test: stabilize flaky tests (#2256) - fix(sidecar): don't double-encode file:// telemetry endpoints (#2230) - fix: last new clippy lint (#2221) ## libdd-tinybytes **Next version:** `1.1.2` **Semver bump:** `patch` **Tag:** `libdd-tinybytes-v1.1.2` ### Commits - chore: moving to workspace-level dependencies, phase 2 (#2270) - refactor: consolidate core dependencies at workspace level (phase 1) (#2253) ## libdd-trace-protobuf **Next version:** `4.0.1` **Semver bump:** `patch` **Tag:** `libdd-trace-protobuf-v4.0.1` ### Commits - refactor: migrate to workspace dependencies, phase 4 (#2296) - chore: moving to workspace-level dependencies, phase 2 (#2270) - refactor: consolidate core dependencies at workspace level (phase 1) (#2253) - feat(stats)!: implement additional metric tags in libdd-trace-stats (#2170) ## libdd-capabilities-impl **Next version:** `4.0.0` **Semver bump:** `major` **Tag:** `libdd-capabilities-impl-v4.0.0` ###⚠️ major bump forced due to: - `libdd-capabilities`: ^2.1.0 → ^3.0.0 ### Commits - feat(telemetry)!: Add Installation signature and AppProduct changes payloads (#2213) - refactor: migrate to workspace dependencies, phase 4 (#2296) - fix(telemetry)!: stop sending Connection: close to the Agent (#2286) - fix(remote-config)!: Expose HttpClientCapability in remote config (#2252) - feat(capabilities)!: Add streaming to http capabilities (#2251) - refactor: consolidate core dependencies at workspace level (phase 1) (#2253) - feat: added file capability [APMSP-3780] (#2240) - feat(telemetry)!: make telemetry worker wasm-compatible for the TraceExporter (#2172) - feat: added environment capability [APMSP-3780] (#2239) ## libdd-http-client **Next version:** `1.0.0` **Semver bump:** `major` **Tag:** `libdd-http-client-v1.0.0` **Warning:** this is an initial release. Please verify that the version and commits included are correct. ## libdd-trace-normalization **Next version:** `3.0.1` **Semver bump:** `patch` **Tag:** `libdd-trace-normalization-v3.0.1` ### Commits - refactor: migrate to workspace dependencies, phase 4 (#2296) - refactor: consolidate core dependencies at workspace level (phase 1) (#2253) ## libdd-remote-config **Next version:** `3.0.0` **Semver bump:** `major` **Tag:** `libdd-remote-config-v3.0.0` ### Commits - refactor(rc)!: make conversion from RemoteConfigProduct back and forth generally available (#2325) - fix(wasm)!: finish the WASM port of remote-config (#2315) - refactor: migrate to workspace dependencies, phase 3 (#2283) - feat(rc): Handle expired config status (#2274) - feat(remote-config)!: add AsmRawResponseBody capability (#2278) - chore: moving to workspace-level dependencies, phase 2 (#2270) - fix(remote-config)!: Expose HttpClientCapability in remote config (#2252) - refactor: consolidate core dependencies at workspace level (phase 1) (#2253) - fix(rc): Make Target fields available again after eaf5ad0 (#2232) - fix: new clippy lints (#2219) ## libdd-trace-utils **Next version:** `10.0.0` **Semver bump:** `major` **Tag:** `libdd-trace-utils-v10.0.0` ###⚠️ major bump forced due to: - `libdd-capabilities`: ^2.1.0 → ^3.0.0 ### Commits - refactor(rc)!: make conversion from RemoteConfigProduct back and forth generally available (#2325) - fix(libdd-trace-utils): apply SpanLink flags masking when v0.5 json encoding (#2314) - refactor: migrate to workspace dependencies, phase 4 (#2296) - feat(data-pipeline)!: add compression option for agentless export (#2205) - refactor: migrate to workspace dependencies, phase 3 (#2283) - refactor(trace-utils)!: split generic TracerHeaderTags (#2279) - chore(lint): update nightly (#2281) - chore: moving to workspace-level dependencies, phase 2 (#2270) - fix(remote-config)!: Expose HttpClientCapability in remote config (#2252) - refactor: consolidate core dependencies at workspace level (phase 1) (#2253) - feat(stats)!: implement additional metric tags in libdd-trace-stats (#2170) - feat(trace-utils)!: add v1 decoder (#2174) - feat(trace-utils)!: add encoder v1 to v04 + refactor (#2145) - feat(otlp): add instrumentation scope metadata (#2235) - fix(trace-utils)!: serialize v0.5 span links and events into meta (#980) ## libdd-ffe **Next version:** `1.0.0` **Semver bump:** `major` **Tag:** `libdd-ffe-v1.0.0` **Warning:** this is an initial release. Please verify that the version and commits included are correct. ## libdd-tracer-flare **Next version:** `1.0.0` **Semver bump:** `major` **Tag:** `libdd-tracer-flare-v1.0.0` **Warning:** this is an initial release. Please verify that the version and commits included are correct. [APMSP-3780]: https://datadoghq.atlassian.net/browse/APMSP-3780?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ [APMSP-3780]: https://datadoghq.atlassian.net/browse/APMSP-3780?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ [APMSP-3780]: https://datadoghq.atlassian.net/browse/APMSP-3780?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: hoolioh <107922352+hoolioh@users.noreply.github.com>
) # Release proposal for libdd-data-pipeline, libdd-telemetry and their dependencies This PR contains version bumps based on public API changes and commits since last release. ## libdd-ddsketch **Next version:** `1.1.1` **Semver bump:** `patch` **Tag:** `libdd-ddsketch-v1.1.1` ### Commits - refactor: migrate to workspace dependencies, phase 4 (#2296) - chore: moving to workspace-level dependencies, phase 2 (#2270) ## libdd-shared-runtime **Next version:** `3.0.0` **Semver bump:** `major` **Tag:** `libdd-shared-runtime-v3.0.0` ###⚠️ major bump forced due to: - `libdd-capabilities`: ^2.1.0 → ^3.0.0 - `libdd-capabilities-impl`: ^3.0.0 → ^4.0.0 ### Commits - feat(shared-runtime): add block_on_with_timeout to BlockingRuntime (#2333) - chore(runtime): restart all workers on invalid state (#2262) - refactor: migrate to workspace dependencies, phase 3 (#2283) - refactor: consolidate core dependencies at workspace level (phase 1) (#2253) ## libdd-dogstatsd-client **Next version:** `5.0.0` **Semver bump:** `major` **Tag:** `libdd-dogstatsd-client-v5.0.0` ### Commits - refactor: consolidate core dependencies at workspace level (phase 1) (#2253) - feat(dogstatsd)!: add shared_runtime buffered sink (#2224) - refactor(statsd)!: make client clonable (#2222) ## libdd-telemetry **Next version:** `7.0.0` **Semver bump:** `major` **Tag:** `libdd-telemetry-v7.0.0` ### Commits - refactor(rc)!: make conversion from RemoteConfigProduct back and forth generally available (#2325) - feat(telemetry)!: Add Installation signature and AppProduct changes payloads (#2213) - fix(telemetry)!: stop sending Connection: close to the Agent (#2286) - refactor: migrate to workspace dependencies, phase 3 (#2283) - chore: moving to workspace-level dependencies, phase 2 (#2270) - refactor: consolidate core dependencies at workspace level (phase 1) (#2253) - fix: drain the mailbox before stopping (#2258) - feat(telemetry)!: make telemetry worker wasm-compatible for the TraceExporter (#2172) - fix(sidecar): don't double-encode file:// telemetry endpoints (#2230) ## libdd-trace-obfuscation **Next version:** `6.0.0` **Semver bump:** `major` **Tag:** `libdd-trace-obfuscation-v6.0.0` ###⚠️ major bump forced due to: - `libdd-trace-utils`: ^9.0.0 → ^10.1.0 ### Commits - refactor: migrate to workspace dependencies, phase 4 (#2296) - chore(lint): update nightly (#2281) - chore: moving to workspace-level dependencies, phase 2 (#2270) - refactor: consolidate core dependencies at workspace level (phase 1) (#2253) - fix: new clippy lints (#2219) ## libdd-trace-stats **Next version:** `7.0.0` **Semver bump:** `major` **Tag:** `libdd-trace-stats-v7.0.0` ###⚠️ major bump forced due to: - `libdd-capabilities`: ^2.1.0 → ^3.0.0 - `libdd-capabilities-impl`: ^3.0.0 → ^4.0.0 - `libdd-dogstatsd-client`: ^4.0.0 → ^5.0.0 - `libdd-telemetry`: ^6.0.0 → ^7.0.0 - `libdd-trace-utils`: ^9.0.0 → ^10.1.0 ### Commits - feat: add v1 isolated bricks for trace exporter (#2280) - refactor: migrate to workspace dependencies, phase 4 (#2296) - feat(data-pipeline)!: add compression option for agentless export (#2205) - feat(stats)!: per-field cardinality limits (#2211) - refactor: migrate to workspace dependencies, phase 3 (#2283) - feat(stats)!: string field length limit (#2267) - feat(trace_exporter)!: add `restart_after_fork` configuration option [APMSP-3843] (#2263) - refactor: consolidate core dependencies at workspace level (phase 1) (#2253) - feat(stats)!: implement additional metric tags in libdd-trace-stats (#2170) - refactor(statsd)!: make client clonable (#2222) - fix(stats-exporter): disable retries for stats payload sends (#2249) - feat(telemetry)!: make telemetry worker wasm-compatible for the TraceExporter (#2172) - fix!: css obfuscation logic (#2203) ## libdd-data-pipeline **Next version:** `8.0.0` **Semver bump:** `major` **Tag:** `libdd-data-pipeline-v8.0.0` ###⚠️ major bump forced due to: - `libdd-capabilities`: ^2.1.0 → ^3.0.0 - `libdd-capabilities-impl`: ^3.0.0 → ^4.0.0 - `libdd-dogstatsd-client`: ^4.0.0 → ^5.0.0 - `libdd-telemetry`: ^6.0.0 → ^7.0.0 - `libdd-trace-stats`: ^6.0.0 → ^7.0.0 - `libdd-trace-utils`: ^9.0.0 → ^10.1.0 ### Commits - fix(trace-exporter): mark OTLP client-computed stats (#2245) - feat(data-pipeline): adds new attributes to OTLP trace metrics export (#2316) - feat: add v1 isolated bricks for trace exporter (#2280) - feat(data-pipeline)!: add flush_and_close to the trace buffer (#2313) - feat(telemetry)!: Add Installation signature and AppProduct changes payloads (#2213) - refactor: migrate to workspace dependencies, phase 4 (#2296) - feat(data-pipeline)!: add compression option for agentless export (#2205) - fix(telemetry)!: stop sending Connection: close to the Agent (#2286) - feat(stats)!: per-field cardinality limits (#2211) - refactor: migrate to workspace dependencies, phase 3 (#2283) - feat(stats)!: string field length limit (#2267) - refactor(trace-utils)!: split generic TracerHeaderTags (#2279) - chore: moving to workspace-level dependencies, phase 2 (#2270) - feat(trace_exporter)!: add `restart_after_fork` configuration option [APMSP-3843] (#2263) - feat(data-pipeline): emit additional_metric_tags on OTLP span duration metric (#2264) - refactor: consolidate core dependencies at workspace level (phase 1) (#2253) - feat(dogstatsd)!: add shared_runtime buffered sink (#2224) - feat(stats)!: implement additional metric tags in libdd-trace-stats (#2170) - refactor(statsd)!: make client clonable (#2222) - feat(telemetry)!: make telemetry worker wasm-compatible for the TraceExporter (#2172) - feat(trace_exporter)!: remove deprecated methods (#2250) - feat(trace-utils)!: add v1 decoder (#2174) - feat(trace-utils)!: add encoder v1 to v04 + refactor (#2145) - feat(otlp): add instrumentation scope metadata (#2235) - fix(trace-utils)!: serialize v0.5 span links and events into meta (#980) - fix!: css obfuscation logic (#2203) - test(data-pipeline): gate telemetry-only tests behind the telemetry feature (#2204) [APMSP-3843]: https://datadoghq.atlassian.net/browse/APMSP-3843?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: hoolioh <107922352+hoolioh@users.noreply.github.com>
Motivation
Since we are sending data to the intake directly, we should be compressing requests as they can be multi MB.
The compression uses the zstd crate, which was already used in trace-utils.
It links to a C library, so it is not usable everyhwere (WASM for instance) and adds ~500KB of artifact size. For these reasons, it is hidden behind a feature flag and should be enabled as required (implementation using the agentless option mostly).
Another option would be rustzstd, an full rust port. Tge library is smaller (300KB) but it is notably slower.. (2x to 5x slower, with worse compression ratios at the same level)
Changes
BREAKING CHANGE:
libdd-trace-utilspublic API changed —send_with_retrynowtakes an additional
CompressionStrategyargument, theSendDataCompressionenum was replaced by
send_with_retry::CompressionStrategy, and the implicitflate2feature was removed from thecompressionfeature set.