feat(data-pipeline)!: Obfuscate v04 spans in agentless context - #2418
Conversation
Clippy Allow Annotation ReportTracked Clippy
By file and crateBy file
By crate
About This ReportThis report tracks Clippy allow annotations for specific rules, showing how they've changed in this PR. Decreasing the number of these annotations generally improves code quality. Panic-inducing macros in particular should be avoided. In the future, this report may become a PR-blocking quality gate. |
|
✅ All CI checks and tests passed. 🎉 All green!🧪 All tests passed 🎯 Code Coverage (details) 🔗 Commit SHA: 79b13e5 | Docs | View more details | Give us feedback! |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 637c473170
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
BenchmarksComparisonBenchmark execution time: 2026-08-31 13:14:41 Comparing candidate commit 79b13e5 in PR branch Found 15 performance improvements and 10 performance regressions! Performance is the same for 143 metrics, 10 unstable metrics.
|
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ad70e61397
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
ichinaski
left a comment
There was a problem hiding this comment.
It's somewhat unclear how much on parity this approach is with the Trace Agent obfuscation config. We should document this somewhere, and also test the schemas are the same.
I also don't fully understand why we want to keep 2 separate implementations here, one for protobuf, the other for raw spans (using JSON endpoint). These 2 paths look also inconsistent at first glance (pb implementation being more complex and more tested).
The protobuf span struct is obsolete and used only by serverless for their extension. I cannot remove the If we ever want to send protobuf spans to the intake APM clients using libdatadog, we will do it by adding a protobuf serializer to the v1 span struct. |
bwoebi
left a comment
There was a problem hiding this comment.
I'm not too happy that it double obfuscates common operations, which have already been obfuscated once for stats computation.
But that's a future optimization, not a blocker. The implementation looks right to me: trivially obfuscating every span, and avoiding reallocating/copying for stuff which clearly doesn't need to be obfuscated.
ekump
left a comment
There was a problem hiding this comment.
A few things that need fixing, but not blocking. Please create Jira tickets for any follow up work.
|
/merge |
|
View all feedbacks in Devflow UI.
It will be processed automatically as soon as GitHub reports it as mergeable. View in MergeQueue UI.
The expected merge time in
|
|
/code blockers |
|
View all feedbacks in Devflow UI.
Checking merge blockers for #2418...
Detected 1 merge blocker(s) to address: 🟠 Pending
|
morrisonlevi
left a comment
There was a problem hiding this comment.
I looked only at the libdd-profiling-ffi/Cargo.toml changes, which are fine.
#2472) # Motivation Credit-card obfuscation only checked the single `card.number` tag, but the trace-agent scans all span meta attributes (with an explicit skip-list) # Changes * Replace the single `TAG_CARD_NUMBER` fast-path in `obfuscate_pb_span` and `obfuscate_v04_span` with a full scan of all `span.meta` entries * Skip keys in the built-in skip-list (`should_obfuscate_cc_key`) and the user-configured `keep_values` set Followup for #2418
…ker, libdd-data-pipeline, li... (#2482) # Release proposal for libdd-capabilities-impl, libdd-crashtracker, libdd-data-pipeline, libdd-ddsketch, libdd-ffe, libdd-http-client, libdd-ipc, libdd-library-config, libdd-live-debugger, libdd-otel-thread-ctx, libdd-remote-config, libdd-shared-runtime, libdd-telemetry, libdd-trace-utils, libdd-tracer-flare and their dependencies This PR contains version bumps based on public API changes and commits since last release. ###⚠️ Crates left out of this proposal affected by its major bumps These publishable workspace crates are not part of this release but their dependency requirement was rewritten on this branch while their published version still requires the old major. If they are a dependency on your deployment not including them in the release could result in duplicate packages or symbol incompatibility. - `libdd-common` `5.2.0` → `6.0.0` affects: `libdd-profiling`, `libdd-sampling` - `libdd-trace-utils` `11.0.0` → `12.0.0` affects: `libdd-sampling` ## libdd-capabilities **Next version:** `3.0.1` **Semver bump:** `patch` **Tag:** `libdd-capabilities-v3.0.1` ### Commits - refactor: migrate HTTP & networking deps to workspace level (phase 4bis) (#2350) - feat: do not entirely disable connection pooling for periodic connections (#2440) ## libdd-common **Next version:** `6.0.0` **Semver bump:** `major` **Tag:** `libdd-common-v6.0.0` ### Commits - refactor: migrate HTTP & networking deps to workspace level (phase 4bis) (#2350) - feat: do not entirely disable connection pooling for periodic connections (#2440) - feat(data-pipeline)!: add agentless stats export (#2309) - feat(data-pipeline): add runtime-independent agentless sending (#2389) - feat(common)!: add HTTPS_PROXY support for hyper_backend (#2421) ## libdd-ipc-macros **Next version:** `1.0.1` **Semver bump:** `patch` **Tag:** `libdd-ipc-macros-v1.0.1` ### Commits - feat(sidecar)!: support appsec helper-rust integration with sidecar (#2310) ## libdd-otel-thread-ctx **Next version:** `1.1.0` **Semver bump:** `minor` **Tag:** `libdd-otel-thread-ctx-v1.1.0` ### Commits - feat(otel-thread-ctx): add update-and-attach operation (#2443) ## libdd-tinybytes **Next version:** `1.1.3` **Semver bump:** `patch` **Tag:** `libdd-tinybytes-v1.1.3` ### Commits - refactor: migrate HTTP & networking deps to workspace level (phase 4bis) (#2350) ## libdd-capabilities-impl **Next version:** `5.0.0` **Semver bump:** `major` **Tag:** `libdd-capabilities-impl-v5.0.0` ###⚠️ major bump forced due to: - `libdd-common`: ^5.1.1 → ^6.0.0 ### Commits - refactor: migrate HTTP & networking deps to workspace level (phase 4bis) (#2350) - feat: do not entirely disable connection pooling for periodic connections (#2440) - feat(data-pipeline): add runtime-independent agentless sending (#2389) ## libdd-http-client **Next version:** `2.0.0` **Semver bump:** `major` **Tag:** `libdd-http-client-v2.0.0` ###⚠️ major bump forced due to: - `libdd-common`: ^5.1.1 → ^6.0.0 ### Commits - refactor: migrate HTTP & networking deps to workspace level (phase 4bis) (#2350) - feat(data-pipeline): add runtime-independent agentless sending (#2389) - feat(common)!: add HTTPS_PROXY support for hyper_backend (#2421) ## libdd-remote-config **Next version:** `5.0.0` **Semver bump:** `major` **Tag:** `libdd-remote-config-v5.0.0` ###⚠️ major bump forced due to: - `libdd-common`: ^5.2.0 → ^6.0.0 ### Commits - fix(remote-config): refresh fetcher identity (#2469) - refactor: migrate HTTP & networking deps to workspace level (phase 4bis) (#2350) - fix(remote-config): reuse injected sleep capability (#2429) ## libdd-shared-runtime **Next version:** `4.0.0` **Semver bump:** `major` **Tag:** `libdd-shared-runtime-v4.0.0` ###⚠️ major bump forced due to: - `libdd-common`: ^5.2.0 → ^6.0.0 ### Commits - fix(shared-runtime): allow disabling worker fork restart (#2464) ## libdd-trace-utils **Next version:** `12.0.0` **Semver bump:** `major` **Tag:** `libdd-trace-utils-v12.0.0` ###⚠️ major bump forced due to: - `libdd-common`: ^5.2.0 → ^6.0.0 ### Commits - feat(trace-utils): add v1-native JSON log encoder brick (#2371) - feat(trace-utils): add v1-native agentless JSON encoder brick (#2370) - refactor: migrate HTTP & networking deps to workspace level (phase 4bis) (#2350) - fix(trace-stats): read OTel HTTP names for the status and method dimensions (#2323) - feat(data-pipeline): emit native trace export telemetry (#2338) - feat(data-pipeline)!: add agentless stats export (#2309) - fix(trace-utils): use vec map dedup when serializing (#2422) - feat(data-pipeline): add runtime-independent agentless sending (#2389) - fix(compression): align zstd behavior across targets (#2400) - feat(trace-utils)!: add from owned to SpanText (#2403) ## libdd-ffe **Next version:** `2.0.0` **Semver bump:** `major` **Tag:** `libdd-ffe-v2.0.0` ###⚠️ major bump forced due to: - `libdd-common`: ^5.1.1 → ^6.0.0 - `libdd-remote-config`: ^3.0.0 → ^4.1.0 - `libdd-trace-protobuf`: ^4.0.1 → ^5.0.0 ### Commits - refactor: migrate HTTP & networking deps to workspace level (phase 4bis) (#2350) - feat(ffe): support arbitrary semver core parts (#2413) - feat(ffe)!: send the split serial id on exposure events [EX-3425] (#2402) - feat(ffe): expose observeFullEvaluationData config-level FFI getter (#2373) - fix(ffe): report rejected flags as parse errors (#2339) - test: skip/shorten slow miri jobs (#2331) ## libdd-dogstatsd-client **Next version:** `6.0.0` **Semver bump:** `major` **Tag:** `libdd-dogstatsd-client-v6.0.0` ###⚠️ major bump forced due to: - `libdd-common`: ^5.2.0 → ^6.0.0 ### Commits - refactor: migrate HTTP & networking deps to workspace level (phase 4bis) (#2350) ## libdd-telemetry **Next version:** `8.0.0` **Semver bump:** `major` **Tag:** `libdd-telemetry-v8.0.0` ###⚠️ major bump forced due to: - `libdd-common`: ^5.2.0 → ^6.0.0 ### Commits - refactor: migrate HTTP & networking deps to workspace level (phase 4bis) (#2350) - refactor(telemetry): avoid doing two separate http requests in stop telemetry (#2435) ## libdd-trace-obfuscation **Next version:** `8.0.0` **Semver bump:** `major` **Tag:** `libdd-trace-obfuscation-v8.0.0` ###⚠️ major bump forced due to: - `libdd-common`: ^5.2.0 → ^6.0.0 - `libdd-trace-utils`: ^11.0.0 → ^12.0.0 ### Commits - fix(trace-obfuscation): scan all span meta for credit-card obfuscation (#2472) - refactor: migrate HTTP & networking deps to workspace level (phase 4bis) (#2350) - feat(data-pipeline)!: Obfuscate v04 spans in agentless context (#2418) ## libdd-tracer-flare **Next version:** `3.0.0` **Semver bump:** `major` **Tag:** `libdd-tracer-flare-v3.0.0` ###⚠️ major bump forced due to: - `libdd-common`: ^5.2.0 → ^6.0.0 - `libdd-trace-utils`: ^11.0.0 → ^12.0.0 ### Commits - refactor: migrate HTTP & networking deps to workspace level (phase 4bis) (#2350) ## libdd-crashtracker **Next version:** `3.0.0` **Semver bump:** `major` **Tag:** `libdd-crashtracker-v3.0.0` ###⚠️ major bump forced due to: - `libdd-common`: ^5.2.0 → ^6.0.0 ### Commits - refactor: migrate HTTP & networking deps to workspace level (phase 4bis) (#2350) - fix(crashtracking): filter out frames above faulting frame (#2428) - feat(sidecar)!: support appsec helper-rust integration with sidecar (#2310) - chore(crashtracking): use RAII remote ptrace API (#2416) - chore(crashtracking): bump libdd-libunwind-sys to v1.0.3 (#2414) ## libdd-data-pipeline-core **Next version:** `1.0.0` **Semver bump:** `major` **Tag:** `libdd-data-pipeline-core-v1.0.0` **Warning:** this is an initial release. Please verify that the version and commits included are correct. ## libdd-trace-stats **Next version:** `9.0.0` **Semver bump:** `major` **Tag:** `libdd-trace-stats-v9.0.0` ###⚠️ major bump forced due to: - `libdd-common`: ^5.2.0 → ^6.0.0 - `libdd-trace-obfuscation`: ^7.0.0 → ^8.0.0 - `libdd-trace-utils`: ^11.0.0 → ^12.0.0 ### Commits - refactor: migrate HTTP & networking deps to workspace level (phase 4bis) (#2350) - fix(trace-stats): read OTel HTTP names for the status and method dimensions (#2323) - feat(data-pipeline)!: add agentless stats export (#2309) - feat(trace-utils)!: add from owned to SpanText (#2403) ## libdd-data-pipeline **Next version:** `10.0.0` **Semver bump:** `major` **Tag:** `libdd-data-pipeline-v10.0.0` ###⚠️ major bump forced due to: - `libdd-common`: ^5.2.0 → ^6.0.0 - `libdd-trace-obfuscation`: ^7.0.0 → ^8.0.0 - `libdd-trace-stats`: ^8.0.0 → ^9.0.0 - `libdd-trace-utils`: ^11.0.0 → ^12.0.0 ### Commits - feat(trace-utils): add v1-native JSON log encoder brick (#2371) - chore: prepare crate for publishing (#2466) - refactor: migrate HTTP & networking deps to workspace level (phase 4bis) (#2350) - fix(trace-stats): read OTel HTTP names for the status and method dimensions (#2323) - fix(data-pipeline): pass obfuscation config to OTLP stats (#2444) - feat(data-pipeline): emit native trace export telemetry (#2338) - feat(data-pipeline): add fork-safe OTLP gRPC trace transport (#2273) - feat(data-pipeline)!: add agentless stats export (#2309) - feat(data-pipeline)!: Obfuscate v04 spans in agentless context (#2418) - feat(data-pipeline): add runtime-independent agentless sending (#2389) - feat(trace-utils)!: add from owned to SpanText (#2403) ## libdd-ipc **Next version:** `2.0.0` **Semver bump:** `major` **Tag:** `libdd-ipc-v2.0.0` ###⚠️ major bump forced due to: - `libdd-common`: ^5.2.0 → ^6.0.0 - `libdd-trace-stats`: ^8.0.0 → ^9.0.0 ### Commits - fix(ipc): drop the signal feature from libdd-ipc (#2431) ## libdd-live-debugger **Next version:** `1.0.0` **Semver bump:** `major` **Tag:** `libdd-live-debugger-v1.0.0` **Warning:** this is an initial release. Please verify that the version and commits included are correct. [EX-3425]: https://datadoghq.atlassian.net/browse/EX-3425?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: hoolioh <107922352+hoolioh@users.noreply.github.com>
…uscation config once (#2474) # Motivation PR #2418 added agentless FFI config setters with two issues flagged in review: * the endpoint setter mutated the handle before validating both inputs * the obfuscation config JSON was parsed twice (once in the setter, again in the builder) * The setters also lacked test coverage. # Changes * FFI setters modify config atomically * Store the parsed ObfuscationConfig on TraceExporterConfig instead of the raw JSON string * Add tests for set_agentless_endpoint, set_agentless_timeout, and set_obfuscation_config
What does this PR do?