feat(trace-utils)!: add encoder v1 to v04 + refactor - #2145
gh-worker-dd-mergequeue-cf854d[bot] merged 23 commits into
Conversation
🎉 All green!🧪 All tests passed 🎯 Code Coverage (details) 🔗 Commit SHA: cbca66a | Docs | Datadog PR Page | Give us feedback! |
Artifact Size Benchmark Reportaarch64-alpine-linux-musl
aarch64-unknown-linux-gnu
libdatadog-x64-windows
libdatadog-x86-windows
x86_64-alpine-linux-musl
x86_64-unknown-linux-gnu
|
yannham
left a comment
There was a problem hiding this comment.
LGTM overall 👍 I think the naming convention brings more clarity indeed.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 5fbf20172c
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
ajgajg1134
left a comment
There was a problem hiding this comment.
Some of the function names are confusing me a bit / don't match the comments?
Also I think some of the v1 attribute types need to be "un-done" differently, specifically key values and lists shouldn't just get put in meta_struct and should be flattened into individual meta/metrics entries.
Once you have that changed, we should validate this with some integration tests against the real agent / intake to make sure the payloads match the agent decode and appear correctly in the UI
Aaalibaba42
left a comment
There was a problem hiding this comment.
Mostly nits and/or styling, most of these can be disregarded
ajgajg1134
left a comment
There was a problem hiding this comment.
The changes look good! The new names are much clearer to me. I do think we need a better integration / system testing story here though, the unit tests do a good job asserting what we think is right here but alignment with the actual trace-agent is harder to be sure of here. Unless that testing step would happen as part of testing actual integration with something like the rust tracer?
Aaalibaba42
left a comment
There was a problem hiding this comment.
For many of the "perf" comments, I didn't look into everything, but I've only really looked at the v04/span_v1.rs encoder, I wager there is more in other parts of the code, maybe try and have an agent scoure for similar issues. We really ought to be mindful of allocations in hotpath, and std structures like HashMaps, HashSets, Vecs, Strings, etc are heap allocated.
BenchmarksComparisonBenchmark execution time: 2026-07-15 13:21:34 Comparing candidate commit cbca66a in PR branch Found 21 performance improvements and 12 performance regressions! Performance is the same for 109 metrics, 0 unstable metrics.
|
Aaalibaba42
left a comment
There was a problem hiding this comment.
I reopened a few things you seem to have closed eagerly without saying why nor fixing the code, but overall it looks good enough after those are either fixed or justified
# What does this PR do? Adds a V1 msgpack decoder in `libdd-trace-utils` and wires the V1 variant through the trace pipeline so any V1 payload can be decoded, inspected, and re-encoded. # Motivation [APMSP-2813](https://datadoghq.atlassian.net/browse/APMSP-2813) : Prerequisite for the sidecar V1-native path. Splitting out the `libdd-trace-utils` / `libdd-data-pipeline` plumbing so it can land before the sidecar wiring. # Additional Notes - Mirrors the existing v04 decoder structure. - `trace_serializer.rs` now dispatches on `(TraceChunks, OutputFormat)`; the existing v0.4 → V1 cross-encode path is preserved. - `trace_utils::collect_trace_chunks` renamed to `convert_trace_chunks_v04_to_v05`. [APMSP-2813]: https://datadoghq.atlassian.net/browse/APMSP-2813?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ⚠️ You can focus on those folders/files only as I merged the others from the PR #2145 to get all the refactors: - libdd-data-pipeline/src/trace_exporter/trace_serializer.rs - libdd-trace-utils/src/msgpack_decoder/ - libdd-trace-utils/src/span/mod.rs - libdd-trace-utils/src/span/vec_map.rs - libdd-trace-utils/src/trace_utils.rs - libdd-trace-utils/src/tracer_payload.rs - libdd-trace-utils/tests/snapshots/compare_v1_full_payload_snapshot_test.json - libdd-trace-utils/tests/test_send_data.rs Co-authored-by: anais.raison <anais.raison@datadoghq.com>
…er-flare (#2332) # Release proposal for libdd-ffe, libdd-http-client, libdd-tracer-flare and their dependencies This PR contains version bumps based on public API changes and commits since last release. ## libdd-capabilities **Next version:** `3.0.0` **Semver bump:** `major` **Tag:** `libdd-capabilities-v3.0.0` ### Commits - fix(telemetry)!: stop sending Connection: close to the Agent (#2286) - feat(capabilities)!: Add streaming to http capabilities (#2251) - feat: added file capability [APMSP-3780] (#2240) - feat: added environment capability [APMSP-3780] (#2239) ## libdd-common **Next version:** `5.1.1` **Semver bump:** `patch` **Tag:** `libdd-common-v5.1.1` ### Commits - feat(telemetry)!: Add Installation signature and AppProduct changes payloads (#2213) - refactor: migrate to workspace dependencies, phase 4 (#2296) - refactor: migrate to workspace dependencies, phase 3 (#2283) - refactor: consolidate core dependencies at workspace level (phase 1) (#2253) - test: stabilize flaky tests (#2256) - fix(sidecar): don't double-encode file:// telemetry endpoints (#2230) - fix: last new clippy lint (#2221) ## libdd-tinybytes **Next version:** `1.1.2` **Semver bump:** `patch` **Tag:** `libdd-tinybytes-v1.1.2` ### Commits - chore: moving to workspace-level dependencies, phase 2 (#2270) - refactor: consolidate core dependencies at workspace level (phase 1) (#2253) ## libdd-trace-protobuf **Next version:** `4.0.1` **Semver bump:** `patch` **Tag:** `libdd-trace-protobuf-v4.0.1` ### Commits - refactor: migrate to workspace dependencies, phase 4 (#2296) - chore: moving to workspace-level dependencies, phase 2 (#2270) - refactor: consolidate core dependencies at workspace level (phase 1) (#2253) - feat(stats)!: implement additional metric tags in libdd-trace-stats (#2170) ## libdd-capabilities-impl **Next version:** `4.0.0` **Semver bump:** `major` **Tag:** `libdd-capabilities-impl-v4.0.0` ###⚠️ major bump forced due to: - `libdd-capabilities`: ^2.1.0 → ^3.0.0 ### Commits - feat(telemetry)!: Add Installation signature and AppProduct changes payloads (#2213) - refactor: migrate to workspace dependencies, phase 4 (#2296) - fix(telemetry)!: stop sending Connection: close to the Agent (#2286) - fix(remote-config)!: Expose HttpClientCapability in remote config (#2252) - feat(capabilities)!: Add streaming to http capabilities (#2251) - refactor: consolidate core dependencies at workspace level (phase 1) (#2253) - feat: added file capability [APMSP-3780] (#2240) - feat(telemetry)!: make telemetry worker wasm-compatible for the TraceExporter (#2172) - feat: added environment capability [APMSP-3780] (#2239) ## libdd-http-client **Next version:** `1.0.0` **Semver bump:** `major` **Tag:** `libdd-http-client-v1.0.0` **Warning:** this is an initial release. Please verify that the version and commits included are correct. ## libdd-trace-normalization **Next version:** `3.0.1` **Semver bump:** `patch` **Tag:** `libdd-trace-normalization-v3.0.1` ### Commits - refactor: migrate to workspace dependencies, phase 4 (#2296) - refactor: consolidate core dependencies at workspace level (phase 1) (#2253) ## libdd-remote-config **Next version:** `3.0.0` **Semver bump:** `major` **Tag:** `libdd-remote-config-v3.0.0` ### Commits - refactor(rc)!: make conversion from RemoteConfigProduct back and forth generally available (#2325) - fix(wasm)!: finish the WASM port of remote-config (#2315) - refactor: migrate to workspace dependencies, phase 3 (#2283) - feat(rc): Handle expired config status (#2274) - feat(remote-config)!: add AsmRawResponseBody capability (#2278) - chore: moving to workspace-level dependencies, phase 2 (#2270) - fix(remote-config)!: Expose HttpClientCapability in remote config (#2252) - refactor: consolidate core dependencies at workspace level (phase 1) (#2253) - fix(rc): Make Target fields available again after eaf5ad0 (#2232) - fix: new clippy lints (#2219) ## libdd-trace-utils **Next version:** `10.0.0` **Semver bump:** `major` **Tag:** `libdd-trace-utils-v10.0.0` ###⚠️ major bump forced due to: - `libdd-capabilities`: ^2.1.0 → ^3.0.0 ### Commits - refactor(rc)!: make conversion from RemoteConfigProduct back and forth generally available (#2325) - fix(libdd-trace-utils): apply SpanLink flags masking when v0.5 json encoding (#2314) - refactor: migrate to workspace dependencies, phase 4 (#2296) - feat(data-pipeline)!: add compression option for agentless export (#2205) - refactor: migrate to workspace dependencies, phase 3 (#2283) - refactor(trace-utils)!: split generic TracerHeaderTags (#2279) - chore(lint): update nightly (#2281) - chore: moving to workspace-level dependencies, phase 2 (#2270) - fix(remote-config)!: Expose HttpClientCapability in remote config (#2252) - refactor: consolidate core dependencies at workspace level (phase 1) (#2253) - feat(stats)!: implement additional metric tags in libdd-trace-stats (#2170) - feat(trace-utils)!: add v1 decoder (#2174) - feat(trace-utils)!: add encoder v1 to v04 + refactor (#2145) - feat(otlp): add instrumentation scope metadata (#2235) - fix(trace-utils)!: serialize v0.5 span links and events into meta (#980) ## libdd-ffe **Next version:** `1.0.0` **Semver bump:** `major` **Tag:** `libdd-ffe-v1.0.0` **Warning:** this is an initial release. Please verify that the version and commits included are correct. ## libdd-tracer-flare **Next version:** `1.0.0` **Semver bump:** `major` **Tag:** `libdd-tracer-flare-v1.0.0` **Warning:** this is an initial release. Please verify that the version and commits included are correct. [APMSP-3780]: https://datadoghq.atlassian.net/browse/APMSP-3780?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ [APMSP-3780]: https://datadoghq.atlassian.net/browse/APMSP-3780?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ [APMSP-3780]: https://datadoghq.atlassian.net/browse/APMSP-3780?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: hoolioh <107922352+hoolioh@users.noreply.github.com>
) # Release proposal for libdd-data-pipeline, libdd-telemetry and their dependencies This PR contains version bumps based on public API changes and commits since last release. ## libdd-ddsketch **Next version:** `1.1.1` **Semver bump:** `patch` **Tag:** `libdd-ddsketch-v1.1.1` ### Commits - refactor: migrate to workspace dependencies, phase 4 (#2296) - chore: moving to workspace-level dependencies, phase 2 (#2270) ## libdd-shared-runtime **Next version:** `3.0.0` **Semver bump:** `major` **Tag:** `libdd-shared-runtime-v3.0.0` ###⚠️ major bump forced due to: - `libdd-capabilities`: ^2.1.0 → ^3.0.0 - `libdd-capabilities-impl`: ^3.0.0 → ^4.0.0 ### Commits - feat(shared-runtime): add block_on_with_timeout to BlockingRuntime (#2333) - chore(runtime): restart all workers on invalid state (#2262) - refactor: migrate to workspace dependencies, phase 3 (#2283) - refactor: consolidate core dependencies at workspace level (phase 1) (#2253) ## libdd-dogstatsd-client **Next version:** `5.0.0` **Semver bump:** `major` **Tag:** `libdd-dogstatsd-client-v5.0.0` ### Commits - refactor: consolidate core dependencies at workspace level (phase 1) (#2253) - feat(dogstatsd)!: add shared_runtime buffered sink (#2224) - refactor(statsd)!: make client clonable (#2222) ## libdd-telemetry **Next version:** `7.0.0` **Semver bump:** `major` **Tag:** `libdd-telemetry-v7.0.0` ### Commits - refactor(rc)!: make conversion from RemoteConfigProduct back and forth generally available (#2325) - feat(telemetry)!: Add Installation signature and AppProduct changes payloads (#2213) - fix(telemetry)!: stop sending Connection: close to the Agent (#2286) - refactor: migrate to workspace dependencies, phase 3 (#2283) - chore: moving to workspace-level dependencies, phase 2 (#2270) - refactor: consolidate core dependencies at workspace level (phase 1) (#2253) - fix: drain the mailbox before stopping (#2258) - feat(telemetry)!: make telemetry worker wasm-compatible for the TraceExporter (#2172) - fix(sidecar): don't double-encode file:// telemetry endpoints (#2230) ## libdd-trace-obfuscation **Next version:** `6.0.0` **Semver bump:** `major` **Tag:** `libdd-trace-obfuscation-v6.0.0` ###⚠️ major bump forced due to: - `libdd-trace-utils`: ^9.0.0 → ^10.1.0 ### Commits - refactor: migrate to workspace dependencies, phase 4 (#2296) - chore(lint): update nightly (#2281) - chore: moving to workspace-level dependencies, phase 2 (#2270) - refactor: consolidate core dependencies at workspace level (phase 1) (#2253) - fix: new clippy lints (#2219) ## libdd-trace-stats **Next version:** `7.0.0` **Semver bump:** `major` **Tag:** `libdd-trace-stats-v7.0.0` ###⚠️ major bump forced due to: - `libdd-capabilities`: ^2.1.0 → ^3.0.0 - `libdd-capabilities-impl`: ^3.0.0 → ^4.0.0 - `libdd-dogstatsd-client`: ^4.0.0 → ^5.0.0 - `libdd-telemetry`: ^6.0.0 → ^7.0.0 - `libdd-trace-utils`: ^9.0.0 → ^10.1.0 ### Commits - feat: add v1 isolated bricks for trace exporter (#2280) - refactor: migrate to workspace dependencies, phase 4 (#2296) - feat(data-pipeline)!: add compression option for agentless export (#2205) - feat(stats)!: per-field cardinality limits (#2211) - refactor: migrate to workspace dependencies, phase 3 (#2283) - feat(stats)!: string field length limit (#2267) - feat(trace_exporter)!: add `restart_after_fork` configuration option [APMSP-3843] (#2263) - refactor: consolidate core dependencies at workspace level (phase 1) (#2253) - feat(stats)!: implement additional metric tags in libdd-trace-stats (#2170) - refactor(statsd)!: make client clonable (#2222) - fix(stats-exporter): disable retries for stats payload sends (#2249) - feat(telemetry)!: make telemetry worker wasm-compatible for the TraceExporter (#2172) - fix!: css obfuscation logic (#2203) ## libdd-data-pipeline **Next version:** `8.0.0` **Semver bump:** `major` **Tag:** `libdd-data-pipeline-v8.0.0` ###⚠️ major bump forced due to: - `libdd-capabilities`: ^2.1.0 → ^3.0.0 - `libdd-capabilities-impl`: ^3.0.0 → ^4.0.0 - `libdd-dogstatsd-client`: ^4.0.0 → ^5.0.0 - `libdd-telemetry`: ^6.0.0 → ^7.0.0 - `libdd-trace-stats`: ^6.0.0 → ^7.0.0 - `libdd-trace-utils`: ^9.0.0 → ^10.1.0 ### Commits - fix(trace-exporter): mark OTLP client-computed stats (#2245) - feat(data-pipeline): adds new attributes to OTLP trace metrics export (#2316) - feat: add v1 isolated bricks for trace exporter (#2280) - feat(data-pipeline)!: add flush_and_close to the trace buffer (#2313) - feat(telemetry)!: Add Installation signature and AppProduct changes payloads (#2213) - refactor: migrate to workspace dependencies, phase 4 (#2296) - feat(data-pipeline)!: add compression option for agentless export (#2205) - fix(telemetry)!: stop sending Connection: close to the Agent (#2286) - feat(stats)!: per-field cardinality limits (#2211) - refactor: migrate to workspace dependencies, phase 3 (#2283) - feat(stats)!: string field length limit (#2267) - refactor(trace-utils)!: split generic TracerHeaderTags (#2279) - chore: moving to workspace-level dependencies, phase 2 (#2270) - feat(trace_exporter)!: add `restart_after_fork` configuration option [APMSP-3843] (#2263) - feat(data-pipeline): emit additional_metric_tags on OTLP span duration metric (#2264) - refactor: consolidate core dependencies at workspace level (phase 1) (#2253) - feat(dogstatsd)!: add shared_runtime buffered sink (#2224) - feat(stats)!: implement additional metric tags in libdd-trace-stats (#2170) - refactor(statsd)!: make client clonable (#2222) - feat(telemetry)!: make telemetry worker wasm-compatible for the TraceExporter (#2172) - feat(trace_exporter)!: remove deprecated methods (#2250) - feat(trace-utils)!: add v1 decoder (#2174) - feat(trace-utils)!: add encoder v1 to v04 + refactor (#2145) - feat(otlp): add instrumentation scope metadata (#2235) - fix(trace-utils)!: serialize v0.5 span links and events into meta (#980) - fix!: css obfuscation logic (#2203) - test(data-pipeline): gate telemetry-only tests behind the telemetry feature (#2204) [APMSP-3843]: https://datadoghq.atlassian.net/browse/APMSP-3843?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: hoolioh <107922352+hoolioh@users.noreply.github.com>
What does this PR do?
Adds a V1 → v0.4 msgpack downgrade encoder so a tracer that emits V1 spans natively can still send traces to an agent that only advertises
/v0.4/traces.Also renames the existing encoder modules/functions so the convention is uniform: parent module = output wire format, file/function suffix = input span type. The four combinations now coexist:
v04/span_v04.rs— v0.4 → v0.4 (native)v04/span_v1.rs— v1 → v0.4 (downgrade, new)v1/span_v04.rs— v0.4 → V1 (upgrade)v1/span_v1.rs— v1 → V1 (native)Motivation
APMSP-2811