Skip to content

fix(otlp): obfuscate malformed json queries and keep key order - #2601

Merged
gh-worker-dd-mergequeue-cf854d[bot] merged 3 commits into
mainfrom
m/otlp-jsonsc
Sep 25, 2026
Merged

gh-worker-dd-mergequeue-cf854d[bot] merged 3 commits into
mainfrom
m/otlp-jsonsc

Conversation

@webern

@webern webern commented Sep 14, 2026 •

Copy link
Copy Markdown
Contributor

Human Summary

Malformed JSON should not bypass obfuscation. We now use the libdatadog provided algorithm which fixes that problem.

AI Summary

This replaces the serde_json::Value rewrite with the JSON obfuscator from libdatadog, a single-pass scan that copies the characters it keeps. Malformed input is obfuscated up to the error and ends in ... instead of being returned with raw values. Valid input keeps its key order and text; whitespace between tokens is dropped, matching the Agent.

SQL obfuscation of a value is passed to the scan as a call-scoped callback, so the SQL configuration this transform resolved reaches it, and a value whose SQL obfuscation fails carries the Agent's failure message rather than ?. A result that comes back empty is treated as a failure too, as the Agent treats it. The scan's obfuscate_into entry point takes caller-owned output and scratch buffers, which are held on the obfuscator and reused across calls; capacity past a retention limit is released after an oversized query so it does not stay held.

Two divergences from the Agent remain. A SQL value with JSON-only escapes (\/, surrogate pairs) is unescaped before obfuscation rather than leaked; a test pins this. A bare top-level value after a complete document is kept rather than obfuscated (DataDog/libdatadog#2577); the Agent also keeps it after an object document. Tests for both cases assert the obfuscated output and are #[ignore]d until that is fixed.

Bumps the pinned libdd-trace-obfuscation revision to pick up the caller-provided JSON transforms.

Change Type

  • Bug fix

How did you test this PR?

Unit tests cover key order, whitespace, nesting, malformed and truncated input, kept subtrees, SQL transformation, escaped keys, multibyte values, multiple documents, and buffer retention after an oversized query. A property test runs arbitrary strings through the obfuscator.

The following checks pass:

  • make fmt
  • cargo check --workspace
  • cargo check --workspace --tests
  • make check-clippy
  • make check-docs
  • make check-deny
  • make check-licenses
  • make check-release-notes
  • make check-unused-deps
  • cargo nextest run -p saluki-components trace_obfuscation (73 tests)

References

@dd-octo-sts dd-octo-sts Bot added area/components Sources, transforms, and destinations. transform/trace-obfuscation Trace Obfuscation synchronous transform. labels Sep 14, 2026
@pr-commenter

pr-commenter Bot commented Sep 14, 2026 •

Copy link
Copy Markdown

Binary Size Analysis (Agent Data Plane)

Baseline: c2dbd79 · Comparison: 0cd5c42 · diff
Analysis Configuration: stripped binaries · Pass/Fail Threshold: +5%
Sizes: 39.42 MiB (baseline) vs 39.53 MiB (comparison)
Size Change: +107.70 KiB (+0.27%)

✅ Binary size difference within threshold

Changes by Module
Module File Size Symbols
core +65.41 KiB 9123
saluki_metrics::mapped::MappedMetric<H> -24.53 KiB 6
saluki_components::destinations::dogstatsd_client_telemetry +23.86 KiB 31
tracing +23.64 KiB 18
saluki_common::resource_tracking::groups -18.71 KiB 20
&mut serde_json -18.41 KiB 95
anon.7e1f65c5eae431c3bb47f731de3df8a7.5.llvm.1234521462112249683 +17.42 KiB 1
anon.7c0ddbe702a5f1897c8c14ad70afcf0e.5.llvm.15030214190652917543 -17.32 KiB 1
libdd_trace_obfuscation +15.02 KiB 19
anon.7e1f65c5eae431c3bb47f731de3df8a7.827.llvm.1234521462112249683 +14.98 KiB 1
anon.7c0ddbe702a5f1897c8c14ad70afcf0e.826.llvm.15030214190652917543 -14.89 KiB 1
otlp_protos::otlp_include::opentelemetry +14.27 KiB 224
anon.7e1f65c5eae431c3bb47f731de3df8a7.825.llvm.1234521462112249683 +13.19 KiB 1
anon.7c0ddbe702a5f1897c8c14ad70afcf0e.824.llvm.15030214190652917543 -13.10 KiB 1
saluki_components::common::datadog +12.28 KiB 501
anon.7c0ddbe702a5f1897c8c14ad70afcf0e.768.llvm.15030214190652917543 -11.49 KiB 1
anon.7e1f65c5eae431c3bb47f731de3df8a7.768.llvm.1234521462112249683 +11.49 KiB 1
saluki_common::cache::CacheBuilder<K,V,W,H> -10.33 KiB 6
prost -10.14 KiB 371
anyhow +9.63 KiB 1074
Detailed Symbol Changes
    FILE SIZE        VM SIZE    
 --------------  -------------- 
  [NEW] +59.5Ki  [NEW] +59.4Ki    saluki_components::common::datadog::io::run_endpoint_io_loop::_{{closure}}::hc401faece882ff90
  +0.4% +53.6Ki  +0.4% +46.5Ki    [29047 Others]
  [NEW] +41.8Ki  [NEW] +41.6Ki    agent_data_plane::cli::run::create_topology::_{{closure}}::h8c94f9968ac23cfd
  [NEW] +41.0Ki  [NEW] +40.9Ki    agent_data_plane::cli::run::handle_run_command::_{{closure}}::h51ae24b73c304ea5
  [NEW] +36.9Ki  [NEW] +36.7Ki    _<saluki_components::forwarders::otlp::OtlpForwarder as saluki_core::components::forwarders::Forwarder>::run::_{{closure}}::he2b5413bbae8ffb4
  [NEW] +31.4Ki  [NEW] +31.2Ki    _<saluki_components::transforms::apm_stats::ApmStats as saluki_core::components::transforms::Transform>::run::_{{closure}}::h4060600910d15de9
  [NEW] +31.3Ki  [NEW] +31.2Ki    agent_data_plane::cli::dogstatsd::run_dogstatsd_command::_{{closure}}::he2168c139b7b48bb
  [NEW] +28.6Ki  [NEW] +28.5Ki    agent_data_plane::dogstatsd_contexts::artifact::for_each_record::h65f4c2779143ed8f
  [NEW] +28.1Ki  [NEW] +27.9Ki    core::ptr::drop_in_place<agent_data_plane::cli::run::handle_run_command::{{closure}}>::hcc9d31b9a61cf7ad
  [NEW] +27.4Ki  [NEW] +27.3Ki    saluki_components::destinations::dogstatsd_client_telemetry::DogStatsDClientTelemetry::record_metric::hc8842a35883e2427
  [NEW] +27.1Ki  [NEW] +26.9Ki    datadog_agent_commons::ipc::client::RemoteAgentClient::connect::_{{closure}}::_{{closure}}::_{{closure}}::hf545857594c46a0c
  [NEW] +26.3Ki  [NEW] +26.1Ki    _<saluki_components::transforms::aggregate::Aggregate as saluki_core::components::transforms::Transform>::run::_{{closure}}::haa421813d5bf4b63
  [DEL] -25.7Ki  [DEL] -25.6Ki    core::ops::function::FnOnce::call_once::hf3344873cea9239f
  [DEL] -27.1Ki  [DEL] -26.9Ki    datadog_agent_commons::ipc::client::RemoteAgentClient::connect::_{{closure}}::_{{closure}}::_{{closure}}::he58de1d27b5c33fc
  [DEL] -28.1Ki  [DEL] -27.9Ki    core::ptr::drop_in_place<agent_data_plane::cli::run::handle_run_command::{{closure}}>::h37442738d8cc1b2d
  [DEL] -28.6Ki  [DEL] -28.5Ki    agent_data_plane::dogstatsd_contexts::artifact::for_each_record::hcd26a7f63a879933
  [DEL] -31.3Ki  [DEL] -31.2Ki    agent_data_plane::cli::dogstatsd::run_dogstatsd_command::_{{closure}}::h7da2c38d2aceef53
  [DEL] -41.0Ki  [DEL] -40.9Ki    agent_data_plane::cli::run::handle_run_command::_{{closure}}::h34f7751c298ec1b0
  [DEL] -41.3Ki  [DEL] -41.1Ki    _<saluki_components::forwarders::otlp::OtlpForwarder as saluki_core::components::forwarders::Forwarder>::run::_{{closure}}::hb436f99d8cac66e9
  [DEL] -41.8Ki  [DEL] -41.7Ki    agent_data_plane::cli::run::create_topology::_{{closure}}::h1c87fd4199d3024b
  [DEL] -60.5Ki  [DEL] -60.3Ki    saluki_components::common::datadog::io::run_endpoint_io_loop::_{{closure}}::ha1cf1b72abf45024
  +0.3%  +107Ki  +0.3%  +100Ki    TOTAL

@pr-commenter

pr-commenter Bot commented Sep 14, 2026 •

Copy link
Copy Markdown

Regression Detector (Agent Data Plane)

Run ID: ca1b16a2-4153-4632-8bd0-7d6ff84c4c84
Baseline: c2dbd79e · Comparison: 0cd5c42b · diff

Optimization Goals: ✅ No significant changes detected

Fine details of change detection per experiment (5)

Experiments configured erratic: true are tagged (ignored) and skipped when determining which experiments regressed or improved. Experiments which are detected as erratic at runtime are tagged (erratic) to flag that the run's sample dispersion was high, but their regression / improvement signal still counts.

experiment goal Δ mean % links
quality_gates_rss_dsd_low memory ⚪ +0.62 metrics profiles logs
quality_gates_rss_idle memory ⚪ +0.42 metrics profiles logs
quality_gates_rss_dsd_medium memory ⚪ +0.19 metrics profiles logs
quality_gates_rss_dsd_ultraheavy memory ⚪ -0.01 metrics profiles logs
quality_gates_rss_dsd_heavy memory ⚪ -0.26 metrics profiles logs
Bounds Checks: ✅ Passed (5)
experiment check replicates observed links
quality_gates_rss_dsd_heavy memory_usage 10/10 ✅ 229 MiB ≤ 250 MiB metrics profiles logs
quality_gates_rss_dsd_low memory_usage 10/10 ✅ 52.3 MiB ≤ 60 MiB metrics profiles logs
quality_gates_rss_dsd_medium memory_usage 10/10 ✅ 92.7 MiB ≤ 100 MiB metrics profiles logs
quality_gates_rss_dsd_ultraheavy memory_usage 10/10 ✅ 388 MiB ≤ 420 MiB metrics profiles logs
quality_gates_rss_idle memory_usage 10/10 ✅ 34.2 MiB ≤ 40 MiB metrics profiles logs
Explanation

A change is flagged as a regression when |Δ mean %| > 5.00% in the regressing direction for its optimization goal AND SMP marks the experiment as a regression (is_regression: true). Improvements use the matching criteria for the improving direction. Experiments configured erratic: true (tagged (ignored)) are skipped outright; experiments detected as erratic at runtime (tagged (erratic)) still count, since that flag describes sample dispersion rather than directional certainty. The Δ mean % cell is colored accordingly: 🟢 = improvement, 🔴 = regression, ⚪ = neutral. Reduction in CPU or memory is an improvement; reduction in ingress throughput is a regression. Experiments tagged (no analysis) show ⚠️ n/a: SMP ran them but produced no analysis, usually because a replicate failed and exhausted its retries. Check the SMP report for that experiment's replicate failures.

@webern webern left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI

Self-review found three SQL-obfuscation edge cases. They are pre-existing or intentional differences and do not appear to block this scanner change. Details are inline.

Comment thread lib/saluki-components/src/transforms/trace_obfuscation/json.rs Outdated
Comment thread lib/saluki-components/src/transforms/trace_obfuscation/json.rs Outdated
Comment thread lib/saluki-components/src/transforms/trace_obfuscation/json.rs Outdated
gh-worker-dd-mergequeue-cf854d Bot pushed a commit to DataDog/libdatadog that referenced this pull request Sep 24, 2026
## Human Summary

While working on DataDog/saluki#2601, I discovered that I couldn’t use libdatadog’s JSON obfuscator as written. Its transformer was a plain function pointer whose only argument was the JSON value. Unlike a closure, a function pointer cannot capture runtime state, so there was no way to pass Saluki’s dynamically loaded SQL obfuscation configuration without putting it in static or global state. Existing users can configure ordinary SQL obfuscation, but the existing JSON transformer API provides no normal way to pass those SQL settings when transforming SQL embedded in JSON. The provided transformer uses default settings.

In fixing this, I decided to take it a little further. Transform failures are now returned as structured information, scan failures use a typed error, and the hot-path API accepts caller-owned output and scratch buffers so repeated calls can reuse their allocations. Transformations return `Cow<str>`, which also avoids forcing an allocation when they can return borrowed text.

The coding agent brought in `thiserror` as a dependency. This is idiomatic and seems like a good call. The scanner's new const fns are `clippy::missing_const_for_fn`, which the workspace enables; they have no runtime effect. `Scanner::new` becoming restart is for buffer reuse. The parse-state stack now survives between passes.

# What does this PR do?

This separates JSON obfuscation policy from transformation behavior. `JsonObfuscatorConfig` now contains only serializable, comparable data. Callers provide transforms per call through generic closures that can capture runtime SQL configuration.

The allocating API is joined by `obfuscate_into`, which reuses caller-owned output and scratch buffers. Scratch capacity is observable and explicitly trimmable so callers can choose their memory-retention policy.

JSON scan errors and transform errors are reported separately. SQL obfuscation now reports an empty result as `SqlObfuscationError::EmptyResult`, and the crate exports the Agent's exact SQL failure replacement. This also pins non-ASCII identifier behavior and fixes the JSON unquoting fallback to retain the literal when unquoting fails.

# Motivation

The stored `fn(&str) -> String` transformer cannot capture runtime configuration or report errors. This prevents Agent-compatible configured SQL transformation inside JSON values.

DataDog/saluki#2601 currently duplicates libdatadog's JSON scanner to work around that API. A call-scoped closure lets Saluki use the shared scanner while retaining its configured SQL behavior, logging, and failure policy.

# Additional Notes

Transform callbacks return `Cow<str>`, allowing borrowed, static, or owned replacements. The callback is generic and allocation-free; no transformer trait or stored trait object is introduced.

A transform error replaces that value with `"?"`, records the error, and continues. Callers that need the Agent's SQL fallback can observe and log the SQL error inside the callback, then return `SQL_OBFUSCATION_FAILURE_REPLACEMENT`.

`JsonObfuscationScratch` retains capacity from the largest prior input until the caller trims or drops it. `retained_capacity` and `trim_to` make that policy explicit.

## Rebase onto #2490

#2490 moved `SqlObfuscateConfig`, `SqlObfuscationMode` and `DbmsKind` from `sql` into `obfuscation_config` and renamed the config struct to `SqlConfig`, and it began deserializing `JsonObfuscator` directly from the Agent's `/info` payload. Resolved as follows:

- Adopted `obfuscation_config::{DbmsKind, SqlConfig, SqlObfuscationMode}` everywhere, including the `obfuscate_with` doc example. No type is reintroduced in `sql`.
- `JsonObfuscatorConfig` keeps `#[serde(default)]` but not `deny_unknown_fields`: #2490 dropped it deliberately, and an `/info` payload from a newer Agent carries fields this struct has no counterpart for. A test pins that forward compatibility.
- #2490's hand-written `PartialEq for JsonObfuscatorConfig`, which existed only to skirt the uncomparable `transformer` field, is replaced by a derive now that the field is gone. The config also derives `Eq`.
- `obfuscate_resource_for_stats` and `obfuscate_pb_span` now consume the `Result` from `obfuscate_sql`: a resource that obfuscates to nothing is left as sent rather than blanked.

## Agent `/info` field names

Checked against the Agent rather than guessed. `pkg/trace/api/info.go` serves a *reduced* view:

```go
type reducedJSONObfuscationConfig struct {
	Enabled  bool     `json:"enabled"`
	KeepKeys []string `json:"keep_keys"`
}
...
oconf.Elasticsearch = reducedJSONObfuscationConfig{Enabled: o.ES.Enabled, KeepKeys: o.ES.KeepValues}
```

So `/info` sends `keep_keys`, which is already this crate's field name, and it does not report the transform set at all. The Agent's own obfuscation config (`pkg/obfuscate`, `apm_config.obfuscation.*`) calls the same two sets `keep_values` and `obfuscate_sql_values`. Rather than rename the Rust fields — `transform_keys` is no longer SQL-specific here, so `obfuscate_sql_values` would be a lie — both Agent spellings are accepted as `#[serde(alias = ...)]`, in the same style as #2490's PascalCase aliases. `obfuscation_config::tests::test_agent_json_obfuscation_field_names` pins all of it.

BREAKING CHANGE: `JsonObfuscatorConfig::transformer` and `JsonStringTransformer` are removed. JSON transforms move to `JsonObfuscator::obfuscate_with` or `JsonObfuscator::obfuscate_into`. SQL obfuscation functions now return `Result`.

# How to test the change?

The following pass on the rebased branch:

```bash
cargo +stable clippy -p libdd-trace-obfuscation --all-targets -- -D warnings
cargo +nightly-2026-07-26 fmt --all -- --check
cargo nextest run -p libdd-trace-obfuscation           # 387 passed
cargo test -p libdd-trace-obfuscation --doc
cargo check -p libdd-trace-stats -p libdd-data-pipeline-core \
  -p libdd-data-pipeline -p libdd-data-pipeline-ffi --all-targets
cargo nextest run -p libdd-data-pipeline -p libdd-trace-stats \
  -E '!test(tracing_integration_tests::)'              # 253 passed
```

The `tracing_integration_tests::` suite needs Docker and was not run locally. `Cargo.lock` gains only `thiserror`, which `LICENSE-3rdparty.csv` already covers, so no regeneration was needed. `cargo deny check` still reports pre-existing workspace advisory and license-policy failures unrelated to this diff.

# References

- Closes: #2541
- Related: #2490
- Related: DataDog/saluki#2601


Co-authored-by: matt.briggs <matt.briggs@datadoghq.com>
@webern
webern force-pushed the m/otlp-jsonsc branch 2 times, most recently from c50aabe to aee6e9e Compare September 25, 2026 08:47
@webern
webern marked this pull request as ready for review September 25, 2026 09:32
@webern
webern requested a review from a team as a code owner September 25, 2026 09:32

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: bbc63bab30

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread lib/saluki-components/src/transforms/trace_obfuscation/json.rs Outdated

@datadog-prod-us1-4 datadog-prod-us1-4 Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bits Code Review: PASS

More details

Malformed JSON is scanned without falling back to the original payload, while valid documents retain key order and use the resolved SQL obfuscation configuration.

Was this helpful? React 👍 or 👎

Open Bits AI session

🤖 Bits Code Review · Commit bbc63ba · @DataDog review to ask questions

Comment thread lib/saluki-components/src/transforms/trace_obfuscation/json.rs Outdated
Comment thread lib/saluki-components/src/transforms/trace_obfuscation/json.rs Outdated
JSON obfuscation parsed the whole query into a `serde_json::Value`, rewrote it,
and re-serialized. A query that was not valid JSON failed to parse and was
returned as sent, so raw values reached the backend, and a valid one came back
with its keys alphabetized, which changes the resource string that stats are
aggregated on.

Obfuscate through the JSON obfuscator in libdatadog instead, which scans in a
single pass and copies the characters it keeps. Keys keep their order and their
text, and a query that stops parsing is obfuscated up to that point and ends in
`...`. Whitespace between tokens is dropped, as the reference implementation
does.

The scan's `obfuscate_into` entry point takes caller-owned output and scratch
buffers, which are held on the obfuscator and reused across calls. SQL
obfuscation of a value is passed as a call-scoped callback, so the SQL
configuration the transform resolved reaches it, and a value whose SQL
obfuscation fails carries the reference implementation's failure message rather
than `?`. A result that comes back empty is treated as a failure too, as the
reference implementation treats it.

The reusable buffers keep the capacity they grow to, so an oversized query would
otherwise stay held for the obfuscator's lifetime. Capacity past a retention
limit is released before the obfuscator is used again; a query within the limit
keeps its reuse.

Bumps libdd-trace-obfuscation to a revision with the caller-provided JSON
transforms.

Refs: #2405
Comment thread lib/saluki-components/src/transforms/trace_obfuscation/json.rs Outdated
@webern
webern requested a review from tobz September 25, 2026 14:19
@gh-worker-dd-mergequeue-cf854d
gh-worker-dd-mergequeue-cf854d Bot merged commit a6ffea2 into main Sep 25, 2026
110 checks passed
@gh-worker-dd-mergequeue-cf854d
gh-worker-dd-mergequeue-cf854d Bot deleted the m/otlp-jsonsc branch September 25, 2026 17:27
dd-octo-sts Bot pushed a commit that referenced this pull request Sep 25, 2026
## Human Summary

Malformed JSON should not bypass obfuscation. We now use the libdatadog provided algorithm which fixes that problem.

## AI Summary

This replaces the `serde_json::Value` rewrite with the JSON obfuscator from libdatadog, a single-pass scan that copies the characters it keeps. Malformed input is obfuscated up to the error and ends in `...` instead of being returned with raw values. Valid input keeps its key order and text; whitespace between tokens is dropped, matching the Agent.

SQL obfuscation of a value is passed to the scan as a call-scoped callback, so the SQL configuration this transform resolved reaches it, and a value whose SQL obfuscation fails carries the Agent's failure message rather than `?`. A result that comes back empty is treated as a failure too, as the Agent treats it. The scan's `obfuscate_into` entry point takes caller-owned output and scratch buffers, which are held on the obfuscator and reused across calls; capacity past a retention limit is released after an oversized query so it does not stay held.

Two divergences from the Agent remain. A SQL value with JSON-only escapes (`\/`, surrogate pairs) is unescaped before obfuscation rather than leaked; a test pins this. A bare top-level value after a complete document is kept rather than obfuscated (DataDog/libdatadog#2577); the Agent also keeps it after an object document. Tests for both cases assert the obfuscated output and are `#[ignore]`d until that is fixed.

Bumps the pinned libdd-trace-obfuscation revision to pick up the caller-provided JSON transforms.

## Change Type
- [x] Bug fix

## How did you test this PR?

Unit tests cover key order, whitespace, nesting, malformed and truncated input, kept subtrees, SQL transformation, escaped keys, multibyte values, multiple documents, and buffer retention after an oversized query. A property test runs arbitrary strings through the obfuscator.

The following checks pass:

- `make fmt`
- `cargo check --workspace`
- `cargo check --workspace --tests`
- `make check-clippy`
- `make check-docs`
- `make check-deny`
- `make check-licenses`
- `make check-release-notes`
- `make check-unused-deps`
- `cargo nextest run -p saluki-components trace_obfuscation` (73 tests)

## References

- Closes: #2405
- Progresses: #2438

Co-authored-by: matt.briggs <matt.briggs@datadoghq.com> a6ffea2
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/components Sources, transforms, and destinations. mergequeue-status: done transform/trace-obfuscation Trace Obfuscation synchronous transform.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Traces(otlp+dd): Obfuscate malformed JSON and preserve key order and formatting

2 participants