feat(obfuscation)!: make json transforms caller-provided

[webern]

Human Summary

While working on DataDog/saluki#2601, I discovered that I couldn’t use libdatadog’s JSON obfuscator as written. Its transformer was a plain function pointer whose only argument was the JSON value. Unlike a closure, a function pointer cannot capture runtime state, so there was no way to pass Saluki’s dynamically loaded SQL obfuscation configuration without putting it in static or global state. Existing users can configure ordinary SQL obfuscation, but the existing JSON transformer API provides no normal way to pass those SQL settings when transforming SQL embedded in JSON. The provided transformer uses default settings.

In fixing this, I decided to take it a little further. Transform failures are now returned as structured information, scan failures use a typed error, and the hot-path API accepts caller-owned output and scratch buffers so repeated calls can reuse their allocations. Transformations return Cow<str>, which also avoids forcing an allocation when they can return borrowed text.

The coding agent brought in thiserror as a dependency. This is idiomatic and seems like a good call. The scanner's new const fns are clippy::missing_const_for_fn, which the workspace enables; they have no runtime effect. Scanner::new becoming restart is for buffer reuse. The parse-state stack now survives between passes.

What does this PR do?

This separates JSON obfuscation policy from transformation behavior. JsonObfuscatorConfig now contains only serializable, comparable data. Callers provide transforms per call through generic closures that can capture runtime SQL configuration.

The allocating API is joined by obfuscate_into, which reuses caller-owned output and scratch buffers. Scratch capacity is observable and explicitly trimmable so callers can choose their memory-retention policy.

JSON scan errors and transform errors are reported separately. SQL obfuscation now reports an empty result as SqlObfuscationError::EmptyResult, and the crate exports the Agent's exact SQL failure replacement. This also pins non-ASCII identifier behavior and fixes the JSON unquoting fallback to retain the literal when unquoting fails.

Motivation

The stored fn(&str) -> String transformer cannot capture runtime configuration or report errors. This prevents Agent-compatible configured SQL transformation inside JSON values.

DataDog/saluki#2601 currently duplicates libdatadog's JSON scanner to work around that API. A call-scoped closure lets Saluki use the shared scanner while retaining its configured SQL behavior, logging, and failure policy.

Additional Notes

Transform callbacks return Cow<str>, allowing borrowed, static, or owned replacements. The callback is generic and allocation-free; no transformer trait or stored trait object is introduced.

A transform error replaces that value with "?", records the error, and continues. Callers that need the Agent's SQL fallback can observe and log the SQL error inside the callback, then return SQL_OBFUSCATION_FAILURE_REPLACEMENT.

JsonObfuscationScratch retains capacity from the largest prior input until the caller trims or drops it. retained_capacity and trim_to make that policy explicit.

Rebase onto #2490

#2490 moved SqlObfuscateConfig, SqlObfuscationMode and DbmsKind from sql into obfuscation_config and renamed the config struct to SqlConfig, and it began deserializing JsonObfuscator directly from the Agent's /info payload. Resolved as follows:

• Adopted obfuscation_config::{DbmsKind, SqlConfig, SqlObfuscationMode} everywhere, including the obfuscate_with doc example. No type is reintroduced in sql.
• JsonObfuscatorConfig keeps #[serde(default)] but not deny_unknown_fields: feat(data-pipeline)!: refactor agent's /info obfuscation config format #2490 dropped it deliberately, and an /info payload from a newer Agent carries fields this struct has no counterpart for. A test pins that forward compatibility.
• feat(data-pipeline)!: refactor agent's /info obfuscation config format #2490's hand-written PartialEq for JsonObfuscatorConfig, which existed only to skirt the uncomparable transformer field, is replaced by a derive now that the field is gone. The config also derives Eq.
• obfuscate_resource_for_stats and obfuscate_pb_span now consume the Result from obfuscate_sql: a resource that obfuscates to nothing is left as sent rather than blanked.

Agent /info field names

Checked against the Agent rather than guessed. pkg/trace/api/info.go serves a reduced view:

type reducedJSONObfuscationConfig struct {
	Enabled  bool     `json:"enabled"`
	KeepKeys []string `json:"keep_keys"`
}
...
oconf.Elasticsearch = reducedJSONObfuscationConfig{Enabled: o.ES.Enabled, KeepKeys: o.ES.KeepValues}

So /info sends keep_keys, which is already this crate's field name, and it does not report the transform set at all. The Agent's own obfuscation config (pkg/obfuscate, apm_config.obfuscation.*) calls the same two sets keep_values and obfuscate_sql_values. Rather than rename the Rust fields — transform_keys is no longer SQL-specific here, so obfuscate_sql_values would be a lie — both Agent spellings are accepted as #[serde(alias = ...)], in the same style as #2490's PascalCase aliases. obfuscation_config::tests::test_agent_json_obfuscation_field_names pins all of it.

BREAKING CHANGE: JsonObfuscatorConfig::transformer and JsonStringTransformer are removed. JSON transforms move to JsonObfuscator::obfuscate_with or JsonObfuscator::obfuscate_into. SQL obfuscation functions now return Result.

How to test the change?

The following pass on the rebased branch:

cargo +stable clippy -p libdd-trace-obfuscation --all-targets -- -D warnings
cargo +nightly-2026-07-26 fmt --all -- --check
cargo nextest run -p libdd-trace-obfuscation           # 387 passed
cargo test -p libdd-trace-obfuscation --doc
cargo check -p libdd-trace-stats -p libdd-data-pipeline-core \
  -p libdd-data-pipeline -p libdd-data-pipeline-ffi --all-targets
cargo nextest run -p libdd-data-pipeline -p libdd-trace-stats \
  -E '!test(tracing_integration_tests::)'              # 253 passed

The tracing_integration_tests:: suite needs Docker and was not run locally. Cargo.lock gains only thiserror, which LICENSE-3rdparty.csv already covers, so no regeneration was needed. cargo deny check still reports pre-existing workspace advisory and license-policy failures unrelated to this diff.

References

• Closes: Obfuscation: an empty SQL obfuscation result is reported as success where the Agent fails, and non-ASCII identifier support is untested #2541
• Related: feat(data-pipeline)!: refactor agent's /info obfuscation config format #2490
• Related: fix(otlp): obfuscate malformed json queries and keep key order saluki#2601

[datadog-datadog-prod-us1]

✅ All CI checks and tests passed.

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

🎯 Code Coverage (details)
• Patch Coverage: 98.85%
• Overall Coverage: 78.61% (+0.31%)

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: c60e9e5 | Docs | View more details | Give us feedback!}

[dd-octo-sts]

Artifact Size Benchmark Report

aarch64-alpine-linux-musl

Artifact	Baseline	Commit	Change
/aarch64-alpine-linux-musl/lib/libdatadog_profiling.so	9.02 MB	9.02 MB	0% (0 B) 👌
/aarch64-alpine-linux-musl/lib/libdatadog_profiling.a	95.97 MB	95.99 MB	+.02% (+24.14 KB) 🔍

aarch64-unknown-linux-gnu

Artifact	Baseline	Commit	Change
/aarch64-unknown-linux-gnu/lib/libdatadog_profiling.so	12.19 MB	12.19 MB	+0% (+184 B) 👌
/aarch64-unknown-linux-gnu/lib/libdatadog_profiling.a	107.35 MB	107.37 MB	+.01% (+21.71 KB) 🔍

libdatadog-x64-windows

Artifact	Baseline	Commit	Change
/libdatadog-x64-windows/debug/dynamic/datadog_profiling_ffi.dll	29.07 MB	29.08 MB	+.03% (+9.00 KB) 🔍
/libdatadog-x64-windows/debug/dynamic/datadog_profiling_ffi.lib	96.08 KB	96.08 KB	0% (0 B) 👌
/libdatadog-x64-windows/debug/dynamic/datadog_profiling_ffi.pdb	191.49 MB	191.55 MB	+.03% (+64.00 KB) 🔍
/libdatadog-x64-windows/debug/static/datadog_profiling_ffi.lib	815.92 MB	817.32 MB	+.17% (+1.40 MB) 🔍
/libdatadog-x64-windows/release/dynamic/datadog_profiling_ffi.dll	9.71 MB	9.72 MB	+.11% (+11.00 KB) 🔍
/libdatadog-x64-windows/release/dynamic/datadog_profiling_ffi.lib	96.08 KB	96.08 KB	0% (0 B) 👌
/libdatadog-x64-windows/release/dynamic/datadog_profiling_ffi.pdb	27.50 MB	27.51 MB	+.05% (+16.00 KB) 🔍
/libdatadog-x64-windows/release/static/datadog_profiling_ffi.lib	55.60 MB	55.62 MB	+.02% (+13.81 KB) 🔍

libdatadog-x86-windows

Artifact	Baseline	Commit	Change
/libdatadog-x86-windows/debug/dynamic/datadog_profiling_ffi.dll	25.42 MB	25.43 MB	+.03% (+8.00 KB) 🔍
/libdatadog-x86-windows/debug/dynamic/datadog_profiling_ffi.lib	97.58 KB	97.58 KB	0% (0 B) 👌
/libdatadog-x86-windows/debug/dynamic/datadog_profiling_ffi.pdb	196.70 MB	196.75 MB	+.02% (+48.00 KB) 🔍
/libdatadog-x86-windows/debug/static/datadog_profiling_ffi.lib	803.55 MB	802.87 MB	--.08% (-699.04 KB) 💪
/libdatadog-x86-windows/release/dynamic/datadog_profiling_ffi.dll	7.52 MB	7.55 MB	+.39% (+30.50 KB) 🔍
/libdatadog-x86-windows/release/dynamic/datadog_profiling_ffi.lib	97.58 KB	97.58 KB	0% (0 B) 👌
/libdatadog-x86-windows/release/dynamic/datadog_profiling_ffi.pdb	29.61 MB	29.62 MB	+.05% (+16.00 KB) 🔍
/libdatadog-x86-windows/release/static/datadog_profiling_ffi.lib	52.55 MB	52.56 MB	+.02% (+14.82 KB) 🔍

x86_64-alpine-linux-musl

Artifact	Baseline	Commit	Change
/x86_64-alpine-linux-musl/lib/libdatadog_profiling.a	85.96 MB	85.98 MB	+.02% (+25.81 KB) 🔍
/x86_64-alpine-linux-musl/lib/libdatadog_profiling.so	10.05 MB	10.05 MB	+.03% (+4.00 KB) 🔍

x86_64-unknown-linux-gnu

Artifact	Baseline	Commit	Change
/x86_64-unknown-linux-gnu/lib/libdatadog_profiling.a	101.83 MB	101.85 MB	+.02% (+23.14 KB) 🔍
/x86_64-unknown-linux-gnu/lib/libdatadog_profiling.so	12.27 MB	12.28 MB	+.03% (+4.20 KB) 🔍

[pr-commenter]

Benchmarks

Comparison

Benchmark execution time: 2026-09-24 12:30:10

Comparing candidate commit c60e9e5 in PR branch m/json-obfuscation-api with baseline commit a5e7164 in branch main.

📊 Benchmarking dashboard

Found 0 performance improvements and 1 performance regressions! Performance is the same for 176 metrics, 0 unstable metrics.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

• 🟩 = significantly better candidate vs. baseline
• 🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'

scenario:vec_map/as_deduped_map/needs_dedup_1_in_4/128

• 🟥 execution_time [+631.589ns; +636.434ns] or [+10.333%; +10.413%]

Benchmark execution time: 2026-09-24 12:23:55

Comparing candidate commit c60e9e5 in PR branch m/json-obfuscation-api with baseline commit a5e7164 in branch main.

📊 Benchmarking dashboard

Found 22 performance improvements and 6 performance regressions! Performance is the same for 137 metrics, 11 unstable metrics.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

• 🟩 = significantly better candidate vs. baseline
• 🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'

scenario:credit_card/is_card_number/ 3782-8224-6310-005

• 🟩 execution_time [-4.027µs; -3.792µs] or [-5.002%; -4.710%]
• 🟩 throughput [+615529.253op/s; +655013.709op/s] or [+4.955%; +5.273%]

scenario:credit_card/is_card_number/ 378282246310005

• 🟩 execution_time [-5.443µs; -5.351µs] or [-7.362%; -7.237%]
• 🟩 throughput [+1056274.670op/s; +1073344.772op/s] or [+7.809%; +7.936%]

scenario:credit_card/is_card_number/378282246310005

• 🟩 execution_time [-5.870µs; -5.802µs] or [-8.287%; -8.191%]
• 🟩 throughput [+1260532.004op/s; +1274728.188op/s] or [+8.928%; +9.029%]

scenario:credit_card/is_card_number/37828224631000521389798

• 🟩 execution_time [-7.292µs; -7.261µs] or [-13.734%; -13.675%]
• 🟩 throughput [+2983857.672op/s; +2997897.623op/s] or [+15.844%; +15.918%]

scenario:credit_card/is_card_number_no_luhn/ 378282246310005

• 🟩 execution_time [-4.207µs; -4.165µs] or [-7.197%; -7.125%]
• 🟩 throughput [+1313188.844op/s; +1326175.760op/s] or [+7.676%; +7.751%]

scenario:credit_card/is_card_number_no_luhn/378282246310005

• 🟩 execution_time [-5.043µs; -4.989µs] or [-9.077%; -8.980%]
• 🟩 throughput [+1776686.439op/s; +1796257.248op/s] or [+9.870%; +9.979%]

scenario:credit_card/is_card_number_no_luhn/37828224631000521389798

• 🟩 execution_time [-7.307µs; -7.277µs] or [-13.759%; -13.703%]
• 🟩 throughput [+2990485.817op/s; +3004018.377op/s] or [+15.881%; +15.953%]

scenario:thread_cpu/alloc_free/system/16

• 🟥 execution_time [+3.507ns; +3.541ns] or [+24.490%; +24.723%]

scenario:thread_cpu/profiler_attached/fast_path_noop/16

• 🟥 execution_time [+2.851ns; +2.856ns] or [+19.701%; +19.742%]

scenario:thread_cpu/profiler_attached/fast_path_noop/256

• 🟥 execution_time [+2.850ns; +2.858ns] or [+19.698%; +19.752%]

scenario:thread_cpu/profiler_attached/fast_path_noop/4096

• 🟥 execution_time [+2.846ns; +2.853ns] or [+19.670%; +19.720%]

scenario:thread_cpu/profiler_attached/fast_path_noop/64

• 🟥 execution_time [+2.850ns; +2.857ns] or [+19.703%; +19.748%]

scenario:thread_cpu/profiler_attached/fast_path_noop/65536

• 🟥 execution_time [+2.854ns; +2.863ns] or [+19.727%; +19.787%]

scenario:thread_cpu/profiler_attached/fast_path_system/16

• 🟩 execution_time [-16.251ns; -16.194ns] or [-36.420%; -36.290%]

scenario:thread_cpu/profiler_attached/fast_path_system/256

• 🟩 execution_time [-15.978ns; -15.915ns] or [-36.011%; -35.869%]

scenario:thread_cpu/profiler_attached/fast_path_system/4096

• 🟩 execution_time [-8.441ns; -8.293ns] or [-7.378%; -7.248%]

scenario:thread_cpu/profiler_attached/fast_path_system/64

• 🟩 execution_time [-15.940ns; -15.873ns] or [-35.943%; -35.792%]

scenario:thread_cpu/profiler_attached/fast_path_system/65536

• 🟩 execution_time [-9.492ns; -9.369ns] or [-8.499%; -8.389%]

scenario:thread_cpu/profiler_attached/slow_path_system/4096

• 🟩 execution_time [-9.090ns; -8.989ns] or [-5.894%; -5.829%]

scenario:trace_buffer/8_senders/no_delay

• 🟩 execution_time [-591.856µs; -399.660µs] or [-9.630%; -6.503%]
• 🟩 throughput [+90343.467op/s; +138512.274op/s] or [+7.683%; +11.779%]

Unstable benchmarks

These benchmarks have a confidence interval too wide to call a change; treat them as noise rather than signal.

scenario:datadog_sample_span/parent_not_sampled_short_circuit/allocated_bytes

• unstable execution_time [-0.000ns; +0.000ns] or [+555.735%; -555.735%]

scenario:datadog_sample_span/parent_sampled_short_circuit/allocated_bytes

• unstable execution_time [-0.000ns; +0.000ns] or [+556.312%; -556.007%]

scenario:flagevaluation_evp/coalescer/typical/100flags_50users_10fields

• unstable execution_time [-6.890µs; +13.229µs] or [-3.724%; +7.151%]

scenario:glob_matcher/ascii_case_insensitive_match/allocated_bytes

• unstable execution_time [-0.000ns; +0.000ns] or [+554.044%; -554.940%]

scenario:glob_matcher/ascii_exact_match/allocated_bytes

• unstable execution_time [-0.000ns; +0.000ns] or [+555.735%; -555.735%]

scenario:glob_matcher/ascii_exact_miss/allocated_bytes

• unstable execution_time [-0.000ns; +0.000ns] or [+555.251%; -555.507%]

scenario:glob_matcher/ascii_wildcard_backtrack_match/allocated_bytes

• unstable execution_time [-0.000ns; +0.000ns] or [+557.291%; -556.468%]

scenario:glob_matcher/ascii_wildcard_heavy_backtrack/allocated_bytes

• unstable execution_time [-0.000ns; +0.000ns] or [+557.442%; -556.539%]

scenario:glob_matcher/ascii_wildcard_question_match/allocated_bytes

• unstable execution_time [-0.000ns; +0.000ns] or [+558.051%; -556.826%]

scenario:glob_matcher/ascii_wildcard_star_match/allocated_bytes

• unstable execution_time [-0.000ns; +0.000ns] or [+556.503%; -556.096%]

scenario:glob_matcher/star_short_circuit/allocated_bytes

• unstable execution_time [-0.000ns; +0.000ns] or [+556.409%; -556.052%]

Candidate

Omitted due to size.

Baseline

Omitted due to size.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c25301ab49

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[Eldolfin]

Looks good 👍🏽

One thing I saw repeated multiple times in this PR is references of types and files from the agent repo which I think we should avoid because they will get stale and don't add much value IMO