Skip to content

feat(apm): add APM relay that proxies to the trace-agent - #2706

Draft
ajgajg1134 wants to merge 2 commits into
andrew.glaude/apm-proxy-configfrom
andrew.glaude/v1-apm-relay
Draft

ajgajg1134 wants to merge 2 commits into
andrew.glaude/apm-proxy-configfrom
andrew.glaude/v1-apm-relay

Conversation

@ajgajg1134

@ajgajg1134 ajgajg1134 commented Sep 28, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Stacked on #2707 (config). Adds ApmRelay, which sits in front of the trace-agent when the experimental APM proxy is enabled:

  • It listens on TCP and/or a Unix domain socket, where the trace-agent normally would. The wiring PR resolves those addresses from apm_config.receiver_port / receiver_socket / apm_non_local_traffic / bind_host.
  • POST /v1.0/traces is handled by ADP: dispatched as Payload::Http on the traces output with headers preserved verbatim, for the ETP decoder that comes later in the stack. 413 over max_payload_size, 400 on other body read errors, empty 200 OK for now. The real response contract (rate_by_service, load shedding) is a follow-up.
  • Everything else, including other methods on /v1.0/traces, is reverse-proxied to proxy_destination (http://host:port or unix:///path):
    • request and response bodies are streamed, not buffered
    • hop-by-hop headers are dropped in both directions
    • no proxy-side request timeout, since the trace-agent's routes each enforce their own (profiling and the EVP proxy run long)
    • requests always go upstream as HTTP/1.1, which the trace-agent speaks, even when the tracer used HTTP/2
    • 502 when the trace-agent can't be reached

The proxy lives in its own relays/apm/proxy.rs.

Known gap: for tracers that send no container headers, the trace-agent derives the container ID from the UDS peer credentials. Behind the proxy, those credentials are ADP's. A TODO in proxy.rs marks where ADP will inject Datadog-Container-ID in a follow-up.

The relay is not wired into the topology yet, so there's no user-visible change.

Change Type

  • Bug fix
  • New feature
  • Non-functional (chore, refactoring, docs)
  • Performance

How did you test this PR?

  • 21 unit tests covering:
    • routing: POST /v1.0/traces is dispatched; other paths and methods are proxied
    • the v1 path's 413, the in-limit dispatch, and the closed-channel 503
    • TCP and UDS listener binding
    • config validation, including invalid proxy destinations (bad schemes, relative socket paths, and URLs with a path, query, fragment or credentials)
    • against a stand-in trace-agent: request and response pass through (method, path, query, headers, body, status), hop-by-hop header stripping, HTTP/2 requests forwarded as HTTP/1.1, forwarding over a UDS destination, 502 when the trace-agent is unreachable, and IPv6 destination parsing
  • make check-all: clean

References

Part of #2438: #2437 (TCP/UDS listeners; named pipe deferred), #2439 (payload size bound), #2450 (routes; everything but v1 served by the trace-agent via the proxy).

🤖 Generated with Claude Code

@ajgajg1134 ajgajg1134 added the changelog/no-changelog No changelog entry needed label Sep 28, 2026
@dd-octo-sts dd-octo-sts Bot added the area/components Sources, transforms, and destinations. label Sep 28, 2026
@pr-commenter

pr-commenter Bot commented Sep 28, 2026 •

Copy link
Copy Markdown

Binary Size Analysis (Agent Data Plane)

Baseline: 60cea0c · Comparison: 36c082c · diff
Analysis Configuration: stripped binaries · Pass/Fail Threshold: +5%
Sizes: 39.50 MiB (baseline) vs 39.55 MiB (comparison)
Size Change: +52.51 KiB (+0.13%)

✅ Binary size difference within threshold

Changes by Module
Module File Size Symbols
saluki_common::task::instrument -26.91 KiB 7
saluki_metrics::mapped::MappedMetric<H> +24.50 KiB 4
saluki_common::resource_tracking::groups +23.41 KiB 7
saluki_components::destinations::dogstatsd_client_telemetry -22.99 KiB 3
anon.a6926c6c21417749435f71dba1b992d8.989.llvm.108194269033826922 +20.80 KiB 1
anon.a6926c6c21417749435f71dba1b992d8.976.llvm.1534094455157536209 -20.54 KiB 1
core +20.51 KiB 1664
anon.d6f5d974d073ad08c76103565ac8e826.11.llvm.2053145391855355513 -17.32 KiB 1
anon.d6f5d974d073ad08c76103565ac8e826.11.llvm.16669733176756370272 +17.24 KiB 1
anon.d6f5d974d073ad08c76103565ac8e826.767.llvm.2053145391855355513 -15.16 KiB 1
anon.d6f5d974d073ad08c76103565ac8e826.767.llvm.16669733176756370272 +15.08 KiB 1
anon.d6f5d974d073ad08c76103565ac8e826.765.llvm.16669733176756370272 +13.28 KiB 1
anon.d6f5d974d073ad08c76103565ac8e826.765.llvm.2053145391855355513 -13.28 KiB 1
tonic_prost -12.05 KiB 11
anon.d6f5d974d073ad08c76103565ac8e826.733.llvm.16669733176756370272 +11.48 KiB 1
anon.d6f5d974d073ad08c76103565ac8e826.733.llvm.2053145391855355513 -11.39 KiB 1
saluki_core::topology::interconnect -11.03 KiB 15
serde_core +10.71 KiB 54
quick_cache +10.13 KiB 49
saluki_components::encoders::datadog -9.68 KiB 48
Detailed Symbol Changes
    FILE SIZE        VM SIZE    
 --------------  -------------- 
  +1.5%  +127Ki  +1.4% +90.2Ki    [12479 Others]
  [NEW] +20.8Ki  [NEW] +20.7Ki    anon.a6926c6c21417749435f71dba1b992d8.989.llvm.108194269033826922
  [NEW] +20.3Ki  [NEW] +20.2Ki    saluki_components::sources::dogstatsd::DogStatsDConfiguration::build_listeners::_{{closure}}::had89af222da99c3d
  [NEW] +17.2Ki  [NEW]     +76    anon.d6f5d974d073ad08c76103565ac8e826.11.llvm.16669733176756370272
  [NEW] +15.1Ki  [NEW]     +81    anon.d6f5d974d073ad08c76103565ac8e826.767.llvm.16669733176756370272
  [NEW] +13.3Ki  [NEW]     +82    anon.d6f5d974d073ad08c76103565ac8e826.765.llvm.16669733176756370272
  [NEW] +11.5Ki  [NEW]     +81    anon.d6f5d974d073ad08c76103565ac8e826.733.llvm.16669733176756370272
 +20e2% +10.7Ki +34e2% +10.7Ki    _<saluki_components::transforms::trace_obfuscation::TraceObfuscation as saluki_core::components::transforms::SynchronousTransform>::transform_buffer::h06aa80c6ed3c8340
  [NEW] +9.01Ki  [NEW]     +81    anon.d6f5d974d073ad08c76103565ac8e826.0.llvm.16669733176756370272
  [DEL] -9.18Ki  [DEL]     -81    anon.d6f5d974d073ad08c76103565ac8e826.0.llvm.2053145391855355513
 -95.0% -9.50Ki -96.6% -9.50Ki    _<saluki_common::task::instrument::InstrumentedTask<F> as core::future::future::Future>::poll::hec924a579853a2bc
 -65.3% -11.3Ki -65.8% -11.3Ki    saluki_components::transforms::trace_obfuscation::sql::obfuscate_sql_string::h630be4eeb9ca738c
  [DEL] -11.4Ki  [DEL]     -81    anon.d6f5d974d073ad08c76103565ac8e826.733.llvm.2053145391855355513
  [DEL] -13.3Ki  [DEL]     -82    anon.d6f5d974d073ad08c76103565ac8e826.765.llvm.2053145391855355513
  [DEL] -15.2Ki  [DEL]     -81    anon.d6f5d974d073ad08c76103565ac8e826.767.llvm.2053145391855355513
  [DEL] -17.3Ki  [DEL]     -76    anon.d6f5d974d073ad08c76103565ac8e826.11.llvm.2053145391855355513
  [DEL] -18.0Ki  [DEL] -17.9Ki    saluki_components::transforms::trace_obfuscation::TraceObfuscation::obfuscate_span::h1d751aeeb1398414
 -77.6% -19.6Ki -78.2% -19.6Ki    _<saluki_components::sources::dogstatsd::DogStatsDConfiguration as saluki_core::components::sources::builder::SourceBuilder>::build::_{{closure}}::h8163573818b7579d
  [DEL] -19.8Ki  [DEL] -19.7Ki    saluki_components::transforms::apm_stats::ApmStats::process_trace::h9892f30ebdf92aca
  [DEL] -20.5Ki  [DEL] -20.5Ki    anon.a6926c6c21417749435f71dba1b992d8.976.llvm.1534094455157536209
  [DEL] -27.3Ki  [DEL] -27.1Ki    saluki_components::destinations::dogstatsd_client_telemetry::DogStatsDClientTelemetry::record_metric::h8b0dc008208f4f49
  +0.1% +52.5Ki  +0.0% +16.1Ki    TOTAL

@pr-commenter

pr-commenter Bot commented Sep 28, 2026 •

Copy link
Copy Markdown

Regression Detector (Agent Data Plane)

Run ID: 04cef9d8-520c-4f6c-8493-39cb425b2c56
Baseline: 60cea0c5 · Comparison: 36c082cf · diff

Optimization Goals: ✅ No significant changes detected

Fine details of change detection per experiment (5)

Experiments configured erratic: true are tagged (ignored) and skipped when determining which experiments regressed or improved. Experiments which are detected as erratic at runtime are tagged (erratic) to flag that the run's sample dispersion was high, but their regression / improvement signal still counts.

experiment goal Δ mean % links
quality_gates_rss_dsd_ultraheavy memory ⚪ +0.45 metrics profiles logs
quality_gates_rss_idle memory ⚪ +0.12 metrics profiles logs
quality_gates_rss_dsd_medium memory ⚪ -0.27 metrics profiles logs
quality_gates_rss_dsd_low memory ⚪ -0.38 metrics profiles logs
quality_gates_rss_dsd_heavy memory ⚪ -0.53 metrics profiles logs
Bounds Checks: ✅ Passed (5)
experiment check replicates observed links
quality_gates_rss_dsd_heavy memory_usage 10/10 ✅ 231 MiB ≤ 250 MiB metrics profiles logs
quality_gates_rss_dsd_low memory_usage 10/10 ✅ 51.8 MiB ≤ 60 MiB metrics profiles logs
quality_gates_rss_dsd_medium memory_usage 10/10 ✅ 91.7 MiB ≤ 100 MiB metrics profiles logs
quality_gates_rss_dsd_ultraheavy memory_usage 10/10 ✅ 390 MiB ≤ 420 MiB metrics profiles logs
quality_gates_rss_idle memory_usage 10/10 ✅ 34.2 MiB ≤ 40 MiB metrics profiles logs
Explanation

A change is flagged as a regression when |Δ mean %| > 5.00% in the regressing direction for its optimization goal AND SMP marks the experiment as a regression (is_regression: true). Improvements use the matching criteria for the improving direction. Experiments configured erratic: true (tagged (ignored)) are skipped outright; experiments detected as erratic at runtime (tagged (erratic)) still count, since that flag describes sample dispersion rather than directional certainty. The Δ mean % cell is colored accordingly: 🟢 = improvement, 🔴 = regression, ⚪ = neutral. Reduction in CPU or memory is an improvement; reduction in ingress throughput is a regression. Experiments tagged (no analysis) show ⚠️ n/a: SMP ran them but produced no analysis, usually because a replicate failed and exhausted its retries. Check the SMP report for that experiment's replicate failures.

@ajgajg1134 ajgajg1134 changed the title feat(apm): add APM relay for v1.0 trace ingress feat(apm): add APM relay that proxies to the trace-agent Sep 28, 2026
@ajgajg1134
ajgajg1134 force-pushed the andrew.glaude/v1-apm-relay branch from 62fad0c to b251256 Compare September 28, 2026 20:52
@ajgajg1134
ajgajg1134 changed the base branch from main to andrew.glaude/apm-proxy-config September 28, 2026 20:52
Introduce `ApmRelayConfiguration` / `ApmRelay`, which sits in front of the
trace-agent: it listens on TCP and/or a Unix domain socket where the
trace-agent normally would, handles Datadog v1.0 (ETP) trace payloads
itself, and proxies every other request to the relocated trace-agent.

- `POST /v1.0/traces` is dispatched as `Payload::Http` on the `traces`
  output, with headers preserved verbatim so tracer metadata reaches the
  decoder. Bodies over `max_payload_size` get 413; other body read errors
  get 400. The response is an empty `200 OK` for now.
- Every other path, and every other method on `/v1.0/traces`, is forwarded
  to `proxy_destination` (`http://host:port` or `unix:///path`). Requests
  and responses are streamed rather than buffered, hop-by-hop headers are
  dropped in both directions, and the proxy imposes no request timeout of
  its own since each trace-agent route enforces its own. An unreachable
  trace-agent is answered with 502.

Proxying hides the tracer's Unix domain socket peer credentials from the
trace-agent, which it uses for origin detection when a tracer sends no
container headers. A TODO marks where ADP will inject the container ID.

The relay takes plain values and states no defaults of its own. It is not
wired into the topology yet.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@ajgajg1134
ajgajg1134 force-pushed the andrew.glaude/v1-apm-relay branch from b251256 to 55dce3d Compare September 28, 2026 21:14
`PathBuf` in the proxy and `tempdir` in the relay tests are only used on Unix, so Windows builds failed on unused
imports under `#![deny(warnings)]`. The UDS listen-address test is also Unix-only, since `/tmp/apm.sock` is not an
absolute path on Windows.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/components Sources, transforms, and destinations. changelog/no-changelog No changelog entry needed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant