feat(crashtracking): send debug log when no data is received at all - #2321
Conversation
📚 Documentation Check Results📦
|
🔒 Cargo Deny Results📦
|
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a8e6a53de3
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
BenchmarksComparisonBenchmark execution time: 2026-08-05 19:19:32 Comparing candidate commit 6dc1fe2 in PR branch Found 0 performance improvements and 0 performance regressions! Performance is the same for 1 metrics, 0 unstable metrics.
|
| cpu_model | git_commit_sha | git_commit_date | git_branch |
|---|---|---|---|
| Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz | 6dc1fe2 | 1785956872 | gyuheon0h/ct-no-data-debug |
| scenario | metric | min | mean ± sd | median ± mad | p75 | p95 | p99 | max | peak_to_median_ratio | skewness | kurtosis | cv | sem | runs | sample_size |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| receiver_entry_point/report/2644 | execution_time | 3.586ms | 3.621ms ± 0.017ms | 3.618ms ± 0.010ms | 3.631ms | 3.650ms | 3.671ms | 3.692ms | 2.04% | 1.142 | 1.947 | 0.47% | 0.001ms | 1 | 200 |
| scenario | metric | 95% CI mean | Shapiro-Wilk pvalue | Ljung-Box pvalue (lag=1) | Dip test pvalue |
|---|---|---|---|---|---|
| receiver_entry_point/report/2644 | execution_time | [3.619ms; 3.623ms] or [-0.065%; +0.065%] | None | None | None |
Baseline
Baseline benchmark details
Group 1
| cpu_model | git_commit_sha | git_commit_date | git_branch |
|---|---|---|---|
| Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz | 9ac89b9 | 1785954810 | main |
| scenario | metric | min | mean ± sd | median ± mad | p75 | p95 | p99 | max | peak_to_median_ratio | skewness | kurtosis | cv | sem | runs | sample_size |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| receiver_entry_point/report/2644 | execution_time | 3.492ms | 3.561ms ± 0.060ms | 3.528ms ± 0.020ms | 3.635ms | 3.663ms | 3.680ms | 3.691ms | 4.62% | 0.763 | -1.122 | 1.68% | 0.004ms | 1 | 200 |
| scenario | metric | 95% CI mean | Shapiro-Wilk pvalue | Ljung-Box pvalue (lag=1) | Dip test pvalue |
|---|---|---|---|---|---|
| receiver_entry_point/report/2644 | execution_time | [3.553ms; 3.569ms] or [-0.234%; +0.234%] | None | None | None |
🎉 All green!🧪 All tests passed 🎯 Code Coverage (details) 🔗 Commit SHA: 6dc1fe2 | Docs | Datadog PR Page | Give us feedback! |
Artifact Size Benchmark Reportaarch64-alpine-linux-musl
aarch64-unknown-linux-gnu
libdatadog-x64-windows
libdatadog-x86-windows
x86_64-alpine-linux-musl
x86_64-unknown-linux-gnu
|
2238baa to
ba607a3
Compare
|
@codex r |
|
To use Codex here, create a Codex account and connect to github. |
|
@codex r |
|
To use Codex here, create a Codex account and connect to github. |
ba607a3 to
f9468bc
Compare
9439a82 to
0c64e83
Compare
0c64e83 to
6dc1fe2
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 6dc1fe271a
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| std::env::set_var( | ||
| "DD_TRACE_AGENT_URL", | ||
| format!("http://{}", listener.local_addr().unwrap()), | ||
| ); |
There was a problem hiding this comment.
Restore the agent URL after the no-data test
When these tests are run by cargo test/libtest or any harness that does not isolate environment per test, this set_var changes a process-global endpoint and never restores it. Other receiver tests now create DebugLogger::new(None, None) and can emit via the env-derived endpoint on timeout/incomplete paths, so they may race with this mock listener or inherit its closed address after this test, causing order-dependent telemetry failures; please guard and restore the variable or serialize env users.
AGENTS.md reference: AGENTS.md:L70-L75
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
nextest isolates env per test
| debug_logger | ||
| .emit_and_wait( | ||
| ReceiverIssue::NoData, | ||
| &builder.uuid.to_string(), | ||
| "Receiver received no data".to_string(), | ||
| LogLevel::Warn, | ||
| ) | ||
| .await; |
There was a problem hiding this comment.
Avoid blocking the long-lived receiver on empty connections
For a long-lived socket receiver (async_receiver_entry_point_unix_socket(..., one_shot=false)), the accept loop handles one connection at a time, so an empty connect/close now awaits this best-effort telemetry upload before accepting another report. If the agent endpoint is slow or unreachable, each empty connection can stall the receiver for the telemetry timeout and delay subsequent crash reports; the no-data signal needs to be spawned or otherwise bounded for that mode.
AGENTS.md reference: AGENTS.md:L70-L75
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
There can only be one report from one process? Or is this referring to sidecar process listening to many processes??
gleocadie
left a comment
There was a problem hiding this comment.
LGTM
leave the AI comments and Rust thingy to experts :)
…mote_config/agentless_fetcher * 'main' of github.com:DataDog/libdatadog: feat(data-pipeline)!: add flush_and_close to the trace buffer (#2313) refactor(rc)!: make conversion from RemoteConfigProduct back and forth generally available (#2325) chore: release v40.0.0 (#2326) feat(datadog-ffe): server-side EVP flagevaluation payload + bincode-safe sidecar delivery (#2117) feat(data-pipeline): set structured span values (#2300) feat(telemetry)!: Add Installation signature and AppProduct changes payloads (#2213) fix(libdd-trace-utils): apply SpanLink flags masking when v0.5 json encoding (#2314) feat(data-pipeline): encode structured values (#2304) feat(crashtracking): send debug log when no data is received at all (#2321) chore: release v39.0.0 (#2324) feat(data-pipeline): add span links FFI (#2305) fix(crashtracking): check fields and exclude uuid for `has_data` (#2322) feat(data-pipeline): add span events FFI (#2301)
# Release proposal for libdd-crashtracker, libdd-gotter and their dependencies This PR contains version bumps based on public API changes and commits since last release. ## libdd-gotter **Next version:** `1.0.0` **Semver bump:** `major` **Tag:** `libdd-gotter-v1.0.0` **Warning:** this is an initial release. Please verify that the version and commits included are correct. ## libdd-crashtracker **Next version:** `2.0.0` **Semver bump:** `major` **Tag:** `libdd-crashtracker-v2.0.0` ###⚠️ major bump forced due to: - `libdd-common`: ^1.0.0 → ^5.2.0 - `libdd-telemetry`: ^1.0.0 → ^7.0.0 ### Commits - feat(crashtracking)!: retrieve c assert message for linux when `__assert_fail` is dynamically loaded (#2268) - test: skip/shorten slow miri jobs (#2331) - feat(crashtracking): send debug log when no data is received at all (#2321) - fix(crashtracking): check fields and exclude uuid for `has_data` (#2322) - refactor: migrate to workspace dependencies, phase 4 (#2296) - refactor: migrate to workspace dependencies, phase 3 (#2283) - chore: moving to workspace-level dependencies, phase 2 (#2270) - refactor: consolidate core dependencies at workspace level (phase 1) (#2253) - test: stabilize flaky tests (#2256) - feat(telemetry)!: make telemetry worker wasm-compatible for the TraceExporter (#2172) - chore(crashtracking): harden multi thread ptrace collection (#2216) - fix(sidecar): don't double-encode file:// telemetry endpoints (#2230) - fix: new clippy lints (#2219) - refactor(sidecar)!: Avoid a dedicated socket for crashtracker (#2179) - refactor(clippy): prefer core and alloc imports (#2196) - refactor(libdd-telemetry)!: avoid leaking libdd-common types in the public API (#2152) - feat(crashtracking): collect all native stacks for unhandled exception and also crashing thread (#2155) - fix(crashtracking): sanitize type and message for unhandled exceptions (#2148) - fix(crashtracking): multi thread collection centos flakes harden (#2113) - chore(crashtracking)!: remove frame count experimental field (#2114) - fix(crashtracking): authenticate peer granted socket ptrace access (#2098) - feat(crashtracking)!: add experimental frame count field (#2099) - fix(crashtracker): increase test_waitall_nohang timeout to 500ms (#2097) - fix(crashtracker): support socket based receiver for all thread collection (#2080) - fix(crashtracker): set failed thread stack collection as incomplete empty stack (#2079) - fix(crashtracking)!: flatten all threads object into a list of `ThreadData` (#2054) - refactor(crashtracker): update imports to linux only (#2036) - fix(crashtracker): move preload logger marking after recursive guard (#2023) - fix(libdd-telemetry): restore previous Cargo.toml version (#1993) - feat!: integrate obfuscation to the stats exporter [APMSP-2764] (#1819) - chore(crashtracking)!: create errorsintake crash ping directly from telemetry (#1963) - feat(telemetry)!: include dependencies and integrations in app-extended-heartbeat (#1962) - feat(crashtracking)!: collect all threads (#1878) - chore(crashtracking): bump libdatadog-libunwind to v1.0.2 (#1942) - chore(crashtracking): default errors intake crash report upload to be on (#1902) - chore(libdd-crashtracker): remove path reference for libdd-libunwind-sys (#1877) - fix(crashtracking): handle new lines in client submitted exception message (#1836) - fix(crashtracker): Fix bin_tests in gitlab (#1832) - feat(crashtracking)!: improve parity between errors intake payload and telemetry intake payload (#1823) - feat(crashtracking)!: emit ucontext registers as structured data (#1787) - fix(ci): use single threaded for all tests that mutate signal state and use non-fatal signal (#1812) - fix(crashtracking): fix SIGCHLD signal guarding while in CT signal handler (#1807) - fix(ci): use single threaded to avoid race conditions for sa guard tests (#1800) - fix(crashtracking): guard sigchld and sigpipe during crashtracker signal handler execution (#1771) - chore(crashtracking): preserve errno for crashtracker (#1767) - chore(crashtracker): use weaker mem ordering for OP_COUNTERS (#1744) - chore(crashtracking): rename target triple to runtime platform (#1747) - chore(ci): give libdd-libunwind-sys its own version (#1743) - chore(crashtracking): add tag for target triple (#1741) - refactor(libdd-crashtracker)!: avoid leaking Endpoint through the public API (#1705) - chore(cargo): bump to 29.0.0 (#1702) - fix(crashtracking): use libunwind to unwind frames (#1663) - chore(deps): bump blazesym to 0.2.3 and blazesym-c to 0.1.7 (#1654) - chore(ci): fix crashtracker receiver binary rpath setting (#1652) - chore(crashtracking): emit a best effort stacktrace for Mac (#1645) - chore(crashtracker): use default-features=false for aws-lc-sys (#1625) - feat(crashtracking): report unhandled exceptions (#1596) - refactor(ddcommon)!: remove direct dependency on hyper client everywhere in common (#1604) - feat(crashtracking): include `Kind` in crash ping and clarify requirements (#1595) - fix(crashtracking): add process_tags to application field (#1576) - ci: update nightly in CI to 2026-02-08 (#1539) - fix(telemetry)!: fix logs payload format [APMSP-2590] (#1498) - chore(crashtracking): bump os_info crate to 3.14 (#1507) - Don't bail (#1494) - feat(crashtracking)!: emit crashing thread name in crash report for linux crashes (#1485) - test(crashtracking): add minimal LD preload test for crashtracker collector (#1428) - chore: release libddcommon-v1.1.0 (#1456) - chore: prepare libdd-telemetry-v2.0.0 (#1457) - [crashtracker] Retrieve panic message when crashing (#1361) - fix(sidecar): AWS lambda also can return EACCESS for shm_open (#1446) - chore(crashtracking): add `is_crash_debug` tag to crashtracker receiver debug logs (#1445) - [crashtracker] Log errors in crashtracker receiver (#1395) - chore: add changelog for every published crate (#1396) - Fix CI (#1389) - [crashtracker] Set OS info in the crash info builder when receiving report (#1388) - crashtracker: support cxx bindings for crashinfo (#1379) [APMSP-2764]: https://datadoghq.atlassian.net/browse/APMSP-2764?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: hoolioh <107922352+hoolioh@users.noreply.github.com>

What does this PR do?
Creates a
DebugLoggerfor use by the receiver. This debug logger has the ability to send telemetry logs always, whetherCrashtrackerConfig,Metadatais received or not. As new data arrives to the receiver, the debug logger self builds itself. This allows us to send debug logs from the receiver even if no data has been received by the receiver yet.Motivation
We have cases where customers crash and the receiver spawns, but there is not even a crash ping.
We should log this. Previously, we waited until
MetadataandCrashtrackerConfigwas received by the receiver before being able to send anything. However, theTelemetryCrashUploadertechnically needs nothing to be able to send logs -- it has fallback paths that derive the endpoint through environment variables. We should take advantage of thisAdditional Notes
Anything else we should know when reviewing?
How to test the change?
Describe here in detail how the change can be validated.