chore(runtime): restart all workers on invalid state by VianneyRuhlmann · Pull Request #2262 · DataDog/libdatadog

VianneyRuhlmann · 2026-07-22T13:52:35Z

Fixes ForkSafeRuntime::after_fork_parent/after_fork_child so a single worker stuck in InvalidState no longer aborts the whole restart loop after a fork.

datadog-datadog-us1-prod · 2026-07-22T14:07:29Z

Tests

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

🎯 Code Coverage (details)
• Patch Coverage: 96.30%
• Overall Coverage: 74.65% (+0.03%)

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: 52b09bc | Docs | Datadog PR Page | Give us feedback!}

pr-commenter · 2026-07-22T14:10:05Z

Benchmarks

Comparison

Benchmark execution time: 2026-07-22 14:05:47

Comparing candidate commit 52b09bc in PR branch vianney/failsafe-after-fork with baseline commit 3081603 in branch main.

Found 0 performance improvements and 2 performance regressions! Performance is the same for 23 metrics, 0 unstable metrics.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

🟩 = significantly better candidate vs. baseline
🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'

scenario:trace_buffer/4_senders/no_delay

🟥 execution_time [+98.912µs; +132.141µs] or [+4.362%; +5.827%]
🟥 throughput [-88114.974op/s; -65781.884op/s] or [-5.543%; -4.138%]

Candidate

Candidate benchmark details

Group 1

cpu_model	git_commit_sha	git_commit_date	git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	`52b09bc`	1784728211	vianney/failsafe-after-fork

scenario	metric	min	mean ± sd	median ± mad	p75	p95	p99	max	peak_to_median_ratio	skewness	kurtosis	cv	sem	runs	sample_size
concentrator/add_spans_to_concentrator	execution_time	8.551ms	8.577ms ± 0.018ms	8.572ms ± 0.013ms	8.589ms	8.610ms	8.626ms	8.628ms	0.66%	0.741	-0.171	0.21%	0.001ms	1	200

scenario	metric	95% CI mean	Shapiro-Wilk pvalue	Ljung-Box pvalue (lag=1)	Dip test pvalue
concentrator/add_spans_to_concentrator	execution_time	[8.574ms; 8.579ms] or [-0.029%; +0.029%]	None	None	None

Group 2

cpu_model	git_commit_sha	git_commit_date	git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	`52b09bc`	1784728211	vianney/failsafe-after-fork

scenario	metric	min	mean ± sd	median ± mad	p75	p95	p99	max	peak_to_median_ratio	skewness	kurtosis	cv	sem	runs	sample_size
trace_buffer/1_senders/10us_delay	execution_time	56.346ms	56.591ms ± 0.145ms	56.561ms ± 0.079ms	56.650ms	56.858ms	57.057ms	57.191ms	1.11%	1.138	1.792	0.25%	0.010ms	1	200
trace_buffer/1_senders/10us_delay	throughput	15736.652op/s	15903.766op/s ± 40.501op/s	15911.918op/s ± 22.161op/s	15930.814op/s	15956.932op/s	15970.999op/s	15972.739op/s	0.38%	-1.119	1.728	0.25%	2.864op/s	1	200
trace_buffer/1_senders/1us_delay	execution_time	50.024ms	50.091ms ± 0.089ms	50.079ms ± 0.014ms	50.094ms	50.157ms	50.199ms	51.247ms	2.33%	11.167	142.096	0.18%	0.006ms	1	200
trace_buffer/1_senders/1us_delay	throughput	17561.953op/s	17967.487op/s ± 31.256op/s	17971.599op/s ± 5.098op/s	17975.494op/s	17984.686op/s	17989.994op/s	17991.232op/s	0.11%	-11.064	140.251	0.17%	2.210op/s	1	200
trace_buffer/1_senders/no_delay	execution_time	337.987µs	343.996µs ± 6.496µs	343.038µs ± 2.320µs	345.661µs	348.766µs	354.329µs	420.261µs	22.51%	8.412	93.975	1.88%	0.459µs	1	200
trace_buffer/1_senders/no_delay	throughput	2141526.991op/s	2617116.456op/s ± 43140.167op/s	2623616.055op/s ± 17626.674op/s	2637814.092op/s	2653655.354op/s	2659627.941op/s	2662828.148op/s	1.49%	-7.084	73.051	1.64%	3050.470op/s	1	200
trace_buffer/2_senders/10us_delay	execution_time	56.979ms	57.310ms ± 0.156ms	57.317ms ± 0.099ms	57.410ms	57.559ms	57.686ms	57.760ms	0.77%	0.077	-0.313	0.27%	0.011ms	1	200
trace_buffer/2_senders/10us_delay	throughput	31163.540op/s	31408.485op/s ± 85.539op/s	31404.046op/s ± 54.120op/s	31460.984op/s	31557.527op/s	31580.283op/s	31590.804op/s	0.59%	-0.063	-0.323	0.27%	6.049op/s	1	200
trace_buffer/2_senders/1us_delay	execution_time	50.063ms	50.125ms ± 0.137ms	50.111ms ± 0.018ms	50.126ms	50.185ms	50.251ms	51.992ms	3.75%	12.715	170.369	0.27%	0.010ms	1	200
trace_buffer/2_senders/1us_delay	throughput	34620.624op/s	35910.797op/s ± 95.013op/s	35920.588op/s ± 12.584op/s	35934.709op/s	35947.104op/s	35954.515op/s	35954.663op/s	0.09%	-12.626	168.732	0.26%	6.718op/s	1	200
trace_buffer/2_senders/no_delay	execution_time	838.230µs	889.887µs ± 20.812µs	887.976µs ± 12.717µs	902.232µs	924.416µs	938.829µs	1025.159µs	15.45%	1.472	7.845	2.33%	1.472µs	1	200
trace_buffer/2_senders/no_delay	throughput	1755826.005op/s	2023796.488op/s ± 46011.681op/s	2027082.418op/s ± 29284.898op/s	2054461.881op/s	2087524.213op/s	2118668.760op/s	2147382.523op/s	5.93%	-1.011	4.849	2.27%	3253.517op/s	1	200
trace_buffer/4_senders/10us_delay	execution_time	57.131ms	57.480ms ± 0.122ms	57.483ms ± 0.079ms	57.555ms	57.669ms	57.757ms	58.064ms	1.01%	0.469	1.995	0.21%	0.009ms	1	200
trace_buffer/4_senders/10us_delay	throughput	62000.184op/s	62630.996op/s ± 132.776op/s	62627.755op/s ± 85.751op/s	62720.689op/s	62831.836op/s	62917.552op/s	63013.006op/s	0.62%	-0.445	1.918	0.21%	9.389op/s	1	200
trace_buffer/4_senders/1us_delay	execution_time	50.115ms	50.179ms ± 0.038ms	50.169ms ± 0.017ms	50.195ms	50.252ms	50.313ms	50.327ms	0.31%	1.393	2.382	0.07%	0.003ms	1	200
trace_buffer/4_senders/1us_delay	throughput	71532.474op/s	71742.522op/s ± 53.786op/s	71757.640op/s ± 24.516op/s	71777.074op/s	71806.094op/s	71821.127op/s	71834.238op/s	0.11%	-1.388	2.361	0.07%	3.803op/s	1	200
trace_buffer/4_senders/no_delay	execution_time	2.134ms	2.383ms ± 0.088ms	2.386ms ± 0.063ms	2.448ms	2.520ms	2.555ms	2.584ms	8.33%	-0.210	-0.343	3.68%	0.006ms	1	200
trace_buffer/4_senders/no_delay	throughput	1392984.678op/s	1512687.175op/s ± 56416.245op/s	1509088.938op/s ± 41072.126op/s	1550404.068op/s	1608373.196op/s	1653927.203op/s	1686784.797op/s	11.78%	0.394	-0.132	3.72%	3989.231op/s	1	200
trace_buffer/8_senders/10us_delay	execution_time	57.572ms	57.917ms ± 0.234ms	57.830ms ± 0.113ms	58.051ms	58.391ms	58.502ms	58.568ms	1.28%	0.988	-0.162	0.40%	0.017ms	1	200
trace_buffer/8_senders/10us_delay	throughput	122934.842op/s	124318.802op/s ± 500.410op/s	124503.152op/s ± 242.162op/s	124673.856op/s	124843.593op/s	124923.227op/s	125061.426op/s	0.45%	-0.978	-0.184	0.40%	35.384op/s	1	200
trace_buffer/8_senders/1us_delay	execution_time	50.210ms	50.305ms ± 0.067ms	50.283ms ± 0.034ms	50.344ms	50.428ms	50.544ms	50.641ms	0.71%	1.635	3.959	0.13%	0.005ms	1	200
trace_buffer/8_senders/1us_delay	throughput	142176.945op/s	143128.536op/s ± 189.659op/s	143188.751op/s ± 97.259op/s	143262.914op/s	143338.952op/s	143364.505op/s	143396.904op/s	0.15%	-1.622	3.886	0.13%	13.411op/s	1	200
trace_buffer/8_senders/no_delay	execution_time	6.031ms	6.586ms ± 0.122ms	6.609ms ± 0.080ms	6.680ms	6.747ms	6.799ms	6.818ms	3.15%	-0.948	1.697	1.85%	0.009ms	1	200
trace_buffer/8_senders/no_delay	throughput	1056097.865op/s	1093551.611op/s ± 20630.652op/s	1089368.663op/s ± 12968.280op/s	1106388.084op/s	1126799.229op/s	1161495.613op/s	1193857.867op/s	9.59%	1.115	2.457	1.88%	1458.807op/s	1	200

scenario	metric	95% CI mean	Shapiro-Wilk pvalue	Ljung-Box pvalue (lag=1)	Dip test pvalue
trace_buffer/1_senders/10us_delay	execution_time	[56.571ms; 56.611ms] or [-0.035%; +0.035%]	None	None	None
trace_buffer/1_senders/10us_delay	throughput	[15898.153op/s; 15909.379op/s] or [-0.035%; +0.035%]	None	None	None
trace_buffer/1_senders/1us_delay	execution_time	[50.078ms; 50.103ms] or [-0.025%; +0.025%]	None	None	None
trace_buffer/1_senders/1us_delay	throughput	[17963.156op/s; 17971.819op/s] or [-0.024%; +0.024%]	None	None	None
trace_buffer/1_senders/no_delay	execution_time	[343.096µs; 344.896µs] or [-0.262%; +0.262%]	None	None	None
trace_buffer/1_senders/no_delay	throughput	[2611137.644op/s; 2623095.268op/s] or [-0.228%; +0.228%]	None	None	None
trace_buffer/2_senders/10us_delay	execution_time	[57.288ms; 57.331ms] or [-0.038%; +0.038%]	None	None	None
trace_buffer/2_senders/10us_delay	throughput	[31396.630op/s; 31420.340op/s] or [-0.038%; +0.038%]	None	None	None
trace_buffer/2_senders/1us_delay	execution_time	[50.106ms; 50.144ms] or [-0.038%; +0.038%]	None	None	None
trace_buffer/2_senders/1us_delay	throughput	[35897.629op/s; 35923.965op/s] or [-0.037%; +0.037%]	None	None	None
trace_buffer/2_senders/no_delay	execution_time	[887.003µs; 892.772µs] or [-0.324%; +0.324%]	None	None	None
trace_buffer/2_senders/no_delay	throughput	[2017419.712op/s; 2030173.265op/s] or [-0.315%; +0.315%]	None	None	None
trace_buffer/4_senders/10us_delay	execution_time	[57.463ms; 57.497ms] or [-0.029%; +0.029%]	None	None	None
trace_buffer/4_senders/10us_delay	throughput	[62612.595op/s; 62649.398op/s] or [-0.029%; +0.029%]	None	None	None
trace_buffer/4_senders/1us_delay	execution_time	[50.174ms; 50.185ms] or [-0.010%; +0.010%]	None	None	None
trace_buffer/4_senders/1us_delay	throughput	[71735.068op/s; 71749.977op/s] or [-0.010%; +0.010%]	None	None	None
trace_buffer/4_senders/no_delay	execution_time	[2.371ms; 2.395ms] or [-0.511%; +0.511%]	None	None	None
trace_buffer/4_senders/no_delay	throughput	[1504868.426op/s; 1520505.924op/s] or [-0.517%; +0.517%]	None	None	None
trace_buffer/8_senders/10us_delay	execution_time	[57.884ms; 57.949ms] or [-0.056%; +0.056%]	None	None	None
trace_buffer/8_senders/10us_delay	throughput	[124249.450op/s; 124388.154op/s] or [-0.056%; +0.056%]	None	None	None
trace_buffer/8_senders/1us_delay	execution_time	[50.295ms; 50.314ms] or [-0.018%; +0.018%]	None	None	None
trace_buffer/8_senders/1us_delay	throughput	[143102.251op/s; 143154.821op/s] or [-0.018%; +0.018%]	None	None	None
trace_buffer/8_senders/no_delay	execution_time	[6.569ms; 6.603ms] or [-0.256%; +0.256%]	None	None	None
trace_buffer/8_senders/no_delay	throughput	[1090692.401op/s; 1096410.821op/s] or [-0.261%; +0.261%]	None	None	None

Baseline

Baseline benchmark details

Group 1

cpu_model	git_commit_sha	git_commit_date	git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	`3081603`	1784719733	main

scenario	metric	min	mean ± sd	median ± mad	p75	p95	p99	max	peak_to_median_ratio	skewness	kurtosis	cv	sem	runs	sample_size
concentrator/add_spans_to_concentrator	execution_time	8.532ms	8.564ms ± 0.017ms	8.559ms ± 0.010ms	8.573ms	8.595ms	8.616ms	8.629ms	0.81%	1.045	1.210	0.19%	0.001ms	1	200

scenario	metric	95% CI mean	Shapiro-Wilk pvalue	Ljung-Box pvalue (lag=1)	Dip test pvalue
concentrator/add_spans_to_concentrator	execution_time	[8.562ms; 8.566ms] or [-0.027%; +0.027%]	None	None	None

Group 2

cpu_model	git_commit_sha	git_commit_date	git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	`3081603`	1784719733	main

scenario	metric	min	mean ± sd	median ± mad	p75	p95	p99	max	peak_to_median_ratio	skewness	kurtosis	cv	sem	runs	sample_size
trace_buffer/1_senders/10us_delay	execution_time	56.337ms	56.632ms ± 0.184ms	56.604ms ± 0.074ms	56.677ms	56.906ms	57.481ms	57.780ms	2.08%	2.755	12.111	0.32%	0.013ms	1	200
trace_buffer/1_senders/10us_delay	throughput	15576.230op/s	15892.113op/s ± 51.235op/s	15899.941op/s ± 20.902op/s	15921.502op/s	15951.129op/s	15956.077op/s	15975.347op/s	0.47%	-2.692	11.651	0.32%	3.623op/s	1	200
trace_buffer/1_senders/1us_delay	execution_time	50.026ms	50.081ms ± 0.033ms	50.079ms ± 0.013ms	50.090ms	50.147ms	50.192ms	50.257ms	0.35%	1.403	4.326	0.07%	0.002ms	1	200
trace_buffer/1_senders/1us_delay	throughput	17908.113op/s	17971.009op/s ± 11.981op/s	17971.529op/s ± 4.602op/s	17977.246op/s	17988.778op/s	17990.168op/s	17990.548op/s	0.11%	-1.394	4.285	0.07%	0.847op/s	1	200
trace_buffer/1_senders/no_delay	execution_time	335.167µs	340.594µs ± 6.204µs	339.785µs ± 1.847µs	342.244µs	344.927µs	351.946µs	417.476µs	22.86%	9.661	116.318	1.82%	0.439µs	1	200
trace_buffer/1_senders/no_delay	throughput	2155813.212op/s	2643189.385op/s ± 41386.685op/s	2648733.458op/s ± 14475.240op/s	2661853.188op/s	2676991.999op/s	2681951.646op/s	2685227.325op/s	1.38%	-8.342	94.547	1.56%	2926.481op/s	1	200
trace_buffer/2_senders/10us_delay	execution_time	56.919ms	57.307ms ± 0.174ms	57.307ms ± 0.134ms	57.436ms	57.592ms	57.727ms	57.776ms	0.82%	0.091	-0.430	0.30%	0.012ms	1	200
trace_buffer/2_senders/10us_delay	throughput	31154.869op/s	31409.844op/s ± 95.589op/s	31409.699op/s ± 73.563op/s	31489.938op/s	31564.853op/s	31593.469op/s	31624.015op/s	0.68%	-0.077	-0.440	0.30%	6.759op/s	1	200
trace_buffer/2_senders/1us_delay	execution_time	50.062ms	50.109ms ± 0.032ms	50.101ms ± 0.017ms	50.122ms	50.169ms	50.220ms	50.252ms	0.30%	1.557	3.444	0.06%	0.002ms	1	200
trace_buffer/2_senders/1us_delay	throughput	35819.817op/s	35921.788op/s ± 22.962op/s	35927.072op/s ± 12.378op/s	35937.444op/s	35949.354op/s	35953.849op/s	35955.091op/s	0.08%	-1.551	3.418	0.06%	1.624op/s	1	200
trace_buffer/2_senders/no_delay	execution_time	849.297µs	899.909µs ± 21.284µs	902.208µs ± 12.591µs	911.942µs	931.111µs	952.856µs	1017.619µs	12.79%	0.758	4.062	2.36%	1.505µs	1	200
trace_buffer/2_senders/no_delay	throughput	1768834.608op/s	2001300.496op/s ± 46695.725op/s	1995106.202op/s ± 27460.365op/s	2030085.273op/s	2076833.889op/s	2102710.892op/s	2119399.511op/s	6.23%	-0.413	2.567	2.33%	3301.886op/s	1	200
trace_buffer/4_senders/10us_delay	execution_time	57.310ms	57.588ms ± 0.126ms	57.565ms ± 0.070ms	57.651ms	57.806ms	57.986ms	58.082ms	0.90%	1.136	2.071	0.22%	0.009ms	1	200
trace_buffer/4_senders/10us_delay	throughput	61980.974op/s	62512.874op/s ± 136.287op/s	62538.043op/s ± 76.652op/s	62606.064op/s	62685.143op/s	62748.571op/s	62816.368op/s	0.45%	-1.118	2.014	0.22%	9.637op/s	1	200
trace_buffer/4_senders/1us_delay	execution_time	50.115ms	50.170ms ± 0.037ms	50.159ms ± 0.015ms	50.179ms	50.254ms	50.284ms	50.293ms	0.27%	1.448	1.806	0.07%	0.003ms	1	200
trace_buffer/4_senders/1us_delay	throughput	71580.618op/s	71756.243op/s ± 52.542op/s	71771.863op/s ± 21.265op/s	71788.139op/s	71814.737op/s	71827.392op/s	71834.452op/s	0.09%	-1.445	1.794	0.07%	3.715op/s	1	200
trace_buffer/4_senders/no_delay	execution_time	2.067ms	2.268ms ± 0.082ms	2.270ms ± 0.056ms	2.324ms	2.396ms	2.445ms	2.471ms	8.87%	-0.094	-0.332	3.59%	0.006ms	1	200
trace_buffer/4_senders/no_delay	throughput	1456728.266op/s	1589635.604op/s ± 57524.789op/s	1585921.764op/s ± 38984.911op/s	1628183.210op/s	1693210.754op/s	1722511.671op/s	1741937.310op/s	9.84%	0.271	-0.298	3.61%	4067.617op/s	1	200
trace_buffer/8_senders/10us_delay	execution_time	57.682ms	58.032ms ± 0.251ms	57.927ms ± 0.147ms	58.259ms	58.485ms	58.553ms	58.647ms	1.24%	0.637	-0.930	0.43%	0.018ms	1	200
trace_buffer/8_senders/10us_delay	throughput	122768.409op/s	124072.811op/s ± 535.573op/s	124293.646op/s ± 315.206op/s	124530.990op/s	124666.327op/s	124776.804op/s	124822.957op/s	0.43%	-0.628	-0.945	0.43%	37.871op/s	1	200
trace_buffer/8_senders/1us_delay	execution_time	50.204ms	50.285ms ± 0.049ms	50.270ms ± 0.022ms	50.308ms	50.379ms	50.469ms	50.531ms	0.52%	1.749	4.490	0.10%	0.003ms	1	200
trace_buffer/8_senders/1us_delay	throughput	142487.126op/s	143183.283op/s ± 138.784op/s	143226.255op/s ± 62.951op/s	143268.358op/s	143345.390op/s	143369.644op/s	143415.644op/s	0.13%	-1.739	4.437	0.10%	9.813op/s	1	200
trace_buffer/8_senders/no_delay	execution_time	5.875ms	6.578ms ± 0.150ms	6.617ms ± 0.079ms	6.684ms	6.760ms	6.785ms	6.829ms	3.21%	-1.366	2.698	2.27%	0.011ms	1	200
trace_buffer/8_senders/no_delay	throughput	1054340.978op/s	1095066.003op/s ± 25823.694op/s	1088148.140op/s ± 13076.452op/s	1103871.556op/s	1143433.352op/s	1189509.008op/s	1225552.497op/s	12.63%	1.577	3.781	2.35%	1826.011op/s	1	200

scenario	metric	95% CI mean	Shapiro-Wilk pvalue	Ljung-Box pvalue (lag=1)	Dip test pvalue
trace_buffer/1_senders/10us_delay	execution_time	[56.607ms; 56.658ms] or [-0.045%; +0.045%]	None	None	None
trace_buffer/1_senders/10us_delay	throughput	[15885.012op/s; 15899.214op/s] or [-0.045%; +0.045%]	None	None	None
trace_buffer/1_senders/1us_delay	execution_time	[50.076ms; 50.085ms] or [-0.009%; +0.009%]	None	None	None
trace_buffer/1_senders/1us_delay	throughput	[17969.349op/s; 17972.670op/s] or [-0.009%; +0.009%]	None	None	None
trace_buffer/1_senders/no_delay	execution_time	[339.734µs; 341.454µs] or [-0.252%; +0.252%]	None	None	None
trace_buffer/1_senders/no_delay	throughput	[2637453.589op/s; 2648925.182op/s] or [-0.217%; +0.217%]	None	None	None
trace_buffer/2_senders/10us_delay	execution_time	[57.283ms; 57.332ms] or [-0.042%; +0.042%]	None	None	None
trace_buffer/2_senders/10us_delay	throughput	[31396.597op/s; 31423.092op/s] or [-0.042%; +0.042%]	None	None	None
trace_buffer/2_senders/1us_delay	execution_time	[50.104ms; 50.113ms] or [-0.009%; +0.009%]	None	None	None
trace_buffer/2_senders/1us_delay	throughput	[35918.606op/s; 35924.970op/s] or [-0.009%; +0.009%]	None	None	None
trace_buffer/2_senders/no_delay	execution_time	[896.959µs; 902.858µs] or [-0.328%; +0.328%]	None	None	None
trace_buffer/2_senders/no_delay	throughput	[1994828.918op/s; 2007772.075op/s] or [-0.323%; +0.323%]	None	None	None
trace_buffer/4_senders/10us_delay	execution_time	[57.571ms; 57.606ms] or [-0.030%; +0.030%]	None	None	None
trace_buffer/4_senders/10us_delay	throughput	[62493.986op/s; 62531.762op/s] or [-0.030%; +0.030%]	None	None	None
trace_buffer/4_senders/1us_delay	execution_time	[50.165ms; 50.175ms] or [-0.010%; +0.010%]	None	None	None
trace_buffer/4_senders/1us_delay	throughput	[71748.962op/s; 71763.525op/s] or [-0.010%; +0.010%]	None	None	None
trace_buffer/4_senders/no_delay	execution_time	[2.256ms; 2.279ms] or [-0.498%; +0.498%]	None	None	None
trace_buffer/4_senders/no_delay	throughput	[1581663.221op/s; 1597607.986op/s] or [-0.502%; +0.502%]	None	None	None
trace_buffer/8_senders/10us_delay	execution_time	[57.997ms; 58.066ms] or [-0.060%; +0.060%]	None	None	None
trace_buffer/8_senders/10us_delay	throughput	[123998.586op/s; 124147.036op/s] or [-0.060%; +0.060%]	None	None	None
trace_buffer/8_senders/1us_delay	execution_time	[50.278ms; 50.292ms] or [-0.013%; +0.013%]	None	None	None
trace_buffer/8_senders/1us_delay	throughput	[143164.049op/s; 143202.517op/s] or [-0.013%; +0.013%]	None	None	None
trace_buffer/8_senders/no_delay	execution_time	[6.558ms; 6.599ms] or [-0.316%; +0.316%]	None	None	None
trace_buffer/8_senders/no_delay	throughput	[1091487.088op/s; 1098644.919op/s] or [-0.327%; +0.327%]	None	None	None

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 52b09bcced

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

chatgpt-codex-connector · 2026-07-22T14:19:20Z

+            if let Err(e) = worker_entry.worker.start(tokio_spawn_fn(&handle)) {
+                error!(
+                    worker_id = worker_entry.id,
+                    "Worker failed to restart after fork in parent: {:?}", e
+                )


Return restart failures after finishing the loop

When a worker is already in InvalidState, this branch logs the failed restart but still lets after_fork_parent return Ok(()); after_fork_child now follows the same pattern. In that scenario the Rust/FFI caller is told the fork hook succeeded even though at least one registered worker is no longer running, so please keep restarting the remaining workers but remember and return a SharedRuntimeError::WorkerError afterward.

AGENTS.md reference: AGENTS.md:L70-L73

Useful? React with 👍 / 👎.

On one hand, this kinda the point of the PR. On another hand, I agree that we now hide information from the caller. Should we keep the first error indeed and return it at the end? Have Result<Vec<(WorkerId, Error)>, _> where we return the non-fatal errors encountered for some workers? Or something else?

I'm not a fan of the silent failure either although there's almost nothing the tracer can do about this issue (worker id isn't meaningful outside of the runtime and you can't know which worker failed). Also It only happens if there's a bug in libdatadog so I don't want to make it to complex to handle on tracer side for this.

dd-octo-sts · 2026-07-22T14:32:50Z

Artifact Size Benchmark Report

aarch64-alpine-linux-musl

Artifact	Baseline	Commit	Change
/aarch64-alpine-linux-musl/lib/libdatadog_profiling.a	88.18 MB	88.18 MB	+0% (+6.57 KB) 👌
/aarch64-alpine-linux-musl/lib/libdatadog_profiling.so	8.13 MB	8.13 MB	+0% (+48 B) 👌

aarch64-unknown-linux-gnu

Artifact	Baseline	Commit	Change
/aarch64-unknown-linux-gnu/lib/libdatadog_profiling.so	10.93 MB	10.93 MB	+0% (+872 B) 👌
/aarch64-unknown-linux-gnu/lib/libdatadog_profiling.a	99.46 MB	99.47 MB	+0% (+7.51 KB) 👌

libdatadog-x64-windows

Artifact	Baseline	Commit	Change
/libdatadog-x64-windows/debug/dynamic/datadog_profiling_ffi.dll	26.59 MB	26.59 MB	+0% (+2.00 KB) 👌
/libdatadog-x64-windows/debug/dynamic/datadog_profiling_ffi.lib	89.18 KB	89.18 KB	0% (0 B) 👌
/libdatadog-x64-windows/debug/dynamic/datadog_profiling_ffi.pdb	191.41 MB	191.38 MB	--.01% (-24.00 KB) 💪
/libdatadog-x64-windows/debug/static/datadog_profiling_ffi.lib	1.04 GB	1.04 GB	+0% (+6.98 KB) 👌
/libdatadog-x64-windows/release/dynamic/datadog_profiling_ffi.dll	8.66 MB	8.66 MB	+.01% (+1.00 KB) 🔍
/libdatadog-x64-windows/release/dynamic/datadog_profiling_ffi.lib	89.18 KB	89.18 KB	0% (0 B) 👌
/libdatadog-x64-windows/release/dynamic/datadog_profiling_ffi.pdb	25.36 MB	25.36 MB	0% (0 B) 👌
/libdatadog-x64-windows/release/static/datadog_profiling_ffi.lib	50.55 MB	50.56 MB	+0% (+3.92 KB) 👌

libdatadog-x86-windows

Artifact	Baseline	Commit	Change
/libdatadog-x86-windows/debug/dynamic/datadog_profiling_ffi.dll	23.16 MB	23.16 MB	+.01% (+3.00 KB) 🔍
/libdatadog-x86-windows/debug/dynamic/datadog_profiling_ffi.lib	90.58 KB	90.58 KB	0% (0 B) 👌
/libdatadog-x86-windows/debug/dynamic/datadog_profiling_ffi.pdb	196.08 MB	196.08 MB	0% (0 B) 👌
/libdatadog-x86-windows/debug/static/datadog_profiling_ffi.lib	1.03 GB	1.03 GB	+0% (+6.42 KB) 👌
/libdatadog-x86-windows/release/dynamic/datadog_profiling_ffi.dll	6.68 MB	6.68 MB	+.01% (+1.00 KB) 🔍
/libdatadog-x86-windows/release/dynamic/datadog_profiling_ffi.lib	90.58 KB	90.58 KB	0% (0 B) 👌
/libdatadog-x86-windows/release/dynamic/datadog_profiling_ffi.pdb	27.26 MB	27.26 MB	0% (0 B) 👌
/libdatadog-x86-windows/release/static/datadog_profiling_ffi.lib	48.13 MB	48.13 MB	+0% (+3.42 KB) 👌

x86_64-alpine-linux-musl

Artifact	Baseline	Commit	Change
/x86_64-alpine-linux-musl/lib/libdatadog_profiling.a	78.64 MB	78.65 MB	+0% (+6.57 KB) 👌
/x86_64-alpine-linux-musl/lib/libdatadog_profiling.so	9.05 MB	9.06 MB	+.08% (+8.06 KB) 🔍

x86_64-unknown-linux-gnu

Artifact	Baseline	Commit	Change
/x86_64-unknown-linux-gnu/lib/libdatadog_profiling.a	94.14 MB	94.14 MB	+0% (+6.53 KB) 👌
/x86_64-unknown-linux-gnu/lib/libdatadog_profiling.so	11.02 MB	11.02 MB	+.04% (+4.63 KB) 🔍

yannham · 2026-07-22T14:48:48Z

+            if let Err(e) = worker_entry.worker.start(tokio_spawn_fn(&handle)) {
+                error!(
+                    worker_id = worker_entry.id,
+                    "Worker failed to restart after fork in parent: {:?}", e
+                )


On one hand, this kinda the point of the PR. On another hand, I agree that we now hide information from the caller. Should we keep the first error indeed and return it at the end? Have Result<Vec<(WorkerId, Error)>, _> where we return the non-fatal errors encountered for some workers? Or something else?

) # Release proposal for libdd-data-pipeline, libdd-telemetry and their dependencies This PR contains version bumps based on public API changes and commits since last release. ## libdd-ddsketch **Next version:** `1.1.1` **Semver bump:** `patch` **Tag:** `libdd-ddsketch-v1.1.1` ### Commits - refactor: migrate to workspace dependencies, phase 4 (#2296) - chore: moving to workspace-level dependencies, phase 2 (#2270) ## libdd-shared-runtime **Next version:** `3.0.0` **Semver bump:** `major` **Tag:** `libdd-shared-runtime-v3.0.0` ### ⚠️ major bump forced due to: - `libdd-capabilities`: ^2.1.0 → ^3.0.0 - `libdd-capabilities-impl`: ^3.0.0 → ^4.0.0 ### Commits - feat(shared-runtime): add block_on_with_timeout to BlockingRuntime (#2333) - chore(runtime): restart all workers on invalid state (#2262) - refactor: migrate to workspace dependencies, phase 3 (#2283) - refactor: consolidate core dependencies at workspace level (phase 1) (#2253) ## libdd-dogstatsd-client **Next version:** `5.0.0` **Semver bump:** `major` **Tag:** `libdd-dogstatsd-client-v5.0.0` ### Commits - refactor: consolidate core dependencies at workspace level (phase 1) (#2253) - feat(dogstatsd)!: add shared_runtime buffered sink (#2224) - refactor(statsd)!: make client clonable (#2222) ## libdd-telemetry **Next version:** `7.0.0` **Semver bump:** `major` **Tag:** `libdd-telemetry-v7.0.0` ### Commits - refactor(rc)!: make conversion from RemoteConfigProduct back and forth generally available (#2325) - feat(telemetry)!: Add Installation signature and AppProduct changes payloads (#2213) - fix(telemetry)!: stop sending Connection: close to the Agent (#2286) - refactor: migrate to workspace dependencies, phase 3 (#2283) - chore: moving to workspace-level dependencies, phase 2 (#2270) - refactor: consolidate core dependencies at workspace level (phase 1) (#2253) - fix: drain the mailbox before stopping (#2258) - feat(telemetry)!: make telemetry worker wasm-compatible for the TraceExporter (#2172) - fix(sidecar): don't double-encode file:// telemetry endpoints (#2230) ## libdd-trace-obfuscation **Next version:** `6.0.0` **Semver bump:** `major` **Tag:** `libdd-trace-obfuscation-v6.0.0` ### ⚠️ major bump forced due to: - `libdd-trace-utils`: ^9.0.0 → ^10.1.0 ### Commits - refactor: migrate to workspace dependencies, phase 4 (#2296) - chore(lint): update nightly (#2281) - chore: moving to workspace-level dependencies, phase 2 (#2270) - refactor: consolidate core dependencies at workspace level (phase 1) (#2253) - fix: new clippy lints (#2219) ## libdd-trace-stats **Next version:** `7.0.0` **Semver bump:** `major` **Tag:** `libdd-trace-stats-v7.0.0` ### ⚠️ major bump forced due to: - `libdd-capabilities`: ^2.1.0 → ^3.0.0 - `libdd-capabilities-impl`: ^3.0.0 → ^4.0.0 - `libdd-dogstatsd-client`: ^4.0.0 → ^5.0.0 - `libdd-telemetry`: ^6.0.0 → ^7.0.0 - `libdd-trace-utils`: ^9.0.0 → ^10.1.0 ### Commits - feat: add v1 isolated bricks for trace exporter (#2280) - refactor: migrate to workspace dependencies, phase 4 (#2296) - feat(data-pipeline)!: add compression option for agentless export (#2205) - feat(stats)!: per-field cardinality limits (#2211) - refactor: migrate to workspace dependencies, phase 3 (#2283) - feat(stats)!: string field length limit (#2267) - feat(trace_exporter)!: add `restart_after_fork` configuration option [APMSP-3843] (#2263) - refactor: consolidate core dependencies at workspace level (phase 1) (#2253) - feat(stats)!: implement additional metric tags in libdd-trace-stats (#2170) - refactor(statsd)!: make client clonable (#2222) - fix(stats-exporter): disable retries for stats payload sends (#2249) - feat(telemetry)!: make telemetry worker wasm-compatible for the TraceExporter (#2172) - fix!: css obfuscation logic (#2203) ## libdd-data-pipeline **Next version:** `8.0.0` **Semver bump:** `major` **Tag:** `libdd-data-pipeline-v8.0.0` ### ⚠️ major bump forced due to: - `libdd-capabilities`: ^2.1.0 → ^3.0.0 - `libdd-capabilities-impl`: ^3.0.0 → ^4.0.0 - `libdd-dogstatsd-client`: ^4.0.0 → ^5.0.0 - `libdd-telemetry`: ^6.0.0 → ^7.0.0 - `libdd-trace-stats`: ^6.0.0 → ^7.0.0 - `libdd-trace-utils`: ^9.0.0 → ^10.1.0 ### Commits - fix(trace-exporter): mark OTLP client-computed stats (#2245) - feat(data-pipeline): adds new attributes to OTLP trace metrics export (#2316) - feat: add v1 isolated bricks for trace exporter (#2280) - feat(data-pipeline)!: add flush_and_close to the trace buffer (#2313) - feat(telemetry)!: Add Installation signature and AppProduct changes payloads (#2213) - refactor: migrate to workspace dependencies, phase 4 (#2296) - feat(data-pipeline)!: add compression option for agentless export (#2205) - fix(telemetry)!: stop sending Connection: close to the Agent (#2286) - feat(stats)!: per-field cardinality limits (#2211) - refactor: migrate to workspace dependencies, phase 3 (#2283) - feat(stats)!: string field length limit (#2267) - refactor(trace-utils)!: split generic TracerHeaderTags (#2279) - chore: moving to workspace-level dependencies, phase 2 (#2270) - feat(trace_exporter)!: add `restart_after_fork` configuration option [APMSP-3843] (#2263) - feat(data-pipeline): emit additional_metric_tags on OTLP span duration metric (#2264) - refactor: consolidate core dependencies at workspace level (phase 1) (#2253) - feat(dogstatsd)!: add shared_runtime buffered sink (#2224) - feat(stats)!: implement additional metric tags in libdd-trace-stats (#2170) - refactor(statsd)!: make client clonable (#2222) - feat(telemetry)!: make telemetry worker wasm-compatible for the TraceExporter (#2172) - feat(trace_exporter)!: remove deprecated methods (#2250) - feat(trace-utils)!: add v1 decoder (#2174) - feat(trace-utils)!: add encoder v1 to v04 + refactor (#2145) - feat(otlp): add instrumentation scope metadata (#2235) - fix(trace-utils)!: serialize v0.5 span links and events into meta (#980) - fix!: css obfuscation logic (#2203) - test(data-pipeline): gate telemetry-only tests behind the telemetry feature (#2204) [APMSP-3843]: https://datadoghq.atlassian.net/browse/APMSP-3843?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: hoolioh <107922352+hoolioh@users.noreply.github.com>

chore(runtime): restart all workers on invalid state

52b09bc

VianneyRuhlmann marked this pull request as ready for review July 22, 2026 14:15

VianneyRuhlmann requested a review from a team as a code owner July 22, 2026 14:15

VianneyRuhlmann mentioned this pull request Jul 22, 2026

fix(shared-runtime)!: fork recovery, shutdown, runtime_handle #1986

Closed

chatgpt-codex-connector Bot reviewed Jul 22, 2026

View reviewed changes

yannham approved these changes Jul 22, 2026

View reviewed changes

gh-worker-dd-devflow-36fce6 Bot added mergequeue-status: queued mergequeue-status: in_progress mergequeue-status: rejected and removed mergequeue-status: queued mergequeue-status: in_progress mergequeue-status: rejected labels Jul 28, 2026

gh-worker-dd-mergequeue-cf854d Bot merged commit 485a0a1 into main Jul 30, 2026
244 of 248 checks passed

gh-worker-dd-devflow-36fce6 Bot removed the mergequeue-status: in_progress label Jul 30, 2026

gh-worker-dd-mergequeue-cf854d Bot deleted the vianney/failsafe-after-fork branch July 30, 2026 09:12

gh-worker-dd-devflow-36fce6 Bot added the mergequeue-status: done label Jul 30, 2026

dd-octo-sts Bot mentioned this pull request Aug 17, 2026

chore(release): proposal for libdd-data-pipeline, libdd-telemetry #2358

Merged

Navigation Menu

chore(runtime): restart all workers on invalid state - #2262

chore(runtime): restart all workers on invalid state#2262
gh-worker-dd-mergequeue-cf854d[bot] merged 1 commit into
mainfrom
vianney/failsafe-after-fork

VianneyRuhlmann commented Jul 22, 2026 •

edited

Loading

Uh oh!

datadog-datadog-us1-prod Bot commented Jul 22, 2026 •

edited by datadog-datadog-prod-us1-2 Bot

Loading

Uh oh!

pr-commenter Bot commented Jul 22, 2026 •

edited

Loading

Explanation

More details about the CI and significant changes

Group 1

Group 2

Group 1

Group 2

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

chatgpt-codex-connector Bot Jul 22, 2026

Uh oh!

yannham Jul 22, 2026

Uh oh!

VianneyRuhlmann Jul 22, 2026

Uh oh!

dd-octo-sts Bot commented Jul 22, 2026

Uh oh!

yannham Jul 22, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

VianneyRuhlmann commented Jul 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

datadog-datadog-us1-prod Bot commented Jul 22, 2026 • edited by datadog-datadog-prod-us1-2 Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pr-commenter Bot commented Jul 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks

Comparison

Explanation

More details about the CI and significant changes

scenario:trace_buffer/4_senders/no_delay

Candidate

Group 1

Group 2

Baseline

Group 1

Group 2

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Jul 22, 2026

Choose a reason for hiding this comment

Uh oh!

yannham Jul 22, 2026

Choose a reason for hiding this comment

Uh oh!

VianneyRuhlmann Jul 22, 2026

Choose a reason for hiding this comment

Uh oh!

dd-octo-sts Bot commented Jul 22, 2026

Artifact Size Benchmark Report

Uh oh!

yannham Jul 22, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

VianneyRuhlmann commented Jul 22, 2026 •

edited

Loading

datadog-datadog-us1-prod Bot commented Jul 22, 2026 •

edited by datadog-datadog-prod-us1-2 Bot

Loading

pr-commenter Bot commented Jul 22, 2026 •

edited

Loading