Conversation
Contributor
|
Is this something that the Datadog agent returns or a curl thing? |
Contributor
Author
|
From what I seen it is |
2 tasks
cataphract
added a commit
that referenced
this pull request
Sep 15, 2026
AI summary of the mechanism: > The crash is a cross-thread use-after-free: > > 1. gRPC executes a PHP CallCredentials callback directly on a native > EventEngine worker thread. > 2. In PHP NTS, executor globals—including the profiler VM > interrupt—are process-wide. The gRPC thread can therefore consume a > pending wall-time profiling interrupt. > > 3. The profiler stack-walks that foreign thread and caches raw ThinStr > pointers in Zend’s function runtime cache. > 4. Those strings are actually owned by a Rust thread_local StringSet > belonging to that specific gRPC worker. > 5. After the worker remains idle, gRPC retires it. Its TLS destructor > frees/unmaps the string arena, but Zend’s runtime cache still contains > the pointers. > 6. A later callback/sample retrieves one of those dangling pointers > and crashes in: > ``` > ThinStr::len > StringSet::get_thin_str > collect_time > zend_call_function > plugin_get_metadata > ``` > The deterministic reproducer is one sufficiently long callback, a > 25-second idle interval allowing worker retirement, then another > callback. > > Forking, AppSec, tracing, payload stress, and allocation profiling are > not required. The profiler must avoid sampling threads that never ran > PHP request initialization and must not store thread-local arena > pointers in runtime-cache slots accessible from other threads. Stacktrace: ``` \#0 0x00007742bf3ee95c in ?? () from /lib/x86_64-linux-gnu/libc.so.6 \#1 0x00007742bf399cc2 in raise () from /lib/x86_64-linux-gnu/libc.so.6 \#2 0x00007742b87c94ec in libdd_crashtracker::collector::signal_handler_manager::chain_signal_handler () at libdatadog/libdd-crashtracker/src/collector/signal_handler_manager.rs:125 \#3 libdd_crashtracker::collector::crash_handler::handle_posix_sigaction () at libdatadog/libdd-crashtracker/src/collector/crash_handler.rs:235 \#4 <signal handler called> \#5 read<[u8; 8]> () at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/core/src/ptr/mod.rs:1380 \#6 read<[u8; 8]> () at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/core/src/ptr/mut_ptr.rs:1356 \#7 len<u8> () at /project/libdatadog/libdd-profiling/src/profiles/collections/thin_str.rs:213 \#8 inline_string_ptr () at /project/libdatadog/libdd-profiling/src/profiles/collections/thin_str.rs:235 \#9 borrow () at /project/libdatadog/libdd-profiling/src/profiles/collections/thin_str.rs:470 \#10 deref () at /project/libdatadog/libdd-profiling/src/profiles/collections/thin_str.rs:453 \#11 get_thin_str () at components-rs/../profiling/src/string_set.rs:117 \#12 get_or_insert<datadog_php::profiling::profiler::stack_walking::detail::handle_function_cache_slot::{closure_env#0}> () at components-rs/../profiling/src/profiler/stack_walking.rs:209 \#13 handle_function_cache_slot () at components-rs/../profiling/src/profiler/stack_walking.rs:466 \#14 collect_call_frame () at components-rs/../profiling/src/profiler/stack_walking.rs:417 \#15 collect_stack_sample_cached () at components-rs/../profiling/src/profiler/stack_walking.rs:362 \#16 0x00007742bea25a55 in {closure#0} () at components-rs/../profiling/src/profiler/stack_walking.rs:393 \#17 {closure#0}<datadog_php::profiling::string_set::StringSet, datadog_php::profiling::profiler::stack_walking::detail::collect_stack_sample::{closure_env#0}, core::result::Result<datadog_php::profiling::profiler::backtrace::Backtrace, datadog_php::profiling::profiler::stack_walking::CollectStackSampleError>> () at components-rs/../profiling/src/lib.rs:533 \#18 map<core::cell::RefMut<datadog_php::profiling::string_set::StringSet>, core::cell::BorrowMutError, core::result::Result<datadog_php::profiling::profiler::backtrace::Backtrace, datadog_php::profiling::profiler::stack_walking::CollectStackSampleError>, datadog_php::profiling::{impl#2}::try_with_borrow_mut::{closure#0}::{closure_env#0}<datadog_php::profiling::string_set::StringSet, datadog_php::profiling::profiler::stack_walking::detail::collect_stack_sample::{closure_env#0}, core::result::Result<datadog_php::profiling::profiler::backtrace::Backtrace, datadog_php::profiling::profiler::stack_walking::CollectStackSampleError>>> () at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/core/src/result.rs:773 \#19 {closure#0}<datadog_php::profiling::string_set::StringSet, datadog_php::profiling::profiler::stack_walking::detail::collect_stack_sample::{closure_env#0}, core::result::Result<datadog_php::profiling::profiler::backtrace::Backtrace, datadog_php::profiling::profiler::stack_walking::CollectStackSampleError>> () at components-rs/../profiling/src/lib.rs:533 \#20 try_with<core::cell::RefCell<datadog_php::profiling::string_set::StringSet>, datadog_php::profiling::{impl#2}::try_with_borrow_mut::{closure_env#0}<datadog_php::profiling::string_set::StringSet, datadog_php::profiling::profiler::stack_walking::detail::collect_stack_sample::{closure_env#0}, core::result::Result<datadog_php::profiling::profiler::backtrace::Backtrace, datadog_php::profiling::profiler::stack_walking::CollectStackSampleError>>, core::result::Result<core::result::Result<datadog_php::profiling::profiler::backtrace::Backtrace, datadog_php::profiling::profiler::stack_walking::CollectStackSampleError>, core::cell::BorrowMutError>> () at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/std/src/thread/local.rs:311 \#21 try_with_borrow_mut<datadog_php::profiling::string_set::StringSet, datadog_php::profiling::profiler::stack_walking::detail::collect_stack_sample::{closure_env#0}, core::result::Result<datadog_php::profiling::profiler::backtrace::Backtrace, datadog_php::profiling::profiler::stack_walking::CollectStackSampleError>> () at components-rs/../profiling/src/lib.rs:532 \#22 collect_stack_sample () at components-rs/../profiling/src/profiler/stack_walking.rs:393 \#23 0x00007742bea2cf55 in collect_stack_sample_timed () at components-rs/../profiling/src/profiler/mod.rs:1158 \#24 0x00007742bea2d087 in collect_time () at components-rs/../profiling/src/profiler/mod.rs:1182 \#25 0x00007742bea35d9a in ddog_php_prof_interrupt_function_wrapper () at components-rs/../profiling/src/wall_time.rs:308 \#26 0x00005b150b5ba079 in zend_interrupt_helper_SPEC () at /root/php/sources/8.5.0/Zend/zend_vm_execute.h:4027 \#27 0x00005b150b69fb6f in execute_ex (ex=0x7742bf013590) at /root/php/sources/8.5.0/Zend/zend_vm_execute.h:116152 \#28 0x00005b150b57cea5 in zend_call_function (fci=0x5b15381ade70, fci_cache=0x5b1538220230) at /root/php/sources/8.5.0/Zend/zend_execute_API.c:1010 \#29 0x00007742be4d7e7e in plugin_get_metadata (ptr=0x7742bf003190, context=..., cb=<optimized out>, user_data=<optimized out>, creds_md=0x77427a7fb0a0, num_creds_md=0x77427a7fb068, status=0x77427a7fb064, error_details=0x77427a7fb058) at /tmp/grpc-php-8.5-release.EUD4iV/src/php/ext/grpc/call_credentials.c:169 \#30 0x00007742be03098f in grpc_plugin_credentials::GetRequestMetadata (this=0x5b153821d660, initial_metadata=..., args=<optimized out>) at /tmp/grpc-php-8.5-release.EUD4iV/src/core/credentials/call/plugin/plugin_credentials.cc:162 \#31 0x00007742be19c295 in grpc_core::ClientAuthFilter::GetMetadataFromCreds (this=<optimized out>, creds=..., md=...) at ./src/core/filter/auth/auth_filters.h:61 \#32 grpc_core::ClientAuthFilter::GetCallCredsMetadata(std::__1::unique_ptr<grpc_metadata_batch, grpc_core::Arena::PooledDeleter>)::{lambda()#1}::operator()()::{lambda()#1}::operator()() const (this=<optimized out>) at ./src/core/filter/auth/auth_filters.h:80 \#33 grpc_core::promise_detail::PromiseLike<grpc_core::Map<grpc_core::ArenaPromise<absl::lts_20250512::StatusOr<std::__1::unique_ptr<grpc_metadata_batch, grpc_core::Arena::PooledDeleter> > >, grpc_core::ClientAuthFilter::GetMetadataFromCreds(grpc_core::RefCountedPtr<grpc_call_credentials>, std::__1::unique_ptr<grpc_metadata_batch, grpc_core::Arena::PooledDeleter>)::{lambda(absl::lts_20250512::StatusOr<std::__1::unique_ptr<grpc_metadata_batch, grpc_core::Arena::PooledDeleter> >)#1}>, void>::PromiseLike<grpc_core::ClientAuthFilter::GetCallCredsMetadata(std::__1::unique_ptr<grpc_metadata_batch, grpc_core::Arena::PooledDeleter>)::{lambda()#1}::operator()()::{lambda()#1}>(std::__1::in_place_t, grpc_core::ClientAuthFilter::GetCallCredsMetadata(std::__1::unique_ptr<grpc_metadata_batch, grpc_core::Arena::PooledDeleter>)::{lambda()#1}::operator()()::{lambda()#1}&&) (this=0x77427a7fb270, f=...) at ./src/core/lib/promise/detail/promise_like.h:219 \#34 grpc_core::promise_detail::PromiseFactoryImpl<grpc_core::promise_detail::OnceToken, grpc_core::ClientAuthFilter::GetCallCredsMetadata(std::__1::unique_ptr<grpc_metadata_batch, grpc_core::Arena::PooledDeleter>)::{lambda()#1}::operator()()::{lambda()#1}>(grpc_core::promise_detail::OnceToken, grpc_core::ClientAuthFilter::GetCallCredsMetadata(std::__1::unique_ptr<grpc_metadata_batch, grpc_core::Arena::PooledDeleter>)::{lambda()#1}::operator()()::{lambda()#1}&&) (f=...) at ./src/core/lib/promise/detail/promise_factory.h:221 ```
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
When flushing traces, if everything work as expected, a "OK" can be present in the response; even in a
terminatemethod for Laravel. In case of an API this will prevent the client to correctly parse the output.