You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
paltest_pal_sxs_test1 failed on SmartOS x64 #35362
The ExceptionInformation[1] comes from the sigsegv_handler, it passes siginfo->si_addr to common_signal_handler which then stores it into the ExceptionInformation[1].
I put the breakpoints in few places in sigsegv_handler and common_signal_handler, they get called as expected. At position src/coreclr/src/pal/src/exception/signal.cpp:516, the failure address (captured from siginfo->si_addr) was 96070, which is the same (wrong) one assigned to ExceptionInformation[1] and get compared with 0x22.
(gdb) c
Continuing.
Thread 2 hit Breakpoint 6, sigsegv_handler (code=11, siginfo=0xfffffc7fef0c3ef0, context=0xfffffc7fef0c3b90) at /home/am11/runtime/src/coreclr/src/pal/src/exception/signal.cpp:516516 if ((failureAddress - (sp - GetVirtualPageSize())) < 2 * GetVirtualPageSize())
(gdb) print failureAddress
$8 = 96070
(gdb) bt
#0 sigsegv_handler (code=11, siginfo=0xfffffc7fef0c3ef0, context=0xfffffc7fef0c3b90) at /home/am11/runtime/src/coreclr/src/pal/src/exception/signal.cpp:516#1 0xfffffc7fef245df6 in __sighndlr () from /lib/64/libc.so.1#2 0xfffffc7fef238c3b in call_user_handler () from /lib/64/libc.so.1#3 0xfffffc7fef238f6e in sigacthandler () from /lib/64/libc.so.1#4 0xffffffffffffffff in ?? ()#5 0x000000000000000b in ?? ()#6 0xfffffc7fef0c3ef0 in ?? ()#7 0x000000000000000f in ?? ()#8 0x0000000000000000 in ?? ()
Could it be the indication of data getting corrupted somewhere in libc?
The strange thing is also that the dlltest2 passes and then dlltest1 fails. They are exactly the same except of the address of failure. So maybe there is something that the first sigsegv screws in the kernel?
The sigsegv_handler is called directly by the kernel, so there should be nothing that libc could corrupt.
Can you try to swap the calls to DllTest1 and 2 to prove that it is not a problem of the DllTest1, but rather the fact that it gets called as the 2nd one?
Oh, I have somehow misread your debugging log, I have thought that the 1st one was succeeding and the 2nd one was failing.
So it seems that the OS is not filling in the siginfo->si_addr. Could it be an OS bug / feature?
--- a/src/coreclr/src/pal/src/exception/signal.cpp+++ b/src/coreclr/src/pal/src/exception/signal.cpp@@ -251,6 +251,7 @@ void SEHCleanupSignals()
restore_signal(SIGTRAP, &g_previous_sigtrap);
restore_signal(SIGFPE, &g_previous_sigfpe);
restore_signal(SIGBUS, &g_previous_sigbus);
+system("echo detaching pal internal sigsegv_handler");
restore_signal(SIGSEGV, &g_previous_sigsegv);
restore_signal(SIGINT, &g_previous_sigint);
restore_signal(SIGQUIT, &g_previous_sigquit);
@@ -505,6 +506,7 @@ Parameters :
--*/
static void sigsegv_handler(int code, siginfo_t *siginfo, void *context)
{
+system("echo pal internal sigsegv_handler");
if (PALIsInitialized())
{
// First check if we have a stack overflow
--- a/src/coreclr/src/pal/tests/palsuite/exception_handling/pal_sxs/test1/exceptionsxs.cpp+++ b/src/coreclr/src/pal/tests/palsuite/exception_handling/pal_sxs/test1/exceptionsxs.cpp@@ -96,8 +96,8 @@ int main(int argc, char *argv[])
// Test catching exceptions in other PAL instances
DllTest2();
- DllTest1();- DllTest2();+// DllTest1();+ // DllTest2();
if (bHandler)
{
on FreeBSD, we get:
PAL_SXS test1 SIGSEGV handler 0x0
Starting pal_sxs test1 DllTest2
pal internal sigsegv_handler
DLLTest2 PASSED
Starting PAL_SXS test1 signal chaining
pal internal sigsegv_handler
pal internal sigsegv_handler
pal_sxs test1: signal handler called
Signal chaining PASSED
on SmartOS, we get:
PAL_SXS test1 SIGSEGV handler 0
Starting pal_sxs test1 DllTest2
pal internal sigsegv_handler
DLLTest2 PASSED
Starting PAL_SXS test1 signal chaining
pal internal sigsegv_handler
pal internal sigsegv_handler
pal internal sigsegv_handler
pal internal sigsegv_handler
pal internal sigsegv_handler
pal internal sigsegv_handler
pal internal sigsegv_handler
pal internal sigsegv_handler
pal internal sigsegv_handler
and it continues, until process is terminated. Looks like the second sigsegv is initiating a never ending loop.
Does the runtime make use of in-process profiling through setitimer(ITIMER_PROF) and SIGPROF at all? If so, you may need to know about illumos bug 11494.
Also I would be very careful using system(3C) in a signal handler, even just for debugging. It is listed with an MT-Level attribute of Unsafe, which as per my reading of attributes(5) means it is not safe for use when multiple threads or signals are at play. If it were safe in a signal handler, I would expect it to be marked Async-Signal-Safe.
Afaict, setitimer is not used in coreclr code; the only use of setitimer is in one of the libunwind's test code, which is passing in upstream repo on SmartOS 2020. Also, SIGPROF is not used in coreclr (mono's POSIX library uses it in one of its APIs, but that is separate).
using system(3C) in a signal handler
Thanks, i have removed it. It was just for a quick sneak peek in the absence of printf/std::cout in that context (as PAL have some of the printer functions redefined, and tests are selectively linked with components).
The overview of this test failure is as follow:
libcoreclrpal.so: PAL library that has generic signal handlers for various signals, including SIGSEGV (and try/catch macros).
libpaltest_pal_sxs_test1_dll2.so: test lib that causes SIGSEGV for testing the handler. links with libcorepal.
paltest_pal_sxs_test1: the main / glue executable program, which links with libpaltest_pal_sxs_test1_dll2. It also defines its own signal handler for SIGSEGV, and expects the signal calls to chain when SIGSEGV is raised from the program's own code, but get stuck in an infinite loop of calling the first handler from PAL repeatedly and never calls the second one; as we can see in truss outputs, SmartOS vs. FreeBSD: paltest_pal_sxs_test1 failed on SmartOS x64 #35362 (comment).
ps - aside from pthread_setschedparam's inability to raise priority after lowering it from same value in case of non-privileged execution, which will likely be fixed upstream (and not too critical for now, as it only causes an assertion to fail in runtime code), this is the last failing (out of 716) PAL test.
To investigate this further, I see that your error message tells you that the expected value was not detected -- but it'd be good to print the value that was detected. In addition, our ucontext_t contains a uc_link member that may be NULL or may be a pointer to a parent signal handling context. It might help to walk up that chain, at least now during debugging, to see if the context you're looking for is further up.
It might help to ask the runtime linker which shared object a particular PC value comes from, too, to see which frame was interrupted; e.g.,
ucontext_t*uc=/* value from handler arguments */;
while (uc!=NULL) {
Dl_info_tdli;
void*pc= (void*)(uintptr_t)uc->uc_mcontext.gregs[REG_PC];
/* report pc */if (dladdr(pc, &dli) !=0) {
/* report dli.dli_fname */
}
uc=uc->uc_link;
}
To investigate this further, I see that your error message tells you that the expected value was not detected -- but it'd be good to print the value that was detected.
I agree, we should emit the faulting address at src/coreclr/src/pal/tests/palsuite/exception_handling/pal_sxs/test1/dlltest2.cpp#67, in the failure message. However, that (error) condition only meets, when we run it under the debugger. The main problem in normal runs (when debugger is not attached) is with the signal handler chaining; if two signal handlers are associated with SIGSEGV, in different objects, then the first one keeps getting fired.
It might help to walk up that chain, at least now during debugging, to see if the context you're looking for is further up.
Thank you. I will try to walk up the chain to find more details. Looks like Linux/BSD also define uc_link in ucontext_t . Is there any well-known difference in Illumos around that area? We have an alias typedef ucontext_t native_context_t; and use that alias in PAL's common_signal_handler.
It might help to ask the runtime linker which shared object a particular PC value comes from, too, to see which frame was interrupted
Could you give some pointers on how -- which linker flag to set in order to obtain this shared object to PC/IP mappings? We are using Solaris native ld (as opposed to gld, which is in PATH of some distros but not necessarily present everywhere).
@jclulow, if you want to give it a try, you can use my branch with libunwind upgrade commit: feature/solaris/pal-test-fixes. On SmartOS 20200408T231825Z x64 userland, my steps are:
sudo pkgin -y install git mozilla-rootcerts cmake icu py37-expat gcc7 gmake gdb-7
git clone https://github.com/am11/runtime --branch feature/solaris/pal-test-fixes --single-branch --depth 1
# skip everything and only build coreclr with PAL tests
runtime/src/coreclr/build-runtime.sh -skipgenerateversion -nopgooptimize \
-cmakeargs -DCLR_CMAKE_BUILD_TESTS=1 -gcc
with seven physical cores, the build took 11m33.128s with clean tree; subsequent runs (without git-clean) are faster.
The test executable paltest_pal_sxs_test1 is in artifacts/obj/coreclr/SunOS.x64.Debug/src/pal/tests/palsuite/exception_handling/pal_sxs/test1 directory.
coreclr linux arm Checked no_tiered_compilation @ (Alpine.314.Arm32.Open)Ubuntu.1804.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.14-helix-arm32v7
coreclr linux arm Checked @ (Alpine.314.Arm32.Open)Ubuntu.1804.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.14-helix-arm32v7
- exception_handling.pal_sxs.test1
Activity
am11 commentedon Apr 24, 2020
With line 67 commented out, the backtraces look like this: https://gist.github.com/am11/2464eff2db3a3ceadd3d56b976ee4a15.
janvorli commentedon Apr 24, 2020
The ExceptionInformation[1] comes from the sigsegv_handler, it passes siginfo->si_addr to common_signal_handler which then stores it into the ExceptionInformation[1].
am11 commentedon Apr 24, 2020
I put the breakpoints in few places in sigsegv_handler and common_signal_handler, they get called as expected. At position src/coreclr/src/pal/src/exception/signal.cpp:516, the failure address (captured from siginfo->si_addr) was 96070, which is the same (wrong) one assigned to
ExceptionInformation[1]and get compared with 0x22.Could it be the indication of data getting corrupted somewhere in libc?
janvorli commentedon Apr 24, 2020
The strange thing is also that the dlltest2 passes and then dlltest1 fails. They are exactly the same except of the address of failure. So maybe there is something that the first sigsegv screws in the kernel?
The sigsegv_handler is called directly by the kernel, so there should be nothing that libc could corrupt.
Can you try to swap the calls to DllTest1 and 2 to prove that it is not a problem of the DllTest1, but rather the fact that it gets called as the 2nd one?
am11 commentedon Apr 24, 2020
I switched the call:
rebuilt and ran the test under debugger without any breakpoints, it failed at same place:
janvorli commentedon Apr 24, 2020
I didn't mean to switch the initialization, I meant switching the calls to DllTest1 and DllTest2 functions.
am11 commentedon Apr 24, 2020
Ah, right. the current order is DllTest2, DllTest1, DllTest2. I changed it to DllTest1, DllTest2, DllTest2 and got these results:
am11 commentedon Apr 24, 2020
DllTest2 is the first one (in master), that is why it was failing.
runtime/src/coreclr/src/pal/tests/palsuite/exception_handling/pal_sxs/test1/exceptionsxs.cpp
Lines 97 to 100 in af36c6d
Now I have put DllTest1 before 2, so DllTest1 is failing. Reordering has no effect.
janvorli commentedon Apr 24, 2020
Oh, I have somehow misread your debugging log, I have thought that the 1st one was succeeding and the 2nd one was failing.
So it seems that the OS is not filling in the siginfo->si_addr. Could it be an OS bug / feature?
am11 commentedon Apr 24, 2020
I tried to isolate the issue, but could not reproduce it with this program: http://sprunge.us/6rXQXO
curl -s http://sprunge.us/6rXQXO | g++ -xc++ -std=c++11 - ; ./a.outcorrectly prints:
Address is: 0x22(followed bySegmentation Fault (core dumped)).am11 commentedon Apr 26, 2020
Another strange thing is that line 65 condition only fails under the debugger, so it is not related to the actual test failure.
I commented out two out of three tests:
// Test catching exceptions in other PAL instances DllTest2(); - DllTest1(); - DllTest2(); +// DllTest1(); +// DllTest2();and got a crash on second sigsegv (in chaining):
(exit code 139)
the truss output looks like this: https://gist.github.com/am11/682e8f2a6b22a551e78004506774047f.
For comparison, here is the output of truss on FreeBSD x64: https://gist.github.com/am11/419ca311e2219c7f033155e2d2a44c6d
am11 commentedon Apr 26, 2020
Added few print messages:
on FreeBSD, we get:
on SmartOS, we get:
and it continues, until process is terminated. Looks like the second sigsegv is initiating a never ending loop.
jclulow commentedon Apr 26, 2020
Does the runtime make use of in-process profiling through
setitimer(ITIMER_PROF)andSIGPROFat all? If so, you may need to know about illumos bug 11494.jclulow commentedon Apr 26, 2020
Also I would be very careful using system(3C) in a signal handler, even just for debugging. It is listed with an MT-Level attribute of Unsafe, which as per my reading of attributes(5) means it is not safe for use when multiple threads or signals are at play. If it were safe in a signal handler, I would expect it to be marked Async-Signal-Safe.
am11 commentedon Apr 26, 2020
Afaict, setitimer is not used in coreclr code; the only use of setitimer is in one of the libunwind's test code, which is passing in upstream repo on SmartOS 2020. Also, SIGPROF is not used in coreclr (mono's POSIX library uses it in one of its APIs, but that is separate).
Thanks, i have removed it. It was just for a quick sneak peek in the absence of printf/std::cout in that context (as PAL have some of the printer functions redefined, and tests are selectively linked with components).
The overview of this test failure is as follow:
libcoreclrpal.so: PAL library that has generic signal handlers for various signals, including SIGSEGV (and try/catch macros).libpaltest_pal_sxs_test1_dll2.so: test lib that causes SIGSEGV for testing the handler. links with libcorepal.paltest_pal_sxs_test1: the main / glue executable program, which links withlibpaltest_pal_sxs_test1_dll2. It also defines its own signal handler for SIGSEGV, and expects the signal calls to chain when SIGSEGV is raised from the program's own code, but get stuck in an infinite loop of calling the first handler from PAL repeatedly and never calls the second one; as we can see in truss outputs, SmartOS vs. FreeBSD: paltest_pal_sxs_test1 failed on SmartOS x64 #35362 (comment).ps - aside from
pthread_setschedparam's inability to raise priority after lowering it from same value in case of non-privileged execution, which will likely be fixed upstream (and not too critical for now, as it only causes an assertion to fail in runtime code), this is the last failing (out of 716) PAL test.jclulow commentedon Apr 28, 2020
To investigate this further, I see that your error message tells you that the expected value was not detected -- but it'd be good to print the value that was detected. In addition, our
ucontext_tcontains auc_linkmember that may beNULLor may be a pointer to a parent signal handling context. It might help to walk up that chain, at least now during debugging, to see if the context you're looking for is further up.It might help to ask the runtime linker which shared object a particular PC value comes from, too, to see which frame was interrupted; e.g.,
am11 commentedon Apr 28, 2020
I agree, we should emit the faulting address at
src/coreclr/src/pal/tests/palsuite/exception_handling/pal_sxs/test1/dlltest2.cpp#67, in the failure message. However, that (error) condition only meets, when we run it under the debugger. The main problem in normal runs (when debugger is not attached) is with the signal handler chaining; if two signal handlers are associated with SIGSEGV, in different objects, then the first one keeps getting fired.Thank you. I will try to walk up the chain to find more details. Looks like Linux/BSD also define
uc_linkinucontext_t. Is there any well-known difference in Illumos around that area? We have an aliastypedef ucontext_t native_context_t;and use that alias in PAL'scommon_signal_handler.Could you give some pointers on how -- which linker flag to set in order to obtain this shared object to PC/IP mappings? We are using Solaris native ld (as opposed to
gld, which is in PATH of some distros but not necessarily present everywhere).am11 commentedon Apr 29, 2020
@jclulow, if you want to give it a try, you can use my branch with libunwind upgrade commit:
feature/solaris/pal-test-fixes. On SmartOS 20200408T231825Z x64 userland, my steps are:sudo pkgin -y install git mozilla-rootcerts cmake icu py37-expat gcc7 gmake gdb-7 git clone https://github.com/am11/runtime --branch feature/solaris/pal-test-fixes --single-branch --depth 1 # skip everything and only build coreclr with PAL tests runtime/src/coreclr/build-runtime.sh -skipgenerateversion -nopgooptimize \ -cmakeargs -DCLR_CMAKE_BUILD_TESTS=1 -gccwith seven physical cores, the build took 11m33.128s with clean tree; subsequent runs (without git-clean) are faster.
The test executable
paltest_pal_sxs_test1is inartifacts/obj/coreclr/SunOS.x64.Debug/src/pal/tests/palsuite/exception_handling/pal_sxs/test1directory.JulieLeeMSFT commentedon Jan 24, 2023
Failed again:
Failed tests:
Error message:
JulieLeeMSFT commentedon Jan 24, 2023
@janvorli, it is blocking outerloop in many runs. PTAL.
cc @mangod9.
am11 commentedon Jan 24, 2023
@JulieLeeMSFT, this is a different OS (illumos), while outerloop failure is in linux-musl-arm. Opened #81113.
am11 commentedon Jul 10, 2024
@gwr, @AustinWise, the PAL tests instructions are here: https://github.com/dotnet/runtime/blob/aed5c225ded0fecf98988ac3736d0f6399a82df3/docs/workflow/testing/coreclr/testing.md#pal-tests-macos-and-linux-only (it is meant to be
Unix onlyinstead of linux/mac; something we can reword along the way)