Sporadic SIGSEGV in macOS [scans] workflow in CI #5023

Activity

1. github-actions

   added

   requires-triage

   on Jul 24, 2026
2. peterxcli commented on Jul 24, 2026

   

   peterxcli

   on Jul 24, 2026

   Last edited by peterxcli

   Member

   More actions

   Issue #5023 is an upstream libhdfs JNIEnv use-after-free, not a Parquet failure or a regression from #5014.

   Evidence

   • I downloaded the failing job’s hs_err_pid22862.log and exact native-lib-macos artifact, then symbolized it:
     • pc = 0
     • lr = hdfsThreadDestructor + 80
     • x8 = 0
   • Disassembly shows hdfsThreadDestructor loads the JNI GetJavaVM function pointer into x8, then executes blr x8. The stale JNIEnv contains a null function entry, producing the jump to address zero.
   • The vulnerable call is exactly (*env)->GetJavaVM. Its null checks cannot detect a freed-but-non-null JNIEnv.
   • This precisely matches Apache HDFS-16021: heap-use-after-free in hdfsThreadDestructorand the earlier HDFS-14488. HDFS-16021 remains open without a fix version, and even current Hadoop trunk still contains the vulnerable destructor implementation.

   The Comet path is:

   fake:// → libhdfs scheme → OpenDAL HDFS → hdrs → hdfs-sys 0.3.0 → vendored libhdfs

   The test explicitly registers fake as a libhdfs scheme in ParquetReadFromFakeHadoopFsSuite.scala. Native routing occurs in is_hdfs_scheme and create_hdfs_object_store. macOS explicitly enables vendored hdrs in Cargo.toml, while Cargo.lock pins hdfs-sys 0.3.0.

   Recommended fix

   Carry or upstream the HDFS-16021 patch: record whether libhdfs itself attached the thread, and only call GetJavaVM/DetachCurrentThread when libhdfs owns that attachment. No released hdfs-sys upgrade currently includes it.

   Assist by LLM

   ❤️React with ❤️1Reacted by Andy Grove

   👀React with 👀1Reacted by Oleks V
3. andygrove

   added

   bugSomething isn't workingSomething isn't working

   priority:highCrashes, panics, segfaults, major functional breakageCrashes, panics, segfaults, major functional breakage

   crashNative engine crash/panic/segfaultNative engine crash/panic/segfault

   area:ciCI/CD, GitHub Actions, build toolingCI/CD, GitHub Actions, build tooling

   area:scanParquet scan / data readingParquet scan / data reading

   and removed

   requires-triage

   on Jul 27, 2026
4. kazuyukitanimura

   modified the milestones: 1.0.0, 1.1.0 on Jul 27, 2026
5. comphead commented on Sep 9, 2026

   

   comphead

   on Sep 9, 2026

   Contributor

   More actions

   Adding a Linux/x86-64/JDK 11 occurrence. This issue is titled and evidenced as macOS, but the same crash reproduces on ubuntu-24.04, which I think widens the scope rather than adding a duplicate.

   Run: 34404348583, job PR Build (Linux) / Spark 3.4, JDK 11, Scala 2.12 [scans], on an unrelated CI-config PR (#5782).

   Same signature

   SIGSEGV (0xb) at pc=0x0000000000000000, pid=1317, tid=4739
   JRE: Zulu11.90+19-CA (11.0.32+9), linux-amd64   (vs Zulu17 / bsd-aarch64 above)
   Problematic frame:  C  0x0000000000000000
   Current thread is native thread
   siginfo: si_signo: 11 (SIGSEGV), si_code: 1 (SEGV_MAPERR), si_addr: 0x0000000000000000
   

   Consistent with @peterxcli's lr = hdfsThreadDestructor+80, x8 = 0 on aarch64:

   • RIP = 0x0, si_addr = 0x0 — jump to address zero.
   • RDI == R15 == 0x00007f6d4003c350. On the SysV x86-64 ABI RDI is the first argument, which is the JNIEnv* position for (*env)->GetJavaVM(env, ...).
   • Stack used is ~4.8 KB of a 4 MB thread stack, i.e. a handful of frames from thread entry. That is what a pthread TLS destructor at thread exit looks like.
   • The return address on top of stack, 0x00007f6ccbc978df, and a deeper entry, 0x00007f6cd0b6f47e, both resolve into the r-xp text segment of /spark/target/tmp/libcomet-4125068328974464746.so (base 0x7f6cca000000, so file offsets 0x1c978df and 0x6b6f47e).

   One caveat: I could not symbolize those offsets, so the hdfsThreadDestructor attribution here is by signature and register shape, not by symbol. The dump is also missing its thread list entirely, because error reporting itself SIGSEGV'd twice inside libjvm.so:

   [error occurred during error reporting (printing register info), id 0xb, SIGSEGV (0xb) at pc=0x00007f6d90c22238]
   [error occurred during error reporting (inspecting top of stack), id 0xb, SIGSEGV (0xb) at pc=0x00007f6d90c22238]
   [thread 17129 also had an error]
   

   Not resource exhaustion: G1 heap was 727 MB used of 1432 MB committed against -Xmx4g, host had 10.1 GB MemAvailable and completely untouched swap.

   Why it is not macOS-specific

   native/core/Cargo.toml:

   default = ["hdfs-opendal"]
   hdfs-opendal = ["opendal", "object_store_opendal", "hdfs-sys"]

   hdfs-sys (with hdfs_3_3, vendoring the libhdfs C that contains the destructor) is a default feature on every platform. Only hdrs = { features = ["vendored"] } is macOS-gated. So every default libcomet build carries the vulnerable hdfsThreadDestructor, and ParquetReadFromFakeHadoopFsSuite is in the [scans] bucket of both pr_build_linux.yml and pr_build_macos.yml.

   The destructor fires in a later suite than the one that arms it

   This may be the more actionable part. In the macOS reports the crash lands right at the ParquetReadFromFakeHadoopFsSuite to ParquetTimestampLtzAsNtzSuite boundary. Here it landed 4m03s into the fork, during CometIcebergNativeSuite, long after the fake-fs suite finished.

   The hs_err command line confirms both suites were in the same JVM, and SparkTestSuite.txt shows the order:

   CometParquetWriterSuite: → ParquetReadV1Suite: → ParquetReadFromFakeHadoopFsSuite:
   → ParquetTimestampLtzAsNtzSuite: → CometNativeReaderSuite: → CometIcebergNativeSuite:   ← crash here
   

   Last test to complete was runtime filtering - join with dynamic partition pruning.

   So the failure is not bounded to the suite that registers the fake libhdfs scheme. That suite attaches libhdfs to whatever pooled threads it touches, and the TLS destructor only runs when one of those threads eventually exits, which can be minutes later in an unrelated suite. Any suite sharing the fork is a candidate crash site, which is probably why this reads as a random [scans] flake rather than something pinned to one test.

   Incidental, and I do not think it is causal, but recording it: the final 9 seconds before the crash show 1269 CodecPool: Got brand-new compressor [.zstd] at ~100/s on a single executor task, meaning compressors are never being returned to the pool. Possibly a separate minor leak.

   Repro artifacts

   The run's crash-logs-*, unit-tests-* and java-test-reports-* artifacts for that job carry hs_err_pid1317.log, unit-tests.log and the surefire output, and native-lib-linux from the same run is the exact binary that crashed, so the offsets above should symbolize directly:

   gh run download 34404348583 --repo apache/datafusion-comet -p '*3.4*scans*'
   gh run download 34404348583 --repo apache/datafusion-comet -n native-lib-linux
   addr2line -f -C -e libcomet.so 0x1c978df 0x6b6f47e

   Nothing here changes the recommended fix; it just argues the title should lose "macOS" and that the priority is a bit higher than a single-platform flake.

   Analysis assisted by LLM.
6. andygrove

   closed this as completedin #5892on Sep 13, 2026
7. andygrove

   marked macOS aarch64 flake: SIGBUS in _pthread_tsd_cleanup after ParquetReadFromFakeHadoopFsSuite #4200 as a duplicate of this issue on Sep 28, 2026