Fix: HUNG-dump signals forked descendants + drains pump before tail by hw-native-sys-bot · Pull Request #765 · hw-native-sys/simpler

hw-native-sys-bot · 2026-05-13T02:37:38Z

Summary

The dispatcher's session-timeout handler in the root conftest.py was
producing empty HUNG groups for the common deadlock pattern in sim
distributed tests. Caught in CI run
25773352155
on PR #763: test_ep_dispatch_combine hung for 585.9 s, the HUNG body
contained only the pytest startup banner and no faulthandler traceback
— zero diagnostic value.

Root cause (two bugs)

Signal only goes to the dispatched pid, not its descendants.
python/simpler/worker.py::_start_hierarchical os.forks
ChipWorker / SubWorker / next-level children at lines 833, 854, 895.
When the actual deadlock is in one of those forked grandchildren,
the dispatched pytest is calmly waiting in waitpid — useful frame
for that pid, but the stuck thread (in the grandchild) sees no
signal and faulthandler never dumps from where the bug lives.
Tail buffer is read without joining the pump thread. After
time.sleep(2.0), the handler did
tail = "".join(rj.output_lines[-200:]) directly. Every other path
in parallel_scheduler.py (_reap_one line 265, the cancel path
line 322) calls rj.pump_thread.join(timeout=2.0) first. Bytes that
the faulthandler signal trampoline had written to the pipe but the
daemon pump hadn't yet spliced into output_lines were lost when
SIGTERM closed the pipe right after.

Fix

New _collect_descendant_pids(pid) in conftest.py that walks
/proc/<pid>/task/*/children BFS. Best-effort: empty list on
non-Linux and on pid races; no exception.
The session-timeout handler now signals the dispatched pid and
every descendant, so the forked grandchild's faulthandler fires
where the deadlock actually is.
HUNG group header now includes descendants=[<pids>] — useful even
when the body itself is sparse.
rj.pump_thread.join(timeout=0.5) before reading the tail buffer,
mirroring the convention from _reap_one / cancel.

Tests

tests/ut/py/test_session_timeout.py:

Test	What it verifies
`test_collect_descendant_pids_sees_fork_tree`	Fork a child that forks a grandchild; both pids appear
`test_collect_descendant_pids_returns_empty_for_dead_pid`	`/proc` walk on a reaped pid returns `[]` (no exception)
`test_session_timeout_surfaces_forked_child_traceback`	End-to-end: real `_install_session_timeout` handler + a forking deadlocking pytest target; HUNG body contains `_grandchild_deadlock_sentinel` frame and header has `descendants=[...]`

All three skip on non-Linux (no /proc).

Reverted the fix locally to confirm the regression tests really exercise
it: without the change, all 3 fail (the importlib load fails on the
helper before reaching the assertions).

Known gap (not fixed here)

pytest's default --capture=fd dup2s the test's stderr onto a
capture pipe that is never flushed when the test hangs. The
descendants fix correctly gets faulthandler to fire in the right
process, but its writes still land in pytest's capture and never
reach the parent's pump unless the dispatched pytest is invoked with
-s. The e2e test uses -s to isolate the descendants/pump logic.

A follow-up could route faulthandler to a pre-capture dup of fd 2
(e.g., os.dup(2) at conftest module load, passed as file=fd to
faulthandler.register). I prototyped this but the timing of pytest's
capture initialization vs. conftest module load is subtle enough that
I want to verify it carefully in a separate PR rather than bundle.

Test plan

python -m pytest tests/ut/py/test_session_timeout.py -v passes
locally (3/3).
Same suite fails 3/3 with the conftest changes reverted (proves
the tests really exercise the fix).
Pre-commit hooks (ruff check, ruff format, pyright,
check-headers) pass on both touched files; no SKIP=.
Full CI run on this branch.

gemini-code-assist

Code Review

This pull request introduces a mechanism to collect and signal descendant processes during a session timeout, ensuring that faulthandler tracebacks from deadlocked sub-processes are captured. It adds a new utility to walk the Linux /proc tree and updates the timeout handler to drain output buffers before reporting hung jobs. Feedback includes correcting the tree traversal from DFS to the documented BFS, optimizing the process lookup complexity, reducing the join timeout for output pump threads to minimize cumulative delays, and refining the cleanup logic in the new unit tests.

…dlock The session-timeout handler in the root conftest sends SIGUSR1 to each running dispatched pytest pid and relies on the child's installed faulthandler trampoline to dump all-thread tracebacks into the HUNG group. Two latent bugs made this useless for the common deadlock pattern in sim distributed tests (e.g. test_ep_dispatch_combine): 1. L3 ``Worker._start_hierarchical`` (python/simpler/worker.py:833,854,895) ``os.fork``s ChipWorker / SubWorker / next-level children. When the real deadlock lives in one of those forked grandchildren, signaling only the dispatched pytest pid hits a process that is calmly waiting in ``waitpid`` — useful frame, but the actual stuck thread (in the grandchild) sees no signal and faulthandler never fires there. 2. After the 2 s drain sleep, the handler read ``rj.output_lines[-200:]`` directly without joining the pump thread, while every other reader in parallel_scheduler.py (``_reap_one`` line 265, the cancel path line 322) does ``pump_thread.join(timeout=2.0)`` first. Bytes that had landed in the OS pipe but not yet in ``output_lines`` were dropped on the floor when SIGTERM closed the pipe right after. Net result for the symptom that motivated this work — the ``test_ep_dispatch_combine`` hang in CI run 25773352155 — was an empty HUNG group body and no actionable diagnostic. Fix: - Add ``_collect_descendant_pids(pid)`` walking ``/proc/<pid>/task/*/children`` in BFS. Best-effort: empty list on non-Linux (macOS sim/UT) or on pid races. The handler signals the dispatched pid AND every descendant. - Annotate the HUNG header with ``descendants=[...]`` so a reviewer can see at a glance that the dispatched pid had forked children and how many — useful even when the body itself is sparse. - Call ``rj.pump_thread.join(timeout=0.5)`` before reading the tail buffer, mirroring the convention from ``_reap_one`` / cancel. Tests (tests/ut/py/test_session_timeout.py): - ``test_collect_descendant_pids_sees_fork_tree`` — fork a child that forks a grandchild, assert both pids appear. - ``test_collect_descendant_pids_returns_empty_for_dead_pid`` — ``/proc`` walk on a reaped pid returns [], no exception. - ``test_session_timeout_surfaces_forked_child_traceback`` — end-to-end repro: spawn a real dispatcher subprocess running a deadlocking forking pytest target, install the real ``_install_session_timeout`` handler, let SIGALRM fire, and assert the HUNG body contains the grandchild's ``_grandchild_deadlock_sentinel`` faulthandler frame and ``descendants=[...]`` in the header. Skipped on non-Linux. Known gap not addressed here: pytest's default ``--capture=fd`` dup2's the test's stderr onto a capture pipe that is never flushed if the test hangs. The descendants fix gets faulthandler firing in the right process, but its writes still land in pytest's capture pipe unless the dispatched pytest is invoked with ``-s`` (the e2e test uses ``-s`` to isolate the descendants/pump logic). Routing faulthandler to a pre-capture dup of fd 2 is a follow-up.

Fixes hw-native-sys#1484 The example carried pypto codegen last refreshed on 2026-06-25, cited a provenance file upstream had deleted (`models/qwen3/14b/decode_layer.py`), and covered 2 of Qwen3-14B's 40 layers. Re-harvested from `decode_fwd.py`'s `decode_fwd_layers` at pypto-lib `45be52c` with `_CHUNK_NLAYERS = 40`, and the README now records that commit plus the ptoas and pto-isa pins so the next drift is detectable rather than archaeological. Scaling to the full model is nearly free. The generated orchestration loops over layers (`for (int64_t i = 0; i < 40; i += 1)`) instead of unrolling them, so going from 2 to 40 moves a trip count and leaves the kernel set the same size. The lib chunks its dispatches because a 40-layer one used to exceed a 2000 ms stream-sync timeout; simpler raised that to 50 s in hw-native-sys#1175 and a 40-layer step is ~40 ms, so the constraint is gone. What did change is attention. Since pypto-lib hw-native-sys#765 and hw-native-sys#796 it is a CANN FusedInferAttentionScore extern rather than generated kernels, folding per-head QK-norm, RoPE, the paged KV write, the flash inner loop and the online softmax into one mixed AIC+2xAIV task behind an FFTS barrier. Seven kernels go away (`rope_qkv`, `qk_gamma`, `qk_recip`, `fa_work_build`, `fa_fused_aic`/`_aiv`, `online_softmax`, `attn_fence`) and the harvested extern arrives under `kernels/paged_attention_cce/`. Its `vendor/` subtree is CANN source under Huawei copyright; the tree is copied verbatim, because `kernel/fai_body.hpp` reaches its dependencies by relative include and splitting it would mean re-patching on every refresh. Verbatim also means our header, formatting and language lint must leave it alone, hence the carve-out. That extern also moves the paged KV layout from NSND to vLLM's BSND (`[page, token, kv_head, dim]`, so `slot_mapping` is directly the row index), which the golden now models for both the attention read and the KV write-back. The previous harvest needed one hand-edit, replacing a `[[block_local]] static` in `fa_fused_aiv` that the AICore loader rejected. That kernel no longer exists and the current codegen emits no such construct, so this harvest is unmodified codegen throughout. Verification: the case passes on device with output and all 40 layers' KV caches matching the torch reference at RTOL=5e-2 / ATOL=1e-1, in ~5 min wall on an idle die, with no ring-sizing overrides — per-layer intermediates live inside that iteration's scope, so the live set does not grow with layer count. The fixture is 38.05 GiB (24.61 weights + 13.44 paged KV) against 64 GiB of HBM. The l0-swimlane docs used this example as their "real SPMD workload", shrinking the generated fa_fused via `--set-arg 0=96` on its work-item count. The extern takes its work distribution from runtime tiling metadata instead, so no `--set-arg` shrinks it and those references are removed rather than repointed. The dfx buffer-capacity row measured at 2 layers is dropped for the same reason: it would now be a stale number rather than a wrong one. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Fixes hw-native-sys#1484 The example carried pypto codegen last refreshed on 2026-06-25, cited a provenance file upstream had deleted (`models/qwen3/14b/decode_layer.py`), and covered 2 of Qwen3-14B's 40 layers. Re-harvested from `decode_fwd.py`'s `decode_fwd_layers` at pypto-lib `45be52c` with `_CHUNK_NLAYERS = 40`, and the README now records that commit plus the ptoas and pto-isa pins so the next drift is detectable rather than archaeological. Scaling to the full model is nearly free. The generated orchestration loops over layers (`for (int64_t i = 0; i < 40; i += 1)`) instead of unrolling them, so going from 2 to 40 moves a trip count and leaves the kernel set the same size. The lib chunks its dispatches because a 40-layer one used to exceed a 2000 ms stream-sync timeout; simpler raised that to 50 s in hw-native-sys#1175 and a 40-layer step is ~40 ms, so the constraint is gone. What did change is attention. Since pypto-lib hw-native-sys#765 and hw-native-sys#796 it is a CANN FusedInferAttentionScore extern rather than generated kernels, folding per-head QK-norm, RoPE, the paged KV write, the flash inner loop and the online softmax into one mixed AIC+2xAIV task behind an FFTS barrier. Seven kernels go away (`rope_qkv`, `qk_gamma`, `qk_recip`, `fa_work_build`, `fa_fused_aic`/`_aiv`, `online_softmax`, `attn_fence`) and the harvested extern arrives under `kernels/vendor/paged_attention_cce/`. That extern needs CANN devkit headers, which nothing could express: every incore compiled against the same fixed include set, even though `KernelCompiler.compile_incore` has always taken `extra_include_dirs`. The scene-test `CALLABLE` now carries a per-incore `extra_include_dirs`, resolved when the kernel is compiled rather than when its module is imported — sim and macOS runners collect a case declaring a CANN dependency without having CANN. Entries may use `$VAR` and may name directories that do not exist here, so one candidate list spans SDK layouts; an unset variable or a set where nothing survived raises, which beats a "file not found" from four includes deep in vendored code. The harvested tree is copied verbatim and must stay that way: `fai_body.hpp` reaches its dependencies by relative include, and reformatting it would make the next refresh diff against our copy instead of against upstream — the exact drift this example exists to expose. It therefore lives under `kernels/vendor/`, and the repo's header, formatting and language lint skip anything below a `vendor/` directory. Keying the carve-out on the directory rather than on this operator's name means the next harvested extern needs no lint change at all. The extern also moves the paged KV layout from NSND to vLLM's BSND (`[page, token, kv_head, dim]`, so `slot_mapping` is directly the row index), which the golden now models for both the attention read and the KV write-back. The previous harvest needed one hand-edit, replacing a `[[block_local]] static` in `fa_fused_aiv` that the AICore loader rejected. That kernel no longer exists and the current codegen emits no such construct, so this harvest is unmodified codegen throughout. At 40 layers the case takes ~5 min against a 38.05 GiB fixture (24.61 weights + 13.44 paged KV, bf16) — over half the onboard sweep's 600 s session budget. It runs as its own CI step on a dedicated device instead, keeping that sweep's short timeout meaningful as a hang detector. Verification: passes on device with output and all 40 layers' KV caches matching the torch reference at RTOL=5e-2 / ATOL=1e-1, with no ring-sizing overrides — per-layer intermediates live inside that iteration's scope, so the live set does not grow with layer count. The l0-swimlane docs used this example as their "real SPMD workload", shrinking the generated fa_fused via `--set-arg 0=96` on its work-item count. The extern takes its work distribution from runtime tiling metadata instead, so no `--set-arg` shrinks it and those references are removed rather than repointed. The dfx buffer-capacity row measured at 2 layers is dropped for the same reason: it would now be a stale number rather than a wrong one. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

) Fixes #1484 The example carried pypto codegen last refreshed on 2026-06-25, cited a provenance file upstream had deleted (`models/qwen3/14b/decode_layer.py`), and covered 2 of Qwen3-14B's 40 layers. Re-harvested from `decode_fwd.py`'s `decode_fwd_layers` at pypto-lib `45be52c` with `_CHUNK_NLAYERS = 40`, and the README now records that commit plus the ptoas and pto-isa pins so the next drift is detectable rather than archaeological. Scaling to the full model is nearly free. The generated orchestration loops over layers (`for (int64_t i = 0; i < 40; i += 1)`) instead of unrolling them, so going from 2 to 40 moves a trip count and leaves the kernel set the same size. The lib chunks its dispatches because a 40-layer one used to exceed a 2000 ms stream-sync timeout; simpler raised that to 50 s in #1175 and a 40-layer step is ~40 ms, so the constraint is gone. What did change is attention. Since pypto-lib #765 and #796 it is a CANN FusedInferAttentionScore extern rather than generated kernels, folding per-head QK-norm, RoPE, the paged KV write, the flash inner loop and the online softmax into one mixed AIC+2xAIV task behind an FFTS barrier. Seven kernels go away (`rope_qkv`, `qk_gamma`, `qk_recip`, `fa_work_build`, `fa_fused_aic`/`_aiv`, `online_softmax`, `attn_fence`) and the harvested extern arrives under `kernels/vendor/paged_attention_cce/`. That extern needs CANN devkit headers, which nothing could express: every incore compiled against the same fixed include set, even though `KernelCompiler.compile_incore` has always taken `extra_include_dirs`. The scene-test `CALLABLE` now carries a per-incore `extra_include_dirs`, resolved when the kernel is compiled rather than when its module is imported — sim and macOS runners collect a case declaring a CANN dependency without having CANN. Entries may use `$VAR` and may name directories that do not exist here, so one candidate list spans SDK layouts; an unset variable or a set where nothing survived raises, which beats a "file not found" from four includes deep in vendored code. The harvested tree is copied verbatim and must stay that way: `fai_body.hpp` reaches its dependencies by relative include, and reformatting it would make the next refresh diff against our copy instead of against upstream — the exact drift this example exists to expose. It therefore lives under `kernels/vendor/`, and the repo's header, formatting and language lint skip anything below a `vendor/` directory. Keying the carve-out on the directory rather than on this operator's name means the next harvested extern needs no lint change at all. The extern also moves the paged KV layout from NSND to vLLM's BSND (`[page, token, kv_head, dim]`, so `slot_mapping` is directly the row index), which the golden now models for both the attention read and the KV write-back. The previous harvest needed one hand-edit, replacing a `[[block_local]] static` in `fa_fused_aiv` that the AICore loader rejected. That kernel no longer exists and the current codegen emits no such construct, so this harvest is unmodified codegen throughout. At 40 layers the case takes ~5 min against a 38.05 GiB fixture (24.61 weights + 13.44 paged KV, bf16) — over half the onboard sweep's 600 s session budget. It runs as its own CI step on a dedicated device instead, keeping that sweep's short timeout meaningful as a hang detector. Verification: passes on device with output and all 40 layers' KV caches matching the torch reference at RTOL=5e-2 / ATOL=1e-1, with no ring-sizing overrides — per-layer intermediates live inside that iteration's scope, so the live set does not grow with layer count. The l0-swimlane docs used this example as their "real SPMD workload", shrinking the generated fa_fused via `--set-arg 0=96` on its work-item count. The extern takes its work distribution from runtime tiling metadata instead, so no `--set-arg` shrinks it and those references are removed rather than repointed. The dfx buffer-capacity row measured at 2 layers is dropped for the same reason: it would now be a stale number rather than a wrong one.

…dlock (hw-native-sys#765) The session-timeout handler in the root conftest sends SIGUSR1 to each running dispatched pytest pid and relies on the child's installed faulthandler trampoline to dump all-thread tracebacks into the HUNG group. Two latent bugs made this useless for the common deadlock pattern in sim distributed tests (e.g. test_ep_dispatch_combine): 1. L3 ``Worker._start_hierarchical`` (python/simpler/worker.py:833,854,895) ``os.fork``s ChipWorker / SubWorker / next-level children. When the real deadlock lives in one of those forked grandchildren, signaling only the dispatched pytest pid hits a process that is calmly waiting in ``waitpid`` — useful frame, but the actual stuck thread (in the grandchild) sees no signal and faulthandler never fires there. 2. After the 2 s drain sleep, the handler read ``rj.output_lines[-200:]`` directly without joining the pump thread, while every other reader in parallel_scheduler.py (``_reap_one`` line 265, the cancel path line 322) does ``pump_thread.join(timeout=2.0)`` first. Bytes that had landed in the OS pipe but not yet in ``output_lines`` were dropped on the floor when SIGTERM closed the pipe right after. Net result for the symptom that motivated this work — the ``test_ep_dispatch_combine`` hang in CI run 25773352155 — was an empty HUNG group body and no actionable diagnostic. Fix: - Add ``_collect_descendant_pids(pid)`` walking ``/proc/<pid>/task/*/children`` in BFS. Best-effort: empty list on non-Linux (macOS sim/UT) or on pid races. The handler signals the dispatched pid AND every descendant. - Annotate the HUNG header with ``descendants=[...]`` so a reviewer can see at a glance that the dispatched pid had forked children and how many — useful even when the body itself is sparse. - Call ``rj.pump_thread.join(timeout=0.5)`` before reading the tail buffer, mirroring the convention from ``_reap_one`` / cancel. Tests (tests/ut/py/test_session_timeout.py): - ``test_collect_descendant_pids_sees_fork_tree`` — fork a child that forks a grandchild, assert both pids appear. - ``test_collect_descendant_pids_returns_empty_for_dead_pid`` — ``/proc`` walk on a reaped pid returns [], no exception. - ``test_session_timeout_surfaces_forked_child_traceback`` — end-to-end repro: spawn a real dispatcher subprocess running a deadlocking forking pytest target, install the real ``_install_session_timeout`` handler, let SIGALRM fire, and assert the HUNG body contains the grandchild's ``_grandchild_deadlock_sentinel`` faulthandler frame and ``descendants=[...]`` in the header. Skipped on non-Linux. Known gap not addressed here: pytest's default ``--capture=fd`` dup2's the test's stderr onto a capture pipe that is never flushed if the test hangs. The descendants fix gets faulthandler firing in the right process, but its writes still land in pytest's capture pipe unless the dispatched pytest is invoked with ``-s`` (the e2e test uses ``-s`` to isolate the descendants/pump logic). Routing faulthandler to a pre-capture dup of fd 2 is a follow-up. Co-authored-by: Chao Wang <26245345+ChaoWao@users.noreply.github.com>

gemini-code-assist Bot reviewed May 13, 2026

View reviewed changes

Comment thread conftest.py Resolved

Comment thread conftest.py Resolved

Comment thread tests/ut/py/test_session_timeout.py Resolved

hw-native-sys-bot force-pushed the fix/hung-dump-descendants-and-pump-drain branch from 59402f9 to 1c9ff75 Compare May 13, 2026 02:50

ChaoWao approved these changes May 13, 2026

View reviewed changes

ChaoWao merged commit 540d924 into hw-native-sys:main May 13, 2026
14 checks passed

ChaoWao deleted the fix/hung-dump-descendants-and-pump-drain branch May 13, 2026 03:30

ChaoWao mentioned this pull request Jul 26, 2026

Update: re-harvest qwen3_14b_decode and scale it to all 40 layers #1506

Merged

Navigation Menu

Repository navigation

Fix: HUNG-dump signals forked descendants + drains pump before tail - #765

Fix: HUNG-dump signals forked descendants + drains pump before tail#765
ChaoWao merged 1 commit into
hw-native-sys:mainfrom
hw-native-sys-bot:fix/hung-dump-descendants-and-pump-drain

hw-native-sys-bot commented May 13, 2026

Uh oh!

gemini-code-assist Bot left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

hw-native-sys-bot commented May 13, 2026

Summary

Root cause (two bugs)

Fix

Tests

Known gap (not fixed here)

Test plan

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants