Skip to content

Fix: prevent poisoned chip-run lane reuse #1918

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ChaoZheng109 merged 1 commit into from
Aug 24, 2026
Merged

Fix: prevent poisoned chip-run lane reuse #1918

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
78 changes: 42 additions & 36 deletions conftest.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1255,19 +1255,17 @@ def pytest_sessionfinish(session, exitstatus): # noqa: ARG001
# physical device. The Worker records actual SDMA dispatch attempts; merely
# registering an unused SDMA callable does not rotate the pool.
#
# The driver-side guard
# (``DeviceRunner::run`` fail-fast on ``device_unusable_``) keeps that cascade
# from wedging, but the Worker stays poisoned.
#
# On a5 the poison survives close()+device-reset for the life of the process
# (an in-process Worker.init rebuild fails with rtStreamCreate 507899; a
# force-reset is unsafe on this shared box), so there is no in-process
# recovery — only a fresh worker process gets a clean device. The native status
# query tells the ``st_worker`` finalizer when to drop a poisoned or
# SDMA-exposed pooled Worker; the hook below also stashes setup/call exceptions
# as a diagnostic fallback. The L2 branch rebuilds where the arch allows it and otherwise
# skips the remaining tests for that runtime (``_l2_poisoned``). Golden
# mismatches / ordinary assertions leave a non-SDMA Worker reusable.
# The driver-side guard (``DeviceRunner::run`` fail-fast on
# ``device_unusable_``) keeps that cascade from wedging. Fatal onboard teardown
# force-resets the exclusively allocated card with ``aclrtResetDeviceForce``
# and probes the new device generation before confirming recovery. A successful
# reset lets the next Worker initialize in the same process; an unconfirmed
# reset or failed rebuild marks the runtime in ``_l2_poisoned`` and falls back
# to a fresh-process retry. For runtime failures, the hook below preserves the
# setup/call exception text and the ``st_worker`` finalizer uses the retirement
# classifier below to decide whether the pooled Worker can be reused. Exception
# wording is therefore part of that decision input, not merely diagnostic.
# Golden mismatches and ordinary assertions leave a non-SDMA Worker reusable.
# ---------------------------------------------------------------------------

# Plain attribute name on the test item — NOT pytest.StashKey()/item.stash,
Expand All @@ -1284,18 +1282,17 @@ def pytest_runtest_makereport(item, call):
yield
# A fixture that depends on st_worker may dispatch during setup. Preserve
# that error before st_worker's teardown finalizer decides whether its
# Worker can return to the session pool. Native lifecycle status below is
# authoritative; this text classification remains a diagnostic fallback.
# Worker can return to the session pool; its retirement decision consumes
# this exception text through the classifier below.
if call.when in {"setup", "call"} and call.excinfo is not None:
setattr(item, _ST_CALL_EXCINFO, call.excinfo)


# Only these error codes mean the device/ACL context is sticky-errored — i.e.
# the worker is poisoned and must be rebuilt/skipped. Matching on the bare
# "<op> failed with code" prefix would also catch ordinary kernel/run failures
# (every non-zero rc is wrapped that way), turning a normal failing test into a
# spurious worker rebuild and, downstream, a misleading runtime-wide skip. So
# we extract the trailing <N> and match only the known poison codes:
# Code-bearing native/init failures retire a Worker only when the code identifies
# a sticky device/ACL context. Matching every "<op> failed with code" message
# would also catch ordinary run failures and turn them into spurious rebuilds.
# This code allowlist is separate from the retirement markers below, which
# identify a non-reusable Worker independently of the originating status code:
# 207001 ACL_ERROR_RT_MEMORY_ALLOCATION (the CI cascade trigger)
# 507000 ACL_ERROR_RT_INTERNAL_ERROR (a5 op-timeout at AICPU stream sync)
# 507015 ACL_ERROR_RT_AICORE_EXCEPTION
Expand All @@ -1314,16 +1311,24 @@ def pytest_runtest_makereport(item, call):
# -1..-999 are device-latched codes and must NOT be listed here — a latched
# SCOPE_DEADLOCK (-1) is an orchestration bug, not a poisoned device, and
# treating it as poison is exactly the spurious rebuild this filter avoids.
# Worker surfaces these as "run/run_prepared/prepare_callable/simpler_init
# failed with code <N>" — never as an AssertionError, so golden mismatches are
# already excluded.
# Worker surfaces these as phase-specific "*_native_run failed with code <N>"
# messages or "simpler_init failed with code <N>" — never as an AssertionError,
# so golden mismatches are already excluded.
_DEVICE_POISON_CODES = frozenset({207001, 507000, 507015, 507018, 507046, 507899, -1000})
_DEVICE_ERROR_CODE_RE = re.compile(r"\b(?:run|run_prepared|prepare_callable|simpler_init) failed with code (-?\d+)\b")
_DEVICE_QUARANTINE_MARKERS = (
_DEVICE_ERROR_CODE_RE = re.compile(
r"(?:(?:prepare|launch|poll|finalize)_native_run|simpler_init) failed with code (-?\d+)\b"
)
_L2_WORKER_RETIREMENT_MARKERS = (
"device is quarantined after a prior Worker could not confirm reset",
"device is already owned by another live ChipWorker in this process",
"device reset was not confirmed, so this process quarantined the device",
"cleanup could not confirm device reset, so this process quarantined the device",
# Poll and finalize failures poison the chip run lane independently of their
# runtime status code. A poisoned lane rejects every later submit on the same
# Worker.
"poll_native_run failed with code",
"finalize_native_run failed with code",
"chip run lane is poisoned",
)


Expand Down Expand Up @@ -1358,24 +1363,25 @@ def retire(self, key):
_L2_TERMINAL_RETAINED_WORKERS: list[object] = []


def _is_device_runtime_error_msg(msg: str) -> bool:
def _requires_l2_worker_retirement_msg(msg: str) -> bool:
match = _DEVICE_ERROR_CODE_RE.search(msg)
return (match is not None and int(match.group(1)) in _DEVICE_POISON_CODES) or any(
marker in msg for marker in _DEVICE_QUARANTINE_MARKERS
marker in msg for marker in _L2_WORKER_RETIREMENT_MARKERS
)


def _is_device_runtime_error(excinfo) -> bool:
def _requires_l2_worker_retirement(excinfo) -> bool:
if excinfo is None or not issubclass(excinfo.type, RuntimeError):
return False
return _is_device_runtime_error_msg(str(excinfo.value))
return _requires_l2_worker_retirement_msg(str(excinfo.value))


def _register_l2_pool_recycle(request, pool, key, poisoned_runtimes):
"""Drop + close a pooled L2 Worker after a device-runtime error.
"""Drop + close a pooled L2 Worker after a terminal runtime/lane error.

Recycling prevents a poisoned-context cascade. A passing test and a
non-device failure (golden mismatch, assertion) keep the Worker pooled.
Recycling prevents a poisoned-context or poisoned-lane cascade. A passing
test and a non-terminal failure (golden mismatch, assertion) keep the Worker
pooled.

NOTE: an AICore op-timeout poisons the device context. `DeviceRunner::
finalize()` force-resets the card (`aclrtResetDeviceForce`, per-card safe
Expand All @@ -1393,8 +1399,8 @@ def _recycle():
worker = pool.get(key)
if worker is None:
return
device_error = _is_device_runtime_error(getattr(request.node, _ST_CALL_EXCINFO, None))
if not device_error:
retirement_required = _requires_l2_worker_retirement(getattr(request.node, _ST_CALL_EXCINFO, None))
if not retirement_required:
return
try:
pool.retire(key)
Expand Down Expand Up @@ -1687,7 +1693,7 @@ def st_worker(request, st_platform, device_pool, _l2_worker_pool, _l2_poisoned):
try:
w.init()
except RuntimeError as e:
if _is_device_runtime_error_msg(str(e)):
if _requires_l2_worker_retirement_msg(str(e)):
_l2_poisoned.add(runtime)
_register_l2_poison_skip(request.node)
try:
Expand Down
39 changes: 19 additions & 20 deletions docs/investigations/2026-06-a5-aicore-op-timeout-cascade.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -103,9 +103,9 @@ verified:

This is what the **force-reset recovery** fix below builds on.

## Fix
## Initial containment fix

Two independent guards, both landed:
Two independent guards landed before force-reset recovery:

1. **Driver fail-fast (root-cause containment)** —
`src/a5/platform/onboard/host/device_runner.{h,cpp}`. On any
Expand All @@ -118,19 +118,18 @@ Two independent guards, both landed:
fails immediately at the run() guard (`code -1`, "Rebuild the Worker"),
not at `halResMap`.

2. **Fixture rebuild-then-skip (CI containment)** — `conftest.py`. A
`pytest_runtest_makereport` hook stashes the call-phase exception; the
`st_worker` L2 path heals **only** on a device-runtime `RuntimeError`
(`simpler_run failed …` / `register_callable failed …` / `DeviceRunner
marked unusable` / `simpler_init failed …`), never on golden mismatches.
The heal `close()`s the pooled `Worker` and drops it so the next test
**rebuilds**. Because the a5 rebuild's `Worker.init()` then fails (device
still poisoned — see above), the create path catches that device error,
marks the runtime poisoned (`_l2_poisoned`), and `pytest.skip`s — that
test and every later one for the runtime — with a clear reason. On an
arch where in-process re-init *does* work, the rebuild succeeds and tests
continue; on a5 they skip cleanly. Either way the worker-wide 507899
failure storm is gone.
2. **Fixture rebuild-then-skip (initial CI containment)** — `conftest.py`. A
`pytest_runtest_makereport` hook stashes the call-phase exception. The error
spellings used by the initial classifier have since been replaced by
`prepare_native_run`, `launch_native_run`, `poll_native_run`, and
`finalize_native_run` failure messages plus the
`chip run lane is poisoned: …` prefix. In both versions, the L2 path retires
the pooled Worker only for a terminal device/Worker failure, never for golden
mismatches. Before force-reset recovery landed, the next a5 `Worker.init()`
still failed on the poisoned device, so the create path marked the runtime
poisoned (`_l2_poisoned`) and skipped the remaining tests. This contained
the worker-wide 507899 failure storm; the follow-up recovery below lets those
tests run in the same process when the force reset succeeds.

Verification (pooled L2 cascade through the real `st_worker` fixture: a hang
class poisons, a noop class follows on the same pool key; plus a normal L2
Expand All @@ -145,11 +144,11 @@ class for no-regression):
Each "after" run is "1 failed, 1 skipped": the one real failure
(TestAHangPoison) stands, every later test is a clean skip.

The skip carries its reason, e.g.: *"L2 Worker.init for runtime
'tensormap_and_ringbuffer' failed with a device-runtime error (simpler_init
failed with code 507899); the device context is not recoverable in-process
after an earlier AICore error — skipping remaining 'tensormap_and_ringbuffer'
L2 tests (a fresh worker process recovers)."*
At this historical containment stage, the skip carried a reason such as:
*"L2 Worker.init for runtime 'tensormap_and_ringbuffer' failed with a
device-runtime error (simpler_init failed with code 507899); the device context
is not recoverable in-process after an earlier AICore error — skipping remaining
'tensormap_and_ringbuffer' L2 tests (a fresh worker process recovers)."*

## Force-reset recovery (2026-06-09)

Expand Down
8 changes: 6 additions & 2 deletions docs/troubleshooting/device-error-codes.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,8 +39,12 @@ above.
At most one of `orch_error_code` (1-11) and `sched_error_code` (100+) is ever
non-zero. `runtime_status` is just the latched code negated.

The exception you see (`RuntimeError: run failed with code <rc>`) is **not** always
the same number:
The native-run exception normally has the form
`RuntimeError: <phase>_native_run failed with code <rc>`, where `<phase>` is
`prepare`, `launch`, `poll`, or `finalize`. When that failure leaves the run lane
terminally poisoned, the same error may carry a
`chip run lane is poisoned:` prefix. The reported `<rc>` is **not** always the
same number:

| Environment | `<rc>` |
| ----------- | ------ |
Expand Down
55 changes: 34 additions & 21 deletions src/common/worker/chip_run_lane.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,28 +41,33 @@ struct ChipRunState {
bool crossed_launch_fence{false};
bool depth_one_fallback{false};
std::exception_ptr error;
std::exception_ptr poison_error;
};

struct ChipRunLaneState {
explicit ChipRunLaneState(ChipWorker &worker) :
worker(&worker),
generations(worker.pipeline_depth(), 0) {}

[[noreturn]] static void rethrow_as_poisoned(const std::exception_ptr &error) {
try {
std::rethrow_exception(error);
} catch (const std::exception &e) {
throw std::runtime_error(std::string("chip run lane is poisoned: ") + e.what());
} catch (...) {
throw std::runtime_error("chip run lane is poisoned by an unknown native failure");
}
}

void require_usable() const {
if (closed) throw std::runtime_error("chip run lane is closed");
if (poison != nullptr) {
try {
std::rethrow_exception(poison);
} catch (const std::exception &e) {
throw std::runtime_error(std::string("chip run lane is poisoned: ") + e.what());
} catch (...) {
throw std::runtime_error("chip run lane is poisoned by an unknown native failure");
}
}
if (poison != nullptr) rethrow_as_poisoned(poison);
}

static void rethrow_run_error(const std::shared_ptr<ChipRunState> &run) {
if (run->error != nullptr) std::rethrow_exception(run->error);
void rethrow_run_error(const std::shared_ptr<ChipRunState> &run) const {
if (run->error == nullptr) return;
if (run->poison_error != nullptr) rethrow_as_poisoned(run->poison_error);
std::rethrow_exception(run->error);
}

bool permits_native_successor(const ChipRunState &predecessor, const CallConfig &successor_config) const {
Expand All @@ -83,16 +88,18 @@ struct ChipRunLaneState {
run->disposition = ChipRunPreparationDisposition::NATIVE_PREPARED;
}

void poison_with(std::exception_ptr error) {
void poison_with(const std::shared_ptr<ChipRunState> &run, std::exception_ptr error) {
if (run->poison_error == nullptr) run->poison_error = error;
if (poison == nullptr) poison = error;
}

void finish(const std::shared_ptr<ChipRunState> &run) noexcept {
try {
worker->finalize_native_run(run->native_run);
} catch (...) {
if (run->error == nullptr) run->error = std::current_exception();
poison_with(std::current_exception());
const std::exception_ptr finalize_error = std::current_exception();
if (run->error == nullptr) run->error = finalize_error;
poison_with(run, finalize_error);
}
run->phase = ChipRunState::Phase::TERMINAL;
if (!fifo.empty() && fifo.front() == run) fifo.pop_front();
Expand All @@ -109,9 +116,11 @@ struct ChipRunLaneState {
if (poison != nullptr) {
if (run->phase == ChipRunState::Phase::PREPARED) {
run->error = poison;
run->poison_error = poison;
finish(run);
} else if (run->phase == ChipRunState::Phase::QUEUED) {
run->error = poison;
run->poison_error = poison;
run->phase = ChipRunState::Phase::TERMINAL;
fifo.pop_front();
}
Expand Down Expand Up @@ -179,7 +188,7 @@ struct ChipRunLaneState {
const std::exception_ptr poll_error = std::current_exception();
finish(target);
if (target->error == nullptr) target->error = poll_error;
poison_with(target->error);
poison_with(target, target->error);
launch_front();
return true;
}
Expand All @@ -193,6 +202,7 @@ struct ChipRunLaneState {
const auto run = fifo.front();
if (poison != nullptr && run->phase == ChipRunState::Phase::QUEUED) {
run->error = poison;
run->poison_error = poison;
run->phase = ChipRunState::Phase::TERMINAL;
fifo.pop_front();
return;
Expand All @@ -205,7 +215,10 @@ struct ChipRunLaneState {
launch_front();
if (run->phase == ChipRunState::Phase::TERMINAL) return;
if (run->phase == ChipRunState::Phase::PREPARED && (!run->activated || poison != nullptr)) {
if (poison != nullptr && run->error == nullptr) run->error = poison;
if (poison != nullptr && run->error == nullptr) {
run->error = poison;
run->poison_error = poison;
}
finish(run);
return;
}
Expand All @@ -214,7 +227,7 @@ struct ChipRunLaneState {
worker->wait_native_run(run->native_run);
} catch (...) {
run->error = std::current_exception();
poison_with(run->error);
poison_with(run, run->error);
}
finish(run);
}
Expand All @@ -235,7 +248,7 @@ struct ChipRunLaneState {
} catch (...) {
const std::exception_ptr wait_error = std::current_exception();
if (front->error == nullptr) front->error = wait_error;
poison_with(front->error);
poison_with(front, front->error);
}
finish(front);
launch_front();
Expand Down Expand Up @@ -268,7 +281,7 @@ bool ChipRun::wait_until(Deadline deadline) {
{
std::lock_guard<std::mutex> lk(lane_->mu);
if (lane_->progress(run_)) {
ChipRunLaneState::rethrow_run_error(run_);
lane_->rethrow_run_error(run_);
return true;
}
// An unbounded waiter has no deadline to end its loop, so polling
Expand All @@ -284,7 +297,7 @@ void ChipRun::activate() {
if (lane_ == nullptr || run_ == nullptr) throw std::runtime_error("empty ChipRun handle");
std::lock_guard<std::mutex> lk(lane_->mu);
if (run_->phase == ChipRunState::Phase::TERMINAL) {
ChipRunLaneState::rethrow_run_error(run_);
lane_->rethrow_run_error(run_);
return;
}
run_->activated = true;
Expand All @@ -308,7 +321,7 @@ void ChipRun::abandon() {
lane_->worker->finalize_native_run(run_->native_run);
} catch (...) {
run_->error = std::current_exception();
lane_->poison_with(run_->error);
lane_->poison_with(run_, run_->error);
}
}
run_->phase = ChipRunState::Phase::TERMINAL;
Expand Down
5 changes: 3 additions & 2 deletions tests/st/aicore_op_timeout/test_aicore_op_timeout.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -88,7 +88,7 @@ def _exercise_aicore_timeout(st_platform, st_device_ids, monkeypatch, tmp_path,
# Acceptable error codes for the STARS-killed AICore op. Device status
# takes precedence when finalization can read it; otherwise the host
# falls back to whichever stream-sync error surfaced first:
# -100 = PTO2 scheduler timeout — the device classified the stalled
# -100 = scheduler timeout — the device classified the stalled
# AIC task before finalization read the shared header.
# 507046 = ACL_ERROR_RT_STREAM_SYNC_TIMEOUT — AICore stream's 4 s
# sync budget fires before AICPU sync notices.
Expand All @@ -107,7 +107,8 @@ def _exercise_aicore_timeout(st_platform, st_device_ids, monkeypatch, tmp_path,
# CANN 9.0.0/driver 26.0.rc1 containment deliberately stops the
# SDMA run stream early so reset precedes DEV_RUNNING_DOWN.
error_codes = r"(-100|507(046|018|015|000))"
with pytest.raises(RuntimeError, match=rf"run failed with code {error_codes}"):
native_run_error = rf"finalize_native_run failed with code {error_codes}\b"
with pytest.raises(RuntimeError, match=native_run_error):
worker.run(handle, None, config)
elapsed = time.monotonic() - t0

Expand Down
Loading
Loading