Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions .claude/skills/test-all-device/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,22 +19,22 @@ Detection / isolation procedures referenced below live in
5. **Read the marker selector out of the same job** you took the timeout from.
`st-onboard-a2a3` is two pytest passes, not one: the sweep deselects
`-m "not sdma"` and a later step runs `-m sdma`. `st-onboard-a5` uses
`-m "not pod"` and does not exclude `sdma`. Take the expression from
`--exclude-level 4` and does not exclude `sdma`. Take the expression from
`ci.yml` rather than assuming, so this skill cannot drift from the job it
reproduces.
Comment thread
ChaoWao marked this conversation as resolved.
6. **Run through `task-submit`** (§E). On a5, the underlying command excludes
only pod tests, so SDMA remains in the sweep:
only level-4 tests, so SDMA remains in the sweep:

```bash
pytest examples tests/st -m "not pod" --platform a5 \
pytest examples tests/st --exclude-level 4 --platform a5 \
--device <range-or-$TASK_DEVICE> --pto-session-timeout <timeout> -v
```

On a platform whose job separates SDMA tests, the underlying sweep command
is:

```bash
pytest examples tests/st -m "not sdma" --platform <platform> \
pytest examples tests/st -m "not sdma" --exclude-level 4 --platform <platform> \
--device <range-or-$TASK_DEVICE> --pto-session-timeout <timeout> -v
```

Expand Down
8 changes: 4 additions & 4 deletions .claude/skills/test-runtime-device/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,15 +20,15 @@ Detection / isolation procedures referenced below live in
`task-submit`, let it pick via `--device auto --device-num <range size>`.
6. **Read the marker selector out of the same job** you took the timeout from.
`st-onboard-a2a3` is two pytest passes: the sweep deselects `-m "not sdma"`
and a later step runs `-m sdma`. `st-onboard-a5` uses `-m "not pod"` and
and a later step runs `-m sdma`. `st-onboard-a5` uses `--exclude-level 4` and
does not exclude `sdma`. Both quarantined tests are
`tensormap_and_ringbuffer`, so the second pass is only needed when
`$ARGUMENTS` names that runtime.
7. **Run through `task-submit`** (§E). On a5, the underlying command excludes
only pod tests, so SDMA remains in the sweep:
only level-4 tests, so SDMA remains in the sweep:

```bash
pytest examples tests/st -m "not pod" --platform a5 --runtime $ARGUMENTS \
pytest examples tests/st --exclude-level 4 --platform a5 --runtime $ARGUMENTS \
--device <range-or-$TASK_DEVICE> \
--pto-session-timeout <timeout> -v
```
Expand All @@ -37,7 +37,7 @@ Detection / isolation procedures referenced below live in
is:

```bash
pytest examples tests/st -m "not sdma" --platform <platform> --runtime $ARGUMENTS \
pytest examples tests/st -m "not sdma" --exclude-level 4 --platform <platform> --runtime $ARGUMENTS \
--device <range-or-$TASK_DEVICE> \
--pto-session-timeout <timeout> -v
```
Expand Down
14 changes: 7 additions & 7 deletions .claude/skills/testing/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,20 +15,20 @@ description: Testing guide and pre-commit testing strategy for simpler. Use when

**Important**: Always read `.github/workflows/ci.yml` first for the current
`--pto-session-timeout` values. Quarantines are marker-based, so mirror the
a2a3 sweep with `-m "not sdma"` rather than copying a path list. PTO-ISA
reproducibility comes from the repo-root `pto_isa.pin`.
a2a3 sweep with `-m "not sdma" --exclude-level 4` rather than copying a path
list. PTO-ISA reproducibility comes from the repo-root `pto_isa.pin`.

**CI does not run one flat sweep on a2a3.** Marked tests are quarantined out of
the general onboard sweep and run in a step of their own, after it, because
they are only correct in isolation. A5 runs the non-pod corpus, including
they are only correct in isolation. A5 runs the corpus below level 4, including
SDMA tests, on both x86_64 and ARM64. Reproducing a2a3 CI means reproducing that
shape — a bare `pytest examples tests/st --platform a2a3` is *not* what CI runs
and will report failures that CI never sees:

| Marker | Tests | CI behavior |
| ------ | ----- | ----------- |
| `@pytest.mark.manual` / `CASES[*]["manual"]` | Standalone pytest tests / individual scene-test cases; optionally scoped to a platform list | Per-PR main sweep: excluded by default on the selected platforms; dedicated DFX steps: included; `daily.yml`: full sweep with `--manual include` |
| `@pytest.mark.sdma` | a2a3: `sdma_async_completion_demo`, `prefetch_async_demo`; a5: `sdma_async_completion_demo` | a2a3: the dedicated SDMA step; a5: included in the non-pod sweep |
| `@pytest.mark.sdma` | a2a3: `sdma_async_completion_demo`, `prefetch_async_demo`; a5: `sdma_async_completion_demo` | a2a3: the dedicated SDMA step; a5: included in the non-network1 sweep |

The a2a3 SDMA demos provision 48 device-only STARS streams, which makes an
AICore fault take ~306 s to tear down instead of ~0.3 s — so they must not
Expand Down Expand Up @@ -74,7 +74,7 @@ pytest examples tests/st --platform a2a3sim \

# All hardware scene tests — mirror ci.yml: deselect the quarantined marker, or
# those tests fail here and nowhere else
pytest examples tests/st -m "not sdma" --platform a2a3 --device <range> \
pytest examples tests/st -m "not sdma" --exclude-level 4 --platform a2a3 --device <range> \
--pto-session-timeout <timeout>

# The quarantined tests, the way CI runs them — same corpus, selected by the
Expand All @@ -83,8 +83,8 @@ pytest examples tests/st -m "not sdma" --platform a2a3 --device <range> \
pytest examples tests/st -m sdma \
--platform a2a3 --device <2 devs> --pto-session-timeout <timeout>

# A5 runs the non-pod corpus, including SDMA tests, on both host architectures.
pytest examples tests/st -m "not pod" --platform a5 --device <range> \
# A5 runs the corpus below level 4, including SDMA tests, on both host architectures.
pytest examples tests/st --exclude-level 4 --platform a5 --device <range> \
--pto-session-timeout <timeout>

# Single runtime
Expand Down
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
name: Run pod pytest across the pair
name: Run network1 pytest across the pair
description: >-
Start the peer's L3 daemon, run pod-marked pytest against it, then stop the
daemon and pull its logs back. pod-stage has already put the tree and venv on
Start the peer's L3 daemon, run network1-marked pytest against it, then stop the
daemon and pull its logs back. network1-stage has already put the tree and venv on
the peer.

inputs:
Expand All @@ -21,37 +21,37 @@ runs:
working-directory: ${{ github.workspace }}
run: |
set -euo pipefail
source "$POD_SSH_HELPER"
DAEMON_PORT="$POD_L3_DAEMON_PORT"
LOCAL_LOGS="$RUN_DIR/pytest/daemon-${POD_REMOTE_MACHINE}"
REMOTE_LOGS="output/pod-ci/pytest/daemon-${POD_REMOTE_MACHINE}"
mkdir -p "$LOCAL_LOGS" "$RUN_DIR/pytest/parent-${POD_MACHINE}/ascend"
source "$NETWORK1_SSH_HELPER"
DAEMON_PORT="$NETWORK1_L3_DAEMON_PORT"
LOCAL_LOGS="$RUN_DIR/pytest/daemon-${NETWORK1_REMOTE_MACHINE}"
REMOTE_LOGS="output/network1-ci/pytest/daemon-${NETWORK1_REMOTE_MACHINE}"
mkdir -p "$LOCAL_LOGS" "$RUN_DIR/pytest/parent-${NETWORK1_MACHINE}/ascend"
{
echo "POD_REMOTE_ENDPOINT=${REMOTE_DAEMON_HOST}:${DAEMON_PORT}"
echo "POD_PYTEST_REMOTE_LOGS=$REMOTE_LOGS"
echo "NETWORK1_REMOTE_ENDPOINT=${REMOTE_DAEMON_HOST}:${DAEMON_PORT}"
echo "NETWORK1_PYTEST_REMOTE_LOGS=$REMOTE_LOGS"
} >> "$GITHUB_ENV"

pod_ssh "
network1_ssh "
pkill -f 'python -m simpler.remote_l3_worker --host ${REMOTE_DAEMON_HOST} --port ${DAEMON_PORT}' || true
" || true

pod_ssh "
network1_ssh "
set -eo pipefail
cd '$REMOTE_WORKDIR'
export PYTHONPATH=\"\${PYTHONPATH:-}\"
export CMAKE_PREFIX_PATH=\"\${CMAKE_PREFIX_PATH:-}\"
source '$POD_REMOTE_CANN_ENV'
source '$NETWORK1_REMOTE_CANN_ENV'
set -u
source .venv/bin/activate
export SIMPLER_SCHEDULER_TIMEOUT_MS='$SIMPLER_SCHEDULER_TIMEOUT_MS'
export SIMPLER_OP_EXECUTE_TIMEOUT_US='$SIMPLER_OP_EXECUTE_TIMEOUT_US'
export SIMPLER_STREAM_SYNC_TIMEOUT_MS='$SIMPLER_STREAM_SYNC_TIMEOUT_MS'
mkdir -p '$REMOTE_LOGS'
export ASCEND_PROCESS_LOG_PATH=\"\$PWD/$REMOTE_LOGS\"
echo '[pod-daemon] machine${POD_REMOTE_MACHINE} listening on ${REMOTE_DAEMON_HOST}:${DAEMON_PORT}'
echo '[network1-daemon] machine${NETWORK1_REMOTE_MACHINE} listening on ${REMOTE_DAEMON_HOST}:${DAEMON_PORT}'
python -m simpler.remote_l3_worker --host '${REMOTE_DAEMON_HOST}' --port '${DAEMON_PORT}'
" > "$LOCAL_LOGS/daemon.ssh.log" 2>&1 &
echo $! > "$RUNNER_TEMP/pod-daemon-pytest.pid"
echo $! > "$RUNNER_TEMP/network1-daemon-pytest.pid"

# A daemon that dies during import never opens the port, so on its own
# the connect probe only reports that after the full wait. The ssh
Expand All @@ -63,7 +63,7 @@ runs:
import sys
import time

ssh_pid = int(open("$RUNNER_TEMP/pod-daemon-pytest.pid").read().strip())
ssh_pid = int(open("$RUNNER_TEMP/network1-daemon-pytest.pid").read().strip())

def ssh_alive():
try:
Expand All @@ -76,7 +76,7 @@ runs:
except OSError:
return False

deadline = time.time() + ${POD_DAEMON_WAIT_S}
deadline = time.time() + ${NETWORK1_DAEMON_WAIT_S}
while time.time() < deadline:
if not ssh_alive():
print(
Expand All @@ -94,7 +94,7 @@ runs:
sys.exit(1)
PYWAIT

- name: Run pod pytest
- name: Run network1 pytest
shell: bash
working-directory: ${{ github.workspace }}
env:
Expand All @@ -103,39 +103,39 @@ runs:
set -eo pipefail
export PYTHONPATH="${PYTHONPATH:-}"
export CMAKE_PREFIX_PATH="${CMAKE_PREFIX_PATH:-}"
source "$POD_CANN_ENV"
source "$NETWORK1_CANN_ENV"
source .venv/bin/activate
set -u
mkdir -p "$RUN_DIR/pytest/parent-${POD_MACHINE}/ascend"
timeout "${POD_SMOKE_TIMEOUT_S}s" python -m pytest $PYTEST_ARGS
mkdir -p "$RUN_DIR/pytest/parent-${NETWORK1_MACHINE}/ascend"
timeout "${NETWORK1_SMOKE_TIMEOUT_S}s" python -m pytest $PYTEST_ARGS

# Runs even when pytest failed, since a device-side failure names its reason
# only in the daemon or ASCEND logs. The staging tree itself stays for
# pod-teardown.
# network1-teardown.
- name: Stop the daemon and collect peer logs
if: always()
shell: bash
working-directory: ${{ github.workspace }}
run: |
set +e
[ -r "${POD_SSH_HELPER:-}" ] || exit 0
source "$POD_SSH_HELPER"
[ -r "${NETWORK1_SSH_HELPER:-}" ] || exit 0
source "$NETWORK1_SSH_HELPER"

PID_FILE="$RUNNER_TEMP/pod-daemon-pytest.pid"
PID_FILE="$RUNNER_TEMP/network1-daemon-pytest.pid"
if [ -r "$PID_FILE" ]; then
kill "$(cat "$PID_FILE")" 2>/dev/null
rm -f "$PID_FILE"
fi
DAEMON_PORT="${POD_L3_DAEMON_PORT:-19073}"
pod_ssh "
DAEMON_PORT="${NETWORK1_L3_DAEMON_PORT:-19073}"
network1_ssh "
pkill -f 'python -m simpler.remote_l3_worker --host ${REMOTE_DAEMON_HOST} --port ${DAEMON_PORT}' || true
for _ in \$(seq 1 20); do
pgrep -f 'python -m simpler.remote_l3_worker --host ${REMOTE_DAEMON_HOST} --port ${DAEMON_PORT}' >/dev/null || break
sleep 0.5
done
"
mkdir -p "$RUN_DIR/pytest/daemon-${POD_REMOTE_MACHINE}"
mkdir -p "$RUN_DIR/pytest/daemon-${NETWORK1_REMOTE_MACHINE}"
rsync -a -e "$RSYNC_SSH" \
"$REMOTE_TARGET:$REMOTE_WORKDIR/${POD_PYTEST_REMOTE_LOGS:-output/pod-ci/pytest/daemon-${POD_REMOTE_MACHINE}}/" \
"$RUN_DIR/pytest/daemon-${POD_REMOTE_MACHINE}/" 2>/dev/null
"$REMOTE_TARGET:$REMOTE_WORKDIR/${NETWORK1_PYTEST_REMOTE_LOGS:-output/network1-ci/pytest/daemon-${NETWORK1_REMOTE_MACHINE}}/" \
"$RUN_DIR/pytest/daemon-${NETWORK1_REMOTE_MACHINE}/" 2>/dev/null
exit 0
Loading
Loading