Fix future hang when a queuing-system job dies without output by jan-janssen · Pull Request #1038 · pyiron/executorlib

jan-janssen · 2026-07-22T05:36:14Z

Summary

Fixes [Bug] job timeout hangs executor #1037: SlurmClusterExecutor/FluxClusterExecutor futures hung forever when the backing queuing-system job died before writing its output file (walltime TIMEOUT, OOM, NODE_FAIL, external scancel).
_check_task_output now falls back to querying the job status via pysqa.QueueAdapter.get_status_of_job when the expected _o.h5 output file is missing, and fails the future with a RuntimeError if the queuing system no longer knows about the job.
The status query is throttled per task (_JOB_STATUS_CHECK_INTERVAL = 30s) to avoid flooding squeue/sacct on every poll of the much faster refresh_rate loop, and is imported lazily so subprocess-only (non-pysqa) task submissions never pay the pysqa import cost.

Test plan

Added unit tests covering: dead job without output fails the future, a still-running job leaves the future pending, the status check is throttled across repeated polls, and subprocess-only (backend=None) tasks never trigger a queuing-system status query.
pytest tests/unit passes (pre-existing failures for missing optional networkx/pygraphviz extras confirmed present on main too, unrelated to this change).

🤖 Generated with Claude Code

Summary by CodeRabbit

Bug Fixes
- Detects queue jobs that stop unexpectedly without producing the expected output file, now marking the task as failed with a clear error instead of leaving it incomplete.
- Adds throttling for repeated queue job status checks to avoid excessive polling.
- Preserves normal behavior when the output appears later and when queue-status integration is not configured.
Tests
- Added unit tests for dead-job handling, still-running jobs, throttling behavior, and ensuring no status lookup occurs when no backend is provided.

FileTaskScheduler only resolved a task's future once its _o.h5 output file appeared, so a job killed by the scheduler (walltime TIMEOUT, OOM, NODE_FAIL, scancel) never wrote that file and future.result() blocked forever. _check_task_output now falls back to querying the job status via pysqa when the output file is missing and fails the future if the job is no longer known to the queuing system. The status query is throttled per task (_JOB_STATUS_CHECK_INTERVAL) and imported lazily so subprocess-only task submissions never pay the pysqa import cost. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

coderabbitai · 2026-07-22T05:36:35Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 6a0e2658-943f-44b2-a5be-e56833cc6543

📥 Commits

Reviewing files that changed from the base of the PR and between b9db96d and bca6164.

📒 Files selected for processing (1)

tests/unit/task_scheduler/file/test_backend.py

🚧 Files skipped from review as they are similar to previous changes (1)

tests/unit/task_scheduler/file/test_backend.py

📝 Walkthrough

Walkthrough

The file scheduler now queries queue status for tasks missing output, throttles repeated checks, fails futures for unknown jobs, and propagates polling state through refresh and shutdown paths. A public pysqa status helper and unit tests were added.

Changes

Dead job detection

Layer / File(s)	Summary
Queue status lookup `src/executorlib/standalone/command_pysqa.py`	Adds `pysqa_get_status_of_job` to return a queue job’s current status through `QueueAdapter`.
Throttled scheduler polling `src/executorlib/task_scheduler/file/shared.py`	Tracks per-task status checks, detects unknown jobs without output, fails their futures, and carries polling state through refresh and shutdown flows.
Dead-job behavior tests `tests/unit/task_scheduler/file/test_backend.py`	Tests dead jobs, running jobs, throttling, and disabled backend polling.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Scheduler
  participant OutputChecker
  participant QueueStatus
  participant Future
  Scheduler->>OutputChecker: inspect task output
  OutputChecker->>QueueStatus: query queue_id
  QueueStatus-->>OutputChecker: status or None
  OutputChecker->>Future: fail when job is unknown and output is absent

Possibly related PRs

pyiron/executorlib#732: Both changes modify the file scheduler execution workflow in shared.py.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly summarizes the main fix: preventing hangs when jobs die without output.
Linked Issues check	✅ Passed	The changes implement the requested detection and failure path for jobs that vanish before writing output, while leaving running jobs pending.
Out of Scope Changes check	✅ Passed	The added tests and skip handling support the fix and reported environment issues; no unrelated changes stand out.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

✨ Finishing Touches

📝 Generate docstrings

Create stacked PR
Commit on current branch

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch fix/1037-slurm-cluster-executor-job-timeout-hang

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

coderabbitai

🧹 Nitpick comments (2)

tests/unit/task_scheduler/file/test_backend.py (2)
266-285: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Test throttle expiry, not only immediate suppression.

The loop on Line 275 proves one lookup for back-to-back polls, but a throttle that never expires would also pass. Mock the scheduler clock, advance it through the 30-second interval, return None on the second lookup, and assert the future fails. This protects against reintroducing the original indefinite-pending behavior.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/unit/task_scheduler/file/test_backend.py` around lines 266 - 285, The
test_check_task_output_status_check_is_throttled test only verifies immediate
suppression, not throttle expiry. Mock the scheduler clock used by
_check_task_output, perform the initial polls, advance time beyond the 30-second
interval, make the next status lookup return None, and assert that the Future
from future_obj fails after the expired-throttle check.
223-246: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Cover the output-arrival race.

This verifies the failure path, but not the required case where the job is absent from the queue and writes its output immediately after the status lookup. Add a test that makes the mocked lookup create a valid output file, then assert successful completion; otherwise a regression can incorrectly fail completed work.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/unit/task_scheduler/file/test_backend.py` around lines 223 - 246, Add a
test alongside test_check_task_output_dead_job_without_output that mocks
pysqa_get_status_of_job to create a valid serialized output file in
cache_directory before returning no job status, then invokes _check_task_output
and asserts the Future completes successfully with the expected result. This
must cover the race where output appears immediately after the status lookup and
preserve successful completion instead of raising RuntimeError.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@tests/unit/task_scheduler/file/test_backend.py`:
- Around line 266-285: The test_check_task_output_status_check_is_throttled test
only verifies immediate suppression, not throttle expiry. Mock the scheduler
clock used by _check_task_output, perform the initial polls, advance time beyond
the 30-second interval, make the next status lookup return None, and assert that
the Future from future_obj fails after the expired-throttle check.
- Around line 223-246: Add a test alongside
test_check_task_output_dead_job_without_output that mocks
pysqa_get_status_of_job to create a valid serialized output file in
cache_directory before returning no job status, then invokes _check_task_output
and asserts the Future completes successfully with the expected result. This
must cover the race where output appears immediately after the status lookup and
preserve successful completion instead of raising RuntimeError.

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 95f01696-151e-44e9-9eea-110efdd1c23f

📥 Commits

Reviewing files that changed from the base of the PR and between 4a10711 and 9beb2dc.

📒 Files selected for processing (3)

src/executorlib/standalone/command_pysqa.py
src/executorlib/task_scheduler/file/shared.py
tests/unit/task_scheduler/file/test_backend.py

jan-janssen · 2026-07-22T07:05:02Z

@copilot On windows I get the following errors, please skip these tests on windows:

======================================================================
ERROR: test_check_task_output_dead_job_without_output (unit.task_scheduler.file.test_backend.TestSharedFunctions.test_check_task_output_dead_job_without_output)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "D:\a\executorlib\executorlib\tests\unit\task_scheduler\file\test_backend.py", line 231, in test_check_task_output_dead_job_without_output
    with patch(
         ~~~~~^
        "executorlib.standalone.command_pysqa.pysqa_get_status_of_job",
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        return_value=None,
        ^^^^^^^^^^^^^^^^^^
    ) as status_mock:
    ^
  File "C:\Users\runneradmin\miniconda3\envs\test\Lib\unittest\mock.py", line 1494, in __enter__
    self.target = self.getter()
                  ~~~~~~~~~~~^^
  File "C:\Users\runneradmin\miniconda3\envs\test\Lib\pkgutil.py", line 473, in resolve_name
    result = getattr(result, p)
AttributeError: module 'executorlib.standalone' has no attribute 'command_pysqa'

======================================================================
ERROR: test_check_task_output_job_still_running (unit.task_scheduler.file.test_backend.TestSharedFunctions.test_check_task_output_job_still_running)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "D:\a\executorlib\executorlib\tests\unit\task_scheduler\file\test_backend.py", line 252, in test_check_task_output_job_still_running
    with patch(
         ~~~~~^
        "executorlib.standalone.command_pysqa.pysqa_get_status_of_job",
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        return_value="running",
        ^^^^^^^^^^^^^^^^^^^^^^^
    ) as status_mock:
    ^
  File "C:\Users\runneradmin\miniconda3\envs\test\Lib\unittest\mock.py", line 1494, in __enter__
    self.target = self.getter()
                  ~~~~~~~~~~~^^
  File "C:\Users\runneradmin\miniconda3\envs\test\Lib\pkgutil.py", line 473, in resolve_name
    result = getattr(result, p)
AttributeError: module 'executorlib.standalone' has no attribute 'command_pysqa'

======================================================================
ERROR: test_check_task_output_no_backend_never_queries_status (unit.task_scheduler.file.test_backend.TestSharedFunctions.test_check_task_output_no_backend_never_queries_status)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "D:\a\executorlib\executorlib\tests\unit\task_scheduler\file\test_backend.py", line 292, in test_check_task_output_no_backend_never_queries_status
    with patch(
         ~~~~~^
        "executorlib.standalone.command_pysqa.pysqa_get_status_of_job",
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ) as status_mock:
    ^
  File "C:\Users\runneradmin\miniconda3\envs\test\Lib\unittest\mock.py", line 1494, in __enter__
    self.target = self.getter()
                  ~~~~~~~~~~~^^
  File "C:\Users\runneradmin\miniconda3\envs\test\Lib\pkgutil.py", line 473, in resolve_name
    result = getattr(result, p)
AttributeError: module 'executorlib.standalone' has no attribute 'command_pysqa'

======================================================================
ERROR: test_check_task_output_status_check_is_throttled (unit.task_scheduler.file.test_backend.TestSharedFunctions.test_check_task_output_status_check_is_throttled)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "D:\a\executorlib\executorlib\tests\unit\task_scheduler\file\test_backend.py", line 271, in test_check_task_output_status_check_is_throttled
    with patch(
         ~~~~~^
        "executorlib.standalone.command_pysqa.pysqa_get_status_of_job",
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        return_value="running",
        ^^^^^^^^^^^^^^^^^^^^^^^
    ) as status_mock:
    ^
  File "C:\Users\runneradmin\miniconda3\envs\test\Lib\unittest\mock.py", line 1494, in __enter__
    self.target = self.getter()
                  ~~~~~~~~~~~^^
  File "C:\Users\runneradmin\miniconda3\envs\test\Lib\pkgutil.py", line 473, in resolve_name
    result = getattr(result, p)
AttributeError: module 'executorlib.standalone' has no attribute 'command_pysqa'

----------------------------------------------------------------------

Copilot · 2026-07-22T07:06:53Z

@copilot On windows I get the following errors, please skip these tests on windows:
======================================================================

...

Done — added @unittest.skipIf(sys.platform == "win32", "pysqa module patching not supported on Windows") to all four affected tests in commit Skip pysqa status tests on Windows.

jan-janssen · 2026-07-22T08:09:35Z

@copilot skip the following tests when pysqa is not available

======================================================================
ERROR: test_check_task_output_dead_job_without_output (unit.task_scheduler.file.test_backend.TestSharedFunctions)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/runner/miniconda3/envs/test/lib/python3.10/unittest/mock.py", line 1248, in _dot_lookup
    return getattr(thing, comp)
AttributeError: module 'executorlib.standalone' has no attribute 'command_pysqa'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/runner/work/executorlib/executorlib/tests/unit/task_scheduler/file/test_backend.py", line 233, in test_check_task_output_dead_job_without_output
    with patch(
  File "/home/runner/miniconda3/envs/test/lib/python3.10/unittest/mock.py", line 1431, in __enter__
    self.target = self.getter()
  File "/home/runner/miniconda3/envs/test/lib/python3.10/unittest/mock.py", line 1618, in <lambda>
    getter = lambda: _importer(target)
  File "/home/runner/miniconda3/envs/test/lib/python3.10/unittest/mock.py", line 1261, in _importer
    thing = _dot_lookup(thing, comp, import_path)
  File "/home/runner/miniconda3/envs/test/lib/python3.10/unittest/mock.py", line 1250, in _dot_lookup
    __import__(import_path)
  File "/home/runner/miniconda3/envs/test/lib/python3.10/site-packages/executorlib/standalone/command_pysqa.py", line 5, in <module>
    from pysqa import QueueAdapter
ModuleNotFoundError: No module named 'pysqa'

======================================================================
ERROR: test_check_task_output_job_still_running (unit.task_scheduler.file.test_backend.TestSharedFunctions)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/runner/miniconda3/envs/test/lib/python3.10/unittest/mock.py", line 1248, in _dot_lookup
    return getattr(thing, comp)
AttributeError: module 'executorlib.standalone' has no attribute 'command_pysqa'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/runner/work/executorlib/executorlib/tests/unit/task_scheduler/file/test_backend.py", line 255, in test_check_task_output_job_still_running
    with patch(
  File "/home/runner/miniconda3/envs/test/lib/python3.10/unittest/mock.py", line 1431, in __enter__
    self.target = self.getter()
  File "/home/runner/miniconda3/envs/test/lib/python3.10/unittest/mock.py", line 1618, in <lambda>
    getter = lambda: _importer(target)
  File "/home/runner/miniconda3/envs/test/lib/python3.10/unittest/mock.py", line 1261, in _importer
    thing = _dot_lookup(thing, comp, import_path)
  File "/home/runner/miniconda3/envs/test/lib/python3.10/unittest/mock.py", line 1250, in _dot_lookup
    __import__(import_path)
  File "/home/runner/miniconda3/envs/test/lib/python3.10/site-packages/executorlib/standalone/command_pysqa.py", line 5, in <module>
    from pysqa import QueueAdapter
ModuleNotFoundError: No module named 'pysqa'

======================================================================
ERROR: test_check_task_output_no_backend_never_queries_status (unit.task_scheduler.file.test_backend.TestSharedFunctions)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/runner/miniconda3/envs/test/lib/python3.10/unittest/mock.py", line 1248, in _dot_lookup
    return getattr(thing, comp)
AttributeError: module 'executorlib.standalone' has no attribute 'command_pysqa'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/runner/work/executorlib/executorlib/tests/unit/task_scheduler/file/test_backend.py", line 297, in test_check_task_output_no_backend_never_queries_status
    with patch(
  File "/home/runner/miniconda3/envs/test/lib/python3.10/unittest/mock.py", line 1431, in __enter__
    self.target = self.getter()
  File "/home/runner/miniconda3/envs/test/lib/python3.10/unittest/mock.py", line 1618, in <lambda>
    getter = lambda: _importer(target)
  File "/home/runner/miniconda3/envs/test/lib/python3.10/unittest/mock.py", line 1261, in _importer
    thing = _dot_lookup(thing, comp, import_path)
  File "/home/runner/miniconda3/envs/test/lib/python3.10/unittest/mock.py", line 1250, in _dot_lookup
    __import__(import_path)
  File "/home/runner/miniconda3/envs/test/lib/python3.10/site-packages/executorlib/standalone/command_pysqa.py", line 5, in <module>
    from pysqa import QueueAdapter
ModuleNotFoundError: No module named 'pysqa'

======================================================================
ERROR: test_check_task_output_status_check_is_throttled (unit.task_scheduler.file.test_backend.TestSharedFunctions)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/runner/miniconda3/envs/test/lib/python3.10/unittest/mock.py", line 1248, in _dot_lookup
    return getattr(thing, comp)
AttributeError: module 'executorlib.standalone' has no attribute 'command_pysqa'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/runner/work/executorlib/executorlib/tests/unit/task_scheduler/file/test_backend.py", line 275, in test_check_task_output_status_check_is_throttled
    with patch(
  File "/home/runner/miniconda3/envs/test/lib/python3.10/unittest/mock.py", line 1431, in __enter__
    self.target = self.getter()
  File "/home/runner/miniconda3/envs/test/lib/python3.10/unittest/mock.py", line 1618, in <lambda>
    getter = lambda: _importer(target)
  File "/home/runner/miniconda3/envs/test/lib/python3.10/unittest/mock.py", line 1261, in _importer
    thing = _dot_lookup(thing, comp, import_path)
  File "/home/runner/miniconda3/envs/test/lib/python3.10/unittest/mock.py", line 1250, in _dot_lookup
    __import__(import_path)
  File "/home/runner/miniconda3/envs/test/lib/python3.10/site-packages/executorlib/standalone/command_pysqa.py", line 5, in <module>
    from pysqa import QueueAdapter
ModuleNotFoundError: No module named 'pysqa'

----------------------------------------------------------------------

Copilot · 2026-07-22T08:11:17Z

@copilot skip the following tests when pysqa is not available
======================================================================

...

Fixed — added a skip_pysqa_test flag (via a try/import pysqa) and extended all four decorators to skip when pysqa is not installed, in addition to the existing Windows skip.

codecov · 2026-07-22T12:39:56Z

Codecov Report

❌ Patch coverage is 90.00000% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 94.15%. Comparing base (faf71ea) to head (bca6164).
⚠️ Report is 3 commits behind head on main.

Files with missing lines	Patch %	Lines
src/executorlib/task_scheduler/file/shared.py	88.88%	3 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1038      +/-   ##
==========================================
- Coverage   94.22%   94.15%   -0.07%     
==========================================
  Files          39       39              
  Lines        2113     2140      +27     
==========================================
+ Hits         1991     2015      +24     
- Misses        122      125       +3

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

coderabbitai Bot reviewed Jul 22, 2026

View reviewed changes

Copilot started work on behalf of jan-janssen July 22, 2026 07:05 View session

Skip pysqa status tests on Windows

b9db96d

Copilot finished work on behalf of jan-janssen July 22, 2026 07:07

Copilot started work on behalf of jan-janssen July 22, 2026 08:10 View session

Skip pysqa tests when pysqa is not installed

bca6164

Copilot finished work on behalf of jan-janssen July 22, 2026 08:11

jan-janssen marked this pull request as draft July 22, 2026 08:53

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Fix future hang when a queuing-system job dies without output#1038

Fix future hang when a queuing-system job dies without output#1038
jan-janssen wants to merge 3 commits into
mainfrom
fix/1037-slurm-cluster-executor-job-timeout-hang

jan-janssen commented Jul 22, 2026 •

edited by coderabbitai Bot

Loading

Uh oh!

coderabbitai Bot commented Jul 22, 2026 •

edited

Loading

Walkthrough

Changes

Sequence Diagram(s)

Uh oh!

coderabbitai Bot left a comment

Uh oh!

jan-janssen commented Jul 22, 2026

Uh oh!

Copilot AI commented Jul 22, 2026

Uh oh!

jan-janssen commented Jul 22, 2026

Uh oh!

Copilot AI commented Jul 22, 2026

Uh oh!

codecov Bot commented Jul 22, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

jan-janssen commented Jul 22, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test plan

Summary by CodeRabbit

Uh oh!

coderabbitai Bot commented Jul 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Sequence Diagram(s)

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

jan-janssen commented Jul 22, 2026

Uh oh!

Copilot AI commented Jul 22, 2026

Uh oh!

jan-janssen commented Jul 22, 2026

Uh oh!

Copilot AI commented Jul 22, 2026

Uh oh!

codecov Bot commented Jul 22, 2026

Codecov Report

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

jan-janssen commented Jul 22, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Jul 22, 2026 •

edited

Loading