Skip to content

Add tests for the AWS Batch executor boto schemas - #72609

Merged
vincbeck merged 1 commit into
apache:mainfrom
sokojh:add-tests-for-batch-boto-schema
Sep 9, 2026
Merged

vincbeck merged 1 commit into
apache:mainfrom
sokojh:add-tests-for-batch-boto-schema

Conversation

@sokojh

@sokojh sokojh commented Sep 7, 2026

Copy link
Copy Markdown

Why

providers/amazon/.../executors/batch/boto_schema.py is on the OVERLOOKED_TESTS
allowlist in airflow-core/tests/unit/always/test_project_structure.py, tracked by
the meta issue #35442.

Unlike most entries on that list, the three schemas in this module are not covered
indirectly either — no file under any providers/*/tests/ directory references
BatchSubmitJobResponseSchema, BatchJobDetailSchema or
BatchDescribeJobsResponseSchema. (For contrast, build_submit_kwargs in the
sibling batch_executor_config.py is exercised from test_batch_executor.py,
which is why this PR is scoped to boto_schema.py alone.)

These schemas parse live AWS Batch API responses, so the behaviour worth pinning
down is the part that is easy to break silently: the data_key mapping from the
API's camelCase to Airflow's snake_case, the unknown = EXCLUDE policy that lets
the schemas tolerate fields AWS adds over time, and the @post_load hook that
makes BatchJobDetailSchema return a BatchJob rather than a plain dict.

How

Adds providers/amazon/tests/unit/amazon/aws/executors/batch/test_boto_schema.py
with one test class per schema, covering for each:

  • the happy-path load and the jobId → job_id renaming,
  • that unknown keys returned by submit_job / describe_jobs are dropped rather
    than raising,
  • that a missing required field raises ValidationError,
  • that BatchJobDetailSchema returns a BatchJob instance with status_reason
    left as None when the API omits it,
  • that BatchDescribeJobsResponseSchema builds one BatchJob per entry, accepts
    an empty jobs list, and propagates a nested job's ValidationError.

No production code is touched.

What

13 tests added. The module's entry is removed from OVERLOOKED_TESTS in the same
PR, as that list fails when a listed test file exists.

Verified locally:

uv run --project providers/amazon pytest \
  providers/amazon/tests/unit/amazon/aws/executors/batch/test_boto_schema.py -v
# 13 passed

uv run --project airflow-core pytest \
  airflow-core/tests/unit/always/test_project_structure.py -q
# 10 passed, 1 xfailed

related: #35442


Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

Generated-by: Claude Code (Opus 5) following the guidelines

@sokojh
sokojh requested a review from o-nikolas as a code owner September 7, 2026 05:05
@boring-cyborg boring-cyborg Bot added area:providers provider:amazon AWS/Amazon - related issues labels Sep 7, 2026
@boring-cyborg

boring-cyborg Bot commented Sep 7, 2026

Copy link
Copy Markdown

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide
Here are some useful points:

  • Pay attention to the quality of your code (ruff, mypy and type annotations). Our prek-hooks will help you with that.
  • In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example Dag that shows how users should use it.
  • Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
  • Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
  • Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
  • Be sure to read the Airflow Coding style.
  • Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
    Apache Airflow is a community-driven project and together we are making it better 🚀.
    In case of doubts contact the developers at:
    Mailing List: dev@airflow.apache.org
    Slack: https://s.apache.org/airflow-slack

The three marshmallow schemas in boto_schema.py sit on the
OVERLOOKED_TESTS allowlist and, unlike most entries there, are not
covered indirectly either - nothing under providers/*/tests references
them. They parse live AWS Batch API responses, so the behaviour worth
pinning down is the data_key renaming from the API's camelCase, the
unknown=EXCLUDE policy that tolerates fields AWS adds over time, and
the post_load hook that returns a BatchJob rather than a plain dict.

Removes the module's entry from OVERLOOKED_TESTS in the same commit, as
that list fails once a listed test file exists.

related: apache#35442

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VKBUyiL28LCzdLsc3kqEoX
@sokojh
sokojh force-pushed the add-tests-for-batch-boto-schema branch from 4681b2b to 0d0ba88 Compare September 7, 2026 05:23
@vincbeck
vincbeck merged commit d9d0d2c into apache:main Sep 9, 2026
82 checks passed
@boring-cyborg

boring-cyborg Bot commented Sep 9, 2026

Copy link
Copy Markdown

Awesome work, congrats on your first merged pull request! You are invited to check our Issue Tracker for additional contributions.

@sokojh
sokojh deleted the add-tests-for-batch-boto-schema branch September 10, 2026 02:15
imrichardwu pushed a commit to imrichardwu/airflow that referenced this pull request Sep 11, 2026
The three marshmallow schemas in boto_schema.py sit on the
OVERLOOKED_TESTS allowlist and, unlike most entries there, are not
covered indirectly either - nothing under providers/*/tests references
them. They parse live AWS Batch API responses, so the behaviour worth
pinning down is the data_key renaming from the API's camelCase, the
unknown=EXCLUDE policy that tolerates fields AWS adds over time, and
the post_load hook that returns a BatchJob rather than a plain dict.

Removes the module's entry from OVERLOOKED_TESTS in the same commit, as
that list fails once a listed test file exists.

related: apache#35442
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:providers provider:amazon AWS/Amazon - related issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants