[py] move complete GitHub Actions test suite to nightly and run only high value smoke tests for PRs

[titusfortner]

🔗 Related Issues

• Part of [🚀 Feature]: Selenium GHA Testing Plan #17539
• Builds on [py] run affected python tests per-browser on PR, full suite on schedule #17643 (affected-target split)
• Mirrors Ruby's [rb] streamline tests on github actions runners #17550
• Builds on [py] run the requirements lock check on RBE with a platform-agnostic lock #18039 (lock-file check moved onto RBE, so nothing in py carries skip-rbe)

💥 What does this PR do?

• Reduces GitHub Actions tests on PRs to minimize redundancies and low marginal value over information obtained from RBE tests
• Keeps full regression test suite for scheduled/manual runs for completeness
• Limits test target run on PRs to only what was calculated as needed by CI workflow check-targets job.

Expected changes for integration tests, per run:

	Trunk (PR and nightly)	This PR, per PR	This PR, nightly
Jobs	9	3	8
Runner minutes	~200	~40	~180
macOS minutes	~23 (2 jobs)	~12 (1 job)	~22 (2 jobs)
Wall clock	~40 min	~22 min	~40 min

🔧 Implementation Notes

• PR tests focus on high value configurations above what RBE provides: os-sensitive (window, screenshot and actions tests), se-manager (driver-finder), and skip-rbe (currently matches nothing in py, but kept for future/parity with other bindings). The ubuntu leg filters se-manager,skip-rbe only: RBE's image runs the same Xvfb and fluxbox setup, so the Linux os-sensitive copies would only differ by pinned versus unpinned browser.
• py_test_suite gains extra_tags so two files inside the globbed common suite can carry os-sensitive without a seventh per-browser suite; an unknown key fails the build so a file rename cannot silently empty the smoke filter.
• The tiers are mutually exclusive by content: smoke also runs on schedule so the flaky parser sees it, and the full jobs pass -os-sensitive,-se-manager,-skip-rbe, the exact complement, so nothing runs twice.
• The GHA remote job is removed; RBE already runs all 127 Grid targets on every PR and nightly, the same arrangement Ruby has had since January.
• Two actions tests now scroll #working into view before the pointer move: Firefox will not move to an element outside the viewport, which the macOS runner's small display exposed on the first smoke run; same fix as Ruby's action_builder_spec in [rb] BiDi get client windows #16211.

🤖 AI assistance

• AI assisted (complete below)
  • Tool(s): Claude Code (Claude Fable 5.1)
  • What was generated: the CI timing analysis, the Bazel and workflow changes, and this description
  • I reviewed all AI output and can explain the change

💡 Additional Considerations

• Dispatching ci.yml manually now runs Python smoke only, as it already does for Ruby; dispatch ci-python.yml directly to get the full matrix.

🔄 Types of changes

• Cleanup (CI)

[qodo-code-review]

PR Summary by Qodo

Split Python CI into per-OS smoke and scheduled full tiers

⚙️ Configuration changes ✨ Enhancement 🐞 Bug fix 🕐 20-40 Minutes

AI Description

• Runs high-value browser smoke tests across Linux, Windows, and macOS on every CI event.
• Reserves complementary build, unit, remote, and browser matrices for scheduled or full manual
 runs.
• Tags OS-sensitive tests and restores Linux Chromium arguments for unpinned browsers.

Diagram

graph TD
  PR["PR or Push"] -->|"calls"| CI["Python CI"] -->|"always"| Smoke["OS Smoke Matrix"] -->|"tag filters"| Tagged["Tagged Tests"] --> Harness["Browser Harness"]
  Schedule["Schedule or Manual"] -->|"runs"| CI -->|"full enabled"| Full["Full Test Matrix"] -->|"excludes smoke"| Complement["Complement Tests"] --> Harness

Loading

High-Level Assessment

The following are alternative approaches to this PR:

1. Keep per-browser smoke jobs

• ➕ Avoids changing the Bazel suite macro
• ➕ Makes browser coverage explicit in workflow YAML
• ➖ Retains redundant jobs and higher runner usage
• ➖ Requires additional suites for narrowly selected test files
• ➖ Scales poorly as smoke categories expand

2. Run the full matrix on every PR

• ➕ Provides maximum pre-merge coverage
• ➕ Avoids tier-selection logic
• ➖ Duplicates substantial RBE coverage
• ➖ Consumes significantly more runner time
• ➖ Produces slower feedback for contributors

Recommendation: Use the proposed tag-driven smoke tier with a mutually exclusive scheduled complement. It preserves high-value cross-platform PR signals while reducing redundant execution, and per-source tagging avoids proliferating browser-specific Bazel suites. The validation of extra_tags keys also limits the risk of silently losing smoke coverage after file renames.

Files changed (4) +76 / -41

Enhancement (1) +10 / -2

suite.bzlSupport validated per-source tags in Python test suites +10/-2

Support validated per-source tags in Python test suites

• Extends py_test_suite with an extra_tags mapping that merges source-specific tags into generated pytest targets. Unknown source keys fail analysis, preventing renamed or removed files from silently dropping smoke coverage.

py/private/suite.bzl

Bug fix (1) +2 / -0

conftest.pyDisable Chromium sandboxing for Linux test sessions +2/-0

Disable Chromium sandboxing for Linux test sessions

• Adds --no-sandbox for Chrome and Edge when running on Linux. This preserves browser startup behavior when smoke jobs use unpinned browsers instead of receiving the flag from pinned browser configuration.

py/conftest.py

Other (2) +64 / -39

ci-python.ymlSplit Python CI into smoke and full execution tiers +52/-38

Split Python CI into smoke and full execution tiers

• Adds a configurable smoke mode and an always-running three-OS smoke matrix filtered to high-value test tags. Build, unit, remote, and full browser jobs now run only for schedules or full invocations and exclude smoke-tagged tests to avoid duplication.

.github/workflows/ci-python.yml

BUILD.bazelMark host-sensitive browser tests for smoke coverage +12/-1

Mark host-sensitive browser tests for smoke coverage

• Defines OS-sensitive test files, applies per-source tags within common browser suites, and tags all action suites as OS-sensitive. These tags drive cross-platform smoke selection without creating additional per-browser suites.

py/BUILD.bazel

[qodo-code-review]

Code Review by Qodo

🐞 Bugs (0) 📘 Rule violations (0) 📜 Skill insights (0)

1. Full runs repeat smoke test sources ✗ Dismissed 🐞 Bug ➹ Performance

Description

//py:test-remote applies the negative os-sensitive filter, but its separately generated
remote-common and remote-actions targets never receive that tag even though they reuse the
smoke-selected source lists. Whenever scheduled or full-dispatch jobs run, Chrome and Firefox rerun
those sources through Grid in addition to smoke, defeating tier exclusivity and adding redundant
browser work.

Code

.github/workflows/ci-python.yml[126]

+        --test_tag_filters=-os-sensitive,-se-manager

Evidence

The local common suites tag the screenshot and window sources individually, and local action suites
tag every action source as OS-sensitive. The remote suites reuse those same sources without
propagating either tag, while the new full remote command relies on the missing tag to exclude them.

py/BUILD.bazel[923-928]
py/BUILD.bazel[1007-1026]
py/BUILD.bazel[1037-1052]
py/BUILD.bazel[1195-1227]
py/BUILD.bazel[1239-1263]
py/BUILD.bazel[1313-1332]
.github/workflows/ci-python.yml[119-127]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The full remote test target reuses OS-sensitive common and action test sources without carrying their `os-sensitive` tag, so its negative tag filter does not exclude them.

## Fix Focus Areas
- py/BUILD.bazel[1197-1227]
- py/BUILD.bazel[1241-1263]
- .github/workflows/ci-python.yml[119-127]

## Recommended Fix
Apply the per-source `os-sensitive` tags to the remote-common suites and the suite-level `os-sensitive` tag to the remote-actions suites. Keep the full remote job's negative filter so these targets are selected only by the smoke tier.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

Context sources

Review mode: 🚀 Fast: This is a single, localized CI filter change adding one tag exclusion, with contained behavioral impact and no high-risk area involved.

Tip of the day

💡 Did you know, you can group findings by type and pick your Finding display, from Minimal to Full

More tips ↗ | Customize Qodo ↗ | Qodo docs ↗

[qodo-code-review]

Code review by qodo was updated up to the latest commit 751cefd

[qodo-code-review]

Code review by qodo was updated up to the latest commit 43dfcc4