Skip to content

Avoid bootstrapping pip in model-only Python test venvs (fixes flaky venv timeout) - #21539

Merged
Mitch Denny (mitchdenny) merged 1 commit into
mainfrom
aspireagent/python-venv-without-pip
Nov 5, 2026
Merged

Mitch Denny (mitchdenny) merged 1 commit into
mainfrom
aspireagent/python-venv-without-pip

Conversation

@davidfowl

@davidfowl David Fowler (davidfowl) commented Oct 31, 2026 •

Copy link
Copy Markdown
Collaborator

Description

Fixes #19542
Also fixes a latent hang in the helper's timeout path.

AddPythonAppTests.PythonResourceSupportsWithReference failed once on windows-latest (run 32375588350, job 96448944641) with Failed to create virtual environment. The test took 2m 06s, and the venv process wrote no stdout or stderr.

Root cause

  • Unnecessary pip bootstrap. Every AddPythonAppTests test that uses CreateTempPythonProject runs a full python -m venv .venv and then pip install, under a fixed 2-minute deadline. Five of those tests only build the app model, and so does the failing one. They need .venv to exist because AddPythonApp adds a -venv-creator resource when it's missing. They never run pip. A full venv runs ensurepip, which starts a pip subprocess that installs and byte-compiles about 1,000 files. Without pip, venv creation copies about 5 files. On a cold Windows runner (Defender, MoUsoCoreWorker and svchost were busy in the heartbeat), this step exceeded 2 minutes. The heartbeat at 14:00:28, 75s into the wait, shows no python process among the top CPU consumers, so the process was blocked on I/O or the OS rather than computing.
  • Timeout path waits for the process anyway. On timeout, the helper read stdout and stderr to EOF before calling Kill. A process that truly hung would therefore block the test until the 20-minute MTP hang timeout instead of failing at 2 minutes. That's why the failure reported 2m 06s: the read waited for python to exit.

Fix (test-only)

  • CreateTempPythonProject(..., withPip: false) is the default for model-only tests. It creates the venv with python -m venv --without-pip .venv and skips pip install. The venv interpreter exists, so the resource model (command path, no venv-creator) is unchanged.
  • Tests that actually run pip still bootstrap it. PythonResourceFinishesSuccessfully starts the app, and its installer resource runs .venv pip. The instrumented test installs requirements.
  • Both process waits now kill on timeout before draining output.

No assertions were changed, and no retries or sleeps were added.

Evidence

Regression test: CreateTempPythonProject_ForModelOnlyTests_CreatesVirtualEnvironmentWithoutBootstrappingPip asserts that the model-only venv has an interpreter and no pip.

  • Before, with main's -m venv .venv: fails, with Model-only test projects should not bootstrap pip, but found '/tmp/bBYQbR/.venv/bin/pip'.
  • After: passes.

Per-test duration (TRX, Linux, Python 3.12), main → this PR:

Test main PR
PythonResourceSupportsWithReference (the flaky test) 2.51s 0.053s
AddPythonAppProducesDockerfileResourceInManifest 2.53s 0.037s
AddPythonApp_SetsResourcePropertiesCorrectly 2.65s 0.039s
AddPythonApp_ObsoleteMethod_StillWorks 2.50s 0.039s
AddPythonAppWithScriptArgs_IncludesTheArguments 2.50s 0.044s
PythonResourceFinishesSuccessfully (still uses pip; starts the app) 6.40s 9.75s (passes)
AddInstrumentedPythonProjectProducesDockerfileResourceInManifest (still uses pip) 9.12s 5.88s

Measured separately: python -m venv .venv writes 1,010 files in 4.27s, while --without-pip writes 5 files in 0.05s. Model-only tests no longer start an ensurepip/pip subprocess, which was the step that exceeded the deadline.

Recurrence: this is the only recorded occurrence. A scan of the last 300 ci.yml runs (2026-10-30 22:08 to 2026-10-31 06:15 UTC; the CI test selector skips Hosting.Python on most runs) found 0 failures in 20 Hosting.Python (windows-latest) jobs. On main, the Windows suite usually takes about 75s (for example, job 124463643298: 1m 15s). The failure is rare and depends on runner conditions, so the evidence above is the removed exposure rather than a local Windows repro.

Validation

  • Aspire.Hosting.Python.Tests (full project, functional PythonResourceFinishesSuccessfully included, which starts the app via DCP): 119 passed, 0 skipped, 3 failed. The 3 failures are WithUv_GeneratesDockerfileInPublishMode* Verify snapshots. They fail identically on main in this environment because the run is as root and Verify scrubs /root/ to {UserProfile}. They're unrelated to this change, and CI is the authority.

Windows CI on this PR (job 124468350194): Hosting.Python (windows-latest) reported 122 succeeded, 0 failed, in 31.6s. On main this suite takes about 75s, so most of the Windows suite time was venv/pip bootstrap that model-only tests didn't need.

Checklist

  • Is this feature complete?
    • Yes. Ready to ship.
  • Are you including unit tests for the changes and scenario tests if relevant?
    • Yes
  • Did you add public API?
    • No
  • Does the change make any security assumptions or guarantees?
    • No

🤖 Posted by AspireAgent (AI Agent)

Model-only AddPythonAppTests only need the .venv directory to exist, but the
shared helper ran a full 'python -m venv' (ensurepip spawns pip and writes
~1000 files) under a 2 minute deadline, which timed out on a cold Windows
runner. Create those venvs with --without-pip, keep pip for tests that run
it, and kill timed-out processes before draining their output.

Fixes #19542

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@github-actions

Copy link
Copy Markdown
Contributor

🚀 Dogfood this PR with:

⚠️ WARNING: Do not do this without first carefully reviewing the code of this PR to satisfy yourself it is safe.

curl -fsSL https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.sh | bash -s -- 21539

Or

  • Run remotely in PowerShell:
iex "& { $(irm https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.ps1) } 21539"

@github-actions github-actions Bot added the needs-area-label An area label is needed to ensure this gets routed to the appropriate area owners label Oct 31, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Tests selector

1 / 100 PR test projects · 0 PR jobs, from 1 changed file.

Selected PR test projects (1 / 100)

Aspire.Hosting.Python.Tests

Selected PR jobs (0)

none


How these were chosen — grouped by what changed

🧪 tests/Aspire.Hosting.Python.Tests/AddPythonAppTests.cs (changed test)
→ 1 directly: Aspire.Hosting.Python.Tests

Job reasons

none


Selection computed for commit 5895beb.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues block the test-only changes that skip unnecessary pip setup in model-only Python environments.

No open issues 🎉

What changed in this PR

Reduces exposure to the flaky Python virtual-environment timeout in #19542 by skipping unnecessary pip setup in Aspire’s model-only Python hosting tests. Changes are test-only.

Changes:

  • Creates model-only virtual environments without pip, while preserving pip setup for tests that need it.
  • Kills timed-out venv creation and package installation processes before draining their output.
  • Adds a regression test verifying that model-only environments contain a Python interpreter but no pip executable.
File Description
tests/​Aspire.Hosting.Python.Tests/​AddPythonAppTests.cs Makes pip setup conditional, fixes timeout handling, and adds regression coverage.

Review effort: Balanced

@davidfowl
David Fowler (davidfowl) marked this pull request as ready for review October 31, 2026 07:01
@mitchdenny
Mitch Denny (mitchdenny) merged commit a56cea3 into main Nov 5, 2026
53 checks passed
@microsoft-github-policy-service microsoft-github-policy-service Bot added this to the 17.0 milestone Nov 5, 2026
@aspire-repo-bot

Copy link
Copy Markdown
Contributor

✅ No documentation update needed.

docs_optional → test_only. Triggered signal: pr_body_has_cli_flag_mention (evidence: PR body mentions python -m venv --without-pip .venv, which is the test helper's internal command, not an Aspire CLI flag). The only changed file is tests/Aspire.Hosting.Python.Tests/AddPythonAppTests.cs, which matches tests/**. No user-facing behavior changed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

needs-area-label An area label is needed to ensure this gets routed to the appropriate area owners

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[CI Failure] Flaky: AddPythonAppTests.PythonResourceSupportsWithReference fails to create Python virtual environment on Windows runner

3 participants