Fix pex3 venv create --link-python for --pex-repository. - #3285
Closed
jasonwbarnett wants to merge 1 commit into
Closed
jasonwbarnett wants to merge 1 commit into
jasonwbarnett wants to merge 1 commit into
Conversation
Populating a venv's sources re-resolved the repository PEX, and that resolve
runs against a local interpreter. For a venv laid out for a foreign platform
the PEX need hold no distribution that interpreter can use, so a PEX built for
the foreign platform alone failed with e.g.:
A distribution for psutil could not be resolved for <local python>.
`_install_from_pex` has already resolved the distributions for the target, so
pass those down to the venv's `pex-repl` script instead of re-resolving. The
`PEX_TOOLS=1 ... venv` path keeps re-resolving, where the PEX and the
interpreter always agree.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012imZdyz1fhVXRNKPjbhSDw
jasonwbarnett
force-pushed
the
venv-create-pex-repository-link-python
branch
from
September 17, 2026 22:56
b7b841d to
07a5c2a
Compare
Member
|
@jasonwbarnett no more Claude please. I haven't had to write an AI policy yet for Pex, but this is the 2nd PR from the altana-ai/pex repo where I can not contribute: :; git push https://github.com/altana-ai/pex HEAD:venv-create-pex-repository-link-python
remote: Permission to altana-ai/pex.git denied to jsirois.
fatal: unable to access 'https://github.com/altana-ai/pex/': The requested URL returned error: 403Now the altana-ai/pex repo thing may or may not have anything to do with Claude, but regardless, two strikes and you're out. I'll do another hand merge to include these: From 45a9df95bde15cda58c4e6058aca126ad0c2992e Mon Sep 17 00:00:00 2001
From: John Sirois <john.sirois@gmail.com>
Date: Thu, 17 Sep 2026 23:29:02 -0700
Subject: [PATCH 1/2] Only resolve once.
---
pex/cli/commands/venv.py | 6 +++---
pex/venv/installer.py | 17 +++++++----------
2 files changed, 10 insertions(+), 13 deletions(-)
diff --git a/pex/cli/commands/venv.py b/pex/cli/commands/venv.py
index 98aaf998..6f746150 100644
--- a/pex/cli/commands/venv.py
+++ b/pex/cli/commands/venv.py
@@ -42,7 +42,7 @@ from pex.venv.installer_configuration import InstallerConfiguration
from pex.venv.virtualenv import Virtualenv
if TYPE_CHECKING:
- from typing import Any, Dict, Iterable, Optional, Sequence, Union
+ from typing import Any, Dict, Optional, Sequence, Union
logger = logging.getLogger(__name__)
@@ -495,7 +495,7 @@ def _install_from_pex(
pex, # type: PEX
installer_configuration, # type: InstallerConfiguration
provenance, # type: Provenance
- distributions, # type: Iterable[Distribution]
+ distributions, # type: Sequence[Distribution]
dest_dir, # type: str
hermetic_scripts, # type: bool
venv=None, # type: Optional[Virtualenv]
@@ -531,9 +531,9 @@ def _install_from_pex(
venv=venv,
pex=pex,
provenance=provenance,
+ activated_dists=distributions,
bin_path=installer_configuration.bin_path,
hermetic_scripts=hermetic_scripts,
- activated_dists=distributions,
)
else:
installer.populate_flat_sources(dst=dest_dir, pex=pex, provenance=provenance)
diff --git a/pex/venv/installer.py b/pex/venv/installer.py
index 2241c75e..33becfc0 100644
--- a/pex/venv/installer.py
+++ b/pex/venv/installer.py
@@ -360,11 +360,11 @@ def populate_venv_sources(
venv, # type: Virtualenv
pex, # type: PEX
provenance, # type: Provenance
+ activated_dists, # type: Sequence[Distribution]
bin_path=BinPath.FALSE, # type: BinPath.Value
hermetic_scripts=True, # type: bool
shebang=None, # type: Optional[str]
set_last_access=True, # type: bool
- activated_dists=None, # type: Optional[Iterable[Distribution]]
):
# type: (...) -> str
@@ -374,11 +374,11 @@ def populate_venv_sources(
target_dir=provenance.target_dir,
venv=venv,
pex=pex,
+ activated_dists=activated_dists,
shebang=shebang,
venv_python=provenance.target_python,
bin_path=bin_path,
set_last_access=set_last_access,
- activated_dists=activated_dists,
)
)
return shebang
@@ -422,10 +422,11 @@ def populate_venv_from_pex(
shebang = provenance.calculate_shebang(hermetic_scripts=hermetic_scripts)
top_level_source_packages = tuple(iter_top_level_source_packages(pex))
+ distributions = tuple(pex.resolve())
if scope in (InstallScope.ALL, InstallScope.DEPS_ONLY):
populate_venv_distributions(
venv=venv,
- distributions=pex.resolve(),
+ distributions=distributions,
copy_mode=copy_mode,
hermetic_scripts=hermetic_scripts,
provenance=provenance,
@@ -436,6 +437,7 @@ def populate_venv_from_pex(
populate_venv_sources(
venv=venv,
pex=pex,
+ activated_dists=distributions,
bin_path=bin_path,
hermetic_scripts=hermetic_scripts,
provenance=provenance,
@@ -747,11 +749,11 @@ def _populate_first_party(
target_dir, # type: str
venv, # type: Virtualenv
pex, # type: PEX
+ activated_dists, # type: Sequence[Distribution]
shebang, # type: str
venv_python, # type: str
bin_path, # type: BinPath.Value
set_last_access, # type: bool
- activated_dists=None, # type: Optional[Iterable[Distribution]]
):
# type: (...) -> Iterator[Tuple[Text, Text]]
@@ -767,12 +769,7 @@ def _populate_first_party(
target_dir=target_dir,
venv=venv,
pex_info=pex_info,
- # N.B.: A venv laid out for a foreign platform cannot re-resolve the PEX: that resolve
- # runs against a local interpreter, which the PEX need hold no distributions for. The
- # caller passes the distributions it resolved for the target instead.
- activated_dists=(
- tuple(activated_dists) if activated_dists is not None else tuple(pex.resolve())
- ),
+ activated_dists=activated_dists,
shebang=shebang,
venv_python=venv_python,
bin_path=bin_path,
--
2.55.0
From 8b25c46ab2317d735f6f20fdcface3a0bd3bc8d0 Mon Sep 17 00:00:00 2001
From: John Sirois <john.sirois@gmail.com>
Date: Thu, 17 Sep 2026 23:30:39 -0700
Subject: [PATCH 2/2] Fix test - psutil 5.9.5 has no aarch64 wheels for Linux
test.
The resulting yolo build using the local interpreter netted native wheels
not compatible with the foreign platform.
---
tests/integration/cli/commands/test_venv_create.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/tests/integration/cli/commands/test_venv_create.py b/tests/integration/cli/commands/test_venv_create.py
index cd235484..cde00c23 100644
--- a/tests/integration/cli/commands/test_venv_create.py
+++ b/tests/integration/cli/commands/test_venv_create.py
@@ -649,7 +649,7 @@ def test_foreign_target_link_python_from_pex_repository(
foreign_platform = cross_arch_platform(3, 10)
pex_repository = tmpdir.join("psutil.pex")
run_pex_command(
- args=["psutil==5.9.5", "--platform", foreign_platform, "-o", pex_repository]
+ args=["psutil==7.2.2", "--platform", foreign_platform, "-o", pex_repository]
).assert_success()
venv_dir = tmpdir.join("venv")
--
2.55.0 |
Member
|
@jasonwbarnett please try out https://github.com/pex-tool/pex/releases/tag/v2.103.2 and close this PR if it meets your needs. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
pex3 venv create --link-pythonfor a foreign platform fails when the venv is installed from a--pex-repository:The distributions themselves install fine — it is populating the venv's sources that fails.
_populate_first_partyre-resolves the repository PEX to build thepex-replscript'sactivated_dists, and that resolve runs against a local interpreter, which a PEX built for the foreign platform alone holds nothing for._install_from_pexhas already resolved the distributions for the target, so this passes those down instead; thePEX_TOOLS=1 ... venvpath keeps re-resolving, where the PEX and the interpreter always agree.This is the combination Pants uses — it always invokes
pex3 venv createwith--pex-repository— so #3279 is unreachable from Pants without it.Repro and verification, on a linux aarch64 host:
Before: the error above. After:
bin/python,bin/python3andbin/python3.13all link to/opt/python/bin/bob, andlib/python3.13/site-packages/psutil/_psutil_linux.abi3.soisELF 64-bit LSB shared object, x86-64.test_foreign_target_link_python_from_pex_repositorycovers it. I could not run it locally — thepy310fixture's CPython 3.10.7 build fails in my environment (ensurepipexits non-zero) — so CI is its first run; the commands above are the same shape by hand.uv run dev-cmd format lint typecheckis green.Written with Claude Code, as was #3279.
🤖 Generated with Claude Code
https://claude.ai/code/session_012imZdyz1fhVXRNKPjbhSDw