Skip to content

Fix pex3 venv create --link-python for --pex-repository. - #3285

Closed
jasonwbarnett wants to merge 1 commit into
pex-tool:mainfrom
altana-ai:venv-create-pex-repository-link-python
Closed

jasonwbarnett wants to merge 1 commit into
pex-tool:mainfrom
altana-ai:venv-create-pex-repository-link-python

Conversation

@jasonwbarnett

Copy link
Copy Markdown
Contributor

pex3 venv create --link-python for a foreign platform fails when the venv is installed from a --pex-repository:

A distribution for psutil could not be resolved for <local python>.
Found 1 distribution for psutil that does not apply:
1.) The wheel tags for psutil 5.9.5 are cp36-abi3-manylinux_2_12_x86_64, ... which do not match
    the supported tags of <local python>: cp313-cp313-manylinux_2_39_aarch64 ...

The distributions themselves install fine — it is populating the venv's sources that fails. _populate_first_party re-resolves the repository PEX to build the pex-repl script's activated_dists, and that resolve runs against a local interpreter, which a PEX built for the foreign platform alone holds nothing for. _install_from_pex has already resolved the distributions for the target, so this passes those down instead; the PEX_TOOLS=1 ... venv path keeps re-resolving, where the PEX and the interpreter always agree.

This is the combination Pants uses — it always invokes pex3 venv create with --pex-repository — so #3279 is unreachable from Pants without it.

Repro and verification, on a linux aarch64 host:

pex psutil==5.9.5 --platform linux_x86_64-cp-313-cp313 -o psutil.pex
pex3 venv create -d venv --pex-repository psutil.pex --platform linux_x86_64-cp-313-cp313 \
    --python-path $(command -v python3.13) --link-python /opt/python/bin/bob

Before: the error above. After: bin/python, bin/python3 and bin/python3.13 all link to /opt/python/bin/bob, and lib/python3.13/site-packages/psutil/_psutil_linux.abi3.so is ELF 64-bit LSB shared object, x86-64.

test_foreign_target_link_python_from_pex_repository covers it. I could not run it locally — the py310 fixture's CPython 3.10.7 build fails in my environment (ensurepip exits non-zero) — so CI is its first run; the commands above are the same shape by hand. uv run dev-cmd format lint typecheck is green.

Written with Claude Code, as was #3279.

🤖 Generated with Claude Code

https://claude.ai/code/session_012imZdyz1fhVXRNKPjbhSDw

Populating a venv's sources re-resolved the repository PEX, and that resolve
runs against a local interpreter. For a venv laid out for a foreign platform
the PEX need hold no distribution that interpreter can use, so a PEX built for
the foreign platform alone failed with e.g.:

    A distribution for psutil could not be resolved for <local python>.

`_install_from_pex` has already resolved the distributions for the target, so
pass those down to the venv's `pex-repl` script instead of re-resolving. The
`PEX_TOOLS=1 ... venv` path keeps re-resolving, where the PEX and the
interpreter always agree.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012imZdyz1fhVXRNKPjbhSDw
@jasonwbarnett
jasonwbarnett force-pushed the venv-create-pex-repository-link-python branch from b7b841d to 07a5c2a Compare September 17, 2026 22:56
@jsirois

jsirois commented Sep 18, 2026

Copy link
Copy Markdown
Member

@jasonwbarnett no more Claude please. I haven't had to write an AI policy yet for Pex, but this is the 2nd PR from the altana-ai/pex repo where I can not contribute:

:; git push https://github.com/altana-ai/pex HEAD:venv-create-pex-repository-link-python
remote: Permission to altana-ai/pex.git denied to jsirois.
fatal: unable to access 'https://github.com/altana-ai/pex/': The requested URL returned error: 403

Now the altana-ai/pex repo thing may or may not have anything to do with Claude, but regardless, two strikes and you're out.

I'll do another hand merge to include these:

From 45a9df95bde15cda58c4e6058aca126ad0c2992e Mon Sep 17 00:00:00 2001
From: John Sirois <john.sirois@gmail.com>
Date: Thu, 17 Sep 2026 23:29:02 -0700
Subject: [PATCH 1/2] Only resolve once.

---
 pex/cli/commands/venv.py |  6 +++---
 pex/venv/installer.py    | 17 +++++++----------
 2 files changed, 10 insertions(+), 13 deletions(-)

diff --git a/pex/cli/commands/venv.py b/pex/cli/commands/venv.py
index 98aaf998..6f746150 100644
--- a/pex/cli/commands/venv.py
+++ b/pex/cli/commands/venv.py
@@ -42,7 +42,7 @@ from pex.venv.installer_configuration import InstallerConfiguration
 from pex.venv.virtualenv import Virtualenv

 if TYPE_CHECKING:
-    from typing import Any, Dict, Iterable, Optional, Sequence, Union
+    from typing import Any, Dict, Optional, Sequence, Union


 logger = logging.getLogger(__name__)
@@ -495,7 +495,7 @@ def _install_from_pex(
     pex,  # type: PEX
     installer_configuration,  # type: InstallerConfiguration
     provenance,  # type: Provenance
-    distributions,  # type: Iterable[Distribution]
+    distributions,  # type: Sequence[Distribution]
     dest_dir,  # type: str
     hermetic_scripts,  # type: bool
     venv=None,  # type: Optional[Virtualenv]
@@ -531,9 +531,9 @@ def _install_from_pex(
                 venv=venv,
                 pex=pex,
                 provenance=provenance,
+                activated_dists=distributions,
                 bin_path=installer_configuration.bin_path,
                 hermetic_scripts=hermetic_scripts,
-                activated_dists=distributions,
             )
         else:
             installer.populate_flat_sources(dst=dest_dir, pex=pex, provenance=provenance)
diff --git a/pex/venv/installer.py b/pex/venv/installer.py
index 2241c75e..33becfc0 100644
--- a/pex/venv/installer.py
+++ b/pex/venv/installer.py
@@ -360,11 +360,11 @@ def populate_venv_sources(
     venv,  # type: Virtualenv
     pex,  # type: PEX
     provenance,  # type: Provenance
+    activated_dists,  # type: Sequence[Distribution]
     bin_path=BinPath.FALSE,  # type: BinPath.Value
     hermetic_scripts=True,  # type: bool
     shebang=None,  # type: Optional[str]
     set_last_access=True,  # type: bool
-    activated_dists=None,  # type: Optional[Iterable[Distribution]]
 ):
     # type: (...) -> str

@@ -374,11 +374,11 @@ def populate_venv_sources(
             target_dir=provenance.target_dir,
             venv=venv,
             pex=pex,
+            activated_dists=activated_dists,
             shebang=shebang,
             venv_python=provenance.target_python,
             bin_path=bin_path,
             set_last_access=set_last_access,
-            activated_dists=activated_dists,
         )
     )
     return shebang
@@ -422,10 +422,11 @@ def populate_venv_from_pex(
     shebang = provenance.calculate_shebang(hermetic_scripts=hermetic_scripts)
     top_level_source_packages = tuple(iter_top_level_source_packages(pex))

+    distributions = tuple(pex.resolve())
     if scope in (InstallScope.ALL, InstallScope.DEPS_ONLY):
         populate_venv_distributions(
             venv=venv,
-            distributions=pex.resolve(),
+            distributions=distributions,
             copy_mode=copy_mode,
             hermetic_scripts=hermetic_scripts,
             provenance=provenance,
@@ -436,6 +437,7 @@ def populate_venv_from_pex(
         populate_venv_sources(
             venv=venv,
             pex=pex,
+            activated_dists=distributions,
             bin_path=bin_path,
             hermetic_scripts=hermetic_scripts,
             provenance=provenance,
@@ -747,11 +749,11 @@ def _populate_first_party(
     target_dir,  # type: str
     venv,  # type: Virtualenv
     pex,  # type: PEX
+    activated_dists,  # type: Sequence[Distribution]
     shebang,  # type: str
     venv_python,  # type: str
     bin_path,  # type: BinPath.Value
     set_last_access,  # type: bool
-    activated_dists=None,  # type: Optional[Iterable[Distribution]]
 ):
     # type: (...) -> Iterator[Tuple[Text, Text]]

@@ -767,12 +769,7 @@ def _populate_first_party(
         target_dir=target_dir,
         venv=venv,
         pex_info=pex_info,
-        # N.B.: A venv laid out for a foreign platform cannot re-resolve the PEX: that resolve
-        # runs against a local interpreter, which the PEX need hold no distributions for. The
-        # caller passes the distributions it resolved for the target instead.
-        activated_dists=(
-            tuple(activated_dists) if activated_dists is not None else tuple(pex.resolve())
-        ),
+        activated_dists=activated_dists,
         shebang=shebang,
         venv_python=venv_python,
         bin_path=bin_path,
--
2.55.0


From 8b25c46ab2317d735f6f20fdcface3a0bd3bc8d0 Mon Sep 17 00:00:00 2001
From: John Sirois <john.sirois@gmail.com>
Date: Thu, 17 Sep 2026 23:30:39 -0700
Subject: [PATCH 2/2] Fix test - psutil 5.9.5 has no aarch64 wheels for Linux
 test.

The resulting yolo build using the local interpreter netted native wheels
not compatible with the foreign platform.
---
 tests/integration/cli/commands/test_venv_create.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tests/integration/cli/commands/test_venv_create.py b/tests/integration/cli/commands/test_venv_create.py
index cd235484..cde00c23 100644
--- a/tests/integration/cli/commands/test_venv_create.py
+++ b/tests/integration/cli/commands/test_venv_create.py
@@ -649,7 +649,7 @@ def test_foreign_target_link_python_from_pex_repository(
     foreign_platform = cross_arch_platform(3, 10)
     pex_repository = tmpdir.join("psutil.pex")
     run_pex_command(
-        args=["psutil==5.9.5", "--platform", foreign_platform, "-o", pex_repository]
+        args=["psutil==7.2.2", "--platform", foreign_platform, "-o", pex_repository]
     ).assert_success()

     venv_dir = tmpdir.join("venv")
--
2.55.0

@jsirois

jsirois commented Sep 18, 2026

Copy link
Copy Markdown
Member

@jasonwbarnett please try out https://github.com/pex-tool/pex/releases/tag/v2.103.2 and close this PR if it meets your needs.

@jasonwbarnett
jasonwbarnett deleted the venv-create-pex-repository-link-python branch September 18, 2026 15:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants