Is Poetry not meant to select a compatible Python Version itself anymore?

[dotressel]

We recently updated our CI to use Poetry 2.4.1 and noticed a different behavior when it creates it's venv. Previously Poetry was smart about selecting a compatible Python version when we didn't run poetry env use ... prior to running poetry install or running any other command.

Now with 2.4.1 and most likely caused by this change (#10685) Poetry will happily run poetry install with an incompatible Python version but will fail on subsequent runs because the active Python version is not compatible.

Question:
Is one supposed to always run poetry env use ... with a compatible Python version prior to setting up the venv?

---

Some more information

OS - Amazon Linux 2023
Default Python Version - 3.9
Installed Python Versions - 3.9, 3.11, 3.13
virtualenvs.use-poetry-python - false

Extract from pyproject.toml for which we see the issue:

[tool.poetry]
name = "foo"
version = "0.0.1"
description = "bar"

[tool.poetry.dependencies]
python = ">=3.11,<4.0"

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

Before updating to 2.4.1

$ if [[ -f pyproject.toml ]]; then poetry install; fi
The currently activated Python version 3.9.25 is not supported by the project (^3.11).
Trying to find and use a compatible version. 
Using python3.13 (3.13.13)

After updating to 2.4.1

Shortened output of initial poetry install without using CI cache. It indicates, that it's creating a venv using Python 3.9.

$ if [[ -f pyproject.toml ]]; then poetry install; fi
Creating virtualenv foo in /<redacted>/.venv
Installing dependencies from lock file
Package operations: 399 installs, 0 updates, 0 removals
  - Installing ...
  - ...
Installing /<redacted>/.venv/lib/python3.9/site-packages/<redacted> over existing file

Output of subsequent poetry install using CI cache.

$ if [[ -f pyproject.toml ]]; then poetry install; fi
Current Python version (3.9.25) is not allowed by the project (>=3.11,<4.0).
Please change python executable via the "env use" command.

[radoering]

This may be an unintended side effect. #10685 only aimed to fix the issue that poetry env use ... just used another version if the chosen version was not compatible. (This command should force a specific version.)

However, I cannot reproduce your issue. I get the following warning the first time and in subsequent runs (if I have not run poetry env use ... before):

The currently activated Python version 3.10 is not supported by the project (>=3.11,<4.0).
Trying to find and use a compatible version. 

Have you run poetry env use ... before? Can you check your envs.toml for entries of the relevant project? (There shouldn't be any.) Can you provide a Dockerfile to reproduce the issue?

[dotressel]

Dockerfile to reproduce (it's quite bloated, but I didn't have time to optimize and minimize as much):

FROM public.ecr.aws/amazonlinux/amazonlinux:2023@sha256:5d271dd33b628252c31c15f5449e50d7c4e9f93badf9342f9b1e2a8d326fbe84

RUN dnf install -y curl-minimal awscli htop iftop iotop vim jq less dnsutils procps telnet unzip zip glibc-langpack-en shadow-utils findutils util-linux && \
    rm -rf /var/cache/dnf

RUN dnf install -y tar wget git gcc docker python3-pip python3.11 python3.11-pip python3.13 python3.13-pip openssl amazon-ecr-credential-helper libatomic && \
    rm -rf /var/cache/dnf

ARG POETRY_VERSION=2.4.1
RUN curl -sSL https://install.python-poetry.org | POETRY_VERSION=$POETRY_VERSION python3.13 - && \
    ln -sf /root/.local/bin/poetry /usr/bin/poetry && \
    poetry config virtualenvs.in-project true

COPY <<EOF /foo/pyproject.toml
[tool.poetry]
name = "foo"
version = "0.0.1"
package-mode = false

[tool.poetry.dependencies]
python = ">=3.11,<4.0"
aws-cdk-lib = "2.265.0"

[[tool.poetry.source]]
name = "pypi-prod"
url = "https://pypi.org/simple"

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"
EOF

Commands inside the container:

❯ docker run -it --rm <image-name>
bash-5.2# cd /foo/
bash-5.2# poetry env list
bash-5.2# python3 --version
Python 3.9.25
bash-5.2# poetry config --list
cache-dir = "/root/.cache/pypoetry"
data-dir = "/root/.local/share/pypoetry"
installer.max-workers = null
installer.no-binary = null
installer.only-binary = null
installer.parallel = true
installer.re-resolve = false
keyring.enabled = true
python.installation-dir = "{data-dir}/python"  # /root/.local/share/pypoetry/python
repositories.pypi-prod.url = "https://pypi.org/simple"
requests.max-retries = 0
solver.lazy-wheel = true
solver.min-release-age = 0
solver.min-release-age-exclude = null
solver.min-release-age-exclude-source = null
system-git-client = false
virtualenvs.create = true
virtualenvs.in-project = true
virtualenvs.options.always-copy = false
virtualenvs.options.no-pip = false
virtualenvs.options.system-site-packages = false
virtualenvs.path = "{cache-dir}/virtualenvs"  # /root/.cache/pypoetry/virtualenvs
virtualenvs.prompt = "{project_name}-py{python_version}"
virtualenvs.use-poetry-python = false
bash-5.2# cat pyproject.toml 
[tool.poetry]
name = "foo"
version = "0.0.1"
package-mode = false

[tool.poetry.dependencies]
python = ">=3.11,<4.0"
aws-cdk-lib = "2.265.0"

[[tool.poetry.source]]
name = "pypi-prod"
url = "https://pypi.org/simple"

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"
bash-5.2# poetry install
Creating virtualenv foo in /foo/.venv
Updating dependencies
Resolving dependencies... (1.3s)

Package operations: 13 installs, 0 updates, 0 removals

  - Installing attrs (26.1.0)
  - Installing six (1.17.0)
  - Installing typing-extensions (4.16.0)
  - Installing cattrs (26.1.0)
  - Installing python-dateutil (2.9.0.post0)
  - Installing publication (0.0.3)
  - Installing typeguard (2.13.3)
  - Installing jsii (1.139.0)
  - Installing aws-cdk-cloud-assembly-schema (54.19.0)
  - Installing constructs (10.8.1)
  - Installing aws-cdk-asset-node-proxy-agent-v6 (2.1.2)
  - Installing aws-cdk-asset-awscli-v1 (2.2.292)
  - Installing aws-cdk-lib (2.265.0): Installing...
  - Installing aws-cdk-lib (2.265.0)

Writing lock file
bash-5.2# ls .venv/bin
activate  activate.csh  activate.fish  activate.nu  activate.ps1  activate.xsh  activate_this.py  pip  pip-3.9  pip3  pip3.9  python  python3  python3.9
bash-5.2# poetry install

Current Python version (3.9.25) is not allowed by the project (>=3.11,<4.0).
Please change python executable via the "env use" command.                                                                                                                                                                
bash-5.2# poetry --version
Poetry (version 2.4.1)
bash-5.2# cat .venv/pyvenv.cfg 
home = /usr/bin
implementation = CPython
version_info = 3.9.25.final.0
version = 3.9.25
executable = /usr/bin/python3.9
command = /root/.local/share/pypoetry/venv/bin/python -m virtualenv /foo/.venv
virtualenv = 21.7.4
prompt = "foo-py3.13"
include-system-site-packages = false
base-prefix = /usr
base-exec-prefix = /usr
base-executable = /usr/bin/python3
bash-5.2# 

As you can see, the initial poetry install happily creates a venv with python3.9 as a base even though the pyproject.toml does not allow it. Subsequent commands fail "correctly".
Hope the sequence of commands gives a clear picture about the environment.

[ShreeChaturvedi]

No, you do not have to run poetry env use first. Auto-selection is still supposed to work after 2.4.1.

With virtualenvs.use-poetry-python=false, Poetry starts from the active interpreter (your system 3.9). If that is outside >=3.11,<4.0, it should print "Trying to find and use a compatible version" and pick 3.13. PR 10685 only changed the explicit path: poetry env use now errors instead of falling back.

The later CI failure is from an existing 3.9 .venv, not from auto-select being removed:

Current Python version (3.9.25) is not allowed by the project (>=3.11,<4.0).
Please change python executable via the "env use" command.

Pin in CI, and drop a cached bad env first if needed:

rm -rf .venv
poetry env use 3.13
poetry install

Also check {cache-dir}/virtualenvs/envs.toml for a leftover project entry. env use is only required when you want a specific version or when an incompatible .venv already exists. If a clean tree still creates 3.9 without the search line, that is unexpected.

[dotressel]

Okay so I suppose what I experience is a bug. There is no existing venv anywhere otherwise poetry would not create a new one in /foo/.venv. The issue is, that it happily creates the new venv with the wrong python version instead of relying on auto-selection.

We have worked around this issue by always using poetry env use before the first poetry command, but we would love to get rid of this and rely on auto-selection again.

[SirHegel]

It is worth separating two things here, because one of them is working as documented and the other one is the actual bug.

What use-poetry-python = false actually promises

From the configuration docs:

By default, Poetry will use the activated Python version to create a new virtual environment. If set to true, the Python version used during Poetry installation is used.

Note what that says and what it does not. It says the activated interpreter. It does not say a compatible interpreter, and it never promised to go looking for one. On your Amazon Linux 2023 box the activated interpreter is 3.9, so with use-poetry-python = false that is what Poetry is told to use — regardless of what requires-python says.

The auto-selection you were relying on before 2.4.1 was not a documented contract. It fell out of the looser condition that #10685 tightened up. The PR description is explicit about the intent:

use-poetry-python should only be relevant if no specific python version has been requested.

So to your direct question — no, you should not need poetry env use as a matter of course. But you do need the activated interpreter to be one your project supports, because that is the input you have configured Poetry to trust.

Three ways to fix the CI, in order of preference

Activate a compatible interpreter before Poetry runs. This is the one that matches what you have configured. If 3.11 is installed, put it on PATH ahead of 3.9 for the job, and everything downstream follows without further Poetry configuration:

export PATH="/usr/bin/python3.11-bin:$PATH"   # or whatever exposes 3.11 as `python`
poetry install

Or flip use-poetry-python to true. Then Poetry uses the interpreter it was installed under and stops caring about the shell's active one. This is the better choice if Poetry itself is installed under a compatible version and you would rather not manage PATH per job.

Or pin it explicitly with poetry env use. This works, but it is the most brittle of the three: it hardcodes a version into your CI that has to be updated by hand every time the supported range moves.

The part that is a genuine bug

poetry install succeeding against an incompatible interpreter and then failing on subsequent commands is not good behaviour regardless of which setting you are using. If the interpreter does not satisfy requires-python, the failure belongs at the point the environment is created, with a message naming the constraint and the interpreter it found — not several commands later, in a state that looks like it installed fine.

@radoering already called this an unintended side effect above. It is worth filing separately from this discussion, because the fix for it is independent of the configuration question: whatever use-poetry-python is set to, creating a venv that the project cannot use should fail loudly at creation time.

That would also have made your upgrade a five-second diagnosis instead of a puzzling one.

[dotressel]

Great answer. Makes a lot more sense now.

Two follow up questions:

1. How do you get poetry to find and select a compatible interpreter version now? For us it is the most convenient way of using poetry, because we cannot make sure, that the active version nor the one we install poetry with is compatible with all our projects. It seems to me, that this is not possible anymore. Also, all your three fixes suggest pinning the active interpreter version in one way or another.
2. How do we go about with the bug that we discovered? I found creating a bug ticket very tedious because there is a lot of information I would have to collect. I don't have that amount of time to invest.

[SirHegel]

Your first question is a fair objection, and checking it properly shows I got part of my answer wrong. Correcting that first, because it changes the advice.

Auto-selection is not gone

I said it was never a documented contract and left the impression it had been removed. It has not been. It is still in EnvManager.create_venv, and it runs whenever no specific interpreter was requested:

specific_python_requested = python is not None
if not python:
    python = Python.get_preferred_python(config=self._poetry.config, io=self._io)
...
if not supported_python.allows(python.patch_version):
    if specific_python_requested:
        raise NoCompatiblePythonVersionFoundError(...)
    self._io.write_error_line(
        f"The currently activated Python version ... is not supported by the project ...\n"
        "Trying to find and use a compatible version."
    )
    python = Python.get_compatible_python(poetry=self._poetry, io=self._io)

Verified on 2.5.0.dev0 with a project requiring a version I do not have installed, on a clean tree:

$ poetry install
The currently activated Python version 3.14.4 is not supported by the project (>=3.15,<4.0).
Trying to find and use a compatible version.
Poetry was unable to find a compatible version. ...

It searched. #10685 only tightened the explicit path — poetry env use now errors instead of silently substituting. That matches what @radoering said, and it is what he was seeing.

So none of my three fixes should be necessary

What actually stops the search is an environment that already exists. A few lines earlier:

if (env.is_venv() and not force) or not create_venv:
    current_python = Version.parse(...)
    if not supported_python.allows(current_python):
        raise InvalidCurrentPythonVersionError(
            ..., note='Please change python executable via the "env use" command.'
        )
    return env

That returns before auto-selection is ever reached. Same project, same Poetry, only difference being a .venv already sitting there:

$ python3 -m venv .venv && poetry install
Current Python version (3.14.4) is not allowed by the project (>=3.15,<4.0).
Please change python executable via the "env use" command.

That is your subsequent-run error, and it is the message telling you to pin — not a sign that you have to. So the thing to fix in CI is the stale environment, not the interpreter selection: make sure the job does not carry a .venv (or a {cache-dir}/virtualenvs entry) created under 3.9 into a run that needs 3.11+. @ShreeChaturvedi's rm -rf .venv is the short version, and if you cache the virtualenvs directory across jobs, the cache key wants the Python version in it.

Keeping use-poetry-python = false is fine. Where I was right is narrower than I made it sound: it means Poetry starts from the activated interpreter — it just does not stop there.

On the bug, and on not having time to write it up

I tried to reproduce the part you described — poetry install succeeding against an incompatible interpreter — and could not, on either path. Clean tree searches; existing incompatible env refuses. So there is a piece of your environment doing something neither branch above explains, and a ticket that cannot reproduce is worse than no ticket.

One thing would settle it, and it is a copy-paste rather than an investigation. On the CI job, before anything else:

rm -rf .venv
poetry config --list | grep -E 'virtualenvs\.(create|in-project|use-poetry-python|path)'
poetry install 2>&1 | head -20

The question is whether that first install prints Trying to find and use a compatible version. If it does and still lands on 3.9, that is a real bug in get_compatible_python and worth filing. If the line never appears, something reached the early-return branch, and the interesting part is what — most likely the job is already inside an activated virtualenv, which makes env.is_venv() true on the very first run.

Paste that output here and I will write the ticket up properly, with the reproduction and the code paths. You should not have to spend an afternoon on it to report something you already found.