compile_pip_requirements does not use credential helper

[jml-derek]

🐞 Bug Report

Affected Rule

The issue is caused by the pip.parse rule with experimental_index_url and experimental_index_url_overrides configurations.

Is this a regression?

I have not tried previous versions. This is my first attempt following the documentation at https://github.com/bazelbuild/rules_python/blob/389431bba6f9a4b46b6cf15dd9cd24a1f52f6e16/docs/pypi-dependencies.md?plain=1#L417.

Description

When using pip.parse with experimental_index_url and experimental_index_url_overrides configured with a credential helper script that directly attaches a token generated from the gcloud CLI, authentication fails. The command still prompts for username and password instead of using the token provided by the helper script.

🔬 Minimal Reproduction

I've created a test repository that demonstrates the issue: https://github.com/derek-jml/test_repo

Steps to reproduce:

1. Clone the repository
2. Run bazel run //:requirements.update
3. Observe that it prompts for username and password instead of using the token

🔥 Exception or Error

When running `bazel run //:requirements.update`, it prompts for username and password authentication instead of using the token provided by the credential helper script.

🌍 Your Environment

Operating System:

[Your OS information]

Output of bazel version:

8.1.1

Rules_python version:

1.2.0

Anything else relevant?
I'm following the documentation for using private PyPI repositories with authentication as described in the pypi-dependencies.md file. The specific section I'm referencing is about using credential helper scripts with experimental_index_url and experimental_index_url_overrides.

The credential helper script generates a token from the gcloud CLI that should be used automatically, but the authentication mechanism appears to ignore this and still prompts for username/password credentials.

[rickeylev]

This potentially sounds like #2640 ?

Getting some more details about when it's prompting for the user/pass would be helpful. If it's because of the pip wheel command, then its the same cause.

Maybe try running with < /dev/null piped in? this would close stdin so the prompt immediately fails and, hopefully, provide a traceback with more details.

[jml-derek]

This potentially sounds like #2640 ?

Getting some more details about when it's prompting for the user/pass would be helpful. If it's because of the pip wheel command, then its the same cause.

Maybe try running with < /dev/null piped in? this would close stdin so the prompt immediately fails and, hopefully, provide a traceback with more details.

there is not a lot of information in bug trace.
//:requirements.update is a compile_pip_requirements target

bazel run --sandbox_debug //:requirements.update < /dev/null
INFO: Analyzed target //:requirements.update (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
Target //:requirements.update up-to-date:
bazel-bin/requirements.update
INFO: Elapsed time: 0.235s, Critical Path: 0.01s
INFO: 1 process: 1 internal.
INFO: Build completed successfully, 1 total action
INFO: Running command line: bazel-bin/requirements.update '--src=_main/requirements.in' _main/requirements_lock.txt //:requirements.update '--resolver=backtracking' --allow-unsafe --generate-hashes
Updating requirements_lock.txt
User for us-central1-python.pkg.dev:
Aborted!

[dougthor42]

Looks like you're missing some quotes in your JSON output from the cred helper. Give this a try:

-echo {"headers": {"Authorization: Bearer $TOKEN"}}
+echo '{"headers": {"Authorization": ["Bearer '$TOKEN'"]}}'

[dougthor42]

Replied in chat, too, but copying here for posterity and for other's benefit.

1. Your cred helper needs to be executable chmod +x tools/artifact_registry/cred_helper.sh. I assume that's just a minor oversight in the example repo.
2. If you're using rules_python to lock your requirements via compile_pip_requirements, then your requirements.in file needs oauth2accesstoken@.

diff --git a/requirements.in b/requirements.in
index aa57009..b7da3a2 100644
--- a/requirements.in
+++ b/requirements.in
@@ -1,3 +1,3 @@
 --extra-index-url https://pypi.org/simple
---extra-index-url https://us-central1-python.pkg.dev/steady-state-msmt-simulations/python-repo/simple
+--extra-index-url https://oauth2accesstoken@us-central1-python.pkg.dev/steady-state-msmt-simulations/python-repo/simple
 opacus

With that change, I get auth failures (as expected, as I'm using a dummy $TOKEN.

$ bazel run //:requirements.update                                                                          
INFO: Analyzed target //:requirements.update (90 packages loaded, 4369 targets configured).
INFO: Found 1 target...                                                                                    
Target //:requirements.update up-to-date:                                                                  
  bazel-bin/requirements.update                                                                            
INFO: Elapsed time: 1.225s, Critical Path: 0.03s                                                           
INFO: 1 process: 4 action cache hit, 1 internal.                                                           
INFO: Build completed successfully, 1 total action                                                         
INFO: Running command line: bazel-bin/requirements.update '--src=_main/requirements.in' _main/requirements_lock.txt //:requirements.update '--resolver=backtracking' --allow-unsafe --generate-hashes
Updating requirements_lock.txt                                                                             
  WARNING: 401 Error, Credentials not correct for https://us-central1-python.pkg.dev/steady-state-msmt-simulations/python-repo/simple/opacus/
  WARNING: 401 Error, Credentials not correct for https://us-central1-python.pkg.dev/steady-state-msmt-simulations/python-repo/simple/numpy/
  WARNING: 401 Error, Credentials not correct for https://us-central1-python.pkg.dev/steady-state-msmt-simulations/python-repo/simple/torch/
  WARNING: 401 Error, Credentials not correct for https://us-central1-python.pkg.dev/steady-state-msmt-simulations/python-repo/simple/scipy/
...

Your lockfile will then also include oauth2accesstoken@ after running the lock:

diff --git a/requirements_lock.txt b/requirements_lock.txt
index 809b052..5a23e8b 100644
--- a/requirements_lock.txt
+++ b/requirements_lock.txt
@@ -4,7 +4,7 @@
 #
 #    bazel run //:requirements.update
 #
---extra-index-url https://us-central1-python.pkg.dev/steady-state-msmt-simulations/python-repo/simple
+--extra-index-url https://oauth2accesstoken@us-central1-python.pkg.dev/steady-state-msmt-simulations/python-repo/simple
 
 filelock==3.17.0 \
     --hash=sha256:533dc2f7ba78dc2f0f531fc6c4940addf7b70a481e269a5a3b93be94ffbe8338 \

[jml-derek]

Root Cause Identified

I believe we've found the issue. There's a fundamental difference in how authentication works between different commands:

bazel run //:requirements.update Authentication

• Does not work with Bazel credential helpers at all
• Requires [keyring](https://cloud.google.com/artifact-registry/docs/python/authentication#keyring) to be installed locally
• When installed, it uses the keyring gcloud toolchain to generate the requirement.txt

Dependency Resolution Authentication

• The credential helper works only when experimental_index_url and experimental_index_url_overrides are set in pip.parse
• This is because these settings control the Bazel downloader's behavior
• If these settings are not provided (meaning we're not using the Bazel downloader), then pip's default downloader is used instead
• The pip downloader will still use keyring in this case, functioning the same way as bazel run //:requirements.update

[aignas]

This is working as intended, I think, because repository_ctx and module_ctx is not used.

However, PRs to integrate are welcome, #2657 may be a good base for that once it is merged.

[aignas]

The credential helper usage is something that is just not possible for the pip-compile rule or equivalents because they are not happening at repository rule time and the execution context is different. What is more the underlying tools do not have the support for credential helper style keyring implementations. I am closing this as unplanned. If uv or pip-compile has a credential helper equivalent we can reopen this issue.

[dougthor42]

+1.

As a workaround, I recommend locking your pip deps outside of Bazel and then just have rules_python update the manifest. Eg:

uv pip compile \
    --output-file=requirements_lock.txt \
    --emit-index-url \
    --generate-hashes \
    --no-strip-extras \
    requirements.in
bazel run //:gazelle_python_manifest.update

[aignas]

With #2657 maybe it will be possible to provide your own uv toolchain and or provide a keyring parameter to the rule. It should be already functional, so if you would like to test, please give us some feedback there.