Repository navigation
[BUG] pyproject.toml: Unicode character in a license file prevents dependencies from being installed #4033
Description
Description
Some licenses, in this specific example, LGPL-2.1, contains Unicode characters. In case of LGPL 2.1 it is U+000c - Form Feed character.
In pyproject.toml having a license file with those characters silently breaks package dependencies installation.
Expected behavior
pip install -e .installs package dependenciespip install -e .[dev]installs optionaldevdependencies group
pip version
23.2.1
Python version
3.10, 3.11
OS
Fedora Linux 38
How to Reproduce
- Create
pyproject.toml
[project]
name = "unicode-bug"
version = "0.1.0"
requires-python = ">=3.10"
license = {file = "LICENSE"}
dependencies = [
"requests"
]
[project.optional-dependencies]
dev = [
"pytest"
]- Create
LICENSEfile with unicode character
$ echo -e '\U000b' > LICENSE
- Try to install the package and optional dependencies - doesn't install dependencies
$ pip install -e .
...
Successfully built unicode-bug
Installing collected packages: unicode-bug
Attempting uninstall: unicode-bug
Found existing installation: unicode-bug 0.1.0
Uninstalling unicode-bug-0.1.0:
Successfully uninstalled unicode-bug-0.1.0
Successfully installed unicode-bug-0.1.0
$ pip install -e .[dev]
...
WARNING: unicode-bug 0.1.0 does not provide the extra 'dev'
...
- Remove unicode character
$ echo "" > LICENSE
- Try to install the package and optional dependencies - installation works
$ pip install -e .
...
Successfully built unicode-bug
Installing collected packages: urllib3, idna, charset-normalizer, certifi, requests, unicode-bug
Attempting uninstall: unicode-bug
Found existing installation: unicode-bug 0.1.0
Uninstalling unicode-bug-0.1.0:
Successfully uninstalled unicode-bug-0.1.0
Successfully installed certifi-2023.7.22 charset-normalizer-3.2.0 idna-3.4 requests-2.31.0 unicode-bug-0.1.0 urllib3-2.0.4
$ pip install -e .[dev]
Successfully built unicode-bug
Installing collected packages: pluggy, packaging, iniconfig, unicode-bug, pytest
Attempting uninstall: unicode-bug
Found existing installation: unicode-bug 0.1.0
Uninstalling unicode-bug-0.1.0:
Successfully uninstalled unicode-bug-0.1.0
Successfully installed iniconfig-2.0.0 packaging-23.1 pluggy-1.3.0 pytest-7.4.0 unicode-bug-0.1.0
Output
No response
Code of Conduct
- I agree to follow the PSF Code of Conduct.
Activity
RonnyPfannschmidt commentedon Aug 29, 2023
What's the output when using a valid build-system section, and what warnings get shown when using build to make the wheel?
yurinnick commentedon Aug 29, 2023
Add
build-systemsection:Install and build the wheel with broken license file:
So no new warnings or errors as far as I can tell.
RonnyPfannschmidt commentedon Aug 29, 2023
I meant the build package,which has easier opt out of build output hiding
yurinnick commentedon Aug 29, 2023
python -m buildyields this output: https://gist.github.com/yurinnick/bf06c9d0f97f9c361b38038f5fe8e072pradyunsg commentedon Aug 29, 2023
FWIW, omitting
build-systemis definitely valid, and passing-vto pip will present the build output. There is no need to usebuildfor it.pfmoore commentedon Aug 29, 2023
Works fine for me on Windows:
(note pytest is installed, so the
devextra was recognised).yurinnick commentedon Aug 29, 2023
Verbose pip install output: https://gist.github.com/yurinnick/9128d68e5f9be9b0b50362122d79346c
yurinnick commentedon Aug 30, 2023
I did some research and my findings so far:
pip._internal.metadata.importlib._dists.DistributionclassDistributionloaded fromunicode_bug.egg_infodirectory has valid metadata [1]Discributionloaded from temporary/tmp/pip-modern-metadata-0q88s1hj/unicode_bug-0.1.0.dist-infohas invalid metadata [2][1]
[2]
uranusjr commentedon Aug 30, 2023
This sounds like a setuptools bug to me. The dist-info directory in the temporary directory is generated by the build backend (setuptools) and pip only reads the result.
pyproject.toml: add initial file
pradyunsg commentedon Aug 30, 2023
This is definitely a setuptools bug. Transferring the issue over. :)
abravalheri commentedon Aug 30, 2023
Hi @yurinnick thank you very much for reporting, I will have a look on this.
This is interesting. Is the PKG-INFO file in the top of the sdist OK? I had a quick view in the code and it seems that we do use UTF-8 to read the file pointed out in
pyproject.toml. And then we use UTF-8 again to write PKG-INFO.One thing that might be related is that setuptools produces
PKG-INFOfiles and those are later transformed intoMETADATAbypypa/wheel. But thenpypa/wheelalso seems to deal with the files using UTF-8.The code contains some places that we use
email.message_from_file(via distutils) to readPKG-INFOwhen building from thesdist, this might be where UTF-8 is not explicitly used. But I am not sure this is used, I need to investigate better.If you build the project skipping the
sdiststep (e.g.python -m build --wheel) is the metadata file still ill-formed?yurinnick commentedon Aug 30, 2023
PKG-INFOinegg-infodirectory has the unicode character, but gets parsed correctly.LICENSEfile fromdist-infodirectory also contains unicode, butMETADATAfile doesn't.Here are the files in the hex form: https://gist.github.com/yurinnick/94fc3d9b19e0bf6270f7dd332f4b0e61
pfmoore commentedon Aug 30, 2023
One thing confuses me. You keep referring to a "unicode character", but the reproducer you gave (which, as I said, works fine for me on Windows) uses the character
\U000b, which is a perfectly ordinary ASCII character - 'LINE TABULATION' (U+000B).Is the problem here caused by the fact that this character is whitespace, and as such is getting stripped at some point due to something removing trailing whitespace from lines?
1 remaining item
abravalheri commentedon Aug 30, 2023
I added some debug statements to see when the transformation is happening and it seems to come from https://github.com/pypa/wheel/blob/0.41.2/src/wheel/bdist_wheel.py#L587:
... serialization_policy = EmailPolicy( # email.policy.EmailPolicy utf8=True, mangle_from_=False, max_line_length=0, ) with open(pkg_info_path, "w", encoding="utf-8") as out: Generator(out, policy=serialization_policy).flatten(pkg_info) # email.generator.Generator ...This is how I obtained the info:
(patch format)
And this is the output (when I run
python -m build --no-isolation1):Now, I am not sure why stdlib's
email.policy.EmailPolicyis replacing the UTF-8 character ifutf8=True... Maybe that is part of the spec in the email message RFC? I am not sure how to tackle this.Maybe it is worth to discuss with
pypa/wheel?Do the
pipmaintainers have any suggestions?Something very similar happens if you replace
\U000bwith\U000c.Footnotes
I use
--no-isolationto force the "patched" version ofwheelto be used instead ofbuilddownloading a new one. ↩pfmoore commentedon Aug 30, 2023
Hang on.
License-Pathisn't valid metadata. So presumably it's a setuptools-specific thing, and bdist_wheel is trying to make it into a valid value for theLicensefield?It looks like bdist_wheel is failing to "properly"1 escape that value, resulting in a metadata file with an unescaped
\n\nin the license field, which is then treated as the end of the headers with everything following being dumped into the description (because it's now the message body).Footnotes
I have no idea what proper escaping would be for a multi-line value, it's not at all well-defined if I recall correctly. ↩
abravalheri commentedon Aug 30, 2023
The
License-Fileis left unprocessed (which is unspecified, but was implemented in setuptools by a contribution in the context of PEP 639 a few years ago1).The
Licensefield is already originally present in thePKG-INFOfile (and it contains the UTF-8 char).When
project.licence.fileis given inpyproject.toml,setuptoolswill read the file (using UTF-8) and populate theLicensecore-metadata (that seems to be what is meant to happen by PEP 621).Footnotes
I initially thought it was a contribution of the original author of PEP 639, but I was mistaken. ↩
abravalheri commentedon Aug 30, 2023
@agronholm do you have any thoughts in the matter? (see #4033 (comment)).
The escaping behaviour seems to be inherited by
bdist_wheelfrom the standard library, but I am not sure if it is intentional or accidental (in the stdlib, sinceutf8=Trueis passed to theEmailPolicy)pfmoore commentedon Aug 30, 2023
Ah. I missed that. Looks like it's translating
\x0b to\n` in that case.Yes:
produces
So that policy doesn't handle arbitrary control characters correctly...
It's worth noting that the specification is (deliberately) vague on how to serialise metadata to a file:
So there's no formal answer to the question "how do we handle data like this?" - other than "don't do that, then" 🙁
pfmoore commentedon Aug 30, 2023
The following is worth noting:
The email module can parse data where header values contain values like
\x0b, but it can't produce them. So it doesn't round-trip correctly.abravalheri commentedon Aug 30, 2023
One minor comment is that the
compat32policy will do a different escaping that could potentially work (I haven't tested)1. Thecompat32is however a bit problematic because it does not support UTF-8.If we replace the
EmailPolicy(utf8=True, ...)withcompat32we are likely to start receiving reports of people's names/readmes not rendering properly on PyPI...Footnotes
It still removes
\x0bbut it does not seem to add a second\n:print(repr(compat32.fold("License", '\x0b\n '))) => 'License: \n'↩pfmoore commentedon Aug 30, 2023
Like it or not, compat32 (for reading) is what the standard says. I omitted an additional part of the spec:
And what does (mostly) work is to read the file using UTF-8 into a string, and then parse with
email.message_from_string.You can't round-trip like this, and the email module (and the spec!) gives you little or no help in writing such a metadata file, so I agree that
bdist_wheelhas a problem with creating a file like this, but if you can create one, it's readable.IMO, the temporary solution here is probably for setuptools to just refuse to allow control characters in a license file. The UTF-8 question is something of a red herring here, as the current behaviour with UTF-8 is fine, it's only control character handling that seems broken.
Longer term, license file handling will have to be dealt with as part of PEP 639, and if the problem of writing arbitrary user-entered text into metadata values other than license and description becomes a problem, someone will need to bite the bullet and propose a standard for how to encode such data (or we switch to a different metadata serialisation format, like JSON...)
agronholm commentedon Aug 30, 2023
So is there some metadata getting used by wheel that should be omitted?
pfmoore commentedon Aug 30, 2023
@agronholm The problem is basically that if
PKG-INFOcontains text likeb"License: \x0b\n \n",bdist_wheelisn't encoding it correctly (i.e., in a way that can be read back).This is because neither the metadata spec nor the email standards really say what to do with control characters in header values, and the stdlib
emailpackage appears to handle it inconsistently.My recommendation is to reject characters like
\x0b- either just fail with a helpful error, or strip them out somehow.agronholm commentedon Aug 30, 2023
Are newlines still allowed though?
abravalheri commentedon Aug 30, 2023
The current stdlib implementation uses
.splitlines()in the header value and then re-joins it. Some control characters will be interpreted as a newline:remove control characters from license
supakeen commentedon Nov 9, 2023
Ran into this issue; it's of note that some standard LICENSE texts will contain characters that end up being incorrectly escaped thus generating a PKG-INFO that's 'broken'.
remove control characters from license