Repository navigation
Synchronizing content with Teleport provider results in 0 deb packages #1233
Description
Version
rubygem-pulp_deb_client-3.3.1-1.el9.noarch
katello-repos-4.15.0-1.el9.noarch
katello-certs-tools-2.10.0-1.el9.noarch
foreman-installer-katello-3.13.0-1.el9.noarch
katello-selinux-5.0.2-1.el9.noarch
katello-common-4.15.0-1.el9.noarch
rubygem-katello-4.15.0-1.el9.noarch
katello-4.15.0-1.el9.noarch
katello-client-bootstrap-1.7.9-2.el9.noarch
rubygem-hammer_cli_katello-1.15.0-0.1.pre.main.20241104082525gitd2c7a35.el9.noarch
Describe the bug
Synchronizing from Teleport repository results in 0 debian packages in Pulp.
To Reproduce
Create a repository.
Set upstream to https://apt.releases.teleport.dev/debian
Set release to bookworm
Either leave component empty, or set it to stable/v17.
Sync.
Expected behavior
I'd expect packages to be synced.
What I see is 0 debian packages.
Additional context
https://apt.releases.teleport.dev/debian/dists/bookworm/InRelease
I can see an InRelease & Release file here. However, most apt repositories allow directory listing, but Teleport's doesn't.
I suspect Pulp requires directory listing, but one of our other sync scripts (https://github.com/apt-mirror/apt-mirror) doesn't require this and syncs Teleport just fine.
I'd expect Pulp to fetch a Release/Package file and grab the files from there, without needing directory listing.
Activity
mdellweg commentedon Feb 14, 2025
Can you try again with a trailing slash on:
"https://apt.releases.teleport.dev/debian/"
As far as i know, pulp does not require directory listings.
hb0nes commentedon Feb 14, 2025
yeah I've been playing around with Nginx and logging what Pulp does and indeed, it seems to fetch Release files directly.
Interesting...
Unfortunately, the trailing slash doesn't do the trick either.
I wonder if someone has a test setup running where they can get it to work or at the very least, figure out where it's going wrong.
hb0nes commentedon Feb 14, 2025
I built my own 'Teleport' repo.
One with component 'stable/v17' and one with component 'stable-v17'.
Pulp does not like components with a slash like that, i.e. 2 directories instead of 1.
component stable-v17 works, but stable/v17 does not.
quba42 commentedon Feb 17, 2025
I get the following log warnings when synchronizing this repo:
Looking at the InRelease file, it looks like the Components field has path prefixes, even though the release file is not stored in a nested location:
I have not encountered this format before, and it looks like pulp_deb can't handle it. I expect this to be pretty complex to fix (without breaking other repo formats).
hb0nes commentedon Feb 17, 2025
It's removing the stable/ part from the component.
If that wouldn't happen, the Release file would be found.
pulp_deb/pulp_deb/app/tasks/synchronizing.py
Line 876 in f801680
If I understand correctly, that is happening here, when referring to plain_component.
What the reasons/implications are for doing and not doing that, I don't know.
For example, changing
The Teleport repo sync works.
But dunno what else will break.
quba42 commentedon Feb 18, 2025
I believe the fix you proposed would break repositories where the Release file "is not in a directory directly beneath dists/" (see here). We do have test coverage that would catch this, so feel free to open a PR with that change (to see if any tests will turn red).
The assumption of the pulp_deb implementation, is that components should only have path prefixes attached to them, if the Release file "is not in a directory directly beneath dists/". However, this is not the case for the teleport repo, which is why our implementation does not work. I assume APT can consume the teleport repo, so there must be a better implementation that works for both repo formats. In other words, this is definitely a bug on our end. I do worry that fixing this (without breaking any other repositories) requires changing very central aspects of the pulp_deb implementation and possibly revisiting some core assumptions. At least if we want to fix this properly for all cases. One could probably add some conditional logic specifically for the case where components have path prefixes, but the release file is not in a nested location as a "quick and dirty" fix.
hb0nes commentedon Feb 18, 2025
That's exactly what I was thinking as well. I'm not a maintainer of this repo and am not familiar with the codebase - it would be too time intensive for me to contribute to this in a meaningful/robust manner. I hope someone else picks it up eventually, even if it is a conditional hack.
Thanks for spending some time on this so far.