Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.
Bug 2271362 - Wget2 blacklists files intended for download
Summary: Wget2 blacklists files intended for download
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: wget2
Version: rawhide
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Michal Ruprich
QA Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2024-03-25 08:39 UTC by Martin Osvald 🛹
Modified: 2024-04-03 02:05 UTC (History)
3 users (show)

Fixed In Version: wget2-2.1.0-8.fc40 wget2-2.1.0-8.el9
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2024-04-02 14:18:04 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github rockdaboot wget2 issues 308 0 None closed --no-parent causes legitimate files to be on a blacklist 2024-04-02 11:18:04 UTC

Description Martin Osvald 🛹 2024-03-25 08:39:03 UTC
Description of problem:

The following command to recursively download all .rpm files doesn't work with wget2 (doesn't work without -A option either):

# wget -q -r -l1 --no-parent --directory-prefix rpms -A ".rpm" https://kojipkgs.fedoraproject.org//work/tasks/8411/115298411/

Worked well with wget1 on rhel39 and earlier.


Version-Release number of selected component (if applicable):

wget2-2.1.0-7.fc40.x86_64


How reproducible:

Always.


Steps to Reproduce:
1. wget -q -r -l1 --no-parent --directory-prefix rpms -A ".rpm" https://kojipkgs.fedoraproject.org//work/tasks/8411/115298411/


Actual results:

No .rpm files downloaded.

Expected results:

.rpm files downloaded like with wget1


Additional info:

With -d option it shows the .rpm files as backlisted:

~~~
# wget -q -r -l1 --no-parent --directory-prefix rpms -A ".rpm" https://kojipkgs.fedoraproject.org//work/tasks/8411/115298411/ -d

... snip ...

25.013602.365 blacklist https://kojipkgs.fedoraproject.org/icons/blank.gif
25.013602.365 blacklist https://kojipkgs.fedoraproject.org/icons/text.gif
25.013602.365 blacklist https://kojipkgs.fedoraproject.org/robots.txt
25.013602.365 blacklist https://kojipkgs.fedoraproject.org/work/tasks/8411/115298411/perl-Cyrus-3.8.2-1.fc41.x86_64.rpm
25.013602.365 blacklist https://kojipkgs.fedoraproject.org/work/tasks/8411/115298411/cyrus-imapd-debuginfo-3.8.2-1.fc41.x86_64.rpm
25.013602.365 blacklist https://kojipkgs.fedoraproject.org/work/tasks/8411/115298411/noarch_rpmdiff.json
25.013602.366 blacklist https://kojipkgs.fedoraproject.org/work/tasks/8411/115298411/state.log
25.013602.366 blacklist https://kojipkgs.fedoraproject.org/work/tasks/8411/115298411/cyrus-imapd-libs-3.8.2-1.fc41.x86_64.rpm
25.013602.366 blacklist https://kojipkgs.fedoraproject.org/work/tasks/8411/115298411/cyrus-imapd-virusscan-3.8.2-1.fc41.x86_64.rpm
25.013602.366 blacklist https://kojipkgs.fedoraproject.org/work/tasks/8411/115298411/build.log
25.013602.366 blacklist https://kojipkgs.fedoraproject.org/work/tasks/8411/115298411/cyrus-imapd-doc-extra-3.8.2-1.fc41.noarch.rpm
25.013602.366 blacklist https://kojipkgs.fedoraproject.org/work/tasks/8411/115298411/cyrus-imapd-3.8.2-1.fc41.src.rpm
25.013602.366 blacklist https://kojipkgs.fedoraproject.org/work/tasks/8411/115298411/cyrus-imapd-devel-3.8.2-1.fc41.x86_64.rpm
25.013602.366 blacklist https://kojipkgs.fedoraproject.org/icons/back.gif
25.013602.366 blacklist https://kojipkgs.fedoraproject.org/work/tasks/8411/115298411/?C=N;O=D
25.013602.366 blacklist https://kojipkgs.fedoraproject.org/work/tasks/8411/115298411/?C=M;O=A
25.013602.366 blacklist https://kojipkgs.fedoraproject.org/work/tasks/8411/
25.013602.366 blacklist https://kojipkgs.fedoraproject.org/work/tasks/8411/115298411/cyrus-imapd-virusscan-debuginfo-3.8.2-1.fc41.x86_64.rpm
25.013602.366 blacklist https://kojipkgs.fedoraproject.org/work/tasks/8411/115298411/?C=D;O=A
25.013602.366 blacklist https://kojipkgs.fedoraproject.org/work/tasks/8411/115298411/cyrus-imapd-utils-3.8.2-1.fc41.x86_64.rpm
25.013602.366 blacklist https://kojipkgs.fedoraproject.org/work/tasks/8411/115298411/mock_output.log
25.013602.366 blacklist https://kojipkgs.fedoraproject.org/icons/unknown.gif
25.013602.366 blacklist https://kojipkgs.fedoraproject.org//work/tasks/8411/115298411/
25.013602.366 blacklist https://kojipkgs.fedoraproject.org/work/tasks/8411/115298411/root.log
25.013602.366 blacklist https://kojipkgs.fedoraproject.org/work/tasks/8411/115298411/cyrus-imapd-utils-debuginfo-3.8.2-1.fc41.x86_64.rpm
25.013602.366 blacklist https://kojipkgs.fedoraproject.org/work/tasks/8411/115298411/hw_info.log
25.013602.366 blacklist https://kojipkgs.fedoraproject.org/work/tasks/8411/115298411/perl-Cyrus-debuginfo-3.8.2-1.fc41.x86_64.rpm
25.013602.366 blacklist https://kojipkgs.fedoraproject.org/work/tasks/8411/115298411/cyrus-imapd-libs-debuginfo-3.8.2-1.fc41.x86_64.rpm
25.013602.366 blacklist https://kojipkgs.fedoraproject.org/work/tasks/8411/115298411/cyrus-imapd-debugsource-3.8.2-1.fc41.x86_64.rpm
25.013602.366 blacklist https://kojipkgs.fedoraproject.org/work/tasks/8411/115298411/?C=S;O=A
25.013602.366 blacklist https://kojipkgs.fedoraproject.org/work/tasks/8411/115298411/cyrus-imapd-3.8.2-1.fc41.x86_64.rpm
~~~

Comment 1 Michal Ruprich 2024-03-25 12:44:35 UTC
Hi Martine,

seems like the files are added to the blacklist because of these lines in the log:

Adding URL: https://kojipkgs.fedoraproject.org/work/tasks/8411/115298411/cyrus-imapd-debugsource-3.8.2-1.fc41.x86_64.rpm
URL 'https://kojipkgs.fedoraproject.org/work/tasks/8411/115298411/cyrus-imapd-debugsource-3.8.2-1.fc41.x86_64.rpm' not followed (parent ascending not allowed)
Adding URL: https://kojipkgs.fedoraproject.org/work/tasks/8411/115298411/cyrus-imapd-devel-3.8.2-1.fc41.x86_64.rpm
URL 'https://kojipkgs.fedoraproject.org/work/tasks/8411/115298411/cyrus-imapd-devel-3.8.2-1.fc41.x86_64.rpm' not followed (parent ascending not allowed)
Adding URL: https://kojipkgs.fedoraproject.org/work/tasks/8411/115298411/cyrus-imapd-doc-extra-3.8.2-1.fc41.noarch.rpm
URL 'https://kojipkgs.fedoraproject.org/work/tasks/8411/115298411/cyrus-imapd-doc-extra-3.8.2-1.fc41.noarch.rpm' not followed (parent ascending not allowed)
Adding URL: https://kojipkgs.fedoraproject.org/work/tasks/8411/115298411/cyrus-imapd-libs-3.8.2-1.fc41.x86_64.rpm
URL 'https://kojipkgs.fedoraproject.org/work/tasks/8411/115298411/cyrus-imapd-libs-3.8.2-1.fc41.x86_64.rpm' not followed (parent ascending not allowed)
Adding URL: https://kojipkgs.fedoraproject.org/work/tasks/8411/115298411/cyrus-imapd-libs-debuginfo-3.8.2-1.fc41.x86_64.rpm
URL 'https://kojipkgs.fedoraproject.org/work/tasks/8411/115298411/cyrus-imapd-libs-debuginfo-3.8.2-1.fc41.x86_64.rpm' not followed (parent ascending not allowed)
Adding URL: https://kojipkgs.fedoraproject.org/work/tasks/8411/115298411/cyrus-imapd-utils-3.8.2-1.fc41.x86_64.rpm
URL 'https://kojipkgs.fedoraproject.org/work/tasks/8411/115298411/cyrus-imapd-utils-3.8.2-1.fc41.x86_64.rpm' not followed (parent ascending not allowed)
Adding URL: https://kojipkgs.fedoraproject.org/work/tasks/8411/115298411/cyrus-imapd-utils-debuginfo-3.8.2-1.fc41.x86_64.rpm
URL 'https://kojipkgs.fedoraproject.org/work/tasks/8411/115298411/cyrus-imapd-utils-debuginfo-3.8.2-1.fc41.x86_64.rpm' not followed (parent ascending not allowed)
Adding URL: https://kojipkgs.fedoraproject.org/work/tasks/8411/115298411/cyrus-imapd-virusscan-3.8.2-1.fc41.x86_64.rpm
URL 'https://kojipkgs.fedoraproject.org/work/tasks/8411/115298411/cyrus-imapd-virusscan-3.8.2-1.fc41.x86_64.rpm' not followed (parent ascending not allowed)
Adding URL: https://kojipkgs.fedoraproject.org/work/tasks/8411/115298411/cyrus-imapd-virusscan-debuginfo-3.8.2-1.fc41.x86_64.rpm
URL 'https://kojipkgs.fedoraproject.org/work/tasks/8411/115298411/cyrus-imapd-virusscan-debuginfo-3.8.2-1.fc41.x86_64.rpm' not followed (parent ascending not allowed)
Adding URL: https://kojipkgs.fedoraproject.org/work/tasks/8411/115298411/hw_info.log
URL 'https://kojipkgs.fedoraproject.org/work/tasks/8411/115298411/hw_info.log' not followed (parent ascending not allowed)
Adding URL: https://kojipkgs.fedoraproject.org/work/tasks/8411/115298411/mock_output.log
URL 'https://kojipkgs.fedoraproject.org/work/tasks/8411/115298411/mock_output.log' not followed (parent ascending not allowed)
Adding URL: https://kojipkgs.fedoraproject.org/work/tasks/8411/115298411/noarch_rpmdiff.json
URL 'https://kojipkgs.fedoraproject.org/work/tasks/8411/115298411/noarch_rpmdiff.json' not followed (parent ascending not allowed)
Adding URL: https://kojipkgs.fedoraproject.org/work/tasks/8411/115298411/perl-Cyrus-3.8.2-1.fc41.x86_64.rpm
URL 'https://kojipkgs.fedoraproject.org/work/tasks/8411/115298411/perl-Cyrus-3.8.2-1.fc41.x86_64.rpm' not followed (parent ascending not allowed)
Adding URL: https://kojipkgs.fedoraproject.org/work/tasks/8411/115298411/perl-Cyrus-debuginfo-3.8.2-1.fc41.x86_64.rpm
URL 'https://kojipkgs.fedoraproject.org/work/tasks/8411/115298411/perl-Cyrus-debuginfo-3.8.2-1.fc41.x86_64.rpm' not followed (parent ascending not allowed)
Adding URL: https://kojipkgs.fedoraproject.org/work/tasks/8411/115298411/root.log
URL 'https://kojipkgs.fedoraproject.org/work/tasks/8411/115298411/root.log' not followed (parent ascending not allowed)
Adding URL: https://kojipkgs.fedoraproject.org/work/tasks/8411/115298411/state25.084207.572 local filename = 'rpms/kojipkgs.fedoraproject.org/work/tasks/8411/115298411/state.log'

When I tried to run this without the --no-parent option, it actually works. But that's definitely another bug, I don't know why it considers files in the final dir as something that would not be accepted because of the no-parent option.

Comment 2 Michal Ruprich 2024-04-02 11:45:16 UTC
Fixed in the Upstream. There was a missing normalization step for the path in the URL. Any // would confuse wget and create a situation where some legitimate files could have ended on the blacklist.

As a workaround before the fix is out, try using --no-parent option and if that does not result in what you actually wanted to achieve, try using --no-directories=0 in your command.

PRs for the fix are here:

F40: https://src.fedoraproject.org/rpms/wget2/pull-request/7

Rawhide: https://src.fedoraproject.org/rpms/wget2/pull-request/6

Comment 3 Fedora Update System 2024-04-02 12:54:42 UTC
FEDORA-EPEL-2024-804d6232a9 (wget2-2.1.0-8.el9) has been submitted as an update to Fedora EPEL 9.
https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2024-804d6232a9

Comment 4 Fedora Update System 2024-04-02 12:54:43 UTC
FEDORA-2024-9129bb0ac3 (wget2-2.1.0-8.fc40) has been submitted as an update to Fedora 40.
https://bodhi.fedoraproject.org/updates/FEDORA-2024-9129bb0ac3

Comment 5 Fedora Update System 2024-04-02 14:18:04 UTC
FEDORA-2024-9129bb0ac3 (wget2-2.1.0-8.fc40) has been pushed to the Fedora 40 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 6 Fedora Update System 2024-04-03 02:05:53 UTC
FEDORA-EPEL-2024-804d6232a9 (wget2-2.1.0-8.el9) has been pushed to the Fedora EPEL 9 stable repository.
If problem still persists, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.