+++ This bug was initially created as a clone of Bug #1637148 +++

Description of problem:
Some yum/dnf repositories contain links to online mirrorlists. Those mirrorlists may use variables like $releasever or $basearch within its URLs. Until Fedora 28 this worked flawlessly. With Fedora 29 beta the DNF variables are NOT resolved any more.

Version-Release number of selected component (if applicable):
DNF version: 3.6.1

How reproducible:
1. Install a repository which uses an online mirrorlist (in my case it's unitedrpms).
2. sudo dnf --disablerepo '*' --enablerepo unitedrpms upgrade --refresh --verbose

Actual results:
Loaded plugins: builddep, config-manager, copr, debug, debuginfo-install, download, generate_completion_cache, needs-restarting, playground, repoclosure, repograph, repomanage, reposync, system-upgrade
DNF version: 3.6.1
cachedir: /var/cache/dnf
repo: downloading from remote: unitedrpms
error: Status code: 404 for https://sourceforge.net/projects/unitedrpms/files/$releasever/$basearch/repodata/repomd.xml (https://sourceforge.net/projects/unitedrpms/files/$releasever/$basearch/repodata/repomd.xml).
error: Status code: 404 for https://osdn.net/projects/unitedrpms/storage/$releasever/$basearch/repodata/repomd.xml/ (https://osdn.net/projects/unitedrpms/storage/$releasever/$basearch/repodata/repomd.xml/).
unitedrpms 29 - x86_64   
Cannot download 'https://raw.githubusercontent.com/UnitedRPMs/unitedrpms/master/mirrorlist_x86_64.txt': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried.
Failed to synchronize cache for repo 'unitedrpms', ignoring this repo.
--> Starting dependency resolution
--> Finished dependency resolution
Dependencies resolved.
Nothing to do.
Complete!

Expected results:
DNF variables within URLs of mirrors shall be resolved/replaced.

e.g.: https://sourceforge.net/projects/unitedrpms/files/$releasever/$basearch/repodata/repomd.xml shall be transformed into https://sourceforge.net/projects/unitedrpms/files/29/x86_64/repodata/repomd.xml

--- Additional comment from Michael on 2018-10-09 11:22:11 CEST ---

This is the mirrorlist whose URLs are not treated properly by DNF in my case:
https://raw.githubusercontent.com/UnitedRPMs/unitedrpms/master/mirrorlist_x86_64.txt

Content:
https://sourceforge.net/projects/unitedrpms/files/$releasever/$basearch/
https://osdn.net/projects/unitedrpms/storage/$releasever/$basearch/

My guess is that many repositories which use that style of mirrorlist are affected.

--- Additional comment from Kamil Páral on 2018-10-09 13:07:34 CEST ---

I can't reproduce this. If I take fedora repos and use baseurl (including both $releasever and $basearch) instead of mirrorlist, dnf still resolves that ok and pulls the metadata.
dnf-3.6.1-1.fc29.noarch

Michael, please upload the repo file that doesn't work for you.

--- Additional comment from Michael on 2018-10-09 13:16:24 CEST ---

Somebody implemented a workaround (!) on UnitedRPMs side already (see https://github.com/UnitedRPMs/unitedrpms/commit/30c505cadaf0145986f7eae938573a32ba7a6316).

However, that doesn't fix the root cause of the issue...

Here's the (original) repo file I was writing about:

/etc/yum.repos.d/unitedrpms.repo:
---------------------------------------------
[unitedrpms]
name=unitedrpms $releasever - $basearch
mirrorlist=https://raw.githubusercontent.com/UnitedRPMs/unitedrpms/master/mirrorlist_x86_64.txt
baseurl=https://sourceforge.net/projects/unitedrpms/files/$releasever/$basearch/
enabled=1
priority=1
metadata_expire=1d
skip_if_unavailable=1
gpgkey=file:///etc/pki/rpm-gpg/URPMS-GPG-PUBLICKEY-Fedora-24
gpgcheck=1
repo_gpgcheck=1
enabled_metadata=1
---------------------------------------------

--- Additional comment from Michael on 2018-10-09 13:23:46 CEST ---

The original state of the repository file, I mentioned, can also be obtained by using the previous version: https://github.com/UnitedRPMs/unitedrpms/releases/download/8/unitedrpms-29-8.fc29.noarch.rpm

--- Additional comment from Kamil Páral on 2018-10-09 13:43:42 CEST ---

OK, also I got confused by metalink vs mirrorlist. Nevertheless, in a repo with this:

mirrorlist=https://raw.githubusercontent.com/UnitedRPMs/unitedrpms/master/mirrorlist_x86_64.txt

I can reproduce the problem. The hyperlink returns this content:

https://sourceforge.net/projects/unitedrpms/files/$releasever/$basearch/
https://osdn.net/projects/unitedrpms/storage/$releasever/$basearch/

at that's later used literally. So it seems that the variables are replaced in baseurl/metalink/mirrorlist values, but not in mirrorlist _content_. I don't know whether that's expected or not (as you say, it worked before). Now we need feedback from DNF devs.

--- Additional comment from Michael on 2018-10-09 13:51:42 CEST ---

Exactly. :-)
The content (URLs of the mirrors) inside the mirrorlist file isn't treated right in my opinion.

I'm very certain that it worked fine the last time I used fedora 28.

--- Additional comment from Neal Gompa on 2018-10-09 14:55:56 CEST ---

I'm not sure this was supposed to work. My understanding is that the mirrorlist text file itself was supposed to be variable substituted, and it would pass in a literal newline separated list of _valid_ URLs.

If this worked before, I'm not sure it was supposed to...

--- Additional comment from Kamil Páral on 2018-10-09 15:03:17 CEST ---

(In reply to Michael from comment #6)
> I'm very certain that it worked fine the last time I used fedora 28.

Confirmed, it works fine with dnf-2.7.5-12.fc28. That doesn't mean it was a documented supported behavior, of course.

--- Additional comment from Neal Gompa on 2018-10-09 17:12:06 CEST ---

(In reply to Neal Gompa from comment #7)
> I'm not sure this was supposed to work. My understanding is that the
> mirrorlist text file itself was supposed to be variable substituted, and it
> would pass in a literal newline separated list of _valid_ URLs.
> 
> If this worked before, I'm not sure it was supposed to...

Sorry, I meant mirrorlist text file URL path, not the file content.

--- Additional comment from Marek Blaha on 2018-10-15 15:44:36 CEST ---

PR https://github.com/rpm-software-management/dnf/pull/1244

--- Additional comment from Geoffrey Marr on 2018-10-15 21:12:51 CEST ---

Discussed during the 2018-10-15 blocker review meeting: [1]

The decision to classify this bug as a "RejectedBlocker" and an "AcceptedFreezeException" was made as this is not believed to affect any official / out-of-the-box repositories and thus does not break any release criteria. As this could benefit some folks on first boot, it is accepted as a freeze exception if the fix is simple and testable.

[1] https://meetbot.fedoraproject.org/fedora-blocker-review/2018-10-15/f29-blocker-review.2018-10-15-16.00.txt

--- Additional comment from Marek Blaha on 2018-11-05 08:53:51 CET ---



--- Additional comment from Marek Blaha on 2018-11-07 07:21:11 CET ---