Created attachment 1712443 [details]
anaconda.log

Created attachment 1712444 [details]
dbus.log

Created attachment 1712445 [details]
dnf.librepo.log

Created attachment 1712446 [details]
packaging.log

Created attachment 1712447 [details]
program.log

Created attachment 1712448 [details]
syslog

Accepted as a Beta Blocker:
https://pagure.io/fedora-qa/blocker-review/issue/35

> AGREED AcceptedBetaBlocker Accepted as a violation of Basic criterion "When using a release-blocking dedicated installer image, the installer must be able to use either HTTP or FTP repositories (or both) as package sources. Release-blocking network install images must default to a valid publicly-accessible package source" [1] , when the 'repository' in question is a mirrorlist and not a direct repo URL. [1] https://fedoraproject.org/wiki/Basic_Release_Criteria#Remote_package_sources

This part of packaging.log looks like the problem to me:

09:22:47,488 DBG packaging: Using the repo configuration: RepoConfigurationData(cost=1000, excluded_packages=[], included_packages=[], name='anaconda-0', proxy='', ssl_configuration=SSLConfigurationData(ca_cert_path='', client_cert_path='', client_key_path=''), ssl_verification_enabled=True, type='MIRRORLIST', url='https://mirrors.fedoraproject.org/mirrorlist?repo=fedora-33&arch=x86_64')
09:22:47,489 DBG packaging: getting release version from tree at  (33)
09:22:47,489 DBG packaging: using treeinfo release version of 33
09:22:47,489 DBG packaging: No base repository found in treeinfo file. Using installation tree root.
09:22:47,489 DBG packaging: releasever from  is 33
09:22:47,489 DBG packaging: Adding new treeinfo repository: Server enabled: True
09:22:47,490 INF packaging: added repo: 'Server' - file:///run/install/sources/mount-0000-cdrom

somewhere in DNFPayload._add_repo we're going from a mirrorlist repo with mirrorlist URL https://mirrors.fedoraproject.org/mirrorlist?repo=fedora-33&arch=x86_64 to what looks like trying to set up a local mount repo with a URI of file:///run/install/sources/mount-0000-cdrom . That doesn't look right at all.

I suspect the payload-related changes Jiri Konecny made between 33.25-1 and 33.25.2-1 are the problem here - probably one of:

a57be7d30897ecf301de673e41d1af975b4f593b - Fix issue that treeinfo repositories were never disabled
1c03016c3906a458a74262eaaffbc732d70d5281 - Move parts together in the DNF repo
0788d3fb8a696fa83a818148f08eb439123cc389 - Get rid of add_disable_repo

Jiri, can you please take a look?

Here's the equivalent section from a test with anaconda-33.25-1.fc33:

15:21:13,175 INF packaging: Configuring the base repo
15:21:13,205 INF packaging: Enabled 'updates'
15:21:13,205 INF packaging: Disabled 'updates-testing'
15:21:13,208 DBG packaging: Using the repo configuration: RepoConfigurationData(cost=1000, excluded_packages=[], included_packages=[], name='anaconda-0', proxy='', ssl_configuration=SSLConfigurationData(ca_cert_path='', client_cert_path='', client_key_path=''), ssl_verification_enabled=True, type='MIRRORLIST', url='https://mirrors.fedoraproject.org/mirrorlist?repo=fedora-33&arch=x86_64')
15:21:13,208 DBG packaging: getting release version from tree at  (rawhide)
15:21:13,208 DBG packaging: using treeinfo release version of rawhide
15:21:13,208 DBG packaging: No base repository found in treeinfo file. Using installation tree root.
15:21:13,208 DBG packaging: releasever from  is rawhide
15:21:13,209 INF packaging: added repo: 'anaconda' - https://mirrors.fedoraproject.org/mirrorlist?repo=fedora-33&arch=x86_64
15:21:13,209 DBG dnf: repo: downloading from remote: anaconda
15:21:17,057 INF packaging: enabled repo: 'anaconda' - https://mirrors.fedoraproject.org/mirrorlist?repo=fedora-33&arch=x86_64 and got repomd

Ah - looking at those logs I actually think https://github.com/rhinstaller/anaconda/commit/6f27de8a38cc7900bb35a4fac4ec258f50207468 is the offending commit here. Things go wrong in 33.25.2 when we get to "Adding new treeinfo repository: Server enabled: True", which is in `_load_treeinfo_repositories` (previously `_add_treeinfo_repositories`), and that commit changes anaconda from doing that *only on the first payload reset* to doing it on *every* payload reset. `_load_treeinfo_repositories` is doing something wrong in this case. I do note this check:

        if base_repo_url is not None:

which is questionable here, because I believe we are passing in `base_repo_url` as an empty string on this route.

Hmm, no, that check isn't the issue - I patched update_base_repo to pass `base_repo_url or None` but it doesn't help. I think what's going on here is that we're trying to reload the repo from the DVD, but it has been umounted in the meantime. There's some key lines in syslog that we don't see in packaging.log. Here's the whole flow around the *first* base repo setup:

syslog:

12:50:47,514 WARNING org.fedoraproject.Anaconda.Modules.Payloads:INFO:program:Running... mount -t iso9660 -o ro /dev/sr0 /run/install/sources/mount-0000-cdrom
12:50:47,532 WARNING org.fedoraproject.Anaconda.Modules.Payloads:DEBUG:program:Return code: 0
12:50:47,532 WARNING org.fedoraproject.Anaconda.Modules.Payloads:INFO:anaconda.modules.payloads.source.cdrom.initialization:using CD-ROM device sr0 mounted at /run/install/sources/mount-0000-cdrom

packaging.log:

12:50:48,883 DBG packaging: Using the repo configuration: RepoConfigurationData(cost=1000, excluded_packages=[], included_packages=[], name='', proxy='', ssl_configuration=SSLConfigurationData(ca_cert_path='', client_cert_path='', client_key_path=''), ssl_verification_enabled=True, type='BASEURL', url='file:///run/install/sources/mount-0000-cdrom')
12:50:48,884 DBG packaging: retrieving treeinfo from file:///run/install/sources/mount-0000-cdrom (proxy: None ; ssl_verify: True)
12:50:48,885 INF packaging: Trying to download '.treeinfo'
12:50:48,887 DBG packaging: Retrieved '.treeinfo' from file:///run/install/sources/mount-0000-cdrom
12:50:48,891 DBG packaging: getting release version from tree at file:///run/install/sources/mount-0000-cdrom (33)
12:50:48,891 DBG packaging: using treeinfo release version of 33
12:50:48,892 DBG packaging: No base repository found in treeinfo file. Using installation tree root.
12:50:48,892 DBG packaging: releasever from file:///run/install/sources/mount-0000-cdrom is 33
12:50:48,893 INF packaging: added repo: 'anaconda' - file:///run/install/sources/mount-0000-cdrom
12:50:48,928 DBG dnf: repo: downloading from remote: anaconda
12:50:49,105 INF packaging: enabled repo: 'anaconda' - ['file:///run/install/sources/mount-0000-cdrom'] and got repomd
12:50:49,106 DBG dnf: repo: using cache for: anaconda
12:50:49,106 DBG packaging: repo anaconda: _sync_metadata success from ['file:///run/install/sources/mount-0000-cdrom']

syslog again:

12:51:42,537 WARNING org.fedoraproject.Anaconda.Modules.Payloads:INFO:program:Running... umount /run/install/sources/mount-0000-cdrom

now back to packaging.log:

08:51:44,613 DBG packaging: Adding new treeinfo repository: Server enabled: True
08:51:44,614 INF packaging: added repo: 'Server' - file:///run/install/sources/mount-0000-cdrom
08:51:44,615 DBG dnf: repo: downloading from remote: Server
08:51:44,616 DBG dnf: error: Curl error (37): Couldn't read a file:// file for file:///run/install/sources/mount-0000-cdrom/repodata/repomd.xml [Couldn't open file /run/install/sources/mount-0000-cdrom/repodata/repomd.xml] (file:///run/install/sources/mount-0000-cdrom/repodata/repomd.xml).

so we mount the repo, set it up successfully, *then unmount it*, then `_load_treeinfo_repositories` runs again and tries to load it again, but without remounting it first.

oh, the timezone gets set in the middle there or something, which is why we suddenly jump "back" four hours. But that is the order stuff actually happens in, I'm fairly sure.

ah, and interestingly, `DNFPayload.update_base_repo` has a `checkmount` arg which looks like it makes sure something like the DVD repo is mounted if necessary, and pyanaconda/ui/gui/spokes/installation_source.py `SourceSpoke.apply` does this:

        if source_changed or repo_changed or self._error:
            payloadMgr.restart_thread(self.payload, checkmount=False)

...and `payloadMgr.restart_thread` winds up doing `payload.update_base_repo` with that `checkmount` value. So I bet that's where we're at here.

yeah, I added a log line and confirmed that during the *first* call to `update_base_repo` (where it works) checkmount is True, but during the *second* call (where it fails) it's False.

PR: https://github.com/rhinstaller/anaconda/pull/2826

This is an old issue but it wasn't a problem before our recent change. The old logic has loaded all additional repositories from the treeinfo just once from the first repository. We have changed the logic to load it every time the repository has changed, however, I forget to change also the metadata loading so it was not cleared and old repository was used as the metadata source. Clear the metadata properly to fix this issue.

FEDORA-2020-daac40f3d3 has been submitted as an update to Fedora 33. https://bodhi.fedoraproject.org/updates/FEDORA-2020-daac40f3d3

FEDORA-2020-daac40f3d3 has been pushed to the Fedora 33 testing repository.
In short time you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2020-daac40f3d3`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2020-daac40f3d3

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

We should be able to check the fix for this in the next Rawhide compose. Testing the F33 update itself is a bit tricky as it requires building a Server DVD...

FEDORA-2020-daac40f3d3 has been submitted as an update to Fedora 33. https://bodhi.fedoraproject.org/updates/FEDORA-2020-daac40f3d3

So Rawhide composes are failing, but they're getting far enough to produce a Server DVD. Here's the latest:
https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20200909.n.0/compose/Server/x86_64/iso/Fedora-Server-dvd-x86_64-Rawhide-20200909.n.0.iso

I'm downloading that to verify the fix now.

Unfortunately that doesn't boot, so can't test. Will try again with the next one.

FEDORA-2020-daac40f3d3 has been pushed to the Fedora 33 testing repository.
In short time you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2020-daac40f3d3`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2020-daac40f3d3

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Rawhide latest compose succeeded, and test passed: https://openqa.fedoraproject.org/tests/661773
so we can confirm the fix for this looks good.

FEDORA-2020-daac40f3d3 has been pushed to the Fedora 33 stable repository.
If problem still persists, please make note of it in this bug report.

FEDORA-2020-384ff75a01 has been submitted as an update to Fedora 33. https://bodhi.fedoraproject.org/updates/FEDORA-2020-384ff75a01

FEDORA-2020-384ff75a01 has been pushed to the Fedora 33 testing repository.
In short time you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2020-384ff75a01`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2020-384ff75a01

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

FEDORA-2020-384ff75a01 has been submitted as an update to Fedora 33. https://bodhi.fedoraproject.org/updates/FEDORA-2020-384ff75a01

FEDORA-2020-384ff75a01 has been pushed to the Fedora 33 testing repository.
In short time you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2020-384ff75a01`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2020-384ff75a01

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

FEDORA-2020-384ff75a01 has been pushed to the Fedora 33 stable repository.
If problem still persists, please make note of it in this bug report.