Bug 2244503

Proposed as a Freeze Exception for 39-final by Fedora user rishi using the blocker tracking app because:

 All of these are regressions arising from this Fedora 39 Change:
https://fedoraproject.org/wiki/Changes/ToolbxReleaseBlocker

It will be sad to have known regressions from a Change that wants to:
"... improve the quality of the containerized interactive command line Toolbx environments on Fedora by adding formal requirements to ensure that they are usable when a new Fedora is released. This will bring them closer to the reliability of similar environments running directly on the host operating system."

Some of these are already fixed for the fedora-toolbox:40 image and there have been no known problems so far.

Finally, these changes only impact the toolbox RPM, and no other part of the distribution.  If necessary, they can be reverted.

The other alternative is to re-spin the fedora-toolbox:39 image  post Fedora 39 GA.

(In reply to Debarshi Ray from comment #0)
> The fedora-container-toolbox.ks file has an obvious typo where the %end and
> %post markers went missing.  This conflates the %pre and %post sections into
> one %pre section.

Here's a pull request for the f39 branch:
https://pagure.io/fedora-kickstarts/pull-request/994

... and it was already merged into main for Rawhide/F40 a few days ago:
https://pagure.io/fedora-kickstarts/pull-request/993

(In reply to Debarshi Ray from comment #0)
> Missing weak dependencies due to the use of --exclude-weakdeps in the
> fedora-container-toolbox.ks file.

Here's one way to fix it:
https://pagure.io/fedora-kickstarts/pull-request/995
https://pagure.io/fedora-kickstarts/pull-request/996

I tried to check which packages are actually missing in fedora-toolbox:39 with:
⬢$ rpm --query --all >packages.txt
⬢$ while read package; do echo "$package"; sudo dnf repoquery --recommends "$package"; echo "--"; done <packages.txt

... and then manually comparing with the content in fedora-toolbox:38.

It seems to me that python-systemd-doc, which is recommended by python3-systemd, is the only package missing in fedora-toolbox:39 when compared with fedora-toolbox:38.  So, we can also fix it with a more localized patch that pulls in only that one RPM.

Discussed at 2023-10-16 FE review meeting: https://meetbot-raw.fedoraproject.org/fedora-blocker-review/2023-10-16/f39-blocker-review.2023-10-16-16.00.html . Accepted as an FE since we do want the release day toolbx container to be 'correct' in these regards.

Thank you for the attention to detail here, Rishi :+1:

Actually today I was quite surprised to find that translations seem to be rpm disabled in all current fedora-toolbox containers?
By /etc/rpm/macros.image-language-conf - I didn't know that was the case...

despite the fact that /etc/rpm/macros.image-language-conf is removed when building fedora-toolbox images.

(In reply to Jens Petersen from comment #6)
> Actually today I was quite surprised to find that translations seem to be
> rpm disabled in all current fedora-toolbox containers?
> By /etc/rpm/macros.image-language-conf - I didn't know that was the case...

Do you remember which packages had their translations missing?

For the OSBS-built images from Dockerfile, I suspect these are packages that are inherited from the base image.  We have a hacky solution where we 'dnf reinstall' those packages, but the package list is manually maintained and hence can go stale.

One simple example is gtk4.

But I just reproduced this in a fresh fedora-toolbox:39 container!:

⬢[petersen@toolbox ~]$ cat /etc/rpm/macros.image-language-conf 
%_install_langs en_US

So the file isn't actually removed!

Maybe the -f should be removed from "rm -f /mnt/sysimage/etc/rpm/macros.image-language-conf", shrug?

(In reply to Debarshi Ray from comment #3)
> (In reply to Debarshi Ray from comment #0)
> > Missing weak dependencies due to the use of --exclude-weakdeps in the
> > fedora-container-toolbox.ks file.
>
> [...]
> 
> It seems to me that python-systemd-doc, which is recommended by
> python3-systemd, is the only package missing in fedora-toolbox:39 when
> compared with fedora-toolbox:38.

mesa-va-drivers, recommended by mesa-dri-drivers, is perhaps a better example of a package that was missing due to the exclusion of weak dependencies.

(In reply to Jens Petersen from comment #9)
> One simple example is gtk4.
> 
> But I just reproduced this in a fresh fedora-toolbox:39 container!:

Aha, ok.  I got alarmed when you said (in comment 6) "translations seem to be rpm disabled in all current fedora-toolbox containers", because I thought that the fedora-toolbox images for F38 and F37 are also impacted.  :)

(In reply to Jens Petersen from comment #9)
> One simple example is gtk4.
> 
> But I just reproduced this in a fresh fedora-toolbox:39 container!:
> 
> ⬢[petersen@toolbox ~]$ cat /etc/rpm/macros.image-language-conf 
> %_install_langs en_US
> 
> So the file isn't actually removed!

This should be fixed by:
https://pagure.io/fedora-kickstarts/pull-request/1002 (main or rawhide)
https://pagure.io/fedora-kickstarts/pull-request/1004 (f39)

> Maybe the -f should be removed from "rm -f
> /mnt/sysimage/etc/rpm/macros.image-language-conf", shrug?

The problem, as described in the above pull requests, was that the %post section of fedora-container-common.ks was reverting what was done in the %pre section of fedora-container-toolbox.ks.

I think that 'rm' invocation can be removed now.  We are no longer using fedora-container-common.ks so there shouldn't be anything to remove.  I didn't do that now to keep the size of the changes as small as possible.

In future, we should also reinstate the built-in tests from the Container/Dockerfile that are currently commented out in fedora-container-toolbox.ks.  It's probably too late for Fedora 39, but we should definitely do that for Fedora 40.

(In reply to Debarshi Ray from comment #11)
> Aha, ok.  I got alarmed when you said (in comment 6) "translations seem to
> be rpm disabled in all current fedora-toolbox containers", because I thought
> that the fedora-toolbox images for F38 and F37 are also impacted.  :)

They (f38 from my testing at the same time) _are_ in some way AFAICT - though I would gladly be wrong!

(In reply to Debarshi Ray from comment #12)
> (In reply to Jens Petersen from comment #9)
> > ⬢[petersen@toolbox ~]$ cat /etc/rpm/macros.image-language-conf 
> > %_install_langs en_US
> > So the file isn't actually removed!
> 
> This should be fixed by:
> https://pagure.io/fedora-kickstarts/pull-request/1002 (main or rawhide)
> https://pagure.io/fedora-kickstarts/pull-request/1004 (f39)

Ah okay cool thanks, then I will hold off on such a PR, at least until yours are merged.

I tested the Fedora-Container-Toolbox build from today morning:
https://koji.fedoraproject.org/koji/buildinfo?buildID=2312442

... by downloading the tarball and:
$ skopeo copy docker-archive:Fedora-Container-Toolbox-39-20231027.n.0.x86_64.tar.xz containers-storage:localhost/fedora-toolbox:39
$ toolbox create --release f39

I can confirm that all the issues I had raised in this bug are fixed to the best of my knowledge.

(In reply to Jens Petersen from comment #13)
> (In reply to Debarshi Ray from comment #11)
> > Aha, ok.  I got alarmed when you said (in comment 6) "translations seem to
> > be rpm disabled in all current fedora-toolbox containers", because I thought
> > that the fedora-toolbox images for F38 and F37 are also impacted.  :)
> 
> They (f38 from my testing at the same time) _are_ in some way AFAICT -
> though I would gladly be wrong!

Just to be sure, the Container/Dockerfile equivalent of the fedora-toolbox images for Fedora < 39:
  * shouldn't have %_install_langs set through /etc/rpm/macros.image-language-conf
  * shouldn't have tsflags=nodocs in /etc/dnf/dnf.conf
  * should have glibc-all-langpacks, not glibc-minimal-langpack

These images had the handicap of being layered on top of the fedora base images.

The base image strips out all documentation, all translations for non-English languages, and had the locale definitions for only C, POSIX and C.UTF-8.  The fedora-toolbox image tries to undo as much of these as possible by:
  RUN rm /etc/rpm/macros.image-language-conf
  RUN sed -i '/tsflags=nodocs/d' /etc/dnf/dnf.conf
  RUN dnf -y swap glibc-minimal-langpack glibc-all-langpacks

... and reinstalling a bunch of packages that are part of the fedora base image and had their documentation and localization stripped out.

This last part about reinstalling the packages was totally ad-hoc.  Every once in a while someone noticed that something is missing from a package, and would add it to the missing-docs file, which would add it to the list of packages that got reinstalled.  We didn't have a way to systematically go through the fedora base image and restore everything that got lost.

I suspect that maybe you noticed something like this?  eg., I recently noticed that the useradd(8), etc. manuals were missing: https://github.com/containers/toolbox/pull/1394

If you found a problem with the fedora-toolbox images for Fedora < 39, then we can definitely rebuild them quickly and issue updates.

(In reply to Debarshi Ray from comment #12)
> In future, we should also reinstate the built-in tests from the
> Container/Dockerfile that are currently commented out in
> fedora-container-toolbox.ks.  It's probably too late for Fedora 39, but we
> should definitely do that for Fedora 40.

Restored the tests: https://pagure.io/fedora-kickstarts/pull-request/1008

I can't reproduce in a fresh fedora-toolbox:38 (so wondering how it happened to me...)

However with fresh fedora-toolbox:39:

petersen@fedora:~$ toolbox enter fedora-toolbox-39
⬢[petersen@toolbox ~]$ cat /etc/rpm/macros.image-language-conf 
%_install_langs en_US

(In reply to Jens Petersen from comment #17)
> I can't reproduce in a fresh fedora-toolbox:38 (so wondering how it happened
> to me...)

The missing documentation and translation due to the minimized fedora base image should be now gone for good in the fedora-toolbox images for Fedora < 39.  We no longer use a hand-maintained list of packages that had their content stripped out by the fedora base image and must be reinstalled, because we found a way to automatically generate it.  Plus, we have a test built into the Container/Dockerfile.

See:
https://github.com/containers/toolbox/issues/603
https://github.com/containers/toolbox/pull/1396
https://bodhi.fedoraproject.org/updates/FEDORA-CONTAINER-2023-56b4fa11b3
https://bodhi.fedoraproject.org/updates/FEDORA-CONTAINER-2023-4b86a3ebfc

Let me know if you find something missing.

> However with fresh fedora-toolbox:39:
> 
> petersen@fedora:~$ toolbox enter fedora-toolbox-39
> ⬢[petersen@toolbox ~]$ cat /etc/rpm/macros.image-language-conf 
> %_install_langs en_US

This looks like a slightly stale registry.fedoraproject.org/fedora-toolbox:39

*** Bug 2248006 has been marked as a duplicate of this bug. ***