Any idea what is new in this compose to add 300 MB to the size of the image? Unless there's a very good reason for it, that seems like too much increase.

So, this looks different from the usual case (where some package got added or dependencies changed). The change happened between 20221125.n.0 and 20221126.n.0 , but 1126.n.0 has no extra packages compared to 1125.n.0 (in fact it has one package *fewer*). The main difference appears to be that /var/cache/dnf has a bunch of stuff in it on 1126.n.0:

[root@localhost-live dnf]# pwd
/var/cache/dnf
[root@localhost-live dnf]# du -h
2.4M	./rawhide-modular-bcea058c95ea8a90/repodata
2.4M	./rawhide-modular-bcea058c95ea8a90
66M	./rawhide-2d95c80a1fa0a67d/repodata
66M	./rawhide-2d95c80a1fa0a67d
36K	./fedora-cisco-openh264-4896e02bbb10d47b/repodata
44K	./fedora-cisco-openh264-4896e02bbb10d47b
156M	.

but nothing in it on 1125.n.0:

[root@localhost-live dnf]# pwd
/var/cache/dnf
[root@localhost-live dnf]# du -h
4.0K	.

not sure *why* yet. The files aren't owned by any package (except one). I suspect something is meant to clean this up post-install but it has stopped doing so, for some reason.

So still haven't fully worked this out, but there's definitely something weird going on and it's not limited to lives. After a fresh install of Fedora-Server-dvd-x86_64-Rawhide-20221126.n.0.iso , /var/tmp is nearly 500M in size, mostly things under /var/tmp/dracut.* . With the previous day's image, Fedora-Server-dvd-x86_64-Rawhide-20221125.n.0.iso , after a fresh install, /var/tmp is basically empty - just 4K of empty directories.

It feels like *something* is meant to clean up stuff under /var after install and...isn't doing it. Or, perhaps, some things under /var are meant to be mounted temporarily during install processes so they'll be clean afterwards, but that isn't happening. But I've no idea what could actually have changed - none of the obvious suspect packages changed between 1125 and 1126. It's pretty weird.

OK, so just a bit more data. This is definitely about the installed package set:

Fedora-Server-dvd-x86_64-Rawhide-20221125.n.0.iso - server package set - no bug
Fedora-Server-dvd-x86_64-Rawhide-20221126.n.0.iso - server package set - BUG
Fedora-Everything-netinst-x86_64-Rawhide-20221129.n.0.iso - minimal 20221125.n.0 package set - no bug
Fedora-Everything-netinst-x86_64-Rawhide-20221129.n.0.iso - minimal 20221126.n.0 package set - no bug
Fedora-Everything-netinst-x86_64-Rawhide-20221129.n.0.iso - server 20221125.n.0 package set - no bug
Fedora-Everything-netinst-x86_64-Rawhide-20221129.n.0.iso - server 20221126.n.0 package set - BUG

so, we get the bug when deploying the Server package set from 1126.n.0 or later, regardless *how* we do it (server DVD install or Everything netinst using a later image). We don't get the bug when deploying a minimal package set or the server package set from 1125.n.0 or earlier, even deploying with a later netinst image.

So whatever's going on, it's down to something in the installed package set. It must be something that's in Server and Workstation (and the other affected package sets...), but not in minimal. I still don't know *what* yet. I guess the next step is to start adding one suspect package (that's in both Server and Workstation 1125.n.0 -> 1126.n.0 diffs) to a kickstart at once, until the bug shows up...

So, I think I've identified the culprit here: it seems to be kexec-tools . If I do a kickstart install of server-product-environment using just the 20221125.n.0 Everything repo, /var/tmp is small. If I add a side repo which contains the newer kexec-tools that landed in 20221126.n.0 - kexec-tools-2.0.25-3.fc38 - /var/tmp grows to over 400M. The only difference in installed packages between the two installs is the version of kexec-tools.

There was quite a lot of change in the kexec-tools package between 2.0.25-2 and 2.0.25-3. None of the changes *directly* affects the package's scriptlets, but the package does do something in %posttrans:

%posttrans
# Try to reset kernel crashkernel value to new default value based on the old
# default value or set up crasherkernel value for osbuild
#
# Note
#  1. Skip ostree systems as they are not supported.
#  2. "[ $1 == 1 ]" in posttrans scriptlet means both install and upgrade. The
#     former case is used to set up crashkernel for osbuild
if [ ! -f /run/ostree-booted ] && [ $1 == 1 ]; then
  kdumpctl _reset-crashkernel-after-update
  rm /tmp/old_default_crashkernel 2>/dev/null
%ifarch ppc64 ppc64le
  rm /tmp/old_default_crashkernel_fadump 2>/dev/null
%endif
  # dnf would complain about the exit code not being 0. To keep it happy,
  # always return 0
  :
fi

and presumably the effects of that differ with the changes in -3. Coiby, can you please take a look and see if you can figure out why kexec-tools seems to be causing this problem? Thanks.

Just to confirm, I can reproduce the same results with a kickstart that just installs @core and kexec-tools . With the 20221125.n.0 repo only, small /var/tmp . With the side repo with kexec-tools-2.0.25-3 added, 400+M in /var/tmp.

Thanks for reporting this bug! I'll try reproducing it and then figuring out the root cause.

Sorry current kexec-tools unintentionally masks dracut's trap handler to clean up /var/tmp/dracut-*. A patch to address this bug has been posted to https://lists.fedoraproject.org/archives/list/kexec@lists.fedoraproject.org/thread/7CX4WYIN57JJP3H6YMBPCIF5X62W43LL/

Fixed in kexec-tools-2.0.25-4.fc38.

Workstation live x86_64 image https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20221207.n.0/compose/Workstation/x86_64/iso/Fedora-Workstation-Live-x86_64-Rawhide-20221207.n.0.iso from compose Fedora-Rawhide-20221207.n.0 is 2393202688 bytes, exceeding the maximum size 2100000000.

(In reply to Fedora QA Tools SIG from comment #10)
> Workstation live x86_64 image
> https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20221207.n.
> 0/compose/Workstation/x86_64/iso/Fedora-Workstation-Live-x86_64-Rawhide-
> 20221207.n.0.iso from compose Fedora-Rawhide-20221207.n.0 is 2393202688
> bytes, exceeding the maximum size 2100000000.

Fedora-Rawhide-20221207.n.0 still uses kexec-tools-2.0.25-3. You should check a compose that uses kexec-tools-2.0.25-4.

It's a bot, I didn't teach it natural language parsing yet :D just unlucky timing, it happened to run on the compose right before your change landed. I'll confirm the fix with the next compose.

I ran the update through openQA's Workstation live image build test manually: https://openqa.stg.fedoraproject.org/tests/overview?distri=fedora&groupid=2&version=38&build=Update-FEDORA-2022-09571c8756 . The generated image is 2072840192 bytes, back under the size limit (just). So looks like this does fix the issue, thanks. Will confirm with the next official compose too.

Confirmed resolved in current composes.

(In reply to Adam Williamson from comment #14)
> Confirmed resolved in current composes.

Thanks for the confirmation!

Workstation live x86_64 image https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20230111.n.0/compose/Workstation/x86_64/iso/Fedora-Workstation-Live-x86_64-Rawhide-20230111.n.0.iso from compose Fedora-Rawhide-20230111.n.0 is 2100852736 bytes, exceeding the maximum size 2100000000.

I downloaded https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20230111.n.0/compose/Workstation/x86_64/iso/Fedora-Workstation-Live-x86_64-Rawhide-20230111.n.0.iso and found /var/tmp/ in LiveOS/squashfs.img/LiveOS/rootfs.img is empty. So it has nothing to do with kexec-tools. 

Btw, compared with Fedora-Workstation-Live-x86_64-Rawhide-20221227.n.0.iso,Fedora-Workstation-Live-x86_64-Rawhide-20230111.n.0.iso has 6MB increase in /usr and 3MB increase in /boot. Notice 20221227.n.0.iso already has a size 1996MB which quite close to 2G,

    [root@ci-vm-10-0-139-141 ~]# du -ms 20221227.n.0/*|sort -h
    75      t3/boot
    5495    t3/usr
    
    [root@ci-vm-10-0-139-141 ~]# du -ms 20230111.n.0./*|sort -h
    72      b3/boot
    5489    b3/usr
    
    [root@ci-vm-10-0-139-141 ~]# du -ms *.iso
    1996    Fedora-Workstation-Live-x86_64-Rawhide-20221227.n.0.iso
    2004    Fedora-Workstation-Live-x86_64-Rawhide-20230111.n.0.iso

According the the comment #17, move back to distribution

Not planning to investigate this, sorry.

This is an automatic F38 blocker under the "The release-blocking images must meet current size requirements." criterion[1]. If you don't want to investigate it, then the other option is for the Workstation WG to raise the maximum size for this image.

[1] (link to F37 because the 38 page hasn't been created yet, apparently) https://fedoraproject.org/wiki/Fedora_37_Beta_Release_Criteria#Image_size_requirements
[2] https://docs.fedoraproject.org/en-US/releases/f38/blocking/

OK for avoidance of doubt, we will almost certainly just raise the size requirement. But we're intentionally keeping the size requirement low so that we get a noisy warning when it gets bigger like this, and people who want to investigate can do so.

The cause is likely something in the base system, as the netinst went oversize at the same time - https://bugzilla.redhat.com/show_bug.cgi?id=2151495 . I usually look into these, but I did not yet as I've been on PTO. I'll look into it today.

So, I forgot I actually did look into this last month, on the Server netinst ticket: https://bugzilla.redhat.com/show_bug.cgi?id=2151497#c4
the findings there are definitely the same for Everything netinst and probably also the problem for Workstation live as well. Basically the main culprits were the kernel and webkitgtk, which both mysteriously got bigger.

Workstation live x86_64 image https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20230121.n.0/compose/Workstation/x86_64/iso/Fedora-Workstation-Live-x86_64-Rawhide-20230121.n.0.iso from compose Fedora-Rawhide-20230121.n.0 is 2110459904 bytes, exceeding the maximum size 2100000000.

Workstation live x86_64 image https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20230126.n.0/compose/Workstation/x86_64/iso/Fedora-Workstation-Live-x86_64-Rawhide-20230126.n.0.iso from compose Fedora-Rawhide-20230126.n.0 is 2115725312 bytes, exceeding the maximum size 2100000000.

Workstation live x86_64 image https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20230202.n.0/compose/Workstation/x86_64/iso/Fedora-Workstation-Live-x86_64-Rawhide-20230202.n.0.iso from compose Fedora-Rawhide-20230202.n.0 is 2172342272 bytes, exceeding the maximum size 2100000000.

Workstation live x86_64 image https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20230205.n.0/compose/Workstation/x86_64/iso/Fedora-Workstation-Live-x86_64-Rawhide-20230205.n.0.iso from compose Fedora-Rawhide-20230205.n.0 is 2173857792 bytes, exceeding the maximum size 2100000000.

This bug appears to have been reported against 'rawhide' during the Fedora Linux 38 development cycle.
Changing version to 38.

Workstation live x86_64 image https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20230208.n.0/compose/Workstation/x86_64/iso/Fedora-Workstation-Live-x86_64-Rawhide-20230208.n.0.iso from compose Fedora-Rawhide-20230208.n.0 is 2172420096 bytes, exceeding the maximum size 2100000000.

Today's is 2078361600 , just barely undersize. Until someone makes something bigger again.

Workstation live x86_64 image https://kojipkgs.fedoraproject.org/compose/branched/Fedora-38-20230216.n.0/compose/Workstation/x86_64/iso/Fedora-Workstation-Live-x86_64-38-20230216.n.0.iso from compose Fedora-38-20230216.n.0 is 2102429696 bytes, exceeding the maximum size 2100000000.

Well, that was nice while it lasted.

Workstation live x86_64 image https://kojipkgs.fedoraproject.org/compose/branched/Fedora-38-20230219.n.0/compose/Workstation/x86_64/iso/Fedora-Workstation-Live-x86_64-38-20230219.n.0.iso from compose Fedora-38-20230219.n.0 is 2103158784 bytes, exceeding the maximum size 2100000000.

Workstation live x86_64 image https://kojipkgs.fedoraproject.org/compose/branched/Fedora-38-20230222.n.0/compose/Workstation/x86_64/iso/Fedora-Workstation-Live-x86_64-38-20230222.n.0.iso from compose Fedora-38-20230222.n.0 is 2120454144 bytes, exceeding the maximum size 2100000000.

So, I poked about a bit on a recent live image just looking for easy wins, and well...gcc is on there.

This seems to be because gdb recommends `gcc-gdb-plugin(x86-64)`, and gcc-gdb-plugin requires gcc. gdb in turn is on there because it's required by anaconda-install-env-deps, i.e. anaconda thinks it needs it in the installer environment. anaconda wants it for its in-line handling/backtracing of installer crashes, which, I guess, fine.

CCing anaconda folks and jkratochvil (gdb) for thoughts on whether we might want to break this chain somewhere. It seems kinda silly to have gcc on the lives purely because anaconda wants to be able to backtrace installer crashes. I think installer images don't include it because we only take hard deps in building the installer images, but for lives we also include recommends.

If we want to keep this dep chain but still don't want gcc on the lives, we could explicitly deselect gcc and gcc-gdb-plugin in the kickstarts, which I think would work. We'd want to make sure we do it for all the lives though, I guess.

jkratochvil seems to be off gdb since 2021, so CCing some other folks from the changelog instead.

I did a test run adding:

-gcc
-gcc-gdb-plugin

to the kickstart, and the results are:

before: 2117496832 Mar  1 23:16 02623223-Fedora-Workstation-Live-x86_64-FEDORA-2023-38a3ae638f.iso
after:  2062307328 Mar  2 00:30 02623394-Fedora-Workstation-Live-x86_64-FEDORA-2023-38a3ae638f.iso

so that definitely would save us the space we need.

Removing gcc and gcc-gdb-plugin is fine. The features in GDB provided by gcc-gdb-plugin are rarely used and may even be broken.

Thanks Kevin, I guess the question is really - should we keep the Recommends: in gdb? Is it really useful to pull that in in most cases? If we still think it's *generally* useful but want to drop them from the live images specifically, we leave the packages as-is and change the live kickstarts. If we think maybe even the soft dependency is *generally* a bit too much, we can remove it from the package (or make it a Suggests).

Or would it perhaps be better as a conditional:

Recommends: gcc-gdb-plugin if gcc

?

I've discussed this matter with Keith Seitz (keiths). Since the GDB compile feature is largely broken and has been so for some time, we think it best to remove the 'Recommends' line for gcc-gdb-plugin from gdb.spec.

I'll make this change for gdb-13.1-1 which should be available in the next day or so.

Awesome, thanks. Can you please make sure to create an update for F38 and mark it as fixing this bug? That's necessary as F38 is in freeze right now and we need the association with this bug (a Beta blocker) to pull it through the freeze. Thanks again!

FEDORA-2023-9e962f87be has been submitted as an update to Fedora 38. https://bodhi.fedoraproject.org/updates/FEDORA-2023-9e962f87be

FEDORA-2023-9e962f87be has been pushed to the Fedora 38 testing repository.

You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2023-9e962f87be

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Can you build this for Rawhide too so it stays ahead of F38 (and we get the benefit of the size reduction)? Thanks!

(In reply to Adam Williamson from comment #44)
> Can you build this for Rawhide too so it stays ahead of F38 (and we get the
> benefit of the size reduction)? Thanks!

Yes, I plan to update rawhide this week.

Workstation live x86_64 image https://kojipkgs.fedoraproject.org/compose/38/Fedora-38-20230307.1/compose/Workstation/x86_64/iso/Fedora-Workstation-Live-x86_64-38_Beta-1.2.iso from compose Fedora-38-20230307.1 is 2118473728 bytes, exceeding the maximum size 2100000000.

Crap. I'll see what went wrong.

OK, somehow the Beta candidate compose didn't get the updated gdb, even though I requested it be pulled in. The Workstation live has gdb-12.1-16.fc38 , so it still pulled in gcc. I'll see if I can find out what happened there.

Looks like it was just a misconfigured compose. There'll be a new one later today.

FEDORA-2023-9e962f87be has been pushed to the Fedora 38 stable repository.
If problem still persists, please make note of it in this bug report.

Setting back to ON_QA so we can confirm the fix in Beta-1.3.

Confirmed fixed in Beta-1.3.

Workstation live x86_64 image https://kojipkgs.fedoraproject.org/compose/branched/Fedora-38-20230315.n.0/compose/Workstation/x86_64/iso/Fedora-Workstation-Live-x86_64-38-20230315.n.0.iso from compose Fedora-38-20230315.n.0 is 2113777664 bytes, exceeding the maximum size 2100000000.

We went back oversize between 20230313.n.0 (which was 2064513024 bytes) and 20230314.n.0 (which was 2114273280 bytes). Looking into it now.

This is the package diff between the 0313.n.0 and 0314.n.0 lives. The obvious thing is we're including a lot more media packages - bits of ffmpeg and associated deps. This is likely caused by a change in gstreamer - note we went from 1.22.0 to 1.22.1 and added gstreamer1-plugin-libav (which wasn't installed before).

--- /tmp/0313	2023-03-15 14:11:14.736768612 -0700
+++ /tmp/0314	2023-03-15 14:11:18.605706563 -0700
@@ -2,3 +2,3 @@
-ImageMagick-1:7.1.0.62-1.fc39.x86_64
-ImageMagick-libs-1:7.1.0.62-1.fc39.x86_64
-LibRaw-0.21.1-2.fc38.x86_64
+ImageMagick-1:7.1.0.62-2.fc39.x86_64
+ImageMagick-libs-1:7.1.0.62-2.fc39.x86_64
+LibRaw-0.21.1-3.fc39.x86_64
@@ -153,0 +154 @@
+cjson-1.7.14-7.fc38.x86_64
@@ -160,0 +162 @@
+codec2-1.0.5-2.fc38.x86_64
@@ -168 +170 @@
-container-selinux-2:2.203.0-1.fc39.noarch
+container-selinux-2:2.204.0-1.fc39.noarch
@@ -175 +177 @@
-cpp-13.0.1-0.6.fc39.x86_64
+cpp-13.0.1-0.7.fc39.x86_64
@@ -186 +188 @@
-cups-1:2.4.2-9.fc39.x86_64
+cups-1:2.4.2-10.fc39.x86_64
@@ -188,2 +190,2 @@
-cups-client-1:2.4.2-9.fc39.x86_64
-cups-filesystem-1:2.4.2-9.fc39.noarch
+cups-client-1:2.4.2-10.fc39.x86_64
+cups-filesystem-1:2.4.2-10.fc39.noarch
@@ -191,2 +193,2 @@
-cups-ipptool-1:2.4.2-9.fc39.x86_64
-cups-libs-1:2.4.2-9.fc39.x86_64
+cups-ipptool-1:2.4.2-10.fc39.x86_64
+cups-libs-1:2.4.2-10.fc39.x86_64
@@ -313,0 +316 @@
+flite-2.2-5.fc39.x86_64
@@ -338,0 +342 @@
+game-music-emu-0.6.3-11.fc38.x86_64
@@ -371,3 +375,3 @@
-git-2.40.0-0.2.rc2.fc39.x86_64
-git-core-2.40.0-0.2.rc2.fc39.x86_64
-git-core-doc-2.40.0-0.2.rc2.fc39.noarch
+git-2.40.0-1.fc39.x86_64
+git-core-2.40.0-1.fc39.x86_64
+git-core-doc-2.40.0-1.fc39.noarch
@@ -500,8 +504,9 @@
-gst-editing-services-1.22.0-2.fc39.x86_64
-gstreamer1-1.22.0-1.fc38.x86_64
-gstreamer1-plugins-bad-free-1.22.0-1.fc38.x86_64
-gstreamer1-plugins-base-1.22.0-1.fc38.x86_64
-gstreamer1-plugins-good-1.22.0-2.fc39.x86_64
-gstreamer1-plugins-good-gtk-1.22.0-2.fc39.x86_64
-gstreamer1-plugins-good-qt-1.22.0-2.fc39.x86_64
-gstreamer1-plugins-ugly-free-1.22.0-1.fc38.x86_64
+gst-editing-services-1.22.1-1.fc39.x86_64
+gstreamer1-1.22.1-1.fc39.x86_64
+gstreamer1-plugin-libav-1.22.1-1.fc39.x86_64
+gstreamer1-plugins-bad-free-1.22.1-1.fc39.x86_64
+gstreamer1-plugins-base-1.22.1-1.fc39.x86_64
+gstreamer1-plugins-good-1.22.1-1.fc39.x86_64
+gstreamer1-plugins-good-gtk-1.22.1-1.fc39.x86_64
+gstreamer1-plugins-good-qt-1.22.1-1.fc39.x86_64
+gstreamer1-plugins-ugly-free-1.22.1-1.fc39.x86_64
@@ -511 +516 @@
-gtk4-4.10.0-3.fc39.x86_64
+gtk4-4.10.0-4.fc39.x86_64
@@ -575,0 +581 @@
+ilbc-3.0.4-4.fc38.x86_64
@@ -581,0 +588 @@
+intel-mediasdk-22.6.4-3.fc38.x86_64
@@ -638,5 +645,5 @@
-kernel-6.3.0-0.rc1.20230310git44889ba56cbb.19.fc39.x86_64
-kernel-core-6.3.0-0.rc1.20230310git44889ba56cbb.19.fc39.x86_64
-kernel-modules-6.3.0-0.rc1.20230310git44889ba56cbb.19.fc39.x86_64
-kernel-modules-core-6.3.0-0.rc1.20230310git44889ba56cbb.19.fc39.x86_64
-kernel-modules-extra-6.3.0-0.rc1.20230310git44889ba56cbb.19.fc39.x86_64
+kernel-6.3.0-0.rc2.23.fc39.x86_64
+kernel-core-6.3.0-0.rc2.23.fc39.x86_64
+kernel-modules-6.3.0-0.rc2.23.fc39.x86_64
+kernel-modules-core-6.3.0-0.rc2.23.fc39.x86_64
+kernel-modules-extra-6.3.0-0.rc2.23.fc39.x86_64
@@ -650,0 +658 @@
+ladspa-1.13-30.fc38.x86_64
@@ -657,0 +666 @@
+leptonica-1.83.1-1.fc38.x86_64
@@ -697,0 +707 @@
+libass-0.17.0-2.fc38.x86_64
@@ -701 +711 @@
-libatomic-13.0.1-0.6.fc39.x86_64
+libatomic-13.0.1-0.7.fc39.x86_64
@@ -702,0 +713,3 @@
+libavcodec-free-6.0-1.fc39.x86_64
+libavfilter-free-6.0-1.fc39.x86_64
+libavformat-free-6.0-1.fc39.x86_64
@@ -703,0 +717 @@
+libavutil-free-6.0-1.fc39.x86_64
@@ -727,0 +742 @@
+libbs2b-3.1.0-30.fc38.x86_64
@@ -739,0 +755 @@
+libchromaprint-1.5.1-8.fc39.x86_64
@@ -788 +804 @@
-libgcc-13.0.1-0.6.fc39.x86_64
+libgcc-13.0.1-0.7.fc39.x86_64
@@ -795 +811 @@
-libgfortran-13.0.1-0.6.fc39.x86_64
+libgfortran-13.0.1-0.7.fc39.x86_64
@@ -805 +821 @@
-libgomp-13.0.1-0.6.fc39.x86_64
+libgomp-13.0.1-0.7.fc39.x86_64
@@ -855,0 +872 @@
+libmodplug-1:0.8.9.0-16.fc38.x86_64
@@ -883,0 +901 @@
+libopenmpt-0.6.9-1.fc39.x86_64
@@ -900,0 +919 @@
+libplacebo-5.229.2-2.fc38.x86_64
@@ -907,0 +927 @@
+libpostproc-free-6.0-1.fc39.x86_64
@@ -917 +937 @@
-libquadmath-13.0.1-0.6.fc39.x86_64
+libquadmath-13.0.1-0.7.fc39.x86_64
@@ -918,0 +939 @@
+librabbitmq-0.13.0-1.fc38.x86_64
@@ -958,0 +980 @@
+librist-0.2.7-1.fc39.x86_64
@@ -969,0 +992 @@
+libshaderc-2023.1-3.fc39.x86_64
@@ -977,0 +1001 @@
+libsodium-1.0.18-11.fc38.x86_64
@@ -992 +1016 @@
-libstdc++-13.0.1-0.6.fc39.x86_64
+libstdc++-13.0.1-0.7.fc39.x86_64
@@ -993,0 +1018,2 @@
+libswresample-free-6.0-1.fc39.x86_64
+libswscale-free-6.0-1.fc39.x86_64
@@ -1023,0 +1050 @@
+libvdpau-1.5-3.fc38.x86_64
@@ -1113,0 +1141 @@
+lpcnetfreedv-0.2-13.fc38.x86_64
@@ -1120,0 +1149 @@
+lv2-1.18.8-5.fc38.x86_64
@@ -1138,0 +1168 @@
+mbedtls-2.28.2-2.fc38.x86_64
@@ -1163 +1193 @@
-mozjs102-102.8.0-1.fc39.x86_64
+mozjs102-102.9.0-1.fc39.x86_64
@@ -1202 +1232 @@
-ntfs-3g-system-compression-1.0-12.fc39.x86_64
+ntfs-3g-system-compression-1.0-13.fc39.x86_64
@@ -1206,0 +1237,3 @@
+ocl-icd-2.3.1-3.fc38.x86_64
+oneVPL-2023.1.3-1.fc39.x86_64
+oneVPL-intel-gpu-23.1.3-1.fc39.x86_64
@@ -1212,0 +1246 @@
+opencore-amr-0.1.6-3.fc38.x86_64
@@ -1216,0 +1251 @@
+openpgm-5.2.122-31.fc38.x86_64
@@ -1218,3 +1253,3 @@
-openssh-9.0p1-12.fc39.1.x86_64
-openssh-clients-9.0p1-12.fc39.1.x86_64
-openssh-server-9.0p1-12.fc39.1.x86_64
+openssh-9.0p1-13.fc39.1.x86_64
+openssh-clients-9.0p1-13.fc39.1.x86_64
+openssh-server-9.0p1-13.fc39.1.x86_64
@@ -1223 +1258 @@
-openvpn-2.6.0-2.fc38.x86_64
+openvpn-2.6.1-1.fc39.x86_64
@@ -1280 +1315 @@
-perl-Git-2.40.0-0.2.rc2.fc39.noarch
+perl-Git-2.40.0-1.fc39.noarch
@@ -1392,2 +1427,2 @@
-python3-crypto-2.6.1-42.fc38.x86_64
-python3-cryptography-39.0.2-1.fc39.x86_64
+python3-crypto-2.6.1-43.fc39.x86_64
+python3-cryptography-39.0.2-2.fc39.x86_64
@@ -1533 +1568 @@
-qt-settings-38.2-1.fc39.noarch
+qt-settings-38.2-2.fc39.noarch
@@ -1570,0 +1606 @@
+rubberband-3.1.2-2.fc38.x86_64
@@ -1619,0 +1656 @@
+spirv-tools-libs-2023.1-2.fc39.x86_64
@@ -1622,0 +1660 @@
+srt-libs-1.5.2-0.rc1.fc39.x86_64
@@ -1663,0 +1702,3 @@
+tesseract-5.3.0-5.fc38.x86_64
+tesseract-langpack-eng-4.1.0-5.fc38.noarch
+tesseract-tessdata-doc-4.1.0-5.fc38.noarch
@@ -1716,0 +1758,2 @@
+vamp-plugin-sdk-2.9.0-7.fc38.x86_64
+vapoursynth-libs-58-4.fc38.x86_64
@@ -1717,0 +1761 @@
+vid.stab-1.1.0-21.20201110gitf9166e9.fc38.x86_64
@@ -1721,0 +1766 @@
+vo-amrwbenc-0.1.3-18.fc38.x86_64
@@ -1791,0 +1837 @@
+xvidcore-1.3.7-9.fc38.x86_64
@@ -1802,0 +1849 @@
+zeromq-4.3.4-6.fc39.x86_64
@@ -1803,0 +1851 @@
+zimg-3.0.4-2.fc38.x86_64
@@ -1807,0 +1856 @@
+zvbi-0.2.35-19.fc38.x86_64

Oh, yikes, I was working on Rawhide not F38 by accident. Analysis is probably the same, though.

F38 diff. It's similar, but not the same. gstreamer1-plugin-libav is pulled in where it wasn't before, and lots of media packages that are probably deps of it - but there wasn't any change to any other gstreamer packages. So something *else* is causing gstreamer1-plugin-libav to be pulled in.

--- /tmp/0313	2023-03-15 14:20:50.148540298 -0700
+++ /tmp/0314	2023-03-15 14:21:10.580212609 -0700
@@ -98 +98 @@
-autocorr-en-1:7.5.0.3-2.fc38.noarch
+autocorr-en-1:7.5.1.2-2.fc38.noarch
@@ -136 +136 @@
-btrfs-progs-6.1.3-1.fc38.x86_64
+btrfs-progs-6.2.1-1.fc38.x86_64
@@ -142,2 +142,2 @@
-cairo-1.17.8-1.fc38.x86_64
-cairo-gobject-1.17.8-1.fc38.x86_64
+cairo-1.17.8-2.fc38.x86_64
+cairo-gobject-1.17.8-2.fc38.x86_64
@@ -153,0 +154 @@
+cjson-1.7.14-7.fc38.x86_64
@@ -160,0 +162 @@
+codec2-1.0.5-2.fc38.x86_64
@@ -167 +169 @@
-conmon-2:2.1.6-3.fc38.x86_64
+conmon-2:2.1.7-2.fc38.x86_64
@@ -223 +225 @@
-distribution-gpg-keys-1.84-1.fc38.noarch
+distribution-gpg-keys-1.85-1.fc38.noarch
@@ -311,0 +314 @@
+flite-2.2-3.fc37.x86_64
@@ -320 +323 @@
-freetype-2.12.1-4.fc38.x86_64
+freetype-2.13.0-2.fc38.x86_64
@@ -336,0 +340 @@
+game-music-emu-0.6.3-11.fc38.x86_64
@@ -365,3 +369,3 @@
-ghostscript-10.0.0-1.fc38.x86_64
-ghostscript-tools-fonts-10.0.0-1.fc38.x86_64
-ghostscript-tools-printing-10.0.0-1.fc38.x86_64
+ghostscript-10.0.0-3.fc38.x86_64
+ghostscript-tools-fonts-10.0.0-3.fc38.x86_64
+ghostscript-tools-printing-10.0.0-3.fc38.x86_64
@@ -441,3 +445,3 @@
-gnutls-3.7.8-11.fc38.x86_64
-gnutls-dane-3.7.8-11.fc38.x86_64
-gnutls-utils-3.7.8-11.fc38.x86_64
+gnutls-3.8.0-2.fc38.x86_64
+gnutls-dane-3.8.0-2.fc38.x86_64
+gnutls-utils-3.8.0-2.fc38.x86_64
@@ -499,0 +504 @@
+gstreamer1-plugin-libav-1.22.0-2.fc38.x86_64
@@ -509 +514 @@
-gtk4-4.10.0-3.fc38.x86_64
+gtk4-4.10.0-4.fc38.x86_64
@@ -534,2 +539,2 @@
-harfbuzz-7.0.0-2.fc38.x86_64
-harfbuzz-icu-7.0.0-2.fc38.x86_64
+harfbuzz-7.0.1-2.fc38.x86_64
+harfbuzz-icu-7.0.1-2.fc38.x86_64
@@ -573,0 +579 @@
+ilbc-3.0.4-4.fc38.x86_64
@@ -579,0 +586 @@
+intel-mediasdk-22.6.4-3.fc38.x86_64
@@ -632,3 +639,3 @@
-kbd-2.5.1-3.fc38.x86_64
-kbd-legacy-2.5.1-3.fc38.noarch
-kbd-misc-2.5.1-3.fc38.noarch
+kbd-2.5.1-5.fc38.x86_64
+kbd-legacy-2.5.1-5.fc38.noarch
+kbd-misc-2.5.1-5.fc38.noarch
@@ -648,0 +656 @@
+ladspa-1.13-30.fc38.x86_64
@@ -655,0 +664 @@
+leptonica-1.83.1-1.fc38.x86_64
@@ -695,0 +705 @@
+libass-0.17.0-2.fc38.x86_64
@@ -700,0 +711,3 @@
+libavcodec-free-6.0-1.fc38.x86_64
+libavfilter-free-6.0-1.fc38.x86_64
+libavformat-free-6.0-1.fc38.x86_64
@@ -701,0 +715 @@
+libavutil-free-6.0-1.fc38.x86_64
@@ -725,0 +740 @@
+libbs2b-3.1.0-30.fc38.x86_64
@@ -737,0 +753 @@
+libchromaprint-1.5.1-8.fc38.x86_64
@@ -808 +824 @@
-libgs-10.0.0-1.fc38.x86_64
+libgs-10.0.0-3.fc38.x86_64
@@ -853,0 +870 @@
+libmodplug-1:0.8.9.0-16.fc38.x86_64
@@ -881,0 +899 @@
+libopenmpt-0.6.8-1.fc38.x86_64
@@ -898,0 +917 @@
+libplacebo-5.229.2-2.fc38.x86_64
@@ -905,0 +925 @@
+libpostproc-free-6.0-1.fc38.x86_64
@@ -916,0 +937 @@
+librabbitmq-0.13.0-1.fc38.x86_64
@@ -922,20 +943,20 @@
-libreoffice-calc-1:7.5.0.3-2.fc38.x86_64
-libreoffice-core-1:7.5.0.3-2.fc38.x86_64
-libreoffice-data-1:7.5.0.3-2.fc38.x86_64
-libreoffice-emailmerge-1:7.5.0.3-2.fc38.x86_64
-libreoffice-filters-1:7.5.0.3-2.fc38.x86_64
-libreoffice-graphicfilter-1:7.5.0.3-2.fc38.x86_64
-libreoffice-gtk3-1:7.5.0.3-2.fc38.x86_64
-libreoffice-gtk4-1:7.5.0.3-2.fc38.x86_64
-libreoffice-help-en-1:7.5.0.3-2.fc38.x86_64
-libreoffice-impress-1:7.5.0.3-2.fc38.x86_64
-libreoffice-langpack-en-1:7.5.0.3-2.fc38.x86_64
-libreoffice-ogltrans-1:7.5.0.3-2.fc38.x86_64
-libreoffice-opensymbol-fonts-1:7.5.0.3-2.fc38.noarch
-libreoffice-pdfimport-1:7.5.0.3-2.fc38.x86_64
-libreoffice-pyuno-1:7.5.0.3-2.fc38.x86_64
-libreoffice-ure-1:7.5.0.3-2.fc38.x86_64
-libreoffice-ure-common-1:7.5.0.3-2.fc38.x86_64
-libreoffice-writer-1:7.5.0.3-2.fc38.x86_64
-libreoffice-x11-1:7.5.0.3-2.fc38.x86_64
-libreoffice-xsltfilter-1:7.5.0.3-2.fc38.x86_64
+libreoffice-calc-1:7.5.1.2-2.fc38.x86_64
+libreoffice-core-1:7.5.1.2-2.fc38.x86_64
+libreoffice-data-1:7.5.1.2-2.fc38.x86_64
+libreoffice-emailmerge-1:7.5.1.2-2.fc38.x86_64
+libreoffice-filters-1:7.5.1.2-2.fc38.x86_64
+libreoffice-graphicfilter-1:7.5.1.2-2.fc38.x86_64
+libreoffice-gtk3-1:7.5.1.2-2.fc38.x86_64
+libreoffice-gtk4-1:7.5.1.2-2.fc38.x86_64
+libreoffice-help-en-1:7.5.1.2-2.fc38.x86_64
+libreoffice-impress-1:7.5.1.2-2.fc38.x86_64
+libreoffice-langpack-en-1:7.5.1.2-2.fc38.x86_64
+libreoffice-ogltrans-1:7.5.1.2-2.fc38.x86_64
+libreoffice-opensymbol-fonts-1:7.5.1.2-2.fc38.noarch
+libreoffice-pdfimport-1:7.5.1.2-2.fc38.x86_64
+libreoffice-pyuno-1:7.5.1.2-2.fc38.x86_64
+libreoffice-ure-1:7.5.1.2-2.fc38.x86_64
+libreoffice-ure-common-1:7.5.1.2-2.fc38.x86_64
+libreoffice-writer-1:7.5.1.2-2.fc38.x86_64
+libreoffice-x11-1:7.5.1.2-2.fc38.x86_64
+libreoffice-xsltfilter-1:7.5.1.2-2.fc38.x86_64
@@ -956,0 +978 @@
+librist-0.2.7-1.fc38.x86_64
@@ -967,0 +990 @@
+libshaderc-2022.2-5.fc38.x86_64
@@ -975,0 +999 @@
+libsodium-1.0.18-11.fc38.x86_64
@@ -991,0 +1016,2 @@
+libswresample-free-6.0-1.fc38.x86_64
+libswscale-free-6.0-1.fc38.x86_64
@@ -1021,0 +1048 @@
+libvdpau-1.5-3.fc38.x86_64
@@ -1107,0 +1135 @@
+lpcnetfreedv-0.2-13.fc38.x86_64
@@ -1114,0 +1143 @@
+lv2-1.18.8-5.fc38.x86_64
@@ -1132,0 +1162 @@
+mbedtls-2.28.2-2.fc38.x86_64
@@ -1200,0 +1231,3 @@
+ocl-icd-2.3.1-3.fc38.x86_64
+oneVPL-2022.2.2-1.fc38.x86_64
+oneVPL-intel-gpu-22.5.3-1.fc38.x86_64
@@ -1206,0 +1240 @@
+opencore-amr-0.1.6-3.fc38.x86_64
@@ -1210,0 +1245 @@
+openpgm-5.2.122-31.fc38.x86_64
@@ -1238 +1273 @@
-passt-0^20230216.g4663ccc-1.fc38.x86_64
+passt-0^20230227.gc538ee8-1.fc38.x86_64
@@ -1528,3 +1563,3 @@
-qt5-qtbase-5.15.8-5.fc38.x86_64
-qt5-qtbase-common-5.15.8-5.fc38.noarch
-qt5-qtbase-gui-5.15.8-5.fc38.x86_64
+qt5-qtbase-5.15.8-6.fc38.x86_64
+qt5-qtbase-common-5.15.8-6.fc38.noarch
+qt5-qtbase-gui-5.15.8-6.fc38.x86_64
@@ -1564,0 +1600 @@
+rubberband-3.1.2-2.fc38.x86_64
@@ -1613,0 +1650 @@
+spirv-tools-libs-2023.1-2.fc38.x86_64
@@ -1616,0 +1654 @@
+srt-libs-1.5.1-4.fc38.x86_64
@@ -1657,0 +1696,3 @@
+tesseract-5.3.0-5.fc38.x86_64
+tesseract-langpack-eng-4.1.0-5.fc38.noarch
+tesseract-tessdata-doc-4.1.0-5.fc38.noarch
@@ -1710,0 +1752,2 @@
+vamp-plugin-sdk-2.9.0-7.fc38.x86_64
+vapoursynth-libs-58-4.fc38.x86_64
@@ -1711,0 +1755 @@
+vid.stab-1.1.0-21.20201110gitf9166e9.fc38.x86_64
@@ -1715,0 +1760 @@
+vo-amrwbenc-0.1.3-18.fc38.x86_64
@@ -1785,0 +1831 @@
+xvidcore-1.3.7-9.fc38.x86_64
@@ -1796,0 +1843 @@
+zeromq-4.3.4-5.fc38.x86_64
@@ -1797,0 +1845 @@
+zimg-3.0.4-2.fc38.x86_64
@@ -1801,0 +1850 @@
+zvbi-0.2.35-19.fc38.x86_64

Oh, I see. It was explicitly added to comps:

https://pagure.io/fedora-comps/c/85ce9364001f0546ac229fc0ebbe968a4197620d?branch=main
https://pagure.io/fedora-comps/c/9f57a685bac8a8279b5791473363ee990db5c9c8?branch=main

Well, it makes things bigger. Adding Neal.

The Workstation WG wanted this to be preloaded, which is why it's now added. Can we raise the limit for this?

If there's no way to trim the deps, that might be the only choice. Or else somebody gets to go spelunking for something else to trim. Can it not be me this time? I did it last time. :D

I agree this package and its dependencies are desired. Let's increase the size limit to 2200000000.

The top 20 packages by size (note, *uncompressed*) on the WS live currently are:

309106573 libreoffice-core 
233261669 firefox 
228747891 glibc-all-langpacks 
203852446 java-17-openjdk-headless 
131434892 linux-firmware 
116851881 ibus 
114389280 llvm-libs 
98675383 cldr-emoji-annotation 
94359944 mesa-dri-drivers 
76992027 webkitgtk6.0 
76452896 webkit2gtk4.1 
67133502 kernel-core 
64613102 google-noto-sans-cjk-vf-fonts 
63522492 kernel-modules 
62712603 geolite2-city 
54666412 gnome-user-docs 
53586942 firefox-langpacks 
51446292 libpinyin-data 
50895934 podman 
46341382 python3-libs 

I don't see any obvious easy wins there. cldr-emoji-annotation might be a target except it's all XML files that I suspect compress very well so probably aren't taking anything like 95M 'really'. It's unfortunate that we need two webkitgtk's but I'm guessing we can't solve that yet? gnome-shell appears to ultimately require both. Java is there because of libreoffice. We already trimmed linux-firmware this cycle, don't think we can get it any smaller (but I'll take a look).

Going down to top 30, there's openblas-openmp which is 40M and only required by gnome-photos (which I think we were talking about dropping anyway?)

There's also qt6-qtdeclarative , which causes me to notice we currently have both qt5 and qt6 on Workstation live. qt5 seems to be there by policy, and I guess as a dep for adwaita-qt5 and qgnomeplatform-qt5 which I guess are about making Qt apps look more native. There don't seem to be any actual apps that require it. qt6 is there as a dep of mediawriter.

I also notice linux-firmware still has about 29M of qualcomm firmware which mostly seems to be for Adreno GPUs, which are only useful on aarch64. We might be able to get them off of x86_64 at least. I've filed https://bugzilla.redhat.com/show_bug.cgi?id=2178852 for that.

I guess we'll drop webkit2gtk4.1, which is pretty close to the top of the top 30, sometime soon hopefully.

Probably not. It's required by gnome-online-accounts and yelp, which are both weakly maintained. Unless somebody decides to try porting them to GTK 4, which is not simple, we'll be stuck with it.

(It might be required by more, but these are the two apps that immediately come to mind.)

The max size was bumped to 2.2G, so this is now 'fixed'.