(In reply to Christian Stadelmann from comment #0)
> Description of problem:
> When installing the webkitgtk4 debuginfo package, it is 1.5 GB to download.
> The /usr/lib/debug/usr/libexec/webkit2gtk-4.0/WebKitPluginProcess2.debug
> file alone is 2.3 GB in size. This seems wrong, can it be smaller?

I think the GTK+ 2 plugin process has its own static-linked copy of WebCore. We *could* cut down on file size by installing a private shared library for the files that don't need GTK+, but that's something that would have to be discussed upstream.

> Debugging webkitgtk4 often does not require debug symbols for the Gtk2
> plugin process. Is it possible to put the Gtk2 plugin debug symbols into a
> separate package that only gets installed if webkitgtk4-plugin-process-gtk2
> is installed?

Hm, I'm surprised that's not already the case....

8 of the The current .debug files for webkitgtk4 (2.17.3) exceed the _dwz_max_die_limit_x86_64 of 110000000 (current redhat-rpm-config) 

Tried locally setting the limit to 200000000, which allowed dwz to halve the size of all those while keeping memory utilization of the dwz to 8-10G in the process.

Maybe we can try
%global _dwz_max_die_limit_x86_64 200000000  
and see how that goes in koji

(In reply to Yanko Kaneti from comment #2)
> 8 of the The current .debug files for webkitgtk4 (2.17.3) exceed the
> _dwz_max_die_limit_x86_64 of 110000000 (current redhat-rpm-config) 
> 
> Tried locally setting the limit to 200000000, which allowed dwz to halve the
> size of all those while keeping memory utilization of the dwz to 8-10G in
> the process.
> 
> Maybe we can try
> %global _dwz_max_die_limit_x86_64 200000000  
> and see how that goes in koji

Thank you for investigation! I tried to make a 2.17.3 scratch build[0] with your suggestions and the debuginfo size decreased from 5 GB (if comparing with original build[1]) to 2.9 GB and that's a significant decrease.

[0] - https://koji.fedoraproject.org/koji/taskinfo?taskID=20027544
[1] - https://koji.fedoraproject.org/koji/buildinfo?buildID=895797

2.9 (for the rpm) still seems a bit too much. Not long ago the installed size of the debuginfo was around 1.5G if I recall correctly.

from the scratch build log:
dwz: ./usr/lib64/libwebkit2gtk-4.0.so.37.22.0-2.17.3-2.fc27.x86_64.debug: Too many DIEs, not optimizing

Not sure why I didn't see that here. I am guessing we could shave a 1G(installed) there by trying 250000000

(In reply to Yanko Kaneti from comment #4)
> 2.9 (for the rpm) still seems a bit too much. Not long ago the installed
> size of the debuginfo was around 1.5G if I recall correctly.
> 
> from the scratch build log:
> dwz: ./usr/lib64/libwebkit2gtk-4.0.so.37.22.0-2.17.3-2.fc27.x86_64.debug:
> Too many DIEs, not optimizing
> 
> Not sure why I didn't see that here. I am guessing we could shave a
> 1G(installed) there by trying 250000000

1.1 with 25000000 set - https://koji.fedoraproject.org/koji/taskinfo?taskID=20041364

I will push it to rawhide first to see how it behaves and then to other branches as well.

I noticed that aarch64 debuginfo package are affected as well. I tried to increase the limit there as well[0], but there were some errors:

dwz: dwz.c:9253: recompute_abbrevs: Assertion `off == cu_size' failed.
/usr/lib/rpm/find-debuginfo.sh: line 418: 25251 Aborted                 (core dumped) dwz $dwz_opts $dwz_files
/usr/lib/rpm/sepdebugcrcfix: Updated 2 CRC32s, 8 CRC32s did match.
cpio: aarch64-redhat-linux-gnu/Source/WebCore/CSSPropertyNames.gperf: Cannot stat: No such file or directory
cpio: aarch64-redhat-linux-gnu/Source/WebCore/CSSValueKeywords.gperf: Cannot stat: No such file or directory
cpio: aarch64-redhat-linux-gnu/Source/WebCore/HTTPHeaderNames.gperf: Cannot stat: No such file or directory
cpio: aarch64-redhat-linux-gnu/Source/WebCore/SelectorPseudoClassAndCompatibilityElementMap.gperf: Cannot stat: No such file or directory
cpio: aarch64-redhat-linux-gnu/Source/WebCore/SelectorPseudoElementTypeMap.gperf: Cannot stat: No such file or directory
cpio: aarch64-redhat-linux-gnu/Source/WebCore/Tokenizer.l: Cannot stat: No such file or directory
cpio: aarch64-redhat-linux-gnu/Source/WebCore/WebCore/xml/XPathGrammar.y: Cannot stat: No such file or directory
cpio: aarch64-redhat-linux-gnu/Source/WebCore/XPathGrammar.cpp: Cannot stat: No such file or directory
cpio: aarch64-redhat-linux-gnu/Source/WebCore/XPathGrammar.hpp: Cannot stat: No such file or directory
cpio: aarch64-redhat-linux-gnu/Source/WebCore/glslang_lex.cpp: Cannot stat: No such file or directory

Maybe it could be caused by not having enough memory in builders? Any idea Peter? It decreased there from ~5.6G to ~4.6G.

[0] - https://koji.fedoraproject.org/koji/taskinfo?taskID=20042012

> Maybe it could be caused by not having enough memory in builders? Any idea
> Peter? It decreased there from ~5.6G to ~4.6G.

Unlikely, both the ARMv7 and aarch64 builders have 24Gb of RAM (all x86 VM builders have 10)

Note that https://kojipkgs.fedoraproject.org/packages/webkitgtk4/2.17.3/1.fc27/aarch64 has a 5.1GB debuginfo rpm, and the respective x86_64 build as well.

Please note that this RPM will be unable to go out soon, since our signing infrastructure cannot handle >4GB at this moment (two places use 32-bit integers to represent size, one of which is koji: https://koji.fedoraproject.org/koji/rpminfo?rpmID=9792187).
While I will be fixing at least the signing infra part, you might want to look at getting new builds for rawhide, since the current ones won't be able to go out soon.

The signing software limitation is recorded as bug #1462565.

(In reply to Patrick Uiterwijk from comment #8)
> While I will be fixing at least the signing infra part, you might want to
> look at getting new builds for rawhide, since the current ones won't be able
> to go out soon.

Building right now as webkitgtk4-2.17.3-2.fc27

(In reply to Michael Catanzaro from comment #1)
> (In reply to Christian Stadelmann from comment #0)
> > Debugging webkitgtk4 often does not require debug symbols for the Gtk2
> > plugin process. Is it possible to put the Gtk2 plugin debug symbols into a
> > separate package that only gets installed if webkitgtk4-plugin-process-gtk2
> > is installed?
> 
> Hm, I'm surprised that's not already the case....

For having separate debuginfo files for subpackage, see this change which has just been announced: https://fedoraproject.org/wiki/Changes/SubpackageAndSourceDebuginfo

I know that the change does not fix the core issue.


Interestingly, on i686, the -debuginfo package is just 70MB: https://kojipkgs.fedoraproject.org//packages/webkitgtk4/2.16.3/1.fc26/i686/

(In reply to Christian Stadelmann from comment #11)
> Interestingly, on i686, the -debuginfo package is just 70MB:
> https://kojipkgs.fedoraproject.org//packages/webkitgtk4/2.16.3/1.fc26/i686/

From the SPEC file:

# Decrease debuginfo even on ix86 because of:
# https://bugs.webkit.org/show_bug.cgi?id=140176

https://src.fedoraproject.org/cgit/rpms/webkitgtk4.git/tree/webkitgtk4.spec#n158

Please note that in the webkitgtk4-2.17.4-1.fc27 build, the x86_64 debuginfo got too big: https://koji.fedoraproject.org/koji/buildinfo?buildID=909563 (https://koji.fedoraproject.org/koji/rpminfo?rpmID=10003418).

Please rebuild epiphany in F27 when this is fixed. It's failing now since it requires 2.17.3.

(In reply to Michael Catanzaro from comment #14)
> Please rebuild epiphany in F27 when this is fixed. It's failing now since it
> requires 2.17.3.

https://koji.fedoraproject.org/koji/taskinfo?taskID=20609183

(In reply to Tomas Popela from comment #15)
> (In reply to Michael Catanzaro from comment #14)
> > Please rebuild epiphany in F27 when this is fixed. It's failing now since it
> > requires 2.17.3.
> 
> https://koji.fedoraproject.org/koji/taskinfo?taskID=20609183

In the end it's https://koji.fedoraproject.org/koji/taskinfo?taskID=20609409 I didn't wait for the webkitgtk4 build to hit the repo

Hmm I see that in the last webkitgtk4 build the size again regressed - for x86_64 the webkitgtk4-debuginfo is ~3.2 GB..

Maybe this could be the cause?:

dwz: dwz.c:9253: recompute_abbrevs: Assertion `off == cu_size' failed.
/usr/lib/rpm/find-debuginfo.sh: line 518:  5059 Aborted                 (core dumped) dwz $dwz_opts ${dwz_files[@]}
/usr/lib/rpm/sepdebugcrcfix: Updated 4 CRC32s, 7 CRC32s did match.

taken from ( https://koji.fedoraproject.org/koji/taskinfo?taskID=21062418 )

Mark, any idea on comment 17?

(In reply to Tomas Popela from comment #18)
> Mark, any idea on comment 17?

(Comment 17):
> Hmm I see that in the last webkitgtk4 build the size again regressed - for
> x86_64 the webkitgtk4-debuginfo is ~3.2 GB..
>
> Maybe this could be the cause?:
> 
> dwz: dwz.c:9253: recompute_abbrevs: Assertion `off == cu_size' failed.
> /usr/lib/rpm/find-debuginfo.sh: line 518:  5059 Aborted                 (core
> dumped) dwz $dwz_opts ${dwz_files[@]}
> /usr/lib/rpm/sepdebugcrcfix: Updated 4 CRC32s, 7 CRC32s did match.
>
> taken from ( https://koji.fedoraproject.org/koji/taskinfo?taskID=21062418 )

Yes, that would certainly explain why the debuginfo would be bigger. The dwz (DWARF compressor) crashed, so no duplication reduction was done.

Why it crashed I don't immediately know. This is an assertion on an invariant that should hold in:

/* Recompute abbrevs for CU.  If any children were collapsed during
   compute_abbrevs, their ->u.p2.die_new_abbrev and ->u.p2.die_new_offset
   fields are no longer available and need to be computed again.  */
static void
recompute_abbrevs (dw_cu_ref cu, unsigned int cu_size)

The assert would trigger if the calculated CU offset at the end doesn't match the given cu_size. I cannot immediately see why/when that would be the case. Maybe Jakub (CCed) immediately recognizes when this could trigger.

Probably best to open a bug against dwz.

This bug appears to have been reported against 'rawhide' during the Fedora 27 development cycle.
Changing version to '27'.

The size is now down to 887MB (single debuginfo on F26) and 316MB (-plugin-process-gtk2-debuginfo on F27), less than 1GB for all -debuginfo subpackages on F27 accumulated. This looks way better, thank you!

(In reply to Mark Wielaard from comment #19)
> Probably best to open a bug against dwz.

Did someone report a bug against dwz and did it get fixed? If yes, can this bug be closed as fixed?

> (In reply to Mark Wielaard from comment #19)
> > Probably best to open a bug against dwz.
> 
> Did someone report a bug against dwz and did it get fixed? If yes, can this
> bug be closed as fixed?

No I didn't filled anything. But I checked the some of the latest webkitgtk4 builds and the dwz didn't crashed there.

(In reply to Christian Stadelmann from comment #21)
> The size is now down to 887MB (single debuginfo on F26) and 316MB
> (-plugin-process-gtk2-debuginfo on F27), less than 1GB for all -debuginfo
> subpackages on F27 accumulated. This looks way better, thank you!
> 
> (In reply to Mark Wielaard from comment #19)
> > Probably best to open a bug against dwz.
> 
> Did someone report a bug against dwz and did it get fixed?

It would be great if somebody could file a dwz report upstream ( https://sourceware.org/bugzilla/enter_bug.cgi?product=dwz ).

Either it is fixed, which would be good to know, or it still needs fixing.

Thanks,
- Tom