Thanks for reporting! Do you have any pointers what might be causing this and where to start looking? At the first sight it looks like something way above my pay grade :o

With mame-0.199-2.fc29.x86_64, I do not see the problem:

$ readelf -S /usr/bin/mame
There are 32 section headers, starting at offset 0xe74d210:

Section Headers:
  [Nr] Name              Type             Address           Offset
       Size              EntSize          Flags  Link  Info  Align
  [ 0]                   NULL             0000000000000000  00000000
       0000000000000000  0000000000000000           0     0     0
  [ 1] .interp           PROGBITS         0000000000000350  00000350
       000000000000001c  0000000000000000   A       0     0     1

Could you try 0.199-4? It was built only a few days back so if something is off with the build system it could be useful information. Thanks!

(In reply to Julian Sikorski from comment #3)
> Could you try 0.199-4? It was built only a few days back so if something is
> off with the build system it could be useful information. Thanks!

mame-0.199-4.fc29.x86_64 is already corrupted.

This points to builds problems as changes between -2 and -4 are minimal. -3 is just a rebuild of -2 if you would like to check. I somehow suspect annobin as it has been causing problems with sections already.
Nick, would you mind having a look? Thanks!

mame-0.199-3.fc29.x86_64 is corrupted as well, so this would point to a toolchain issue.

Mark noted some dwz crashes in the build log:

extracting debug info from /builddir/build/BUILDROOT/mame-0.199-1.fc27.x86_64/usr/bin/chdman
extracting debug info from /builddir/build/BUILDROOT/mame-0.199-1.fc27.x86_64/usr/bin/floptool
munmap_chunk(): invalid pointer
/usr/lib/rpm/find-debuginfo.sh: line 231:  6729 Aborted                 (core dumped) eu-strip --remove-comment $r $g ${keep_remove_args} -f "$1" "$2"
dwz: dwz.c:9899: read_dwarf: Assertion `data != NULL && data->d_buf != NULL' failed.
/usr/lib/rpm/find-debuginfo.sh: line 522:  6748 Aborted                 (core dumped) dwz $dwz_opts ${dwz_files[@]}
/usr/lib/rpm/sepdebugcrcfix: Updated 7 CRC32s, 9 CRC32s did match.

This is from <https://kojipkgs.fedoraproject.org//packages/mame/0.199/1.fc27/data/logs/x86_64/build.log>, so the issue goes *way* back.  It could be that the more recent corruption is indirectly related to that.

I checked the previous build logs for dwz crashes and found the following: while 0.199 and higher have those across all Fedora releases and at least on x86_64 and aarch64, 0.198 and 0.197 were not affected by the dwz crash - at least on rawhide.

I also had a closer look at 0.200-1.fc29 and it turns out that dwz only crashes on aarch64 and x86_64, but not on armv7hl, s390x or i686. Incidentally this matches the architectures which have the default symbols level left at -g2:

%ifarch %{arm} %{ix86} riscv64 s390x
RPM_OPT_FLAGS=$(echo $RPM_OPT_FLAGS | sed "s@-g@-g1@")

Moreover, prior to 0.199 -g1 was being used across the board. mame is a big project, looks like compiling it with -g stretches dwz beyond its capabilities.
Florian, do you have the ability to check the binary using readelf on any of the architectures on which dwz does not crash? I do not have any VMs set up at the moment unfortunately.

(In reply to Julian Sikorski from comment #9)

> Florian, do you have the ability to check the binary using readelf on any of
> the architectures on which dwz does not crash? I do not have any VMs set up
> at the moment unfortunately.

You can use eu-readelf from elfutils for cross-architecture checking (or use cross-binutils).  eu-readelf -S shows basically the same corrupted output.

Looks like it is really arch-specific:
$ readelf -S mame-i686 
There are 8648 section headers, starting at offset 0xfe2fb1c:

Section Headers:
  [Nr] Name              Type            Addr     Off    Size   ES Flg Lk Inf Al
  [ 0]                   NULL            00000000 000000 000000 00      0   0  0
  [ 1] .interp           PROGBITS        000001b4 0001b4 000013 00   A  0   0  1
  [ 2] .note.ABI-tag     NOTE            000001c8 0001c8 000020 00   A  0   0  4
  [ 3] .note.gnu.build-i NOTE            000001e8 0001e8 000024 00   A  0   0  4
  [ 4] .gnu.hash         GNU_HASH        0000020c 00020c 0001d0 04   A  5   0  4
  [ 5] .dynsym           DYNSYM          000003dc 0003dc 005270 10   A  6   1  4
  [ 6] .dynstr           STRTAB          0000564c 00564c 008dde 00   A  0   0  1
  [ 7] .gnu.version      VERSYM          0000e42a 00e42a 000a4e 02   A  5   0  2
  [ 8] .gnu.version_r    VERNEED         0000ee78 00ee78 000420 00   A  6  13  4
  [ 9] .rel.dyn          REL             0000f298 00f298 c67578 08   A  5   0  4
  [10] .rel.plt          REL             00c76810 c76810 001f40 08  AI  5  25  4
  [11] .init             PROGBITS        00c79000 c79000 000020 00  AX  0   0  4
  [12] .plt              PROGBITS        00c79020 c79020 003e90 04  AX  0   0 16
  [13] .text             PROGBITS        00c7ceb0 c7ceb0 7a86908 00  AX  0   0 16

Is this a bug in dwz? Or should I dial the symbols back to -g1 and call it a day?

I checked further and the corrupted section header table in mame-0.200-1.fc29 completely overlaps with dwz crashes: corrupted on aarch64 and x86_64, fine on armv7hl, i686 and s390x. Using -g (as opposed to downgrading to -g1) is what seems to be causing the crash.

I am mostly convinced this is caused by eu-strip, which is ran before dwz.

eu-strip doesn't correctly handle a large number (> 65280) of sections.
It might still be that dwz also has a bug, but it doesn't have a fighting chance if it gets handed a corrupted file by eu-strip.

So I think this is a sub-bug of https://bugzilla.redhat.com/show_bug.cgi?id=1608549#c20 as Adam already pointed out.

Is it worth investigating why there are so many sections in the first place? mame-0.199-1.fc28 has 32 whereas 0.200-1.fc29 (even ones where they are not corrupted) have much more than that. For example, armv7hl has 8644 sections but all except 0-26 and 8642-8644 are .gnu.build.attrib

(In reply to Julian Sikorski from comment #14)
> Is it worth investigating why there are so many sections in the first place?

Yes, sorry. We actually do know. It is annobin. And it isn't technically a bad thing. Except that various tools cannot handle thousands of sections. You can work around it for now by adding %undefine _annotated_build to you spec.

(In reply to Mark Wielaard from comment #15)
> (In reply to Julian Sikorski from comment #14)
> > Is it worth investigating why there are so many sections in the first place?
> 
> Yes, sorry. We actually do know. It is annobin. And it isn't technically a
> bad thing. Except that various tools cannot handle thousands of sections.
> You can work around it for now by adding %undefine _annotated_build to you
> spec.

This shouldn't be necessary.  Newer binutils will produce just one section in the output.

However, we are currently dealing with some binutils breakage related to this change, so the new build will have to wait for a bit.

/usr/bin/mame in mame-0.200-2.fc29.i686.rpm looks like this:

There are 32 section headers, starting at offset 0xfdae320:

Section Headers:
[Nr] Name                 Type         Addr     Off    Size   ES Flags Lk Inf Al
[ 0]                      NULL         00000000 000000 000000  0        0   0  0
[ 1] .interp              PROGBITS     000001b4 0001b4 000013  0 A      0   0  1
[ 2] .note.ABI-tag        NOTE         000001c8 0001c8 000020  0 A      0   0  4
[ 3] .note.gnu.build-id   NOTE         000001e8 0001e8 000024  0 A      0   0  4
[ 4] .gnu.hash            GNU_HASH     0000020c 00020c 0001d0  4 A      5   0  4
[ 5] .dynsym              DYNSYM       000003dc 0003dc 005270 16 A      6   1  4
[ 6] .dynstr              STRTAB       0000564c 00564c 008dde  0 A      0   0  1
[ 7] .gnu.version         GNU_versym   0000e42a 00e42a 000a4e  2 A      5   0  2
[ 8] .gnu.version_r       GNU_verneed  0000ee78 00ee78 000420  0 A      6  13  4
[ 9] .rel.dyn             REL          0000f298 00f298 c67578  8 A      5   0  4
[10] .rel.plt             REL          00c76810 c76810 001f40  8 AI     5  26  4
[11] .init                PROGBITS     00c79000 c79000 000020  0 AX     0   0  4
[12] .plt                 PROGBITS     00c79020 c79020 003e90  4 AX     0   0 16
[13] .text                PROGBITS     00c7ceb0 c7ceb0 7a86908  0 AX     0   0 16
[14] .fini                PROGBITS     087037b8 87037b8 000014  0 AX     0   0  4
[15] .rodata              PROGBITS     08704000 8704000 2073f64  0 A      0   0 32
[16] .qtversion           PROGBITS     0a777f68 a777f68 000008  0 A      0   0  8
[17] .gnu.build.attributes NOTE         0a777f70 de759c4 19f28e4  0        0   0  4
[18] .eh_frame_hdr        PROGBITS     0a777f70 a777f70 354c04  0 A      0   0  4
[19] .eh_frame            PROGBITS     0aaccb74 aaccb74 20ac700  0 A      0   0  4
[20] .gcc_except_table    PROGBITS     0cb79274 cb79274 3183cc  0 A      0   0  4
[21] .tbss                NOBITS       0ce928d4 ce918d4 00000c  0 WAT    0   0  4
[22] .init_array          INIT_ARRAY   0ce928d4 ce918d4 005cc8  4 WA     0   0  4
[23] .fini_array          FINI_ARRAY   0ce9859c ce9759c 000004  4 WA     0   0  4
[24] .data.rel.ro         PROGBITS     0ce985a0 ce975a0 daad44  0 WA     0   0 32
[25] .dynamic             DYNAMIC      0dc432e4 dc422e4 0001d8  8 WA     6   0  4
[26] .got                 PROGBITS     0dc434bc dc424bc 002b40  4 WA     0   0  4
[27] .data                PROGBITS     0dc46000 dc45000 2309c4  0 WA     0   0 32
[28] .bss                 NOBITS       0de769e0 de759c4 71c5a8  0 WA     0   0 32
[29] .gnu_debuglink       PROGBITS     00000000 f8682a8 000024  0        0   0  4
[30] .gnu_debugdata       PROGBITS     00000000 f8682cc 545f10  0        0   0  1
[31] .shstrtab            STRTAB       00000000 fdae1dc 000143  0        0   0  1

So things should be back to normal, relatively speaking.  That 27 MiB notes section is still very worrisome.

(In reply to Florian Weimer from comment #17)
> /usr/bin/mame in mame-0.200-2.fc29.i686.rpm looks like this:
> 
> There are 32 section headers, starting at offset 0xfdae320:
> [...]
> So things should be back to normal, relatively speaking.

That is good. I still will fix eu-strip so we won't trip over this in the future if for some reason we do have to deal with many sections.

>  That 27 MiB notes section is still very worrisome.

Yes. And ELF NOTES will be copied (not stripped) in the .debug file too (because they are expected to be small and useful in both the regular and debug file). So that is actually twice as bad. But that does seem to be a bug for annobin.

On x86_64 dwz still crashes but the section header table seems to survive this:

$ readelf -S mame
There are 31 section headers, starting at offset 0xec0be88:

Section Headers:
  [Nr] Name              Type             Address           Offset
       Size              EntSize          Flags  Link  Info  Align
  [ 0]                   NULL             0000000000000000  00000000
       0000000000000000  0000000000000000           0     0     0
  [ 1] .interp           PROGBITS         00000000000002e0  000002e0
       000000000000001c  0000000000000000   A       0     0     1
  [ 2] .note.ABI-tag     NOTE             00000000000002fc  000002fc
       0000000000000020  0000000000000000   A       0     0     4
  [ 3] .note.gnu.build-i NOTE             000000000000031c  0000031c
       0000000000000024  0000000000000000   A       0     0     4
  [ 4] .gnu.hash         GNU_HASH         0000000000000340  00000340
       00000000000003d8  0000000000000000   A       5     0     8
  [ 5] .dynsym           DYNSYM           0000000000000718  00000718
       0000000000007aa0  0000000000000018   A       6     1     8
  [ 6] .dynstr           STRTAB           00000000000081b8  000081b8
       0000000000008ca1  0000000000000000   A       0     0     1
  [ 7] .gnu.version      VERSYM           0000000000010e5a  00010e5a
       0000000000000a38  0000000000000002   A       5     0     2
  [ 8] .gnu.version_r    VERNEED          0000000000011898  00011898
       0000000000000390  0000000000000000   A       6    13     8
  [ 9] .rela.dyn         RELA             0000000000011c28  00011c28
       000000000252c000  0000000000000018   A       5     0     8
  [10] .rela.plt         RELA             000000000253dc28  0253dc28
       0000000000005d00  0000000000000018  AI       5    26     8
  [11] .init             PROGBITS         0000000002544000  02544000
       0000000000000017  0000000000000000  AX       0     0     4
  [12] .plt              PROGBITS         0000000002544020  02544020
       0000000000003e10  0000000000000010  AX       0     0     16
  [13] .text             PROGBITS         0000000002547e30  02547e30
       0000000007492c84  0000000000000000  AX       0     0     16
  [14] .fini             PROGBITS         00000000099daab4  099daab4
       0000000000000009  0000000000000000  AX       0     0     4
  [15] .rodata           PROGBITS         00000000099db000  099db000
       0000000002134e84  0000000000000000   A       0     0     32
  [16] .qtversion        PROGBITS         000000000bb0fe88  0bb0fe88
       0000000000000010  0000000000000000   A       0     0     8
  [17] .gnu.build.attrib NOTE             000000000bb0fe98  0bb0fe98
       0000000001b5bac0  0000000000000000           0     0     4
  [18] .eh_frame_hdr     PROGBITS         000000000bb0fe98  0bb0fe98
       0000000000354df4  0000000000000000   A       0     0     4
  [19] .eh_frame         PROGBITS         000000000be64c90  0be64c90
       0000000001032c20  0000000000000000   A       0     0     8
  [20] .gcc_except_table PROGBITS         000000000ce978b0  0ce978b0
       0000000000306924  0000000000000000   A       0     0     4
  [21] .tbss             NOBITS           000000000d19f348  0d19e348
       0000000000000014  0000000000000000 WAT       0     0     8
  [22] .init_array       INIT_ARRAY       000000000d19f348  0d19e348
       000000000000b990  0000000000000008  WA       0     0     8
  [23] .fini_array       FINI_ARRAY       000000000d1aacd8  0d1a9cd8
       0000000000000008  0000000000000008  WA       0     0     8
  [24] .data.rel.ro      PROGBITS         000000000d1aace0  0d1a9ce0
       0000000001603e18  0000000000000000  WA       0     0     32
  [25] .dynamic          DYNAMIC          000000000e7aeaf8  0e7adaf8
       00000000000003b0  0000000000000010  WA       6     0     8
  [26] .got              PROGBITS         000000000e7aeea8  0e7adea8
       0000000000002140  0000000000000008  WA       0     0     8
  [27] .data             PROGBITS         000000000e7b1000  0e7b0000
       000000000045bd2c  0000000000000000  WA       0     0     32
  [28] .bss              NOBITS           000000000ec0cd40  0ec0bd2c
       0000000000bc9410  0000000000000000  WA       0     0     32
  [29] .gnu_debuglink    PROGBITS         0000000000000000  0ec0bd2c
       0000000000000024  0000000000000000           0     0     4
  [30] .shstrtab         STRTAB           0000000000000000  0ec0bd50
       0000000000000136  0000000000000000

FYI - I have now made a few small improvements to objcopy's --merge-notes option.  With this improved version I was able to shrink the size of the .gnu.build.attributes section in the mame executable down from 0x1b2bb08 to 0x751e38.  This is about a 60% reduction, which is good, but there is still room for improvement.

This bug appears to have been reported against 'rawhide' during the Fedora 29 development cycle.
Changing version to '29'.

elfutils-0.174-1.fc29 has been submitted as an update to Fedora 29. https://bodhi.fedoraproject.org/updates/FEDORA-2018-32c8599fe1

elfutils-0.174-1.fc29 has been pushed to the Fedora 29 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2018-32c8599fe1

elfutils-0.174-1.fc28 has been submitted as an update to Fedora 28. https://bodhi.fedoraproject.org/updates/FEDORA-2018-1eec1f0d17

elfutils-0.174-1.fc28 has been pushed to the Fedora 28 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2018-1eec1f0d17

elfutils-0.174-1.fc29 has been pushed to the Fedora 29 stable repository. If problems still persist, please make note of it in this bug report.

elfutils-0.174-1.fc28 has been pushed to the Fedora 28 stable repository. If problems still persist, please make note of it in this bug report.