Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.
Bug 1608390
Summary: | eu-strip might corrupted section header table | ||
---|---|---|---|
Product: | [Fedora] Fedora | Reporter: | Florian Weimer <fweimer> |
Component: | elfutils | Assignee: | Mark Wielaard <mjw> |
Status: | CLOSED ERRATA | QA Contact: | Fedora Extras Quality Assurance <extras-qa> |
Severity: | unspecified | Docs Contact: | |
Priority: | unspecified | ||
Version: | 29 | CC: | aoliva, belegdol, fche, jakub, me, mjw, nickc |
Target Milestone: | --- | ||
Target Release: | --- | ||
Hardware: | Unspecified | ||
OS: | Unspecified | ||
Whiteboard: | |||
Fixed In Version: | elfutils-0.174-1.fc29 elfutils-0.174-1.fc28 | Doc Type: | If docs needed, set a value |
Doc Text: | Story Points: | --- | |
Clone Of: | Environment: | ||
Last Closed: | 2018-09-30 23:24:55 UTC | Type: | Bug |
Regression: | --- | Mount Type: | --- |
Documentation: | --- | CRM: | |
Verified Versions: | Category: | --- | |
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
Cloudforms Team: | --- | Target Upstream Version: | |
Embargoed: |
Description
Florian Weimer
2018-07-25 12:33:46 UTC
Thanks for reporting! Do you have any pointers what might be causing this and where to start looking? At the first sight it looks like something way above my pay grade :o With mame-0.199-2.fc29.x86_64, I do not see the problem: $ readelf -S /usr/bin/mame There are 32 section headers, starting at offset 0xe74d210: Section Headers: [Nr] Name Type Address Offset Size EntSize Flags Link Info Align [ 0] NULL 0000000000000000 00000000 0000000000000000 0000000000000000 0 0 0 [ 1] .interp PROGBITS 0000000000000350 00000350 000000000000001c 0000000000000000 A 0 0 1 Could you try 0.199-4? It was built only a few days back so if something is off with the build system it could be useful information. Thanks! (In reply to Julian Sikorski from comment #3) > Could you try 0.199-4? It was built only a few days back so if something is > off with the build system it could be useful information. Thanks! mame-0.199-4.fc29.x86_64 is already corrupted. This points to builds problems as changes between -2 and -4 are minimal. -3 is just a rebuild of -2 if you would like to check. I somehow suspect annobin as it has been causing problems with sections already. Nick, would you mind having a look? Thanks! mame-0.199-3.fc29.x86_64 is corrupted as well, so this would point to a toolchain issue. Mark noted some dwz crashes in the build log: extracting debug info from /builddir/build/BUILDROOT/mame-0.199-1.fc27.x86_64/usr/bin/chdman extracting debug info from /builddir/build/BUILDROOT/mame-0.199-1.fc27.x86_64/usr/bin/floptool munmap_chunk(): invalid pointer /usr/lib/rpm/find-debuginfo.sh: line 231: 6729 Aborted (core dumped) eu-strip --remove-comment $r $g ${keep_remove_args} -f "$1" "$2" dwz: dwz.c:9899: read_dwarf: Assertion `data != NULL && data->d_buf != NULL' failed. /usr/lib/rpm/find-debuginfo.sh: line 522: 6748 Aborted (core dumped) dwz $dwz_opts ${dwz_files[@]} /usr/lib/rpm/sepdebugcrcfix: Updated 7 CRC32s, 9 CRC32s did match. This is from <https://kojipkgs.fedoraproject.org//packages/mame/0.199/1.fc27/data/logs/x86_64/build.log>, so the issue goes *way* back. It could be that the more recent corruption is indirectly related to that. I checked the previous build logs for dwz crashes and found the following: while 0.199 and higher have those across all Fedora releases and at least on x86_64 and aarch64, 0.198 and 0.197 were not affected by the dwz crash - at least on rawhide. I also had a closer look at 0.200-1.fc29 and it turns out that dwz only crashes on aarch64 and x86_64, but not on armv7hl, s390x or i686. Incidentally this matches the architectures which have the default symbols level left at -g2: %ifarch %{arm} %{ix86} riscv64 s390x RPM_OPT_FLAGS=$(echo $RPM_OPT_FLAGS | sed "s@-g@-g1@") Moreover, prior to 0.199 -g1 was being used across the board. mame is a big project, looks like compiling it with -g stretches dwz beyond its capabilities. Florian, do you have the ability to check the binary using readelf on any of the architectures on which dwz does not crash? I do not have any VMs set up at the moment unfortunately. (In reply to Julian Sikorski from comment #9) > Florian, do you have the ability to check the binary using readelf on any of > the architectures on which dwz does not crash? I do not have any VMs set up > at the moment unfortunately. You can use eu-readelf from elfutils for cross-architecture checking (or use cross-binutils). eu-readelf -S shows basically the same corrupted output. Looks like it is really arch-specific: $ readelf -S mame-i686 There are 8648 section headers, starting at offset 0xfe2fb1c: Section Headers: [Nr] Name Type Addr Off Size ES Flg Lk Inf Al [ 0] NULL 00000000 000000 000000 00 0 0 0 [ 1] .interp PROGBITS 000001b4 0001b4 000013 00 A 0 0 1 [ 2] .note.ABI-tag NOTE 000001c8 0001c8 000020 00 A 0 0 4 [ 3] .note.gnu.build-i NOTE 000001e8 0001e8 000024 00 A 0 0 4 [ 4] .gnu.hash GNU_HASH 0000020c 00020c 0001d0 04 A 5 0 4 [ 5] .dynsym DYNSYM 000003dc 0003dc 005270 10 A 6 1 4 [ 6] .dynstr STRTAB 0000564c 00564c 008dde 00 A 0 0 1 [ 7] .gnu.version VERSYM 0000e42a 00e42a 000a4e 02 A 5 0 2 [ 8] .gnu.version_r VERNEED 0000ee78 00ee78 000420 00 A 6 13 4 [ 9] .rel.dyn REL 0000f298 00f298 c67578 08 A 5 0 4 [10] .rel.plt REL 00c76810 c76810 001f40 08 AI 5 25 4 [11] .init PROGBITS 00c79000 c79000 000020 00 AX 0 0 4 [12] .plt PROGBITS 00c79020 c79020 003e90 04 AX 0 0 16 [13] .text PROGBITS 00c7ceb0 c7ceb0 7a86908 00 AX 0 0 16 Is this a bug in dwz? Or should I dial the symbols back to -g1 and call it a day? I checked further and the corrupted section header table in mame-0.200-1.fc29 completely overlaps with dwz crashes: corrupted on aarch64 and x86_64, fine on armv7hl, i686 and s390x. Using -g (as opposed to downgrading to -g1) is what seems to be causing the crash. I am mostly convinced this is caused by eu-strip, which is ran before dwz. eu-strip doesn't correctly handle a large number (> 65280) of sections. It might still be that dwz also has a bug, but it doesn't have a fighting chance if it gets handed a corrupted file by eu-strip. So I think this is a sub-bug of https://bugzilla.redhat.com/show_bug.cgi?id=1608549#c20 as Adam already pointed out. Is it worth investigating why there are so many sections in the first place? mame-0.199-1.fc28 has 32 whereas 0.200-1.fc29 (even ones where they are not corrupted) have much more than that. For example, armv7hl has 8644 sections but all except 0-26 and 8642-8644 are .gnu.build.attrib (In reply to Julian Sikorski from comment #14) > Is it worth investigating why there are so many sections in the first place? Yes, sorry. We actually do know. It is annobin. And it isn't technically a bad thing. Except that various tools cannot handle thousands of sections. You can work around it for now by adding %undefine _annotated_build to you spec. (In reply to Mark Wielaard from comment #15) > (In reply to Julian Sikorski from comment #14) > > Is it worth investigating why there are so many sections in the first place? > > Yes, sorry. We actually do know. It is annobin. And it isn't technically a > bad thing. Except that various tools cannot handle thousands of sections. > You can work around it for now by adding %undefine _annotated_build to you > spec. This shouldn't be necessary. Newer binutils will produce just one section in the output. However, we are currently dealing with some binutils breakage related to this change, so the new build will have to wait for a bit. /usr/bin/mame in mame-0.200-2.fc29.i686.rpm looks like this: There are 32 section headers, starting at offset 0xfdae320: Section Headers: [Nr] Name Type Addr Off Size ES Flags Lk Inf Al [ 0] NULL 00000000 000000 000000 0 0 0 0 [ 1] .interp PROGBITS 000001b4 0001b4 000013 0 A 0 0 1 [ 2] .note.ABI-tag NOTE 000001c8 0001c8 000020 0 A 0 0 4 [ 3] .note.gnu.build-id NOTE 000001e8 0001e8 000024 0 A 0 0 4 [ 4] .gnu.hash GNU_HASH 0000020c 00020c 0001d0 4 A 5 0 4 [ 5] .dynsym DYNSYM 000003dc 0003dc 005270 16 A 6 1 4 [ 6] .dynstr STRTAB 0000564c 00564c 008dde 0 A 0 0 1 [ 7] .gnu.version GNU_versym 0000e42a 00e42a 000a4e 2 A 5 0 2 [ 8] .gnu.version_r GNU_verneed 0000ee78 00ee78 000420 0 A 6 13 4 [ 9] .rel.dyn REL 0000f298 00f298 c67578 8 A 5 0 4 [10] .rel.plt REL 00c76810 c76810 001f40 8 AI 5 26 4 [11] .init PROGBITS 00c79000 c79000 000020 0 AX 0 0 4 [12] .plt PROGBITS 00c79020 c79020 003e90 4 AX 0 0 16 [13] .text PROGBITS 00c7ceb0 c7ceb0 7a86908 0 AX 0 0 16 [14] .fini PROGBITS 087037b8 87037b8 000014 0 AX 0 0 4 [15] .rodata PROGBITS 08704000 8704000 2073f64 0 A 0 0 32 [16] .qtversion PROGBITS 0a777f68 a777f68 000008 0 A 0 0 8 [17] .gnu.build.attributes NOTE 0a777f70 de759c4 19f28e4 0 0 0 4 [18] .eh_frame_hdr PROGBITS 0a777f70 a777f70 354c04 0 A 0 0 4 [19] .eh_frame PROGBITS 0aaccb74 aaccb74 20ac700 0 A 0 0 4 [20] .gcc_except_table PROGBITS 0cb79274 cb79274 3183cc 0 A 0 0 4 [21] .tbss NOBITS 0ce928d4 ce918d4 00000c 0 WAT 0 0 4 [22] .init_array INIT_ARRAY 0ce928d4 ce918d4 005cc8 4 WA 0 0 4 [23] .fini_array FINI_ARRAY 0ce9859c ce9759c 000004 4 WA 0 0 4 [24] .data.rel.ro PROGBITS 0ce985a0 ce975a0 daad44 0 WA 0 0 32 [25] .dynamic DYNAMIC 0dc432e4 dc422e4 0001d8 8 WA 6 0 4 [26] .got PROGBITS 0dc434bc dc424bc 002b40 4 WA 0 0 4 [27] .data PROGBITS 0dc46000 dc45000 2309c4 0 WA 0 0 32 [28] .bss NOBITS 0de769e0 de759c4 71c5a8 0 WA 0 0 32 [29] .gnu_debuglink PROGBITS 00000000 f8682a8 000024 0 0 0 4 [30] .gnu_debugdata PROGBITS 00000000 f8682cc 545f10 0 0 0 1 [31] .shstrtab STRTAB 00000000 fdae1dc 000143 0 0 0 1 So things should be back to normal, relatively speaking. That 27 MiB notes section is still very worrisome. (In reply to Florian Weimer from comment #17) > /usr/bin/mame in mame-0.200-2.fc29.i686.rpm looks like this: > > There are 32 section headers, starting at offset 0xfdae320: > [...] > So things should be back to normal, relatively speaking. That is good. I still will fix eu-strip so we won't trip over this in the future if for some reason we do have to deal with many sections. > That 27 MiB notes section is still very worrisome. Yes. And ELF NOTES will be copied (not stripped) in the .debug file too (because they are expected to be small and useful in both the regular and debug file). So that is actually twice as bad. But that does seem to be a bug for annobin. On x86_64 dwz still crashes but the section header table seems to survive this: $ readelf -S mame There are 31 section headers, starting at offset 0xec0be88: Section Headers: [Nr] Name Type Address Offset Size EntSize Flags Link Info Align [ 0] NULL 0000000000000000 00000000 0000000000000000 0000000000000000 0 0 0 [ 1] .interp PROGBITS 00000000000002e0 000002e0 000000000000001c 0000000000000000 A 0 0 1 [ 2] .note.ABI-tag NOTE 00000000000002fc 000002fc 0000000000000020 0000000000000000 A 0 0 4 [ 3] .note.gnu.build-i NOTE 000000000000031c 0000031c 0000000000000024 0000000000000000 A 0 0 4 [ 4] .gnu.hash GNU_HASH 0000000000000340 00000340 00000000000003d8 0000000000000000 A 5 0 8 [ 5] .dynsym DYNSYM 0000000000000718 00000718 0000000000007aa0 0000000000000018 A 6 1 8 [ 6] .dynstr STRTAB 00000000000081b8 000081b8 0000000000008ca1 0000000000000000 A 0 0 1 [ 7] .gnu.version VERSYM 0000000000010e5a 00010e5a 0000000000000a38 0000000000000002 A 5 0 2 [ 8] .gnu.version_r VERNEED 0000000000011898 00011898 0000000000000390 0000000000000000 A 6 13 8 [ 9] .rela.dyn RELA 0000000000011c28 00011c28 000000000252c000 0000000000000018 A 5 0 8 [10] .rela.plt RELA 000000000253dc28 0253dc28 0000000000005d00 0000000000000018 AI 5 26 8 [11] .init PROGBITS 0000000002544000 02544000 0000000000000017 0000000000000000 AX 0 0 4 [12] .plt PROGBITS 0000000002544020 02544020 0000000000003e10 0000000000000010 AX 0 0 16 [13] .text PROGBITS 0000000002547e30 02547e30 0000000007492c84 0000000000000000 AX 0 0 16 [14] .fini PROGBITS 00000000099daab4 099daab4 0000000000000009 0000000000000000 AX 0 0 4 [15] .rodata PROGBITS 00000000099db000 099db000 0000000002134e84 0000000000000000 A 0 0 32 [16] .qtversion PROGBITS 000000000bb0fe88 0bb0fe88 0000000000000010 0000000000000000 A 0 0 8 [17] .gnu.build.attrib NOTE 000000000bb0fe98 0bb0fe98 0000000001b5bac0 0000000000000000 0 0 4 [18] .eh_frame_hdr PROGBITS 000000000bb0fe98 0bb0fe98 0000000000354df4 0000000000000000 A 0 0 4 [19] .eh_frame PROGBITS 000000000be64c90 0be64c90 0000000001032c20 0000000000000000 A 0 0 8 [20] .gcc_except_table PROGBITS 000000000ce978b0 0ce978b0 0000000000306924 0000000000000000 A 0 0 4 [21] .tbss NOBITS 000000000d19f348 0d19e348 0000000000000014 0000000000000000 WAT 0 0 8 [22] .init_array INIT_ARRAY 000000000d19f348 0d19e348 000000000000b990 0000000000000008 WA 0 0 8 [23] .fini_array FINI_ARRAY 000000000d1aacd8 0d1a9cd8 0000000000000008 0000000000000008 WA 0 0 8 [24] .data.rel.ro PROGBITS 000000000d1aace0 0d1a9ce0 0000000001603e18 0000000000000000 WA 0 0 32 [25] .dynamic DYNAMIC 000000000e7aeaf8 0e7adaf8 00000000000003b0 0000000000000010 WA 6 0 8 [26] .got PROGBITS 000000000e7aeea8 0e7adea8 0000000000002140 0000000000000008 WA 0 0 8 [27] .data PROGBITS 000000000e7b1000 0e7b0000 000000000045bd2c 0000000000000000 WA 0 0 32 [28] .bss NOBITS 000000000ec0cd40 0ec0bd2c 0000000000bc9410 0000000000000000 WA 0 0 32 [29] .gnu_debuglink PROGBITS 0000000000000000 0ec0bd2c 0000000000000024 0000000000000000 0 0 4 [30] .shstrtab STRTAB 0000000000000000 0ec0bd50 0000000000000136 0000000000000000 FYI - I have now made a few small improvements to objcopy's --merge-notes option. With this improved version I was able to shrink the size of the .gnu.build.attributes section in the mame executable down from 0x1b2bb08 to 0x751e38. This is about a 60% reduction, which is good, but there is still room for improvement. This bug appears to have been reported against 'rawhide' during the Fedora 29 development cycle. Changing version to '29'. elfutils-0.174-1.fc29 has been submitted as an update to Fedora 29. https://bodhi.fedoraproject.org/updates/FEDORA-2018-32c8599fe1 elfutils-0.174-1.fc29 has been pushed to the Fedora 29 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2018-32c8599fe1 elfutils-0.174-1.fc28 has been submitted as an update to Fedora 28. https://bodhi.fedoraproject.org/updates/FEDORA-2018-1eec1f0d17 elfutils-0.174-1.fc28 has been pushed to the Fedora 28 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2018-1eec1f0d17 elfutils-0.174-1.fc29 has been pushed to the Fedora 29 stable repository. If problems still persist, please make note of it in this bug report. elfutils-0.174-1.fc28 has been pushed to the Fedora 28 stable repository. If problems still persist, please make note of it in this bug report. |