Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.
Bug 1164221
Summary: | test fails on ppc64 | ||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|
Product: | [Fedora] Fedora | Reporter: | Dan Horák <dan> | ||||||||
Component: | glibc | Assignee: | Carlos O'Donell <codonell> | ||||||||
Status: | CLOSED EOL | QA Contact: | Fedora Extras Quality Assurance <extras-qa> | ||||||||
Severity: | unspecified | Docs Contact: | |||||||||
Priority: | unspecified | ||||||||||
Version: | 21 | CC: | arjun.is, ashankar, bkabrda, codonell, fweimer, jakub, jcajka, jstribny, law, mmorsi, mnewsome, mtasaka, pfrankli, s, tagoh, vanmeeuwen+fedora, vondruch | ||||||||
Target Milestone: | --- | ||||||||||
Target Release: | --- | ||||||||||
Hardware: | ppc64 | ||||||||||
OS: | Unspecified | ||||||||||
Whiteboard: | |||||||||||
Fixed In Version: | Doc Type: | Bug Fix | |||||||||
Doc Text: | Story Points: | --- | |||||||||
Clone Of: | Environment: | ||||||||||
Last Closed: | 2015-12-02 04:58:41 UTC | Type: | Bug | ||||||||
Regression: | --- | Mount Type: | --- | ||||||||
Documentation: | --- | CRM: | |||||||||
Verified Versions: | Category: | --- | |||||||||
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||||
Cloudforms Team: | --- | Target Upstream Version: | |||||||||
Embargoed: | |||||||||||
Bug Depends On: | |||||||||||
Bug Blocks: | 1071880 | ||||||||||
Attachments: |
|
Description
Dan Horák
2014-11-14 11:33:59 UTC
There used to by such error: https://bugs.ruby-lang.org/issues/7312 Is there chance that glibc detection fails? Otherwise this might be the offending commit: https://github.com/ruby/ruby/commit/def5eab9e10120de90ec2d2dac1828b1cffb31c1 Actually glibc detection fails, but in bit unexpected way at `#{glibcpath}` in up mentioned test case. It returns empty string, although path to libc(/lib64/libc.so.6) is correct(even passed explicitly`/lib64...`). Run outside ruby returns expected result. Also running 'ordinary' executables(`/bin/ls`) in test case yields expected results. Replacing version detection with expected version('2.20'.split(... ) in test makes it finish successfully. I have tried to scratch build ruby-2.1.4 for f20, surprisingly this test case passes(build fails elsewhere on ppc(64)). http://ppc.koji.fedoraproject.org/koji/taskinfo?taskID=2190978 I have noted glibc version difference, 2.18 x 2.20 (contains ppc related changes(as ^^ is affecting only ppc64...)). Oh my, I was not aware that they again reverted [2] my patches proposed in [1] (In reply to Jakub Čajka from comment #2) > I have tried to scratch build ruby-2.1.4 for f20, surprisingly this test > case passes(build fails elsewhere on ppc(64)). > > http://ppc.koji.fedoraproject.org/koji/taskinfo?taskID=2190978 > > I have noted glibc version difference, 2.18 x 2.20 (contains ppc related > changes(as ^^ is affecting only ppc64...)). Do I read it correctly, that this test is passing with current glibc? [1] https://bugs.ruby-lang.org/issues/7312 [2] https://bugs.ruby-lang.org/issues/7828 (In reply to Vít Ondruch from comment #3) > Do I read it correctly, that this test is passing with current glibc? Nope, test is passing wit old one(f20 2.18 pass,f21 2.20 fail). There is more differences(gcc,...), this is one that stood out (bug's related to glibc detection..., and it's failing only on ppc64...). Bit unrelated: build of ruby failed recently on ppc64le with(passed successfully last time, see OP): 5) Failure: OpenSSL::TestPKCS7#test_signed [/builddir/build/BUILD/ruby-2.1.4/test/openssl/test_pkcs7.rb:47]: Failed assertion, no message given. http://ppc.koji.fedoraproject.org/koji/taskinfo?taskID=2189041 No change in buildroot, it seems as timing issue. Created attachment 966856 [details]
Back traces
To start again, my previous comments are bit confusing... Sorry for that...
Actual problem seems to be caused by invocation of `/lib64/libc.so.6` in test/ruby/test_m17n_comp.rb:736, which leads to segmentation fault (witch is not reported by ruby), and consequent crypt failure(empty string as salt :749) with EINVAL.
It seems that, build fails only on ppc64 with glibc-2.19 and newer(tried 2.19.90-36, 2.20.90-10, and f21 2.20-5), although I was unsuccessful reproducing it outside ruby build tests. Using older glibc-2.18.90-27, or building for f20(glibc-2.18) results in successful build.
(2.20.90-10 built with disabled valgrind tests)
Running other "normal" executables for example `/bin/ls`, doesn't result in seg fault.
Please see attachment for back traces from gdb.
Could someone from glibc please check this bug if it's glibc bug? Thanks! (In reply to Jakub Čajka from comment #6) > Could someone from glibc please check this bug if it's glibc bug? > Thanks! We need a cut down reproducer please. Please provide as small a test case as possible that shows the problem and that can be run outside of the ruby testsuite. Single test case can be simple run such as: make test-all TESTS="-v -n test_str_crypt test/ruby/test_m17n_comb.rb" Hello, thank you for checking this bug. I have looked in to it more and I finally managed to find reproducer outside ruby. Steps to reproduce: assuming ppc64 machine with Fedora 21 and glibc-2.19.90-19>(last successful build of ruby used 2.19.90-19, also it doesn't crash in f20 with glibc-2.18) packages used: glibc-2.20-5.fc21.ppc64p7 ruby-libs-2.1.2-23.fc21.ppc64 steps to reproduce: <mock-chroot>[root@power /]# LD_PRELOAD=/lib64/libruby.so.2.1.0 <mock-chroot>[root@power /]# export LD_PRELOAD <mock-chroot>[root@power /]# /lib64/libc.so.6 Segmentation fault (core dumped) without LD_PRELOAD : <mock-chroot>[root@power /]# /lib64/libc.so.6 GNU C Library (GNU libc) stable release version 2.20, by Roland McGrath et al. Copyright (C) 2014 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. Compiled by GNU CC version 4.9.1 20140912 (Red Hat 4.9.1-9). Available extensions: The C stubs add-on version 2.1.2. crypt add-on version 2.1 by Michael Glad and others GNU Libidn by Simon Josefsson Native POSIX Threads Library by Ulrich Drepper et al BIND-8.2.3-T5B RT using linux kernel aio libc ABIs: UNIQUE IFUNC For bug reporting instructions, please see: <http://www.gnu.org/software/libc/bugs.html>. Segfault seems to be triggered by LD_PRELOAD-ing library (in case of ruby build it is libruby). It can be reproduce by preloading some libraries (libruby,) libssl, libssh..., but not others libselinux, libsepol, libc, libbz...(all chosen randomly). (In reply to Jakub Čajka from comment #9) > <mock-chroot>[root@power /]# LD_PRELOAD=/lib64/libruby.so.2.1.0 > <mock-chroot>[root@power /]# export LD_PRELOAD > <mock-chroot>[root@power /]# /lib64/libc.so.6 > Segmentation fault (core dumped) Can you run this through gdb and valgrind and get a stack trace at the point of the failure please? Created attachment 979709 [details]
valgrind
(gdb) run
Starting program: /usr/lib64/libc.so.6
Program received signal SIGSEGV, Segmentation fault.
allocate_dtv (result=0x3fffffffe810) at dl-tls.c:327
327 dtv[0].counter = dtv_length;
(gdb) bt
#0 allocate_dtv (result=0x3fffffffe810) at dl-tls.c:327
#1 _dl_allocate_tls_storage () at dl-tls.c:391
#2 0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)
(gdb)
(In reply to Jakub Čajka from comment #11) > Created attachment 979709 [details] > valgrind > > (gdb) run > Starting program: /usr/lib64/libc.so.6 > > Program received signal SIGSEGV, Segmentation fault. > allocate_dtv (result=0x3fffffffe810) at dl-tls.c:327 > 327 dtv[0].counter = dtv_length; > (gdb) bt > #0 allocate_dtv (result=0x3fffffffe810) at dl-tls.c:327 > #1 _dl_allocate_tls_storage () at dl-tls.c:391 > #2 0x0000000000000000 in ?? () > Backtrace stopped: previous frame inner to this frame (corrupt stack?) > (gdb) Excellent work. That's exactly the kind of backtrace I need to pinpoint a possible solution. This looks like one of 2 upstream problems: Bug 13862 - Reuse of cached stack can cause bounds overrun of thread DTV https://sourceware.org/bugzilla/show_bug.cgi?id=13862 Bug 17621 - DTV update for Static TLS dlopened modules is racy https://sourceware.org/bugzilla/show_bug.cgi?id=17621 I think you are seeing bug 13862. Can you try upstream commit d8dd00805b8f3a011735d7a407097fb1c408d867 and see if it fixes the issue for you? One is fixed, the other Alex is fixing (or has a patch already). Alex won't be back for several weeks, so this is going ot have to wait (unless it's bug 13862 and you verify it fixes your issue). Using latest glibc(glibc-2.20.90-18) from fedora22(should include fix for Bug 13862, seems to help a bit). Suggested upstream commit doesn't seems to make a difference. Without upstream commit: (gdb) run Starting program: /usr/lib64/libc.so.6 Program received signal SIGSEGV, Segmentation fault. 0x00003fffffffec10 in ?? () (gdb) bt #0 0x00003fffffffec10 in ?? () #1 0x00003fffb7fb8e4c in resolve_ifunc (sym_map=0x3fffb7ff1f08, map=0x3fffb7ff3de8, value=<optimized out>) at ../sysdeps/powerpc/powerpc64/dl-machine.h:630 #2 elf_machine_rela (skip_ifunc=<optimized out>, reloc_addr_arg=0x3fffb7e5d900, version=<optimized out>, sym=<optimized out>, reloc=0x3fffb7c6d4c0, map=0x3fffb7ff3de8) at ../sysdeps/powerpc/powerpc64/dl-machine.h:672 #3 elf_dynamic_do_Rela (skip_ifunc=<optimized out>, lazy=<optimized out>, nrelative=<optimized out>, relsize=<optimized out>, reladdr=<optimized out>, map=0x3fffb7ff3de8) at do-rel.h:137 #4 _dl_relocate_object (scope=0x3fffb7ff4160, reloc_mode=<optimized out>, consider_profiling=<optimized out>) at dl-reloc.c:264 #5 0x00003fffb7fa6e68 in dl_main (phdr=<optimized out>, phnum=<optimized out>, user_entry=<optimized out>, auxv=<optimized out>) at rtld.c:2070 #6 0x00003fffb7fc9dd4 in _dl_sysdep_start (start_argptr=<optimized out>, dl_main=@0x3fffb7ff0120: 0x3fffb7fa47f0 <dl_main>) at ../elf/dl-sysdep.c:249 #7 0x00003fffb7fa4038 in _dl_start_final (arg=arg@entry=0x3ffffffff640, info=info@entry=0x3ffffffff0c0) at rtld.c:306 #8 0x00003fffb7fa84b0 in _dl_start (arg=0x3ffffffff640) at rtld.c:414 #9 0x00003fffb7fa37f0 in ._start () from /lib64/ld64.so.1 with upstream commit: (gdb) run Starting program: /usr/lib64/libc.so.6 Program received signal SIGSEGV, Segmentation fault. 0x00003fffffffec10 in ?? () (gdb) bt #0 0x00003fffffffec10 in ?? () #1 0x00003fffb7fb8e4c in resolve_ifunc (sym_map=0x3fffb7ff1f08, map=0x3fffb7ff3de8, value=<optimized out>) at ../sysdeps/powerpc/powerpc64/dl-machine.h:630 #2 elf_machine_rela (skip_ifunc=<optimized out>, reloc_addr_arg=0x3fffb7e5d900, version=<optimized out>, sym=<optimized out>, reloc=0x3fffb7c6d4c0, map=0x3fffb7ff3de8) at ../sysdeps/powerpc/powerpc64/dl-machine.h:672 #3 elf_dynamic_do_Rela (skip_ifunc=<optimized out>, lazy=<optimized out>, nrelative=<optimized out>, relsize=<optimized out>, reladdr=<optimized out>, map=0x3fffb7ff3de8) at do-rel.h:137 #4 _dl_relocate_object (scope=0x3fffb7ff4160, reloc_mode=<optimized out>, consider_profiling=<optimized out>) at dl-reloc.c:264 #5 0x00003fffb7fa6e68 in dl_main (phdr=<optimized out>, phnum=<optimized out>, user_entry=<optimized out>, auxv=<optimized out>) at rtld.c:2070 #6 0x00003fffb7fc9dd4 in _dl_sysdep_start (start_argptr=<optimized out>, dl_main=@0x3fffb7ff0120: 0x3fffb7fa47f0 <dl_main>) at ../elf/dl-sysdep.c:249 #7 0x00003fffb7fa4038 in _dl_start_final (arg=arg@entry=0x3ffffffff640, info=info@entry=0x3ffffffff0c0) at rtld.c:306 #8 0x00003fffb7fa84b0 in _dl_start (arg=0x3ffffffff640) at rtld.c:414 #9 0x00003fffb7fa37f0 in ._start () from /lib64/ld64.so.1 Created attachment 980049 [details]
valgring output
glibc-2.20.90-18 with suggested commit
(In reply to Jakub Čajka from comment #13) > Program received signal SIGSEGV, Segmentation fault. > 0x00003fffffffec10 in ?? () > (gdb) bt > #0 0x00003fffffffec10 in ?? () > #1 0x00003fffb7fb8e4c in resolve_ifunc (sym_map=0x3fffb7ff1f08, > map=0x3fffb7ff3de8, value=<optimized out>) at > ../sysdeps/powerpc/powerpc64/dl-machine.h:630 > #2 elf_machine_rela (skip_ifunc=<optimized out>, > reloc_addr_arg=0x3fffb7e5d900, version=<optimized out>, sym=<optimized out>, > reloc=0x3fffb7c6d4c0, map=0x3fffb7ff3de8) > at ../sysdeps/powerpc/powerpc64/dl-machine.h:672 > #3 elf_dynamic_do_Rela (skip_ifunc=<optimized out>, lazy=<optimized out>, > nrelative=<optimized out>, relsize=<optimized out>, reladdr=<optimized out>, > map=0x3fffb7ff3de8) at do-rel.h:137 > #4 _dl_relocate_object (scope=0x3fffb7ff4160, reloc_mode=<optimized out>, > consider_profiling=<optimized out>) at dl-reloc.c:264 > #5 0x00003fffb7fa6e68 in dl_main (phdr=<optimized out>, phnum=<optimized > out>, user_entry=<optimized out>, auxv=<optimized out>) at rtld.c:2070 > #6 0x00003fffb7fc9dd4 in _dl_sysdep_start (start_argptr=<optimized out>, > dl_main=@0x3fffb7ff0120: 0x3fffb7fa47f0 <dl_main>) at ../elf/dl-sysdep.c:249 > #7 0x00003fffb7fa4038 in _dl_start_final (arg=arg@entry=0x3ffffffff640, > info=info@entry=0x3ffffffff0c0) at rtld.c:306 > #8 0x00003fffb7fa84b0 in _dl_start (arg=0x3ffffffff640) at rtld.c:414 > #9 0x00003fffb7fa37f0 in ._start () from /lib64/ld64.so.1 This is an GNU indirect function resolution failure, either a bug in glibc or a corrupt library. This is very different from the original backtrace you present. Exactly what hardware are you using on ppc64? Is it stable? Well cooled? I originally reproduced it on power7+ KVM guest and just tested it on power8 bare metal with same results. HW should be OK. Just checked on them. Hope this answers your question. Please note that this still fails the same way with different preloaded libraries. Fails with libssl, libssh, libruby prelaoded(chose one) but not with libbz for example. This message is a reminder that Fedora 21 is nearing its end of life. Approximately 4 (four) weeks from now Fedora will stop maintaining and issuing updates for Fedora 21. It is Fedora's policy to close all bug reports from releases that are no longer maintained. At that time this bug will be closed as EOL if it remains open with a Fedora 'version' of '21'. Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, simply change the 'version' to a later Fedora version. Thank you for reporting this issue and we are sorry that we were not able to fix it before Fedora 21 is end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora, you are encouraged change the 'version' to a later Fedora version prior this bug is closed as described in the policy above. Although we aim to fix as many bugs as possible during every release's lifetime, sometimes those efforts are overtaken by events. Often a more recent Fedora release includes newer upstream software that fixes bugs or makes them obsolete. Fedora 21 changed to end-of-life (EOL) status on 2015-12-01. Fedora 21 is no longer maintained, which means that it will not receive any further security or bug fix updates. As a result we are closing this bug. If you can reproduce this bug against a currently maintained version of Fedora please feel free to reopen this bug against that version. If you are unable to reopen this bug, please file a new report against the current release. If you experience problems, please add a comment to this bug. Thank you for reporting this bug and we are sorry it could not be fixed. |