Just going to try with elfutils 162 as I noticed we're behind on aarch64

Same issue with elfutils-0.162-2.fc23

http://arm.koji.fedoraproject.org/koji/taskinfo?taskID=3037374

Adding Mark too

I don't believe this is an elfutils issue. elfutils isn't used during the build/link, only afterwards to post-process the binaries/debug-info. Given the symbol this refers to "__stack_chk_guard" which is used for -fstack-protector it might also be some interplay between gcc and glibc. If I remember right there was some change how to refer to that symbol on aarch64. https://sourceware.org/ml/libc-alpha/2013-12/msg00248.html But that might be too old and might not have anything to do with the issue.

Hi Peter,

  I tried to reproduce this problem on one of the mustang boards (running FC21) but qemu will not even start to build because of unsolvable missing dependencies:

        glusterfs-api-devel >= 3.4.0 is needed by qemu-2:2.3.0-8.fc23.aarch64
	glusterfs-devel >= 3.4.0 is needed by qemu-2:2.3.0-8.fc23.aarch64
	gtk3-devel is needed by qemu-2:2.3.0-8.fc23.aarch64
	vte3-devel is needed by qemu-2:2.3.0-8.fc23.aarch64

Would it be possible for you to create a tarball of your build tree that I can use to reproduce the program locally ?

Cheers
  Nick

> Would it be possible for you to create a tarball of your build tree that I
> can use to reproduce the program locally ?

Well it's building with all the F-23 deps so I suspect you'll need a full chroot. You should be able to do:

fedpkg clone qemu
cd qemu
fedpkg mockbuild

And that should built it exactly like in koji

Nick: did that work or do you need some more help?`

Hi Peter,

  Yes and no.  I had to add a "-a" parameter to the "fedpkg clone qemu" command, but that was easy.  Then I ran the mockbuild command, and sure enough I saw the problem appear.  Now all I have to do is work out how to get inside the mock environment so that I can add some debugging to the linker and find out exactly what is going wrong.  Do you have any idea how I can do that ?  (IE get into the mock environment so that I can start playing)

Cheers
  Nick

https://fedoraproject.org/wiki/Using_Mock_to_test_package_builds

This gives a good overview. You might be better off doing some of the mock bits manually so it doesn't get cleaned up and then you can get into it with --shell

Nick: let me know if you've got any other issues/queries

Any status update?

Hi Peter,

  I have been trying, struggling really, to find an environment where I can reproduce the bug and then debug it.  I tried following the directions on the link you provided, but the process always fails at the step where I try to start the mock shell:

  [...]
  File "/usr/lib/python2.7/site-packages/mockbuild/buildroot.py", line 442, in _setup_devices
    os.symlink("/proc/self/mounts", self.make_chroot_path('etc', 'mtab'))
OSError: [Errno 17] File exists

  This happens both when I use "mock --noclean install qemu" and if I try "mock --copyin qemu ; mock --install <qemu's dependency rpms>" instead.

  Is there a way to start the shell and tell that it is OK if paths already exist ?

Cheers
  Nick

Just trying to get you a recipe to sort it out for you

Hi Peter,

  FYI I have now been able to successfully build qemu 2.3.0 on the apm-mustang-ev3-05.lab.eng.brq.redhat.com board running FC22.

  The problem still exists when building qemu on the mustang-02.farm.hsv.redhat.com board.  

  This makes me wonder if it is a glibc problem rather than a linker problem (since the error messages are about glibc symbols).  The beaker board has binutils 2.25-5.fc22 and glibc 2.21-5.fc22 installed on it.  The farm board has binutils 2.25-10.fc23 and glibc 2.21.90-14.fc23 installed on it.

  I am going to try building the rawhide binutils on the beaker board and then linking qemu with that...

Cheers
  Nick

So I don't see the issue on F-22 at all. The last time we built qemu on F-22 was June 9th.

I see this as purely a F-23 issue and there's quite a few changes between F-22 and F-23 from the hardening flags, new c++ standards etc. so as to where it lies I'm not sure but to properly reproduce it I honestly believe you need a full F-23 system not just random bits of it.

Nick: so if you have mock 1.2.10 it should all work now. If "yum upgrade" doesn't work try the following:

F-21:
rpm -Uvh http://arm.koji.fedoraproject.org//packages/mock/1.2.10/1.fc21/noarch/mock-1.2.10-1.fc21.noarch.rpm

F-22:
rpm -Uvh http://arm.koji.fedoraproject.org//packages/mock/1.2.10/1.fc22/noarch/mock-1.2.10-1.fc22.noarch.rpm

Then I can do:

/usr/bin/mock -r fedora-rawhide-aarch64 --yum --rebuild --no-cleanup-after qemu-2.3.0-10.fc23.src.rpm
/usr/bin/mock -r fedora-rawhide-aarch64 --yum --shell --no-cleanup-after

And get to a shell with the full build env failure. In the chroot a "cd /builddir/build/" will get you to the standard rpmbuild directory structure with what you're looking for likely in /builddir/build/BUILD/qemu-2.3.0/

I have one I prepared earlier that I'll keep around if you want me to grab anything from it too

Hi Peter,

  OK, I have installed the F22 1.2.10-1 mock package, and the few other dependencies that it needed.  But now:

  mock -r fedora-rawhide-aarch64 --init" 

fails with:

  ERROR: Command /usr/bin/yum-deprecated is not available. Either install 
  package containing this command or run mock with --yum or --dnf to 
  overwrite config value. However this may lead to different dependency 
  solving!

I could not find a yum-deprecated package so I added the --dnf option instead and that appears to work.  But then when I try to build rpm using: 

  /usr/bin/mock -r fedora-rawhide-aarch64 --dnf --rebuild --no-cleanup-after qemu-2.3.0-10.fc23.src.rpm

I get:

  FileNotFoundError: [Errno 2] No such file or directory: 'qemu-2.3.0-10.fc23.src.rpm'

Which is weird because that file is definitely present in my current directory.  So I guessed that maybe I need to install the source rpm first, before trying to build it.  So I ran:

  /usr/bin/mock -r fedora-rawhide-aarch64 --dnf --install --no-cleanup-after qemu-2.3.0-10.fc23.src.rpm

which fails with:

  Error: nothing provides SDL2-devel needed by qemu-2:2.3.0-10.fc23.src

*sigh*

So next I ran:

  /usr/bin/mock -r fedora-rawhide-aarch64 --dnf --install --no-cleanup-after http://arm.koji.fedoraproject.org/packages/SDL2/2.0.3/5.fc22/aarch64/SDL2-2.0.3-5.fc22.aarch64.rpm

and then:

/usr/bin/mock -r fedora-rawhide-aarch64 --dnf --install --no-cleanup-after http://arm.koji.fedoraproject.org/packages/SDL2/2.0.3/5.fc22/aarch64/SDL2-devel-2.0.3-5.fc22.aarch64.rpm

but that needs alsa-lib-devel.  So now I am installing that.  I suspect that it is going to be a long day...

 
Cheers
  Nick

> I could not find a yum-deprecated package so I added the --dnf option
> instead and that appears to work.  But then when I try to build rpm using: 

"dnf install yum" should sort that out, but you'd need to then make it " mock -r fedora-rawhide-aarch64 --init --yum"

>   /usr/bin/mock -r fedora-rawhide-aarch64 --dnf --rebuild --no-cleanup-after
> qemu-2.3.0-10.fc23.src.rpm
> 
> I get:
> 
>   FileNotFoundError: [Errno 2] No such file or directory:
> 'qemu-2.3.0-10.fc23.src.rpm'
> 
> Which is weird because that file is definitely present in my current
> directory.  So I guessed that maybe I need to install the source rpm first,
> before trying to build it.  So I ran:

That's exactly what I did and it worked just find, have you tried fully qualified path?

>   /usr/bin/mock -r fedora-rawhide-aarch64 --dnf --install --no-cleanup-after
> qemu-2.3.0-10.fc23.src.rpm
> 
> which fails with:
> 
>   Error: nothing provides SDL2-devel needed by qemu-2:2.3.0-10.fc23.src
> 
> *sigh*

We've had a few issues with the mirrors so maybe try the following steps to ensure all your caches are clean:

dnf clean all
dnf install -y yum
/usr/bin/mock -r fedora-rawhide-aarch64 --clean
/usr/bin/mock -r fedora-rawhide-aarch64 --yum --rebuild --no-cleanup-after qemu--Right-NVR.src.rpm
/usr/bin/mock -r fedora-rawhide-aarch64 --yum --shell --no-cleanup-after

Nick: the issues with the mirrors should be all cleaned up no so where ever you all you should be good with the process above now.

Sorry, it's been a bit of a mission :-(

Created attachment 1044972 [details]
Suppress stack protection for AArch64.

Hi Peter,

  Good news/Bad news...

  Good news: I can now reproduce and debug the bug.
  Bad news: It is not a binutils bug at all.

  It is a disconnect between GCC and GLIBC.  GCC thinks that the stack protection feature is available for the AArch64.  GLIBC is not providing the necessary support.  (I do not know why.  I suspect that GLIBC is switching from a global stack protection mechanism to a per-thread stack protection mechanism, but that GCC has not been updated to follow this).

  Good news: I have a patch that disables building with stack protection for the AArch64.  (See uploaded).  With this patch applied qemu-2.3.0-12 successfully builds in the mock environment on mustang-2

  Bad news: Qemu will be running without stack protection on the AArch64.  Plus in theory you ought to check each time there is a new release of gcc or glibc for the AArch64 to see if the stack protection support can be re-enabled.

Cheers
  Nick

Thanks nick for your help here, much appreciated. Re-routing to glibc as it appears that that is where we're missing bits.

(In reply to Nick Clifton from comment #19)
> Created attachment 1044972 [details]
> Suppress stack protection for AArch64.
> 
> I suspect that GLIBC is switching
> from a global stack protection mechanism to a per-thread stack protection
> mechanism, but that GCC has not been updated to follow this).
>

eh? aarch64 doesn't support any other mechanism beyond a global __stack_chk... i'll take a look at this today, if nobody else does.

also, it's suspicious that there's relocation errors on stdout as well. that's a pretty clear indication that there's a red herring here.

whatever the latest glibc is indeed providing __stack_chk_guard in ld.so...

jkkm@dreadnought:~/junk% aarch64-linux-gnu-objdump -t ./lib64/ld-2.21.90.so | grep stack_chk_
000000000002fc60 g     O .data.rel.ro   0000000000000008              __stack_chk_guard
jkkm@dreadnought:~/junk% ll glibc-2.21.90-14.fc23.aarch64.rpm 
-rw-r--r-- 1 jkkm jkkm 3722976 May 30 07:02 glibc-2.21.90-14.fc23.aarch64.rpm

Created attachment 1045506 [details]
Proposed patch

Hi Peter,

  So first of all, apologies to Kyke - he is right, the problem is not with glibc.
It is a combination of a bug in the binutils - not reporting the error properly - and a misfeature in qemu - compiling without PIC enabled.

  The bug in the binutils was that it not reporting erroneous PC-relative relocations against dynamic symbols, and suggesting that the file concerned ought to be recompiled with -fPIC enabled.

  The misfeature in qemu is that is not compiling the aarch64 target with -fPIC enabled.  (My guess is that -fPIC did not work for aarch64, and hence it was disabbled in the qemu spec.  It is working now however).

  So ... I have uploaded a patch to update the qemu spec and add -fPIC.  With this applied the rpm builds just fine.

  I will also apply a patch to the rawhide binutils rpm to add in the missing error message.

Cheers
  Nick

Why does QEMU need to use -fPIC?  It already uses it for shared libraries, and use -fPIE for everything else.

Peter, please set needinfo and wait a day before using provenpackager powers.

Passing back to binutils since glibc contains a global default symbol for __stack_chk_guard which should be accessible to any executable being built with the normal gcc compile driver.

I agree with Paolo, that none of this should require qemu to be built with -fPIC since it's an executable.

The problem might be in the way in which qemu is linked. For example if the linking is done by hand and misses the fact that it is a required step to link against the symbols offered by ld.so

The default libc.so is actually a linker script which does the right GROUP/AS_NEEDED for the linker.

For example:

# cat /lib64/libc.so
/* GNU ld script
   Use the shared library, but some functions are only in
   the static library, so try that secondarily.  */
OUTPUT_FORMAT(elf64-littleaarch64)
GROUP ( /lib64/libc.so.6 /usr/lib64/libc_nonshared.a  AS_NEEDED ( /lib/ld-linux-aarch64.so.1 ) )

How is qemu being linked?

nickc, you don't *ever* need to apologize to me. i'm just happy to try to help. (and thought it was suspicious more wasn't broken if glibc was indeed not providing __stack_chk_guard.)

Carlos,

comment 0 mentions qemu-img, and its linking is pretty vanilla.  Notably it doesn't use libtool.

c++ -I/usr/include/pixman-1   -DHAS_LIBSSH2_SFTP_FSYNC -fPIE -DPIE ${long_list_of_more_irrelevant_cflags} -fstack-protector-strong -O2 -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -pthread -I/usr/include/glib-2.0 -I/usr/lib/glib-2.0/include  -g  -Wl,--warn-common -Wl,-z,relro -Wl,-z,now -pie -g -Wl,--build-id -pie -Wl,-z,relro -Wl,-z,now  -o qemu-img qemu-img.o ${long_list_of_o_files_all_built_with_fPIE} ${long_list_of_libraries}

For the full command line, look at https://kojipkgs.fedoraproject.org//packages/qemu/2.3.0/12.fc23/data/logs/armv7hl/build.log.

All the -Wl, flags come from Fedora's build machinery, not from QEMU's configure script.

-fstack-protector-strong, -fPIE and -pie however come from QEMU's configure script.

Hi Paolo,

> Why does QEMU need to use -fPIC?  It already uses it for shared libraries, 
> and use -fPIE for everything else.

It may well be that using -fPIE is sufficient - I did not test that.  

But the problem is that currently qemu for the aarch64 is being built without -fPIC or -fPIE being specified.  And for some reason the aarch64 architecture is being singled out for this treatment.  All other target architectures are being compiled with -pie enabled.

I guess that until F23 this used to work.  But now gcc for the aarch64 is generating stack protection code that will only work if compiled with -fPIC (or maybe -fPIE).  Hence the problem.

Cheers
  Nick

(In reply to Nick Clifton from comment #29)
> I guess that until F23 this used to work.  But now gcc for the aarch64 is
> generating stack protection code that will only work if compiled with -fPIC
> (or maybe -fPIE).  Hence the problem.

If the compiler is generating PIC accesses then the simplest solution is to fix the qemu build.

Assigning to qemu.

Could the qemu folks comments on why -fPIE and -fPIC are not used on aarch64 for qemu?

(In reply to Carlos O'Donell from comment #30)
> (In reply to Nick Clifton from comment #29)
> > I guess that until F23 this used to work.  But now gcc for the aarch64 is
> > generating stack protection code that will only work if compiled with -fPIC
> > (or maybe -fPIE).  Hence the problem.
> 
> If the compiler is generating PIC accesses then the simplest solution is to
> fix the qemu build.
> 
> Assigning to qemu.
> 
> Could the qemu folks comments on why -fPIE and -fPIC are not used on aarch64
> for qemu?

It is, now (since yesterday):

commit 1ec8e52bb21abc6cb8ce92f3063edf01a82139c5
Author: Peter Robinson <pbrobinson>
Date:   Thu Jul 2 16:32:23 2015 +0100

    Build aarch64 with -fPIC (rhbz 1232499)

As for why it wasn't before, it's just an accident of history.  I
originally enabled -fPIC back in February this year, but I only added
it to !aarch64, for reasons that are only really obvious if you look
at the patch:

http://pkgs.fedoraproject.org/cgit/qemu.git/commit/?id=6c3741c2769a21542a34716fa9188e520887a803

Summed up as "convenience".

Anyway, it's -fPIC and -fPIE on every architecture right now.

(In reply to Richard W.M. Jones from comment #31)
> (In reply to Carlos O'Donell from comment #30)
> > (In reply to Nick Clifton from comment #29)
> > > I guess that until F23 this used to work.  But now gcc for the aarch64 is
> > > generating stack protection code that will only work if compiled with -fPIC
> > > (or maybe -fPIE).  Hence the problem.
> > 
> > If the compiler is generating PIC accesses then the simplest solution is to
> > fix the qemu build.
> > 
> > Assigning to qemu.
> > 
> > Could the qemu folks comments on why -fPIE and -fPIC are not used on aarch64
> > for qemu?
> 
> It is, now (since yesterday):
> 
> commit 1ec8e52bb21abc6cb8ce92f3063edf01a82139c5
> Author: Peter Robinson <pbrobinson>
> Date:   Thu Jul 2 16:32:23 2015 +0100
> 
>     Build aarch64 with -fPIC (rhbz 1232499)
> 
> As for why it wasn't before, it's just an accident of history.  I
> originally enabled -fPIC back in February this year, but I only added
> it to !aarch64, for reasons that are only really obvious if you look
> at the patch:
> 
> http://pkgs.fedoraproject.org/cgit/qemu.git/commit/
> ?id=6c3741c2769a21542a34716fa9188e520887a803
> 
> Summed up as "convenience".
> 
> Anyway, it's -fPIC and -fPIE on every architecture right now.

Sorry that's not accurate.  There's another commit in the middle
which removed -fPIC and -fPIE for !aarch64, seemingly by accident:

http://pkgs.fedoraproject.org/cgit/qemu.git/commit/?id=749c3c43c328bfbb7db8fdacd35e0d36582839c6

And also aarch64 is only -fPIC, not -fPIE.

I'll see if qemu builds when I enable both ...

This build (if successful) should use -fPIC and -fPIE on
every arch.

http://koji.fedoraproject.org/koji/taskinfo?taskID=10284840

Using -fPIC and -fPIE makes no sense, those options are mutually exclusive, the last one wins.

As I understand the (very thin) documentation on the subject,
since qemu mainly builds executables (OK, except libcacard),
it's a good thing that -fPIE accidentally wins.

Carlos, QEMU is using -fPIE on all architectures for the executables, but apparently -fPIC is necessary instead for the stack protector on aarch64 so Peter's patch was correct after all.  But we should add a comment.

Richard, the patch you committed makes no sense.  First, what Jakub said.  Second, PIE is already enabled with --enable-pie.  Third, PIC should not be added to --extra-cflags, aarch64 is a special case.  Please revert, and next time please use a scratch build.

> gcc for the aarch64 is generating stack protection code that will only work
> if compiled with -fPIC

Moving to gcc.  If this is the case, it should at least add the option automatically.

So what is the small self-contained testcase that shows that for -fstack-protector you need -fPIC?
gcc during its %check compiles/runs thousands of tests with -fstack-protector-strong and without -fPIC.

This bug appears to have been reported against 'rawhide' during the Fedora 23 development cycle.
Changing version to '23'.

(As we did not run this process for some time, it could affect also pre-Fedora 23 development
cycle bugs. We are very sorry. It will help us with cleanup during Fedora 23 End Of Life. Thank you.)

More information and reason for this action is here:
https://fedoraproject.org/wiki/BugZappers/HouseKeeping/Fedora23

Paolo: status?

I am building a QEMU with -fPIE and seeing what happens.

http://koji.fedoraproject.org/koji/taskinfo?taskID=16567239 is built with PIE and seems to work fine (can run kvm-unit-tests at least).  I'll push the patch into Rawhide tomorrow.