Created attachment 1030584 [details]
spec fix for aarch64

Systemd builds with this patch applied.

http://arm.koji.fedoraproject.org/koji/taskinfo?taskID=3009857 is test build

Moving back to binutils. The patch to the systemd spec is a work around not a fix to the actual problem

Hi Marcin,

  Is this problem reproducible ?

  I ask because I find it very strange that the problem was introduced by the 2.25-8 revision of the binutils package.  The only change from 2.25-7 to 2.25-8 was to add a "Requires: coreutils" to the binutils.spec file.  That's it.  This change should have had no effect at all on any of the binutils builds, including the aarch64 build.

Cheers
  Nick

>   Is this problem reproducible ?

Yes, completely

>   I ask because I find it very strange that the problem was introduced by
> the 2.25-8 revision of the binutils package.  The only change from 2.25-7 to
> 2.25-8 was to add a "Requires: coreutils" to the binutils.spec file.  That's
> it.  This change should have had no effect at all on any of the binutils
> builds, including the aarch64 build.

No, it's not that but the Release field actually doesn't align with the changelog NVR field numberings.

It's was the introduction of the gold linker that caused the problems.

Hi Peter, Hi Marcin,

  Can one of you guys point me at a build environment where I can reproduce this problem ?  I have been trying to build systemd-220 on the  mustang-02.farm.hsv.redhat.com board, but to no avail.  It has a dependencies upon:
 
        firewalld-filesystem is needed by systemd-220-4.fc23.aarch64
        libseccomp-devel is needed by systemd-220-4.fc23.aarch64

which cannot be resolved. :-(

Cheers
  Nick

Nick: login to mustang and then "mock --uniqueext hrw -r fedora-rawhide-aarch64 shell" should drop you into chroot with all build dependencies installed.

In /builddir/ you have Fedora sources of binutils and systemd already checked out and all build dependencies are installed.

dnf is available inside of this chroot.

>   Can one of you guys point me at a build environment where I can reproduce
> this problem ?  I have been trying to build systemd-220 on the 
> mustang-02.farm.hsv.redhat.com board, but to no avail.  It has a
> dependencies upon:

A fully updated rawhide should be fine now. we're pretty up to date for builds.

Status update:

  I have not (yet) been able to reproduce the bug reported in this BZ, but I have been able to produce a similar problem:

 22698 Segmentation fault (core dumped) ( /bin/sh ../../libtool --mode=execute ./gudev-scan )

This happens during the build2 stage of the systemd build.  It turns out that gudev-scan is just a script that launches the lt-gudev-san executable in the .libs subdirectory.  This is the program that seg-faults.  Examining it with file shows:

  % file lt-gudev-scan
  lt-gudev-scan: ELF 64-bit LSB executable, ARM aarch64, version 1
  (SYSV), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, for GNU/Linux 3.7.0, BuildID[sha1]=6880f357fcda86e711e9e4778c4a3a5600ff6a86, not stripped

But, using ldd shows:

  % ldd lt-gudev-scan
    not a dynamic executable

So I think that this is the same problem that we have with BZ 1215546, ie the aarch64 gold linker produced binaries that are incompatible with Fedora, but which do work with other Linux variants.

  https://bugzilla.redhat.com/show_bug.cgi?id=1215546

The problem I have with that BZ, and now this one, is that I have no idea *why* the loader is rejecting these binaries.  If I knew that I could probably fix gold.  But I do not have any familiarity with the loader and I do not know how to force it to tell me what is wrong.

So maybe there are two gold bugs here, or maybe it is just one bug, manifesting itself in two different ways.

Anyway that is where I am at the moment.  Any help or ideas would be much appreciated.

Cheers
  Nick

PS.  I have confirmed that I can build the systemd rpm with the bfd linker installed as the default linker.

We're in the process of removing gudev from systemd, so the problem might partially solve itself, it is only the gudev part which is causing problems. http://koji.fedoraproject.org/koji/buildinfo?buildID=644608 is the first build of libgudev as a separate package.

libgudev builds fine locally:

Wrote: /builddir/build/RPMS/libgudev-230-1.fc23.aarch64.rpm
Wrote: /builddir/build/RPMS/libgudev-devel-230-1.fc23.aarch64.rpm
Wrote: /builddir/build/RPMS/libgudev-debuginfo-230-1.fc23.aarch64.rpm

> So I think that this is the same problem that we have with BZ 1215546, ie
> the aarch64 gold linker produced binaries that are incompatible with Fedora,
> but which do work with other Linux variants.

I think the main way that Fedora varies from other distros on aarch64 is the use of 64K page sizes rather than 4K. This caused us issues when running ARMv7 virt on aarch64 and needed a fix in binutils. Not sure if this might be related?

Hi Peter,

  You are a genius sir.  The lack of a 64K page size does indeed seem to be the problem.  A small patch to the binutils rpm, (plus installing the newly built gold linker once the rpm has been created), fixes the building of systemd.  At least in the mock build environment on mustang-2.

Marcin, Peter - please try binutils-2.25-10.fc23 or binutils-2.25-8.fc22 and let me know if gold really is working now.

Cheers
  Nick

Awesome! Building on aarch64 now, will try a scratch of systemd once done

http://arm.koji.fedoraproject.org/koji/taskinfo?taskID=3031585

(In reply to Nick Clifton from comment #12)
>   You are a genius sir.  The lack of a 64K page size does indeed seem to be
> the problem.  A small patch to the binutils rpm, (plus installing the newly
> built gold linker once the rpm has been created), fixes the building of
> systemd.  At least in the mock build environment on mustang-2.

Nice, I think Jakub also pointed to the page size in bug 1215546,
which I will try to test now.