This might be related to the shuffling with gcc and glibc on ppc64le because the IFUNC support was broken. It could go away during or after the mass-rebuild once webkit gets rebuilt.

same problem when building evolution-data-server-3.25.4-1.fc27

/usr/lib64/libwebkit2gtk-4.0.so.37: undefined reference to `std::__once_callable.11'
/usr/lib64/libwebkit2gtk-4.0.so.37: undefined reference to `std::__once_call.11'

glibc team - I suppose we just need to rebuild the webkit package, right?

I had a chat on #fedora-admin and puiterwijk pointed me to bug #1456261, which prevents update of webkitgtk4 in rawhide. Note the version evolution-data-server build in rawhide [1] picked was webkitgtk4-2.16.2-2.fc27, which is neither the latest unstable version (due to bug #1456261), nor the latest stable version.

Thus, from my point of view, either the issue with debuginfo will be fixed soon, or the rawhide should get the latest stable version to not block the rest of the system from update.

An interesting thing, from my point of view, is that this missing symbol issue strikes only on ppc64le, but not anywhere else (see [1]), while the package versions I checked are the same on x86_64 and ppc64le (it was a rough check, nothing in-deep).

[1] https://koji.fedoraproject.org/koji/taskinfo?taskID=20570888

libstdc++.so.6 comes from gcc, reassinging.

The problem is:
 /* Define to 1 if the target supports thread-local storage. */
-#define _GLIBCXX_HAVE_TLS 1
+/* #undef _GLIBCXX_HAVE_TLS */
 
in between gcc-7.1.1-4.fc26 and gcc-7.1.1-5.fc27.  As there is no related change between -4 and -5 in this, this must be again some glibc headers related change in rawhide.

(In reply to Jakub Jelinek from comment #5)
> The problem is:
>  /* Define to 1 if the target supports thread-local storage. */
> -#define _GLIBCXX_HAVE_TLS 1
> +/* #undef _GLIBCXX_HAVE_TLS */
>  
> in between gcc-7.1.1-4.fc26 and gcc-7.1.1-5.fc27.  As there is no related
> change between -4 and -5 in this, this must be again some glibc headers
> related change in rawhide.

I think this is derived from the run-time check here:

AC_DEFUN([GCC_CHECK_TLS], [
  AC_REQUIRE([AC_CANONICAL_HOST])
  GCC_ENABLE(tls, yes, [], [Use thread-local storage])
  AC_CACHE_CHECK([whether the target supports thread-local storage],
                 gcc_cv_have_tls, [
    AC_RUN_IFELSE([__thread int a; int b; int main() { return a = b; }],
      [dnl If the test case passed with dynamic linking, try again with
       dnl static linking, but only if static linking is supported (not
       dnl on Solaris 10).  This fails with some older Red Hat releases.
      chktls_save_LDFLAGS="$LDFLAGS"

I suspect the run-time check failed due to the bug we were trying to fix.  The build log is garbled due to the concurrency (maybe you should switch to “make -O”?), but I see some instances of

checking whether the target supports thread-local storage... no

Which, I think, should not happen.  It does not in the old logs from gcc-7.1.1-2.fc27.

I'm not sure if the configure check will pass with the fixed glibc and the current gcc in rawhide.  If not, you'll have to build with an older version, or override the check to always succeed.

I've tried to reproduce it today but it works now:
https://koji.fedoraproject.org/koji/getfile?taskID=20573264&volume=DEFAULT&name=build.log
This was a scratch build with hacked up spec file so that it would just stop after bootstrap and uuencode libstdc++-v3/config.log
Today it was with glibc-2.25.90-25.fc27, the problematic build on ppc64le was against glibc-2.25.90-22.fc27.
So have you fixed some bug on the glibc side?

(In reply to Jakub Jelinek from comment #7)
> I've tried to reproduce it today but it works now:
> https://koji.fedoraproject.org/koji/
> getfile?taskID=20573264&volume=DEFAULT&name=build.log
> This was a scratch build with hacked up spec file so that it would just stop
> after bootstrap and uuencode libstdc++-v3/config.log
> Today it was with glibc-2.25.90-25.fc27, the problematic build on ppc64le
> was against glibc-2.25.90-22.fc27.
> So have you fixed some bug on the glibc side?

We changed the TCB initialization for statically-linked binaries, and considering that there's a check for TLS with -static, we might have already fixed it.

(In reply to Florian Weimer from comment #8)
> (In reply to Jakub Jelinek from comment #7)
> > I've tried to reproduce it today but it works now:
> > https://koji.fedoraproject.org/koji/
> > getfile?taskID=20573264&volume=DEFAULT&name=build.log
> > This was a scratch build with hacked up spec file so that it would just stop
> > after bootstrap and uuencode libstdc++-v3/config.log
> > Today it was with glibc-2.25.90-25.fc27, the problematic build on ppc64le
> > was against glibc-2.25.90-22.fc27.
> > So have you fixed some bug on the glibc side?
> 
> We changed the TCB initialization for statically-linked binaries, and
> considering that there's a check for TLS with -static, we might have already
> fixed it.

Yes on ppc64le for statically-linked binaries we now initialize the TCB and TLS *before* running the IFUNC resolvers, which was done to support the libgcc_s.so/.a IFUNC which uses a builtin to check the TCB's AT_PLATFORM/AT_HWCAP  entries. This is a ppc64le-only difference.

All statically-linked applications in ppc64le were broken until we fixed the bug. This is reason we had to do the circular builds last week to get ready for the mass rebuild.

*** Bug 1472424 has been marked as a duplicate of this bug. ***

I did a scratch build of webkitgtk4 against current Rawhide
(changing nothing else) and it appears to have fixed emacs on
ppc64le.  My scratch build is here:

https://koji.fedoraproject.org/koji/taskinfo?taskID=20602070

I intend to bump and rebuild the real webkitgtk4 package now.

(In reply to Richard W.M. Jones from comment #11)
> I intend to bump and rebuild the real webkitgtk4 package now.

It'll possibly fail during package signing, as it did so for several latest builds:
https://koji.fedoraproject.org/koji/rpminfo?rpmID=10003418
which is an "info" link of x86_64 build of debuginfo subpackage at
https://koji.fedoraproject.org/koji/buildinfo?buildID=909563

That's why this depends on bug #1456261.

(In reply to Milan Crha from comment #12)
> (In reply to Richard W.M. Jones from comment #11)
> > I intend to bump and rebuild the real webkitgtk4 package now.
> 
> It'll possibly fail during package signing

It's completed..

Rebuilding webkitgtk4 or any other affected package is just wrong.
This is a gcc bug caused by glibc bug, the real fix is to rebuild gcc, not the other packages.  I'm trying to do that since Monday, but it takes long due to the s390x builders being broken because of a binutils bug.
If you rebuilt anything in order to "cure" this issue, you'll need to rebuild it again once fixed gcc is in the buildroots.

(In reply to Milan Crha from comment #12)
> (In reply to Richard W.M. Jones from comment #11)
> > I intend to bump and rebuild the real webkitgtk4 package now.
> 
> It'll possibly fail during package signing, as it did so for several latest
> builds:
> https://koji.fedoraproject.org/koji/rpminfo?rpmID=10003418
> which is an "info" link of x86_64 build of debuginfo subpackage at
> https://koji.fedoraproject.org/koji/buildinfo?buildID=909563
> 
> That's why this depends on bug #1456261.

Does this explain why the webkitgtk4 package which is pulled
into the buildroot is the much older version (2.16.2-2.fc27)
and not the latest version (2.17.4-3.fc27)?

(In reply to Richard W.M. Jones from comment #15)
> (In reply to Milan Crha from comment #12)
> > That's why this depends on bug #1456261.
> 
> Does this explain why the webkitgtk4 package which is pulled
> into the buildroot is the much older version (2.16.2-2.fc27)
> and not the latest version (2.17.4-3.fc27)?

Yes, it's the reason. I've been told so on IRC too (see comment #3).

(In reply to Richard W.M. Jones from comment #15)
> (In reply to Milan Crha from comment #12)
> > (In reply to Richard W.M. Jones from comment #11)
> > > I intend to bump and rebuild the real webkitgtk4 package now.
> > 
> > It'll possibly fail during package signing, as it did so for several latest
> > builds:
> > https://koji.fedoraproject.org/koji/rpminfo?rpmID=10003418
> > which is an "info" link of x86_64 build of debuginfo subpackage at
> > https://koji.fedoraproject.org/koji/buildinfo?buildID=909563
> > 
> > That's why this depends on bug #1456261.
> 
> Does this explain why the webkitgtk4 package which is pulled
> into the buildroot is the much older version (2.16.2-2.fc27)
> and not the latest version (2.17.4-3.fc27)?

It does pull latest version otherwise the epiphany build won't succeeded - (but I manually ran koji wait-repo webkitgtk4-2.17.4-3.fc27 f27-build)

According to Jakub (comment #14), it's not correct to rebuild everything (what depends on webkitgtk4) now, thus I'm still on hold with evolution packages.

That the webkitgtk4-2.17.4-3.fc27 finally got into the build root is because it worked with the package/debuginfo signing. Each version before it had been broken (too large debuginfo) and 2.16.2 had been used in rawhide till now (and
on Monday [1]).

[1] https://kojipkgs.fedoraproject.org//work/tasks/910/20570910/root.log
    which is from:
    https://koji.fedoraproject.org/koji/taskinfo?taskID=20570888

libstdc++.so.6 in libstdc++-7.1.1-6.fc27.ppc64le contains the TLS symbols again.
Packages that used to refer to those symbols and were rebuilt with the broken gcc-7.1.1-5.fc27.ppc64le should be rebuilt.

Thanks for rebuilding GCC.  Do we have a test for which packages
might be affected?  Would this work?

  readelf -s /usr/lib64/<library>.so.? |
      fgrep '_ZSt11__once_call.11'

I rebuilt webkitgtk4 before the GCC fix, and it's now working for me.
I don't know if that's just luck
.