Created attachment 1670312 [details]
example of JVM crash (from a local junit mock build on fedora-rawhide-x86_64)

Attached a locally produced crash when building junit in mock with "-r fedora-rawhide-x86_64".

The log indicates a bug in surefire plugin. Better assign this bug to there, or generally mockito executed by maven.
Wouldn't wonder if surefire isn't able to parse any character(s) got from the maven xml. Just a guessing because there are reported issues (partly fixed) with surefire and unicode (umlauts) or other special characters.

Shorten backtrace for the relevant thread (0x00007f2eb031b000) only:

 1285       4       java.lang.CharacterDataLatin1::toLowerCase (39 bytes)
 nmethod 1285 0x00007f2ea15e0450 code [0x00007f2ea15e05a0, 0x00007f2ea15e0678]
 1289       4       org.apache.maven.surefire.util.internal.StringUtils::escapeToPrintable (152 bytes)
 nmethod 1289 0x00007f2ea15e7350 code [0x00007f2ea15e7520, 0x00007f2ea15e7b60]
 1314       4       sun.reflect.ClassFileAssembler::emitConstantPoolUTF8 (50 bytes)
 nmethod 1314 0x00007f2ea160c190 code [0x00007f2ea160c3a0, 0x00007f2ea160ced0]
 1349       4       java.lang.Class::getName (21 bytes)
 nmethod 1349 0x00007f2ea15c76d0 code [0x00007f2ea15c7820, 0x00007f2ea15c78c8]
 1322       4       java.lang.String::indexOf (7 bytes)
 nmethod 1322 0x00007f2ea1608750 code [0x00007f2ea16088a0, 0x00007f2ea1608a18]


That's clearly a bug with unicode. Can you assign the complete maven build file?

openjdk itself FTBFS in F32, so I tend to say this bug depends on bug #1795268.

We're also seeing this when running an existing (already built) Java package in Fedora Rawhide. See:

https://pagure.io/dogtagpki/issue/3130

This seems to be the "common core" of all crashes I've looked at:

# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007f82a66cdfbe, pid=120, tid=0x00007f82a48f8700
#
# JRE version: OpenJDK Runtime Environment (8.0_242-b08) (build 1.8.0_242-b08)
# Java VM: OpenJDK 64-Bit Server VM (25.242-b08 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# V  [libjvm.so+0x701fbe]  JNIHandleBlock::oops_do(OopClosure*)+0xae

Everything else seems to not be reproducible.

During package builds, this crash affects mostly tests, probably just because those exercises the JVM most.
But it seems to be triggered randomly during any execution of Java code.

(In reply to Raphael Groner from comment #3)
…
> That's clearly a bug with unicode. Can you assign the complete maven build
> file?

Can you please answer my question?

After some investigation, it looks like these crashes originate somewhere in memory allocation / Garbage Collection, for example (from the junit crash):

Current thread (0x00007f2eb0066000):  GCTaskThread [stack: 0x00007f2e9fa35000,0x00007f2e9fb35000] [id=190]

Stack: [0x00007f2e9fa35000,0x00007f2e9fb35000],  sp=0x00007f2e9fb33560,  free space=1017k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0x701fbe]  JNIHandleBlock::oops_do(OopClosure*)+0xae
V  [libjvm.so+0xae8b6c]  JavaThread::oops_do(OopClosure*, CLDClosure*, CodeBlobClosure*)+0x3c
V  [libjvm.so+0x9a633a]  ThreadRootsTask::do_it(GCTaskManager*, unsigned int)+0x7a
V  [libjvm.so+0x5d8274]  GCTaskThread::run()+0x1d4
V  [libjvm.so+0x92fd78]  java_start(Thread*)+0x1c8

And I've just been able to reproduce this crash with the pending fedora 32 update as well:
https://bodhi.fedoraproject.org/updates/FEDORA-2020-af190951f6

(In reply to Fabio Valentini from comment #9)
> And I've just been able to reproduce this crash with the pending fedora 32
> update as well:
> https://bodhi.fedoraproject.org/updates/FEDORA-2020-af190951f6

OK, so this means I fail to update my packages based on java as long it's jdk8, especially jpype (see bug #1791951 and bug #1808625).

(In reply to Raphael Groner from comment #10)
> (In reply to Fabio Valentini from comment #9)
> > And I've just been able to reproduce this crash with the pending fedora 32
> > update as well:
> > https://bodhi.fedoraproject.org/updates/FEDORA-2020-af190951f6
> 
> OK, so this means I fail to update my packages based on java as long it's
> jdk8, especially jpype (see bug #1791951 and bug #1808625).

Yep, this is blocking any package updates for builds that are affected by this crash.
Which is pretty bad, given how often Java packages get CVEs or need other critical fixes.

For now, you should still be able to build things on fedora 32, though.
The broken f32 update has not been pushed to stable yet, and there's no buildroot override for it.

Not sure if this is due to the GCC10 fix or something else. Assigning to aph to take a look.

Proposing this as a prioritized bug, since it's blocking a lot of updates in rawhide, and is blocking inclusion of future OpenJDK 8 security updates in fedora 32+.

Created attachment 1670862 [details]
hserr pid from 17.3.2020

 while true ; do mock -r fedora-rawhide-x86_64 java-runtime-decompiler-3.0-6.fc33.src.rpm   ; done

Failed 3 times in 10 builds Attached the trace.

We could try to propose as a blocker for F32 Final.

I have reproduced this problem and I'm working on it.

The crash itself can't be a blocker for F32, as the broken Java did not make it through updates-testing (I've just unpushed it). But 'we need a newer Java for security fixes' could be, if any of the security issues is 'important' or higher on the Red Hat scale:

https://fedoraproject.org/wiki/Fedora_32_Final_Release_Criteria#Security_bugs

it would be best to have a bug that is actually for such a security issue and propose *that* as the blocker, though (then have this block that).

(In reply to Adam Williamson from comment #17)
…
> it would be best to have a bug that is actually for such a security issue
> and propose *that* as the blocker, though (then have this block that).

My packages are not security risky, obviously. Maybe take a look generally in (unfixed) jdk8 CVE?

(In reply to Adam Williamson from comment #17)
> The crash itself can't be a blocker for F32, as the broken Java did not make
> it through updates-testing (I've just unpushed it). But 'we need a newer
> Java for security fixes' could be, if any of the security issues is
> 'important' or higher on the Red Hat scale:
> 
> https://fedoraproject.org/wiki/Fedora_32_Final_Release_Criteria#Security_bugs
> 
> it would be best to have a bug that is actually for such a security issue
> and propose *that* as the blocker, though (then have this block that).

Sorry, can you be more specific as to what was 'unpushed'?

The bug here isn't due to a change in the OpenJDK package. It's due to the way GCC has started compiling code that has been in OpenJDK since its inception.

Andrew Haley has located the bug and a fix: https://bugs.openjdk.java.net/browse/JDK-8241296

If it doesn't show up on older builds, it's because they were built before GCC 10 was added. Any further rebuilds will need this fix to avoid this bug.

I'm going to work on getting it into rawhide now.

(In reply to Andrew John Hughes from comment #19)
> Sorry, can you be more specific as to what was 'unpushed'?

The fedora 32 update for java-1.8.0-openjdk which has this crasher bug has been "unpushed":
https://bodhi.fedoraproject.org/updates/FEDORA-2020-af190951f6#comment-1286568

Which means it cannot be "pushed" into stable repositories now.

> The bug here isn't due to a change in the OpenJDK package. It's due to the
> way GCC has started compiling code that has been in OpenJDK since its
> inception.
> 
> Andrew Haley has located the bug and a fix:
> https://bugs.openjdk.java.net/browse/JDK-8241296
> 
> If it doesn't show up on older builds, it's because they were built before
> GCC 10 was added. Any further rebuilds will need this fix to avoid this bug.
> 
> I'm going to work on getting it into rawhide now.

That's great news. Thank you!

Once it lands in rawhide, I'll check it out and confirm whether it fixes the issues I've been seeing.
If it indeed does, then the update should probably go into fedora 32 as well (since it has currently an outdated openjdk8 version).

Thanks. Maybe try to file a pull request.

Andrew Haley is also the downstream packager, so I would just relax and expect that he's perfectly capable of bringing the fix downstream once he's ready :) If it's not done in a day or two we can start worrying about it I guess.

sorry, I meant both the Andrews work on the downstream package, so between them they're certainly capable of getting the fix landed.

(In reply to Andrew John Hughes from comment #19)
> The bug here isn't due to a change in the OpenJDK package. It's due to the
> way GCC has started compiling code that has been in OpenJDK since its
> inception.
> 
> Andrew Haley has located the bug and a fix:
> https://bugs.openjdk.java.net/browse/JDK-8241296

And it's now double-approved upstream, so we're good to go. I'll also
create some backport requests for 8u and 11u tomorrow.

(In reply to Fabio Valentini from comment #20)
> (In reply to Andrew John Hughes from comment #19)
> > Sorry, can you be more specific as to what was 'unpushed'?
> 
> The fedora 32 update for java-1.8.0-openjdk which has this crasher bug has
> been "unpushed":
> https://bodhi.fedoraproject.org/updates/FEDORA-2020-af190951f6#comment-
> 1286568
> 
> Which means it cannot be "pushed" into stable repositories now.
> 
> > The bug here isn't due to a change in the OpenJDK package. It's due to the
> > way GCC has started compiling code that has been in OpenJDK since its
> > inception.
> > 
> > Andrew Haley has located the bug and a fix:
> > https://bugs.openjdk.java.net/browse/JDK-8241296
> > 
> > If it doesn't show up on older builds, it's because they were built before
> > GCC 10 was added. Any further rebuilds will need this fix to avoid this bug.
> > 
> > I'm going to work on getting it into rawhide now.
> 
> That's great news. Thank you!
> 
> Once it lands in rawhide, I'll check it out and confirm whether it fixes the
> issues I've been seeing.
> If it indeed does, then the update should probably go into fedora 32 as well
> (since it has currently an outdated openjdk8 version).

Scratch build for rawhide is underway now: https://koji.fedoraproject.org/koji/taskinfo?taskID=42635480

Changes are in my local fork and I'll post a PR once that build succeeds: https://src.fedoraproject.org/fork/ahughes/rpms/java-1.8.0-openjdk/c/1b81e9374ae1c3d473461f20737119f489aa4b4c?branch=master

Also already completed the same for java-11-openjdk: https://koji.fedoraproject.org/koji/taskinfo?taskID=42633444

That old version of OpenJDK 8 is what I was worried about. We need to get an updated build into Fedora 32. The b06 version currently there is an early access build and doesn't include the last batch of security fixes from January (see the "Security Fixes" section of http://bitly.com/oj8u242). We were blocked from doing the later updates in what was then rawhide once GCC 10 went in (see bug #1795268). This would mean that Fedora 32 would regress from the version in Fedora 31.

If I need to file an additional blocker bug to make that happen, just let me know.

Nope, just merge the master branch into f32 after rawhide is done, and then build and submit a bodhi update as usual. there are no freezes in effect for feodra 32 right now, so there's nothing else you need to do (except wait a few days or for positive feedback, before it can be pushed to stable).

Final freeze is scheduled for 2020-04-07; as long as you get the update submitted for stable before then it should be OK.

If all goes well, I should be able to submit the update today.

If you send a build to Rawhide at the same time, we'll find out with the next Rawhide compose whether things are OK (openQA runs on each Rawhide compose, and this bug always causes the FreeIPA tests to fail).

I was about to send the build to rawhide, but it seems my previous build on rawhide, and my scratch build of this change, both failed on s390x:

https://koji.fedoraproject.org/koji/taskinfo?taskID=42635501
https://koji.fedoraproject.org/koji/taskinfo?taskID=42634995

It seems the current rawhide JDK is not stable enough to build OpenJDK on s390x. I'm not sure if this is the same bug as this or something else. It doesn't seem obviously the same:

# A fatal error has been detected by the Java Runtime Environment:
#
#  Internal Error (os_linux_zero.cpp:261), pid=3765835, tid=0x000003ffa4bd2900
#  fatal error: caught unhandled signal 11

but that could be because this is the zero assembler port. armv7hl uses that too, incidentally, and has built.

It looks like the OpenJDK bootstrap mechanism does not perform what I would describe as a bootstrap. It's more of a minimal test build.

I would expect a bootstrap to do a minimal build with the system compiler, and then a full build using that build. That places minimum pressure on the system install and fully tests the one being built.

Instead, OpenJDK's bootstrapping just uses the just-built JDK to build a minimal image again, and then goes back to using the system JDK to build docs and demos, which is where it fails on s390x.

I'm going to look at changing the RPM to do a more usual bootstrap, as we used to do with java-1.{6,7}.0-openjdk. This should tell us a) if this fix resolves the problem and b) make it less likely from broken JDKs to end up being used to bootstrap in future.

Is there a way we can roll back rawhide to an earlier JDK as with F32? The one used by F32 seems to be the common ancestor that worked before GCC 10 builds were used.

Well. Technically yes, but practically no. =) By policy Rawhide doesn't go backwards, once a build has made it into a successful compose we do not go backwards from it. In theory we *could* (it's just a matter of untagging all broken builds, then the next compose will include the older working build), but we don't do that. (It also wouldn't fix installed Rawhide systems on uodate ordinarily; users doing 'dnf update' would not be downgraded. You'd only be downgraded if you did 'dnf distro-sync'.)

If you can't figure out a way to get an s390x build through, we'd be left with only the options of leaving it broken or excluding s390x from the build (which would fix it on other arches, but mean s390x didn't have the package *at all* any more, until we got it building again).

Looks like my changes to the bootstrapping process worked. It's built on rawhide:

https://koji.fedoraproject.org/koji/taskinfo?taskID=42694611

f32 build seems to have stalled on s390x. I'm going to kick off another to see if that has more luck:

https://koji.fedoraproject.org/koji/taskinfo?taskID=42704865

New f32 build: https://koji.fedoraproject.org/koji/taskinfo?taskID=42716235

Thanks! I can confirm that the crashes I saw on rawhide are now gone.

Great!
Now just hope this f32 build completes: https://koji.fedoraproject.org/koji/taskinfo?taskID=42716912

Should be fixed now.

You could edit the new build into the existing update and then mark it as fixing this bug.

FEDORA-2020-42e79d74f1 has been submitted as an update to Fedora 32. https://bodhi.fedoraproject.org/updates/FEDORA-2020-42e79d74f1

I've triggered the openQA tests on this update, we'll see how they go.

FEDORA-2020-42e79d74f1 has been pushed to the Fedora 32 stable repository.
If problem still persists, please make note of it in this bug report.

(In reply to Fedora Update System from comment #40)
> FEDORA-2020-42e79d74f1 has been pushed to the Fedora 32 stable repository.
> If problem still persists, please make note of it in this bug report.

Thanks for the great work. Could you additionally fix those errors (marked as red in taskotron( given about rpmlint[¹] and rpmgrill[²]? Sorry for my late response after received enough positive karma for the autopush. Should I open a new bug for that?

¹ https://taskotron.fedoraproject.org/artifacts/all/a271f662-6d3e-11ea-9431-525400364adf/tests.yml/java-1.8.0-openjdk-1.8.0.242.b08-1.fc32.log

² https://taskotron.fedoraproject.org/artifacts/all/a2bc370e-6d3e-11ea-a1da-525400364adf/tests.yml/rpmgrill.json

I beleive I am still seeing the seg fault in the java-1.8.0-openjdk-aarch32 package -- is there still an update pending?

For example as seen in this task: https://koji.fedoraproject.org/koji/taskinfo?taskID=42782568

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0xb5582e7c, pid=19031, tid=0xa25e3460
#
# JRE version: OpenJDK Runtime Environment (8.0_242-b07) (build 1.8.0_242-b07)
# Java VM: OpenJDK Client VM (25.242-b07 mixed mode linux-aarch32 )
# Problematic frame:
# V  [libjvm.so+0x330e7c]  JNIHandleBlock::oops_do(OopClosure*)+0xa8
#
# Core dump written. Default location: /builddir/build/BUILD/netty-netty-4.1.13.Final/core or core.19031
#
# An error report file with more information is saved as:
# /builddir/build/BUILD/netty-netty-4.1.13.Final/hs_err_pid19031.log
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
#

(In reply to Mat Booth from comment #42)
> I beleive I am still seeing the seg fault in the java-1.8.0-openjdk-aarch32
> package -- is there still an update pending?

Yes. It's a different package[1]. Please file a new bug for it, component java-1.8.0-openjdk-aarch32.

[1] https://src.fedoraproject.org/rpms/java-1.8.0-openjdk-aarch32

Please see bug 1818078

(In reply to Raphael Groner from comment #41)
> (In reply to Fedora Update System from comment #40)
> > FEDORA-2020-42e79d74f1 has been pushed to the Fedora 32 stable repository.
> > If problem still persists, please make note of it in this bug report.
> 
> Thanks for the great work. Could you additionally fix those errors (marked
> as red in taskotron( given about rpmlint[¹] and rpmgrill[²]? Sorry for my
> late response after received enough positive karma for the autopush. Should
> I open a new bug for that?
> 
> ¹
> https://taskotron.fedoraproject.org/artifacts/all/a271f662-6d3e-11ea-9431-
> 525400364adf/tests.yml/java-1.8.0-openjdk-1.8.0.242.b08-1.fc32.log
> 
> ²
> https://taskotron.fedoraproject.org/artifacts/all/a2bc370e-6d3e-11ea-a1da-
> 525400364adf/tests.yml/rpmgrill.json

Most of these are long-term and are deliberate. They probably need whitelisting in some way. If I "fixed" them, the result would be a completely broken JDK.  Specifically, the $ORIGIN issues are required for the JDK to find its libraries in its directory tree, which is not in a standard location indexed by the linker (and can't be while supporting multiple JDKs)

The symlinks are resolved by other packages e.g. to use system copies of timezone data & security certificates.

The explicit dependencies are due to OpenJDK dynamically loading some libraries at runtime rather than linking against them (e.g. CUPS, libXcomposite)

So only the description & typos are worthy of actually being fixed.

Yes, this warrants a separate bug as it's not related to this crash. It probably also applies to the other OpenJDK packages (java-11-openjdk, java-latest-openjdk)