Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.

Bug 1799087

Summary: java-11-openjdk: [s390x] FTBFS in Fedora rawhide/f32 due to GCC 10
Product: [Fedora] Fedora Reporter: Fedora Release Engineering <releng>
Component: java-11-openjdkAssignee: Severin Gehwolf <sgehwolf>
Status: CLOSED DUPLICATE QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 32CC: ahughes, ashundi, bugproxy, dan, hannsj_uhl, jerboaa, jhuttana, jvanek, sgehwolf
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: java-11-openjdk-11.0.7.2-0.3.ea.fc33 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-03-25 19:25:17 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 467765, 1750908, 1799531    
Attachments:
Description Flags
build.log
none
root.log
none
state.log none

Description Fedora Release Engineering 2020-02-06 15:36:02 UTC 
java-11-openjdk failed to build from source in Fedora rawhide/f32

https://koji.fedoraproject.org/koji/taskinfo?taskID=41213309


For details on the mass rebuild see:

https://fedoraproject.org/wiki/Fedora_32_Mass_Rebuild
Please fix java-11-openjdk at your earliest convenience and set the bug's status to
ASSIGNED when you start fixing it. If the bug remains in NEW state for 8 weeks,
java-11-openjdk will be orphaned. Before branching of Fedora 33,
java-11-openjdk will be retired, if it still fails to build.

For more details on the FTBFS policy, please visit:
https://fedoraproject.org/wiki/Fails_to_build_from_source
Comment 1 Fedora Release Engineering 2020-02-06 15:36:15 UTC 
Created attachment 1658259 [details]
build.log

file build.log too big, will only attach last 32768 bytes
Comment 2 Fedora Release Engineering 2020-02-06 15:36:17 UTC 
Created attachment 1658260 [details]
root.log

file root.log too big, will only attach last 32768 bytes
Comment 3 Fedora Release Engineering 2020-02-06 15:36:19 UTC 
Created attachment 1658261 [details]
state.log
Comment 4 Severin Gehwolf 2020-02-06 20:07:19 UTC 
We weren't seeing this issue prior GCC 10 in build roots.

Symptom looks like this:

Exception in thread "main" java.lang.ExceptionInInitializerError
Exception in thread "main" java.lang.ExceptionInInitializerError
	at jdk.jlink/jdk.tools.jmod.JmodTask.getMessage(JmodTask.java:1547)
	at jdk.jlink/jdk.tools.jmod.JmodTask.handleOptions(JmodTask.java:1284)
	at jdk.jlink/jdk.tools.jmod.JmodTask.run(JmodTask.java:191)
	at jdk.jlink/jdk.tools.jmod.Main.main(Main.java:34)
Caused by: java.security.PrivilegedActionException: java.lang.ClassNotFoundException: jdk.tools.jmod.resources.spi.jmodProvider
	at java.base/java.security.AccessController.doPrivileged(Native Method)
	at java.base/java.util.ResourceBundle.getResourceBundleProviderType(ResourceBundle.java:1913)
	at java.base/java.util.ResourceBundle.getServiceLoader(ResourceBundle.java:1880)
	at java.base/java.util.ResourceBundle$CacheKey.getProviders(ResourceBundle.java:699)
	at java.base/java.util.ResourceBundle$CacheKey.hasProviders(ResourceBundle.java:706)
	at java.base/java.util.ResourceBundle.loadBundle(ResourceBundle.java:1809)
	at java.base/java.util.ResourceBundle.findBundle(ResourceBundle.java:1774)
	at java.base/java.util.ResourceBundle.findBundle(ResourceBundle.java:1728)
	at java.base/java.util.ResourceBundle.findBundle(ResourceBundle.java:1728)
	at java.base/java.util.ResourceBundle.getBundleImpl(ResourceBundle.java:1662)
	at java.base/java.util.ResourceBundle.getBundleImpl(ResourceBundle.java:1582)
	at java.base/java.util.ResourceBundle.getBundleImpl(ResourceBundle.java:1556)
	at java.base/java.util.ResourceBundle.getBundle(ResourceBundle.java:932)
	at jdk.jlink/jdk.tools.jmod.JmodTask$ResourceBundleHelper.<clinit>(JmodTask.java:1559)
	... 4 more
Caused by: java.lang.ClassNotFoundException: jdk.tools.jmod.resources.spi.jmodProvider
	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581)
	at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
	at java.base/java.lang.Class.forName0(Native Method)
	at java.base/java.lang.Class.forName(Class.java:398)
	at java.base/java.util.ResourceBundle$3.run(ResourceBundle.java:1918)
	at java.base/java.util.ResourceBundle$3.run(ResourceBundle.java:1914)
	... 18 more
	at jdk.jlink/jdk.tools.jmod.JmodTask.getMessage(JmodTask.java:1547)
	at jdk.jlink/jdk.tools.jmod.JmodTask.handleOptions(JmodTask.java:1284)
	at jdk.jlink/jdk.tools.jmod.JmodTask.run(JmodTask.java:191)
	at jdk.jlink/jdk.tools.jmod.Main.main(Main.java:34)
Caused by: java.security.PrivilegedActionException: java.lang.ClassNotFoundException: jdk.tools.jmod.resources.spi.jmodProvider
	at java.base/java.security.AccessController.doPrivileged(Native Method)
	at java.base/java.util.ResourceBundle.getResourceBundleProviderType(ResourceBundle.java:1913)
	at java.base/java.util.ResourceBundle.getServiceLoader(ResourceBundle.java:1880)
	at java.base/java.util.ResourceBundle$CacheKey.getProviders(ResourceBundle.java:699)
	at java.base/java.util.ResourceBundle$CacheKey.hasProviders(ResourceBundle.java:706)
	at java.base/java.util.ResourceBundle.loadBundle(ResourceBundle.java:1809)
	at java.base/java.util.ResourceBundle.findBundle(ResourceBundle.java:1774)
	at java.base/java.util.ResourceBundle.findBundle(ResourceBundle.java:1728)
	at java.base/java.util.ResourceBundle.findBundle(ResourceBundle.java:1728)
	at java.base/java.util.ResourceBundle.getBundleImpl(ResourceBundle.java:1662)
	at java.base/java.util.ResourceBundle.getBundleImpl(ResourceBundle.java:1582)
	at java.base/java.util.ResourceBundle.getBundleImpl(ResourceBundle.java:1556)
	at java.base/java.util.ResourceBundle.getBundle(ResourceBundle.java:932)
	at jdk.jlink/jdk.tools.jmod.JmodTask$ResourceBundleHelper.<clinit>(JmodTask.java:1559)
	... 4 more
Caused by: java.lang.ClassNotFoundException: jdk.tools.jmod.resources.spi.jmodProvider
	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581)
	at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
	at java.base/java.lang.Class.forName0(Native Method)
	at java.base/java.lang.Class.forName(Class.java:398)
	at java.base/java.util.ResourceBundle$3.run(ResourceBundle.java:1918)
	at java.base/java.util.ResourceBundle$3.run(ResourceBundle.java:1914)
	... 18 more
+ exitcode=1
Comment 5 Ben Cotton 2020-02-11 16:38:08 UTC 
This bug appears to have been reported against 'rawhide' during the Fedora 32 development cycle.
Changing version to 32.
Comment 6 Severin Gehwolf 2020-02-12 19:21:53 UTC 
slowdebug build (no opt) works.

$ uname -m
s390x
$  ./images/jdk/bin/java -version
openjdk version "11.0.6" 2020-01-14
OpenJDK Runtime Environment 18.9 (slowdebug build 11.0.6+10)
OpenJDK 64-Bit Server VM 18.9 (slowdebug build 11.0.6+10, mixed mode)
Comment 7 Severin Gehwolf 2020-02-13 11:42:22 UTC 
Interestingly enough, fastdebug build works too:

$ uname -m
s390x
$ ./build/images/jdk/bin/java -version
openjdk version "11.0.6" 2020-01-14
OpenJDK Runtime Environment 18.9 (fastdebug build 11.0.6+10)
OpenJDK 64-Bit Server VM 18.9 (fastdebug build 11.0.6+10, mixed mode)
Comment 8 Severin Gehwolf 2020-02-13 14:54:09 UTC 
What do we know so far?

- slowdebug/fastdebug builds work fine on s390x
- release build fails on first invocation of jmod when
  trying to create interim jmods.
  "jmod --help" is actually sufficient to reproduce.

Using a libjvm.so from a fastdebug build in a partial 'release' build fixes the problem.

Thus, the problem seems to be in libjvm.so of a release build.
Comment 9 Severin Gehwolf 2020-02-17 19:59:03 UTC 
Like JDK 8, compiling with these options "fixes" the build:

-fno-tree-coalesce-vars
-fno-tree-copy-prop
-fno-tree-dce
-fno-tree-dominator-opts
-fno-tree-dse
-fno-tree-forwprop
-fno-tree-fre
-fno-tree-loop-distribute-patterns
-fno-tree-loop-distribution
-fno-tree-loop-vectorize
-fno-tree-partial-pre
-fno-tree-phiprop
-fno-tree-pre
-fno-tree-pta
-fno-tree-scev-cprop
-fno-tree-sink
-fno-tree-slp-vectorize
-fno-tree-slsr
-fno-tree-sra
-fno-tree-switch-conversion
-fno-tree-tail-merge
-fno-tree-ter
-fno-tree-vrp
-fno-unit-at-a-time
-fno-unswitch-loops
-fno-vect-cost-model
-fno-version-loops-for-strides

It also seems that the issue is in the C1 compiler. -Xint works, -XX:-TieredCompilation too.

One of the objects getting compiled badly is c1_Canonicalizer.o (it's not the only one, though).

Determined with:
https://github.com/jerboaa/hotspot-tools-find-bad-object/commit/08e9c374ecf6629fb944c1840d37d4879d50c994
Comment 10 Severin Gehwolf 2020-02-26 18:17:19 UTC 
PR to get java-11-openjdk building again. This is really ugly and I don't like it :-/ Anyway:

https://src.fedoraproject.org/rpms/java-11-openjdk/pull-request/68
Comment 11 Severin Gehwolf 2020-02-27 13:54:42 UTC 
Built in rawhide, but I'll keep this bug open for root-cause analysis.

https://koji.fedoraproject.org/koji/buildinfo?buildID=1471503
Comment 12 Dan Horák 2020-03-12 14:39:25 UTC 
Do you need any help from the IBM s390x experts?
Comment 13 Severin Gehwolf 2020-03-12 14:57:17 UTC 
(In reply to Dan Horák from comment #12)
> Do you need any help from the IBM s390x experts?

Sure! Would love to figure out where things go wrong. Whether it's GCC 10 or some hotspot UB. All I know is in the comments of this bug. We currently work around by disabling a bunch of optimizations, which isn't great.
Comment 14 IBM Bug Proxy 2020-03-12 15:32:41 UTC 
------- Comment From Andreas.Krebbel.com 2020-03-12 11:20 EDT-------
Jakub recently fixed a nasty bug in combine which triggered miscompiles on S/390 (e.g. git). Was that patch already part of your GCC when you tried to build the package?

------- Comment From Andreas.Krebbel.com 2020-03-12 11:28 EDT-------
This was the patch from Jakub:

commit 73dc4ae47418aef2eb470b8f71cef57dce37349e
Author: Jakub Jelinek <jakub>
Date:   Tue Feb 25 13:56:47 2020 +0100

combine: Fix find_split_point handling of constant store into ZERO_EXTRACT [PR93908]
Comment 15 Severin Gehwolf 2020-03-12 17:02:55 UTC 
(In reply to IBM Bug Proxy from comment #14)
> ------- Comment From Andreas.Krebbel.com 2020-03-12 11:20 EDT-------
> Jakub recently fixed a nasty bug in combine which triggered miscompiles on
> S/390 (e.g. git). Was that patch already part of your GCC when you tried to
> build the package?

Thanks! When I looked at the gcc builds today in koji this bugfix caught my eye. Unfortunately, our builds were done before that fix went in. I'll try again today without the disabled OPT flags on s390x and see how far we get. Will report back here once I have that going.
Comment 16 Severin Gehwolf 2020-03-13 17:34:44 UTC 
Here is a scratch build with the GCC flags for s390x reverted. Took longer than expected since make 4.3 landed since in rawhide which needed another patch.

https://koji.fedoraproject.org/koji/taskinfo?taskID=42450266

I'm seeing this in root.log for s390x which is the GCC version which should have the fix above:

DEBUG util.py:598:   libgcc                     s390x  10.0.1-0.9.fc33                  build  72 k

We'll see how that goes.

Built from this branch:
https://src.fedoraproject.org/fork/jerboaa/rpms/java-11-openjdk/commits/gcc_10_fixes_new
Comment 17 Severin Gehwolf 2020-03-24 13:06:56 UTC 
(In reply to Severin Gehwolf from comment #16)
> Here is a scratch build with the GCC flags for s390x reverted. Took longer
> than expected since make 4.3 landed since in rawhide which needed another
> patch.
> 
> https://koji.fedoraproject.org/koji/taskinfo?taskID=42450266

OK that one passed. It looks like this is a duplicate of bug 1799408. I'll prepare a pull request removing the workaround.
Comment 18 Severin Gehwolf 2020-03-24 13:19:32 UTC 
https://src.fedoraproject.org/rpms/java-11-openjdk/pull-request/72
Comment 19 Severin Gehwolf 2020-03-25 19:25:17 UTC 
Fixed in rawhide. Marking as duplicate of 1799408 as the actual fix was a GCC 10 bugfix.

*** This bug has been marked as a duplicate of bug 1799408 ***