Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.

Bug 1208369

Summary: java-1.8.0-openjdk: Memory leak
Product: [Fedora] Fedora Reporter: Mikolaj Izdebski <mizdebsk>
Component: java-1.8.0-openjdkAssignee: Severin Gehwolf <sgehwolf>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: rawhideCC: ahughes, dbhole, jerboaa, jvanek, mizdebsk, omajid, puntogil, sgehwolf
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Unspecified   
Whiteboard:
Fixed In Version: java-1.8.0-openjdk-1.8.0.45-36.b13.fc22 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-05-03 17:23:15 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1208319    
Attachments:
Description Flags
hs_err log
none
gdb commands file in order to get to the error condition
none
Proposed patch for working around type.o miscompilation.
none
Proposed patch fixing signed integer overflow in type.cpp none

Description Mikolaj Izdebski 2015-04-02 06:14:47 UTC
Description of problem:
Latest OpenJDK in rawhide has OOM problem when building maven-scm.

Version-Release number of selected component (if applicable):
java-1.8.0-openjdk-devel-1.8.0.40-26.b25.fc23.x86_64
not reproducible with java-1.8.0-openjdk-devel-1.8.0.40-19.b12.fc22.x86_64

Steps to Reproduce:
1. install java-1.8.0-openjdk-devel-1.8.0.40-26.b25.fc23.x86_64 in rawhide
2. wget http://archive.apache.org/dist/maven/maven-3/3.3.1/binaries/apache-maven-3.3.1-bin.tar.gz
3. tar xf apache-maven-3.3.1-bin.tar.gz
4. wget http://repo1.maven.org/maven2/org/apache/maven/scm/maven-scm/1.9.4/maven-scm-1.9.4-source-release.zip
5. unzip maven-scm-1.9.4-source-release.zip
6. cd ./maven-scm-1.9.4/
7. ulimit -Sv 20000000
8. \time -pv ../apache-maven-3.3.1/bin/mvn -V verify

Actual results:
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (malloc) failed to allocate 32744 bytes for ChunkPool::allocate

Expected results:
Build succeeds.

Comment 1 Mikolaj Izdebski 2015-04-02 06:24:22 UTC
This is urgent - it breaks large number of Fedora Java builds [1] and makes Java applications unusable.

[1] http://koschei.cloud.fedoraproject.org/groups/java?order_by=state%2Cname

Comment 2 Severin Gehwolf 2015-04-02 08:10:50 UTC
These two builds are the first OpenJDK builds compiled with GCC 5: java-1.8.0-openjdk-devel-1.8.0.40-26.b25.fc2{2,3}.x86_64

Comment 3 Severin Gehwolf 2015-04-02 08:22:56 UTC
(In reply to Severin Gehwolf from comment #2)
> These two builds are the first OpenJDK builds compiled with GCC 5:
> java-1.8.0-openjdk-devel-1.8.0.40-26.b25.fc2{2,3}.x86_64

Both landed yesterday, since java-1.8.0-openjdk was FTBFS due to bug 1206656 until yesterday.

Comment 4 Severin Gehwolf 2015-04-02 08:59:37 UTC
According to [1] F22 has old ABI. F23 breaks this.

[1] http://fedoramagazine.org/gcc-5-in-fedora-whats-an-abi-and-what-happens-when-we-change-it/

Comment 5 gil cattaneo 2015-04-02 09:41:15 UTC
Created attachment 1010090 [details]
hs_err log

Comment 6 jiri vanek 2015-04-02 09:55:39 UTC
Few more observations. After yesterday build - 26.f22/f23 after long pause  because of (arm32 bug 1206656)  they do not build itself and are causing this OOM

Note
packages -26.fc22 fails on both f21 and f23 (no rebuild, just install)
Also 25.fc21 works ok on on f21 but  on f23 it fails.

Comment 7 Severin Gehwolf 2015-04-02 10:24:44 UTC
Failed tasks of java-1.8.0-openjdk trying to build itself with a java-1.8.0-openjdk boot JDK compiled with GCC 5:

F22:     https://koji.fedoraproject.org/koji/taskinfo?taskID=9398527
rawhide: https://koji.fedoraproject.org/koji/taskinfo?taskID=9398358

Comment 8 Severin Gehwolf 2015-04-02 11:05:27 UTC
This seems to be a problem of java-1.8.0-openjdk built with GCC 5. I've reproduced this issue on F22 too. Offending builds have been untagged.

$ koji latest-build f23-build java-1.8.0-openjdk
Build                                     Tag                   Built by
----------------------------------------  --------------------  ----------------
java-1.8.0-openjdk-1.8.0.40-19.b12.fc22   f23                   sharkcz
$ koji latest-build f22-build java-1.8.0-openjdk
Build                                     Tag                   Built by
----------------------------------------  --------------------  ----------------
java-1.8.0-openjdk-1.8.0.40-19.b12.fc22   f22                   sharkcz

java-1.8.0-openjdk-1.8.0.40-19.b12.fc22 was the last one built with GCC 4.9.

Comment 9 Severin Gehwolf 2015-04-02 14:46:57 UTC
A smaller reproducer seems this (TestCrypoLevel.java comes from java-1.8.0-openjdk sources):

$ /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.40-26.b25.fc22.x86_64/bin/javac -d . TestCryptoLevel.java

Comment 10 Severin Gehwolf 2015-04-02 14:51:14 UTC
On my F22 VM which has 2GB of memory allocated I then see this:

$ \time -pv javac -d . TestCryptoLevel.java
	Command being timed: "javac -d . TestCryptoLevel.java"
	User time (seconds): 4.35
	System time (seconds): 3.67
	Percent of CPU this job got: 70%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 0:11.38
	Average shared text size (kbytes): 0
	Average unshared data size (kbytes): 0
	Average stack size (kbytes): 0
	Average total size (kbytes): 0
	Maximum resident set size (kbytes): 1874600
	Average resident set size (kbytes): 0
	Major (requiring I/O) page faults: 23780
	Minor (reclaiming a frame) page faults: 763237
	Voluntary context switches: 27466
	Involuntary context switches: 2841
	Swaps: 0
	File system inputs: 1212272
	File system outputs: 8
	Socket messages sent: 0
	Socket messages received: 0
	Signals delivered: 0
	Page size (bytes): 4096
	Exit status: 0

That javac is /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.40-26.b25.fc22.x86_64/bin/javac.

Note the astonishing number of 23780 major page faults and a max resident size of 1874600 KB, or about 1.8 GB of memory.

Comment 11 Severin Gehwolf 2015-04-02 14:55:57 UTC
Contrast this to the same run on F21 with javac from java-1.8.0-openjdk-devel-1.8.0.40-21.b25.fc21.x86_64

$ \time -pv javac -d . TestCryptoLevel.java
	Command being timed: "javac -d . TestCryptoLevel.java"
	User time (seconds): 0.92
	System time (seconds): 0.04
	Percent of CPU this job got: 229%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.42
	Average shared text size (kbytes): 0
	Average unshared data size (kbytes): 0
	Average stack size (kbytes): 0
	Average total size (kbytes): 0
	Maximum resident set size (kbytes): 62792
	Average resident set size (kbytes): 0
	Major (requiring I/O) page faults: 0
	Minor (reclaiming a frame) page faults: 18192
	Voluntary context switches: 1777
	Involuntary context switches: 28
	Swaps: 0
	File system inputs: 0
	File system outputs: 8
	Socket messages sent: 0
	Socket messages received: 0
	Signals delivered: 0
	Page size (bytes): 4096
	Exit status: 0

Comment 12 Mikolaj Izdebski 2015-04-02 14:58:44 UTC
(In reply to Severin Gehwolf from comment #10)
> Note the astonishing number of 23780 major page faults and a max resident
> size of 1874600 KB, or about 1.8 GB of memory.

On my system after running for 40 s it reached max RSS of 14 GiB, virtual memory allocation of 25 GiB:
 PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
8406 kojan     20   0 24.812g 0.014t   6256 S   1.4 90.0   0:41.37 java

These are typical symptoms of large memory leak.

Comment 13 Severin Gehwolf 2015-04-02 15:20:11 UTC
Running the same reproducer again on F22 with a GCC 4.9 produced build I get this.

$ rpm -q java-1.8.0-openjdk-devel
java-1.8.0-openjdk-devel-1.8.0.40-19.b12.fc22.x86_64

$ \time -pv javac -d . TestCryptoLevel.java
	Command being timed: "javac -d . TestCryptoLevel.java"
	User time (seconds): 0.69
	System time (seconds): 0.04
	Percent of CPU this job got: 158%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.46
	Average shared text size (kbytes): 0
	Average unshared data size (kbytes): 0
	Average stack size (kbytes): 0
	Average total size (kbytes): 0
	Maximum resident set size (kbytes): 58400
	Average resident set size (kbytes): 0
	Major (requiring I/O) page faults: 0
	Minor (reclaiming a frame) page faults: 15293
	Voluntary context switches: 820
	Involuntary context switches: 513
	Swaps: 0
	File system inputs: 1728
	File system outputs: 8
	Socket messages sent: 0
	Socket messages received: 0
	Signals delivered: 0
	Page size (bytes): 4096
	Exit status: 0

Comment 14 Severin Gehwolf 2015-04-02 15:21:08 UTC
(In reply to Mikolaj Izdebski from comment #12)
> (In reply to Severin Gehwolf from comment #10)
> > Note the astonishing number of 23780 major page faults and a max resident
> > size of 1874600 KB, or about 1.8 GB of memory.
> 
> On my system after running for 40 s it reached max RSS of 14 GiB, virtual
> memory allocation of 25 GiB:
>  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
> 8406 kojan     20   0 24.812g 0.014t   6256 S   1.4 90.0   0:41.37 java

This is running "javac -d . TestCryptoLevel.java" I take it?

Comment 15 Mikolaj Izdebski 2015-04-02 16:29:27 UTC
(In reply to Severin Gehwolf from comment #14)
> (In reply to Mikolaj Izdebski from comment #12)
> > (In reply to Severin Gehwolf from comment #10)
> > > Note the astonishing number of 23780 major page faults and a max resident
> > > size of 1874600 KB, or about 1.8 GB of memory.
> > 
> > On my system after running for 40 s it reached max RSS of 14 GiB, virtual
> > memory allocation of 25 GiB:
> >  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
> > 8406 kojan     20   0 24.812g 0.014t   6256 S   1.4 90.0   0:41.37 java
> 
> This is running "javac -d . TestCryptoLevel.java" I take it?

No, this was with my reproducer. Your reproducer doesn't cause OOM on my system as apparently I have more RAM. When trying to compile TestCryptoLevel.java javac succeeds on my rawhide system, but allocates over 25 GiB virtual memory (RSS 15 GiB):

$ ulimit -Sv 30000000

$ \time -pv schedtool -B -e nice javac -d . TestCryptoLevel.java
	Command being timed: "schedtool -B -e nice javac -d . TestCryptoLevel.java"
	User time (seconds): 27.33
	System time (seconds): 4.03
	Percent of CPU this job got: 230%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 0:13.59
	Average shared text size (kbytes): 0
	Average unshared data size (kbytes): 0
	Average stack size (kbytes): 0
	Average total size (kbytes): 0
	Maximum resident set size (kbytes): 14973324
	Average resident set size (kbytes): 0
	Major (requiring I/O) page faults: 26
	Minor (reclaiming a frame) page faults: 3758815
	Voluntary context switches: 98037
	Involuntary context switches: 1683
	Swaps: 0
	File system inputs: 5280
	File system outputs: 72
	Socket messages sent: 0
	Socket messages received: 0
	Signals delivered: 0
	Page size (bytes): 4096
	Exit status: 0

$ ls -go TestCryptoLevel*
-rw-rw-r--. 1 2013 Apr  2 18:20 TestCryptoLevel.class
-rw-rw-r--. 1 2477 Apr  2 18:18 TestCryptoLevel.java

$ rpm -qa|grep openjdk
java-1.8.0-openjdk-headless-1.8.0.40-26.b25.fc23.x86_64
java-1.8.0-openjdk-devel-1.8.0.40-26.b25.fc23.x86_64
java-1.8.0-openjdk-1.8.0.40-26.b25.fc23.x86_64

$ cat /etc/system-release
Fedora release 23 (Rawhide)

Comment 16 Severin Gehwolf 2015-04-02 16:58:28 UTC
Numbers for a local slowdebug rebuild look better: about 103MB vs 1.8GB of a release build. Also a lot less major page faults.

$ \time -pv javac -d . TestCryptoLevel.java 
	Command being timed: "javac -d . TestCryptoLevel.java"
	User time (seconds): 5.02
	System time (seconds): 0.05
	Percent of CPU this job got: 179%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 0:02.82
	Average shared text size (kbytes): 0
	Average unshared data size (kbytes): 0
	Average stack size (kbytes): 0
	Average total size (kbytes): 0
	Maximum resident set size (kbytes): 103880
	Average resident set size (kbytes): 0
	Major (requiring I/O) page faults: 0
	Minor (reclaiming a frame) page faults: 31590
	Voluntary context switches: 502
	Involuntary context switches: 560
	Swaps: 0
	File system inputs: 0
	File system outputs: 8
	Socket messages sent: 0
	Socket messages received: 0
	Signals delivered: 0
	Page size (bytes): 4096
	Exit status: 0

$ java -version
openjdk version "1.8.0_40-debug"
OpenJDK Runtime Environment (build 1.8.0_40-debug-b25)
OpenJDK 64-Bit Server VM (build 25.40-b25-debug, mixed mode)

Comment 17 Omair Majid 2015-04-02 18:46:56 UTC
I built the fastdebug variant and javac is reproducibly crashing:

$ /home/omajid/java-1.8.0-openjdk/java-1.8.0-openjdk-1.8.0.40-27.b25.fc22.x86_64/jdk8/build/jdk8.build/images/j2sdk-image/bin/java -version
openjdk version "1.8.0_40-fastdebug"
OpenJDK Runtime Environment (build 1.8.0_40-fastdebug-b25)
OpenJDK 64-Bit Server VM (build 25.40-b25-fastdebug, mixed mode)
$ /home/omajid/java-1.8.0-openjdk/java-1.8.0-openjdk-1.8.0.40-27.b25.fc22.x86_64/jdk8/build/jdk8.build/images/j2sdk-image/bin/javac -d . /home/omajid/java-1.8.0-openjdk/TestCryptoLevel.java
# To suppress the following error report, specify this argument
# after -XX: or in .hotspotrc:  SuppressErrorAt=/phaseX.cpp:1486
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  Internal Error (/home/omajid/java-1.8.0-openjdk/java-1.8.0-openjdk-1.8.0.40-27.b25.fc22.x86_64/jdk8/hotspot/src/share/vm/opto/phaseX.cpp:1486), pid=23668, tid=140306751637248
#  assert(t0->isa_int()->_widen <= t->isa_int()->_widen) failed: widen increases
#
# JRE version: OpenJDK Runtime Environment (8.0_40-b25) (build 1.8.0_40-fastdebug-b25)
# Java VM: OpenJDK 64-Bit Server VM (25.40-b25-fastdebug mixed mode linux-amd64 compressed oops)
# Core dump written. Default location: /home/omajid/java-1.8.0-openjdk/core or core.23668 (max size 100000000 kB). To ensure a full core dump, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /home/omajid/java-1.8.0-openjdk/hs_err_pid23668.log
#
# Compiler replay data is saved as:
# /home/omajid/java-1.8.0-openjdk/replay_pid23668.log
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
#
Current thread is 140306751637248
Dumping core ...
Aborted (core dumped)

Comment 18 Omair Majid 2015-04-02 18:49:07 UTC
Also, -Xint works, but -Xcomp fails:

$ /home/omajid/java-1.8.0-openjdk/java-1.8.0-openjdk-1.8.0.40-27.b25.fc22.x86_64/jdk8/build/jdk8.build/images/j2sdk-image/bin/javac -J-Xint -d . /home/omajid/java-1.8.0-openjdk/TestCryptoLevel.java

$ /home/omajid/java-1.8.0-openjdk/java-1.8.0-openjdk-1.8.0.40-27.b25.fc22.x86_64/jdk8/build/jdk8.build/images/j2sdk-image/bin/javac -J-Xcomp -d . /home/omajid/java-1.8.0-openjdk/TestCryptoLevel.java
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (malloc) failed to allocate 1342177312 bytes for Chunk::new
# An error report file with more information is saved as:
# /home/omajid/java-1.8.0-openjdk/hs_err_pid23842.log
#
# Compiler replay data is saved as:
# /home/omajid/java-1.8.0-openjdk/replay_pid23842.log

Comment 19 Severin Gehwolf 2015-04-14 08:45:54 UTC
(In reply to Omair Majid from comment #18)
> Also, -Xint works, but -Xcomp fails:
> 
> $
> /home/omajid/java-1.8.0-openjdk/java-1.8.0-openjdk-1.8.0.40-27.b25.fc22.
> x86_64/jdk8/build/jdk8.build/images/j2sdk-image/bin/javac -J-Xint -d .
> /home/omajid/java-1.8.0-openjdk/TestCryptoLevel.java
> 
> $
> /home/omajid/java-1.8.0-openjdk/java-1.8.0-openjdk-1.8.0.40-27.b25.fc22.
> x86_64/jdk8/build/jdk8.build/images/j2sdk-image/bin/javac -J-Xcomp -d .
> /home/omajid/java-1.8.0-openjdk/TestCryptoLevel.java
> #
> # There is insufficient memory for the Java Runtime Environment to continue.
> # Native memory allocation (malloc) failed to allocate 1342177312 bytes for
> Chunk::new
> # An error report file with more information is saved as:
> # /home/omajid/java-1.8.0-openjdk/hs_err_pid23842.log
> #
> # Compiler replay data is saved as:
> # /home/omajid/java-1.8.0-openjdk/replay_pid23842.log

Reproduced, thanks. So this is starting to look like a JIT bug caused by GCC 5. FWIW, the plain java command line is:

----------------------------

JDK=/path/to/j2sdk-image
$JDK/bin/java \
 -classpath $JDK/lib/tools.jar:$JDK/classes \
 -Dapplication.home=$JDK \
 -Xms8m \
 com.sun.tools.javac.Main -d . TestCryptoLevel.java

----------------------------

Comment 20 Severin Gehwolf 2015-04-14 15:12:42 UTC
Something is going wrong when java.lang.String::hashCode is getting compiled with C2. Reproducer on a F22 fastdebug build is:

JDK=/path/to/j2sdk-image
$JDK/bin/java \
 -classpath $JDK/lib/tools.jar:$JDK/classes \
 -Dapplication.home=$JDK \
 -Xms8m \
 -XX:+PrintCompilation \
 -XX:-TieredCompilation \
 -XX:CICompilerCount=1 \
 com.sun.tools.javac.Main -d . TestCryptoLevel.java

Comment 21 Severin Gehwolf 2015-04-14 15:15:46 UTC
With -XX:+TracePhaseCCP turned on as well and using the reproducer in comment 20 I get this output until the assertion triggers:

     83    1             java.lang.String::hashCode (55 bytes)
bottom		 0	Root	===  0  52  103  116  189  [[ 0  1  3  22  23  35  39  47  91  98  111  139  127  190  213  309  311  313  316 ]] 
{0:control, 1:abIO, 2:memory, 3:rawptr:BotPTR, 4:return_address, 5:java/lang/String:NotNull:exact *}	 3	Start	===  3  0  [[ 3  5  6  7  8  9  10 ]]  #{0:control, 1:abIO, 2:memory, 3:rawptr:BotPTR, 4:return_address, 5:java/lang/String:NotNull:exact *}
long:16		 22	ConL	===  0  [[ 24  287  224  251 ]]  #long:16
int:0		 23	ConI	===  0  [[ 26  56  258  63  261  112  99  112  99  264  215 ]]  #int:0
long:12		 35	ConL	===  0  [[ 36  54 ]]  #long:12
NULL		 39	ConP	===  0  [[ 40  48 ]]  #NULL
int:-10		 47	ConI	===  0  [[ 48 ]]  #int:-10
int:1		 91	ConI	===  0  [[ 92  105  312  291  228  229  255  256  281  292 ]]  #int:1
int:-122	 98	ConI	===  0  [[ 99 ]]  #int:-122
int:-130	 111	ConI	===  0  [[ 112 ]]  #int:-130
int:-28		 139	ConI	===  0  [[]]  #int:-28
int:5		 127	ConI	===  0  [[ 194  221  248  284 ]]  #int:5
int:0		 190	ConI	===  0  [[]]  #int:0
int:-1		 213	ConI	===  0  [[ 214  306 ]]  #int:-1
long:1		 309	ConL	===  0  [[]]  #long:1
int:2		 311	ConI	===  0  [[ 196 ]]  #int:2
long:2		 313	ConL	===  0  [[]]  #long:2
control		 5	Parm	===  3  [[ 28 ]] Control !jvms: String::hashCode @ bci:-1
abIO		 6	Parm	===  3  [[ 112  48  189  99 ]] I_O !orig=[87] !jvms: String::hashCode @ bci:-1
memory		 7	Parm	===  3  [[ 187  187  99  16  25  150  37  112  48  55  184  223  250  286 ]] Memory  Memory: @BotPTR *+bot, idx=Bot; !orig=[88],[120] !jvms: String::hashCode @ bci:-1
rawptr:BotPTR	 8	Parm	===  3  [[ 189  103  99  112  116  52  48 ]] FramePtr !jvms: String::hashCode @ bci:-1
return_address	 9	Parm	===  3  [[ 189  48  99  112 ]] ReturnAdr !jvms: String::hashCode @ bci:-1
long:18		 316	ConL	===  0  [[ 148 ]]  #long:18
java/lang/String:NotNull:exact *	 10	Parm	===  3  [[ 36  36  112  24  24  99 ]] Parm0: java/lang/String:NotNull:exact *  Oop:java/lang/String:NotNull:exact * !jvms: String::hashCode @ bci:-1
int		 281	Opaque1	=== _  91  55  [[ 253 ]] 
{0:top, 1:top}	 28	If	===  5  27  [[ 29  30 ]] P=0.336000, C=250.000000 !jvms: String::hashCode @ bci:6
int		 92	Opaque1	=== _  91  [[ 93 ]]  !jvms: String::hashCode @ bci:33
java/lang/String:NotNull:exact+16 *	 24	AddP	=== _  10  10  22  [[ 25  184 ]]   Oop:java/lang/String:NotNull:exact+16 * !jvms: String::hashCode @ bci:1
java/lang/String:NotNull:exact+12 * [narrow]	 36	AddP	=== _  10  10  35  [[ 37 ]]   Oop:java/lang/String:NotNull:exact+12 * [narrow] !jvms: String::hashCode @ bci:10
int		 105	Opaque1	=== _  91  [[ 106 ]]  !jvms: String::hashCode @ bci:33
bool		 93	Conv2B	=== _  92  [[ 94 ]]  !jvms: String::hashCode @ bci:33
narrowoop: char[int:>=0]:exact *	 37	LoadN	=== _  7  36  [[ 38 ]]  @java/lang/String:exact+12 * [narrow], name=value, idx=5; #narrowoop: char[int:>=0]:exact * !orig=[156] !jvms: String::hashCode @ bci:10
int		 25	LoadI	=== _  7  24  [[ 63  26 ]]  @java/lang/String:exact+16 *, name=hash, idx=4; #int !jvms: String::hashCode @ bci:1
int:-1..1	 26	CmpI	=== _  25  23  [[ 27 ]]  !jvms: String::hashCode @ bci:6
char[int:>=0]:exact *	 38	DecodeN	=== _  37  [[ 53  40 ]]  #char[int:>=0]:exact * !orig=[157] !jvms: String::hashCode @ bci:10
bool		 27	Bool	=== _  26  [[ 28 ]] [ne] !jvms: String::hashCode @ bci:6
memory		 16	MergeMem	=== _  1  7  1  187  [[ 189 ]]  { - N187:java/lang/String:exact+16 * }  Memory: @BotPTR *+bot, idx=Bot;
int:-1..1	 40	CmpP	=== _  38  39  [[ 41 ]]  !orig=[158] !jvms: String::hashCode @ bci:13
{0:control, 1:control}	 28	If	===  5  27  [[ 29  30 ]] P=0.336000, C=250.000000 !jvms: String::hashCode @ bci:6
control		 29	IfTrue	===  28  [[ 33 ]] #1 !jvms: String::hashCode @ bci:6
control		 33	Region	===  33  239  59  29  [[ 33  189  63  187 ]]  !jvms: String::hashCode @ bci:53
bottom		 189	Return	===  33  6  16  8  9 returns 63  [[ 0 ]] 
bool		 106	Conv2B	=== _  105  [[ 107 ]]  !jvms: String::hashCode @ bci:33
bool		 41	Bool	=== _  40  [[ 44 ]] [ne] !orig=[159] !jvms: String::hashCode @ bci:13
control		 30	IfFalse	===  28  [[ 44 ]] #0 !jvms: String::hashCode @ bci:6
memory		 187	Phi	===  33  184  7  7  [[ 16 ]]  #memory  Memory: @java/lang/String:exact+16 *, name=hash, idx=4; !orig=[186] !jvms: String::hashCode @ bci:53
int		 63	Phi	===  33  240  23  25  [[ 189 ]]  #int !jvms: String::hashCode @ bci:14
{0:control, 1:control}	 44	If	===  30  41  [[ 45  46 ]] P=0.999999, C=-1.000000 !jvms: String::hashCode @ bci:13
control		 45	IfTrue	===  44  [[ 58  53 ]] #1 !jvms: String::hashCode @ bci:13
{0:top, 1:top}	 58	If	===  45  57  [[ 59  60 ]] P=0.289157, C=166.000000 !jvms: String::hashCode @ bci:14
control		 46	IfFalse	===  44  [[ 48 ]] #0 !jvms: String::hashCode @ bci:13
char[int:>=0]:NotNull:exact *	 53	CastPP	===  45  38  [[ 54  54  315  315  148  112  99  224  227  227  251  254  254  287  290  290 ]]  #char[int:>=0]:NotNull:exact * !orig=[170] !jvms: String::hashCode @ bci:13
char[int:>=0]:NotNull:exact+12 *	 54	AddP	=== _  53  53  35  [[ 55 ]]  !orig=[171] !jvms: String::hashCode @ bci:13
{0:control, 1:abIO, 2:memory, 3:rawptr:BotPTR, 4:return_address}	 48	CallStaticJava	===  46  6  7  8  9 ( 47  1  1  1  1  39 ) [[ 49 ]] # Static uncommon_trap(reason='null_check' action='maybe_recompile')  void ( int ) C=0.000100 String::hashCode @ bci:13 !jvms: String::hashCode @ bci:13
int:>=0		 55	LoadRange	=== _  7  54  [[ 56  211  282  215  214  226  244  281 ]]  @bottom[int:>=0]+12 * [narrow], idx=6; #int:>=0 !orig=[172] !jvms: String::hashCode @ bci:13
bool		 56	CmpI	=== _  55  23  [[ 57 ]]  !jvms: String::hashCode @ bci:14
int		 282	Opaque2	=== _  55  [[ 306 ]] 
bool		 215	CmpU	=== _  55  23  [[ 217 ]]  !orig=[208]
int:-1..max-1	 214	AddI	=== _  55  213  [[ 211 ]]  !orig=[210]
int:-1..1	 211	CmpU	=== _  214  55  [[ 212 ]] 
control		 49	Proj	===  48  [[ 52 ]] #0 !jvms: String::hashCode @ bci:13
bool		 212	Bool	=== _  211  [[ 205 ]] [lt]
bool		 217	Bool	=== _  215  [[ 218 ]] [le]
int		 306	AddI	=== _  282  213  [[ 273  197 ]]  !orig=[283]
bottom		 52	Halt	===  49  1  1  8  1  [[ 0 ]]  !jvms: String::hashCode @ bci:13
bool		 57	Bool	=== _  56  [[ 58 ]] [le] !jvms: String::hashCode @ bci:14
int		 273	Opaque1	=== _  306  [[ 274 ]] 
{0:control, 1:control}	 58	If	===  45  57  [[ 59  60 ]] P=0.289157, C=166.000000 !jvms: String::hashCode @ bci:14
control		 59	IfTrue	===  58  [[ 33 ]] #1 !jvms: String::hashCode @ bci:14
control		 60	IfFalse	===  58  [[ 218 ]] #0 !jvms: String::hashCode @ bci:14
{0:control, 1:control}	 218	If	===  60  217  [[ 219  220 ]] P=0.000001, C=-1.000000 !orig=[202],94 !jvms: String::hashCode @ bci:33
control		 219	IfFalse	===  218  [[ 205 ]] #0 !orig=[203]
{0:control, 1:control}	 205	If	===  219  212  [[ 206  207 ]] P=0.999999, C=-1.000000 !orig=94 !jvms: String::hashCode @ bci:33
control		 207	IfFalse	===  205  [[ 201 ]] #0
control		 220	IfTrue	===  218  [[ 201 ]] #1 !orig=[204]
control		 206	IfTrue	===  205  [[ 94  150  223  250  286 ]] #1
{0:control, 1:control}	 94	If	===  206  93  [[ 95  104 ]] P=0.999999, C=-1.000000 !jvms: String::hashCode @ bci:33
control		 201	Region	===  201  95  220  207  [[ 201  99 ]] 
{0:control, 1:abIO, 2:memory, 3:rawptr:BotPTR, 4:return_address}	 99	CallStaticJava	===  201  6  7  8  9 ( 98  10  23  53  23 ) [[ 100 ]] # Static uncommon_trap(reason='predicate' action='maybe_recompile')  void ( int ) C=0.000100 String::hashCode @ bci:33 !jvms: String::hashCode @ bci:33
control		 104	IfTrue	===  94  [[ 107 ]] #1 !jvms: String::hashCode @ bci:33
control		 95	IfFalse	===  94  [[ 201 ]] #0 !jvms: String::hashCode @ bci:33
{0:control, 1:control}	 107	If	===  104  106  [[ 108  117 ]] P=0.999999, C=-1.000000 !jvms: String::hashCode @ bci:33
control		 108	IfFalse	===  107  [[ 112 ]] #0 !jvms: String::hashCode @ bci:33
control		 100	Proj	===  99  [[ 103 ]] #0 !jvms: String::hashCode @ bci:33
control		 117	IfTrue	===  107  [[ 259 ]] #1 !jvms: String::hashCode @ bci:33
control		 259	CountedLoop	===  259  117  263  [[ 258  259  261  262  264 ]] inner stride: 1 pre of N200 !orig=[200],[191],[118] !jvms: String::hashCode @ bci:33
int:0		 261	Phi	===  259  23  249  [[ 260 ]]  #int !orig=121 !jvms: String::hashCode @ bci:33
{0:control, 1:abIO, 2:memory, 3:rawptr:BotPTR, 4:return_address}	 112	CallStaticJava	===  108  6  7  8  9 ( 111  10  23  53  23 ) [[ 113 ]] # Static uncommon_trap(reason='loop_limit_check' action='maybe_recompile')  void ( int ) C=0.000100 String::hashCode @ bci:33 !jvms: String::hashCode @ bci:33
bottom		 103	Halt	===  100  1  1  8  1  [[ 0 ]]  !jvms: String::hashCode @ bci:33
int:0		 264	Phi	===  259  23  248  [[ 260 ]]  #int !orig=192,[128] !jvms: String::hashCode @ bci:36
int:0		 260	SubI	=== _  264  261  [[ 249 ]]  !orig=129 !jvms: String::hashCode @ bci:36
int:0		 258	Phi	===  259  23  256  [[ 256  257 ]]  #int #tripcount !orig=[122] !jvms: String::hashCode @ bci:33
control		 113	Proj	===  112  [[ 116 ]] #0 !jvms: String::hashCode @ bci:33
{0:top, 1:top}	 262	CountedLoopEnd	===  259  252  [[ 263  272 ]] [lt] P=0.500000, C=8524.000000 !orig=199,[175] !jvms: String::hashCode @ bci:30
int:1		 256	AddI	=== _  258  91  [[ 253  258  270  274  279 ]]  !orig=196,[152] !jvms: String::hashCode @ bci:42
int:-1..1	 253	CmpI	=== _  256  281  [[ 252 ]]  !orig=197,[173] !jvms: String::hashCode @ bci:30
long:0		 257	ConvI2L	=== _  258  [[ 255 ]]  #long:0..maxint-1 !orig=[146] !jvms: String::hashCode @ bci:39
bottom		 116	Halt	===  113  1  1  8  1  [[ 0 ]]  !jvms: String::hashCode @ bci:33
bool		 252	Bool	=== _  253  [[ 262 ]] [lt] !orig=198,[174] !jvms: String::hashCode @ bci:30
int:-1..1	 274	CmpI	=== _  256  273  [[ 275 ]] 
long:0		 255	LShiftL	=== _  257  91  [[ 254 ]]  !orig=[147] !jvms: String::hashCode @ bci:39
char[int:>=0]:NotNull:exact *	 254	AddP	=== _  53  53  255  [[ 251 ]]  !orig=[149] !jvms: String::hashCode @ bci:39
{0:control, 1:control}	 262	CountedLoopEnd	===  259  252  [[ 263  272 ]] [lt] P=0.500000, C=8524.000000 !orig=199,[175] !jvms: String::hashCode @ bci:30
control		 263	IfTrue	===  262  [[ 259 ]] #1 !orig=176 !jvms: String::hashCode @ bci:30
bool		 258	Phi	===  259  23  256  [[ 256  257 ]]  #int #tripcount !orig=[122] !jvms: String::hashCode @ bci:33
int:1..2	 256	AddI	=== _  258  91  [[ 253  258  270  274  279 ]]  !orig=196,[152] !jvms: String::hashCode @ bci:42
char[int:>=0]:NotNull:exact[0] *	 251	AddP	=== _  53  254  22  [[ 250 ]]  !orig=148 !jvms: String::hashCode @ bci:39
bool		 275	Bool	=== _  274  [[ 276 ]] [lt]
control		 272	IfFalse	===  262  [[ 276 ]] #0
long:0..1	 257	ConvI2L	=== _  258  [[ 255 ]]  #long:0..maxint-1 !orig=[146] !jvms: String::hashCode @ bci:39
char		 250	LoadUS	===  206  7  251  [[ 249 ]]  @char[int:>=0]:exact+any *, idx=7; #char !orig=150 !jvms: String::hashCode @ bci:39
{0:control, 1:control}	 276	If	===  272  275  [[ 265  277 ]] P=0.999999, C=-1.000000
control		 265	IfFalse	===  276  [[ 266 ]] #0 !orig=242
control		 266	Region	===  266  265  242  [[ 266  246  267  268  270 ]] 
{0:top, 1:top}	 246	If	===  266  245  [[ 177  247 ]] P=0.500000, C=-1.000000
long:0..2	 255	LShiftL	=== _  257  91  [[ 254 ]]  !orig=[147] !jvms: String::hashCode @ bci:39
int:0..2	 258	Phi	===  259  23  256  [[ 256  257 ]]  #int #tripcount !orig=[122] !jvms: String::hashCode @ bci:33
char		 249	AddI	=== _  260  250  [[ 248  261  268  297 ]]  !orig=151,[182] !jvms: String::hashCode @ bci:40
control		 277	IfTrue	===  276  [[ 295  279 ]] #1
int:1..2	 270	Phi	===  266  256  196  [[ 278  243 ]]  #int
char		 268	Phi	===  266  249  151  [[ 234  240 ]]  #int
int:1..3	 256	AddI	=== _  258  91  [[ 253  258  270  274  279 ]]  !orig=196,[152] !jvms: String::hashCode @ bci:42
int:0..2097120	 248	LShiftI	=== _  249  127  [[ 264  267  300 ]]  !orig=194,[128] !jvms: String::hashCode @ bci:36
char		 261	Phi	===  259  23  249  [[ 260 ]]  #int !orig=121 !jvms: String::hashCode @ bci:33
control		 295	CountedLoop	===  295  277  176  [[ 294  295  297  199  300 ]] inner stride: 2 main of N295 !orig=[200],[191],[118] !jvms: String::hashCode @ bci:33
{0:top, 1:top}	 199	CountedLoopEnd	===  295  198  [[ 242  176 ]] [lt] P=0.986232, C=8524.000000 !orig=[175] !jvms: String::hashCode @ bci:30
int:0..2097120	 300	Phi	===  295  248  194  [[ 296 ]]  #int !orig=192,[128] !jvms: String::hashCode @ bci:36
int:0..3	 258	Phi	===  259  23  256  [[ 256  257 ]]  #int #tripcount !orig=[122] !jvms: String::hashCode @ bci:33
char[int:>=0]:NotNull:exact+any *	 254	AddP	=== _  53  53  255  [[ 251 ]]  !orig=[149] !jvms: String::hashCode @ bci:39
long:0..3	 257	ConvI2L	=== _  258  [[ 255 ]]  #long:0..maxint-1 !orig=[146] !jvms: String::hashCode @ bci:39
char		 297	Phi	===  295  249  151  [[ 296 ]]  #int !orig=121 !jvms: String::hashCode @ bci:33
int:1..3	 279	CastII	===  277  256  [[ 294 ]]  #int:<=max-1 carry dependency
int		 243	Opaque1	=== _  270  [[ 244 ]] 
int:1..3	 270	Phi	===  266  256  196  [[ 278  243 ]]  #int
int:0..2097120	 267	Phi	===  266  248  194  [[ 237 ]]  #int
int:-65535..0	 260	SubI	=== _  264  261  [[ 249 ]]  !orig=129 !jvms: String::hashCode @ bci:36
int:0..2097120	 264	Phi	===  259  23  248  [[ 260 ]]  #int !orig=192,[128] !jvms: String::hashCode @ bci:36
int:-65535..2097120	 260	SubI	=== _  264  261  [[ 249 ]]  !orig=129 !jvms: String::hashCode @ bci:36
int:1..4	 256	AddI	=== _  258  91  [[ 253  258  270  274  279 ]]  !orig=196,[152] !jvms: String::hashCode @ bci:42
char[int:>=0]:NotNull:exact+any *	 251	AddP	=== _  53  254  22  [[ 250 ]]  !orig=148 !jvms: String::hashCode @ bci:39
long:0..6	 255	LShiftL	=== _  257  91  [[ 254 ]]  !orig=[147] !jvms: String::hashCode @ bci:39
int:-65535..2097120	 296	SubI	=== _  300  297  [[ 285 ]]  !orig=129 !jvms: String::hashCode @ bci:36
int:-1..1	 244	CmpI	=== _  243  55  [[ 245 ]] 
int:1..3	 294	Phi	===  295  279  196  [[ 292  293  307  196 ]]  #int #tripcount !orig=[122] !jvms: String::hashCode @ bci:33
int:2..4	 292	AddI	=== _  294  91  [[]]  !orig=196,[152] !jvms: String::hashCode @ bci:42
long:1..3	 293	ConvI2L	=== _  294  [[ 291 ]]  #long:0..maxint-1 !orig=[146] !jvms: String::hashCode @ bci:39
long:2..6	 291	LShiftL	=== _  293  91  [[ 290 ]]  !orig=[147] !jvms: String::hashCode @ bci:39
int:-65535..2162655	 249	AddI	=== _  260  250  [[ 248  261  268  297 ]]  !orig=151,[182] !jvms: String::hashCode @ bci:40
int:1..4	 279	CastII	===  277  256  [[ 294 ]]  #int:<=max-1 carry dependency
int:1..4	 270	Phi	===  266  256  196  [[ 278  243 ]]  #int
int:0..4:w	 258	Phi	===  259  23  256  [[ 256  257 ]]  #int #tripcount !orig=[122] !jvms: String::hashCode @ bci:33
bool		 245	Bool	=== _  244  [[ 246 ]] [lt]
int:3..5	 196	AddI	=== _  294  311  [[ 294  197  270 ]]  !orig=[152] !jvms: String::hashCode @ bci:42
long:1..3	 307	ConvI2L	=== _  294  [[ 312 ]]  #long:-1..maxint-2
int:-1..1	 197	CmpI	=== _  196  306  [[ 198 ]]  !orig=[173] !jvms: String::hashCode @ bci:30
bool		 198	Bool	=== _  197  [[ 199 ]] [lt] !orig=[174] !jvms: String::hashCode @ bci:30
char[int:>=0]:NotNull:exact+any *	 290	AddP	=== _  53  53  291  [[ 287 ]]  !orig=[149] !jvms: String::hashCode @ bci:39
int:-65535..2162655:w	 297	Phi	===  295  249  151  [[ 296 ]]  #int !orig=121 !jvms: String::hashCode @ bci:33
int:1..4	 294	Phi	===  295  279  196  [[ 292  293  307  196 ]]  #int #tripcount !orig=[122] !jvms: String::hashCode @ bci:33
int:-65535..2162655:w	 268	Phi	===  266  249  151  [[ 234  240 ]]  #int
long:0..4:w	 257	ConvI2L	=== _  258  [[ 255 ]]  #long:0..maxint-1 !orig=[146] !jvms: String::hashCode @ bci:39
int:1..5:w	 256	AddI	=== _  258  91  [[ 253  258  270  274  279 ]]  !orig=196,[152] !jvms: String::hashCode @ bci:42
int:-65535..2162655:w	 261	Phi	===  259  23  249  [[ 260 ]]  #int !orig=121 !jvms: String::hashCode @ bci:33
int:-2097120..69204960	 248	LShiftI	=== _  249  127  [[ 264  267  300 ]]  !orig=194,[128] !jvms: String::hashCode @ bci:36
{0:control, 1:control}	 246	If	===  266  245  [[ 177  247 ]] P=0.500000, C=-1.000000
int:1..5:w	 270	Phi	===  266  256  196  [[ 278  243 ]]  #int
long:2..6	 312	LShiftL	=== _  307  91  [[ 315 ]] 
int:2..5	 292	AddI	=== _  294  91  [[]]  !orig=196,[152] !jvms: String::hashCode @ bci:42
long:1..4	 293	ConvI2L	=== _  294  [[ 291 ]]  #long:0..maxint-1 !orig=[146] !jvms: String::hashCode @ bci:39
long:1..4	 307	ConvI2L	=== _  294  [[ 312 ]]  #long:-1..maxint-2
int:0..5:ww	 258	Phi	===  259  23  256  [[ 256  257 ]]  #int #tripcount !orig=[122] !jvms: String::hashCode @ bci:33
int:-2097120..69204960:w	 264	Phi	===  259  23  248  [[ 260 ]]  #int !orig=192,[128] !jvms: String::hashCode @ bci:36
int:1..6:ww	 256	AddI	=== _  258  91  [[ 253  258  270  274  279 ]]  !orig=196,[152] !jvms: String::hashCode @ bci:42
int:1..6:ww	 270	Phi	===  266  256  196  [[ 278  243 ]]  #int
{0:control, 1:control}	 199	CountedLoopEnd	===  295  198  [[ 242  176 ]] [lt] P=0.986232, C=8524.000000 !orig=[175] !jvms: String::hashCode @ bci:30
char[int:>=0]:NotNull:exact+any *	 287	AddP	=== _  53  290  22  [[ 286 ]]  !orig=148 !jvms: String::hashCode @ bci:39
int:-2162655..2162655:w	 296	SubI	=== _  300  297  [[ 285 ]]  !orig=129 !jvms: String::hashCode @ bci:36
int:3..6	 196	AddI	=== _  294  311  [[ 294  197  270 ]]  !orig=[152] !jvms: String::hashCode @ bci:42
long:0..8:w	 255	LShiftL	=== _  257  91  [[ 254 ]]  !orig=[147] !jvms: String::hashCode @ bci:39
int:1..6:ww	 279	CastII	===  277  256  [[ 294 ]]  #int:<=max-1 carry dependency
int:-4259775..69270495:w	 260	SubI	=== _  264  261  [[ 249 ]]  !orig=129 !jvms: String::hashCode @ bci:36
int:-2097120..69204960:w	 300	Phi	===  295  248  194  [[ 296 ]]  #int !orig=192,[128] !jvms: String::hashCode @ bci:36
control		 247	IfTrue	===  246  [[ 232  278 ]] #1
int:1..6:ww	 278	CastII	===  247  270  [[ 231 ]]  #int carry dependency
char[int:>=0]:NotNull:exact+any *	 315	AddP	=== _  53  53  312  [[ 148 ]] 
control		 177	IfFalse	===  246  [[ 239 ]] #0 !jvms: String::hashCode @ bci:30
long:2..8	 291	LShiftL	=== _  293  91  [[ 290 ]]  !orig=[147] !jvms: String::hashCode @ bci:39
long:2..8	 312	LShiftL	=== _  307  91  [[ 315 ]] 
int:-2097120..69204960:w	 267	Phi	===  266  248  194  [[ 237 ]]  #int
control		 232	CountedLoop	===  232  247  236  [[ 231  232  234  235  237 ]] inner stride: 1 post of N200 !orig=[200],[191],[118] !jvms: String::hashCode @ bci:33
int:0..6:www	 258	Phi	===  259  23  256  [[ 256  257 ]]  #int #tripcount !orig=[122] !jvms: String::hashCode @ bci:33
control		 176	IfTrue	===  199  [[ 295 ]] #1 !jvms: String::hashCode @ bci:30
char		 286	LoadUS	===  206  7  287  [[ 285 ]]  @char[int:>=0]:exact+any *, idx=7; #char !orig=150 !jvms: String::hashCode @ bci:39
int:-2162655..2228190:w	 285	AddI	=== _  296  286  [[ 284  129 ]]  !orig=151,[182] !jvms: String::hashCode @ bci:40
int:1..6:ww	 294	Phi	===  295  279  196  [[ 292  293  307  196 ]]  #int #tripcount !orig=[122] !jvms: String::hashCode @ bci:33
control		 242	IfFalse	===  199  [[ 266 ]] #0
long:0..6:www	 257	ConvI2L	=== _  258  [[ 255 ]]  #long:0..maxint-1 !orig=[146] !jvms: String::hashCode @ bci:39
int:-4259775..69336030:w	 249	AddI	=== _  260  250  [[ 248  261  268  297 ]]  !orig=151,[182] !jvms: String::hashCode @ bci:40
int:-4259775..69270495:w	 296	SubI	=== _  300  297  [[ 285 ]]  !orig=129 !jvms: String::hashCode @ bci:36
int:-2097120..69204960:w	 237	Phi	===  232  267  221  [[ 233 ]]  #int !orig=192,[128] !jvms: String::hashCode @ bci:36
{0:top, 1:top}	 235	CountedLoopEnd	===  232  225  [[ 236  238 ]] [lt] P=0.500000, C=8524.000000 !orig=199,[175] !jvms: String::hashCode @ bci:30
char[int:>=0]:NotNull:exact+any *	 148	AddP	=== _  53  315  316  [[ 150 ]]  !jvms: String::hashCode @ bci:39
control		 239	Region	===  239  238  177  [[ 239  33  184  240 ]] 
int:-65535..2162655:w	 234	Phi	===  232  268  222  [[ 233 ]]  #int !orig=121 !jvms: String::hashCode @ bci:33
int:2..7:ww	 292	AddI	=== _  294  91  [[]]  !orig=196,[152] !jvms: String::hashCode @ bci:42
long:1..6:ww	 293	ConvI2L	=== _  294  [[ 291 ]]  #long:0..maxint-1 !orig=[146] !jvms: String::hashCode @ bci:39
long:1..6:ww	 307	ConvI2L	=== _  294  [[ 312 ]]  #long:-1..maxint-2
int:1..6:ww	 231	Phi	===  232  278  229  [[ 229  230 ]]  #int #tripcount !orig=[122] !jvms: String::hashCode @ bci:33
int:1..7:www	 256	AddI	=== _  258  91  [[ 253  258  270  274  279 ]]  !orig=196,[152] !jvms: String::hashCode @ bci:42
int:-4259775..69336030:ww	 297	Phi	===  295  249  151  [[ 296 ]]  #int !orig=121 !jvms: String::hashCode @ bci:33
int:-69204960..71302080:w	 284	LShiftI	=== _  285  127  [[ 129 ]]  !orig=194,[128] !jvms: String::hashCode @ bci:36
int:3..8:ww	 196	AddI	=== _  294  311  [[ 294  197  270 ]]  !orig=[152] !jvms: String::hashCode @ bci:42
int:1..8:www	 270	Phi	===  266  256  196  [[ 278  243 ]]  #int
long:0..12:www	 255	LShiftL	=== _  257  91  [[ 254 ]]  !orig=[147] !jvms: String::hashCode @ bci:39
int:-4259775..69336030:ww	 261	Phi	===  259  23  249  [[ 260 ]]  #int !orig=121 !jvms: String::hashCode @ bci:33
int:-4259775..69336030:w	 285	AddI	=== _  296  286  [[ 284  129 ]]  !orig=151,[182] !jvms: String::hashCode @ bci:40
int:-4259775..69270495:w	 233	SubI	=== _  237  234  [[ 222 ]]  !orig=129 !jvms: String::hashCode @ bci:36
char		 150	LoadUS	===  206  7  148  [[ 151 ]]  @char[int:>=0]:exact+any *, idx=7; #char !jvms: String::hashCode @ bci:39
int		 248	LShiftI	=== _  249  127  [[ 264  267  300 ]]  !orig=194,[128] !jvms: String::hashCode @ bci:36
int:-65535..2162655:w	 240	Phi	===  239  222  268  [[ 63  184 ]]  #int

[error output omitted]

Comment 22 Severin Gehwolf 2015-04-14 15:20:46 UTC
Created attachment 1014401 [details]
gdb commands file in order to get to the error condition

Comment 23 Severin Gehwolf 2015-04-14 15:23:26 UTC
Invoke gdb as follows and you should get to the point where the assertion triggers. The rhbz_1208369_gdb_commands.txt file is the one coming from comment 22.

JDK=/path/to/jdk \
gdb -x rhbz_1208369_gdb_commands.txt --args \
 $JDK/bin/java \
 -classpath $JDK/lib/tools.jar:$JDK/classes \
 -Dapplication.home=$JDK \
 -Xms8m \
 -XX:+PrintCompilation \
 -XX:+TracePhaseCCP \
 -XX:-TieredCompilation \
 -XX:CICompilerCount=1 \
 com.sun.tools.javac.Main -d . ../../TestCryptoLevel.java

Comment 24 Severin Gehwolf 2015-04-14 15:26:43 UTC
example gdb back trace once you hit the breakpoint in src/share/vm/opto/type.hpp:1488.

(gdb) bt
#0  ccp_type_widens (t0=<optimized out>, t=<optimized out>) at /home/sgehwolf/Documents/java-1.8.0-openjdk/java-1.8.0-openjdk-1.8.0.40-27.b25.fc22.x86_64/jdk8/hotspot/src/share/vm/opto/phaseX.cpp:1486
#1  PhaseCCP::analyze (this=<optimized out>) at /home/sgehwolf/Documents/java-1.8.0-openjdk/java-1.8.0-openjdk-1.8.0.40-27.b25.fc22.x86_64/jdk8/hotspot/src/share/vm/opto/phaseX.cpp:1513
#2  0x00007ffff6d602f9 in Compile::Optimize (this=this@entry=0x7fffd1842d10)
    at /home/sgehwolf/Documents/java-1.8.0-openjdk/java-1.8.0-openjdk-1.8.0.40-27.b25.fc22.x86_64/jdk8/hotspot/src/share/vm/opto/compile.cpp:2173
#3  0x00007ffff6d63820 in Compile::Compile (this=<optimized out>, ci_env=<optimized out>, compiler=<optimized out>, target=<optimized out>, osr_bci=<optimized out>, subsume_loads=<optimized out>, 
    do_escape_analysis=true, eliminate_boxing=true) at /home/sgehwolf/Documents/java-1.8.0-openjdk/java-1.8.0-openjdk-1.8.0.40-27.b25.fc22.x86_64/jdk8/hotspot/src/share/vm/opto/compile.cpp:856
#4  0x00007ffff6c1753a in C2Compiler::compile_method (this=0x7ffff0150c10, env=0x7fffd18439e0, target=0x7fffc402b5c0, entry_bci=-1)
    at /home/sgehwolf/Documents/java-1.8.0-openjdk/java-1.8.0-openjdk-1.8.0.40-27.b25.fc22.x86_64/jdk8/hotspot/src/share/vm/opto/c2compiler.cpp:121
#5  0x00007ffff6d71273 in CompileBroker::invoke_compiler_on_method (task=task@entry=0x7ffff0192480)
    at /home/sgehwolf/Documents/java-1.8.0-openjdk/java-1.8.0-openjdk-1.8.0.40-27.b25.fc22.x86_64/jdk8/hotspot/src/share/vm/compiler/compileBroker.cpp:1993
#6  0x00007ffff6d734ff in CompileBroker::compiler_thread_loop ()
    at /home/sgehwolf/Documents/java-1.8.0-openjdk/java-1.8.0-openjdk-1.8.0.40-27.b25.fc22.x86_64/jdk8/hotspot/src/share/vm/compiler/compileBroker.cpp:1815
#7  0x00007ffff7684be0 in JavaThread::thread_main_inner (this=this@entry=0x7ffff0161800)
    at /home/sgehwolf/Documents/java-1.8.0-openjdk/java-1.8.0-openjdk-1.8.0.40-27.b25.fc22.x86_64/jdk8/hotspot/src/share/vm/runtime/thread.cpp:1698
#8  0x00007ffff7684e16 in JavaThread::run (this=0x7ffff0161800) at /home/sgehwolf/Documents/java-1.8.0-openjdk/java-1.8.0-openjdk-1.8.0.40-27.b25.fc22.x86_64/jdk8/hotspot/src/share/vm/runtime/thread.cpp:1678
#9  0x00007ffff745feb2 in java_start (thread=0x7ffff0161800) at /home/sgehwolf/Documents/java-1.8.0-openjdk/java-1.8.0-openjdk-1.8.0.40-27.b25.fc22.x86_64/jdk8/hotspot/src/os/linux/vm/os_linux.cpp:828
#10 0x000000368ea07555 in start_thread (arg=0x7fffd1844700) at pthread_create.c:333
#11 0x000000368ef01f3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Comment 25 Omair Majid 2015-04-14 21:10:59 UTC
(In reply to Omair Majid from comment #17)
> I built the fastdebug variant and javac is reproducibly crashing:
> 
> $
> /home/omajid/java-1.8.0-openjdk/java-1.8.0-openjdk-1.8.0.40-27.b25.fc22.
> x86_64/jdk8/build/jdk8.build/images/j2sdk-image/bin/java -version
> openjdk version "1.8.0_40-fastdebug"
> OpenJDK Runtime Environment (build 1.8.0_40-fastdebug-b25)
> OpenJDK 64-Bit Server VM (build 25.40-b25-fastdebug, mixed mode)
> $
> /home/omajid/java-1.8.0-openjdk/java-1.8.0-openjdk-1.8.0.40-27.b25.fc22.
> x86_64/jdk8/build/jdk8.build/images/j2sdk-image/bin/javac -d .
> /home/omajid/java-1.8.0-openjdk/TestCryptoLevel.java

Building with "-fno-tree-vrp" added to C(XX)FLAGS makes it work for me.

$ java-1.8.0-openjdk-1.8.0.40-27.b25.fc22.x86_64/jdk8/build/jdk8.build/images/j2sdk-image/bin/java -version
openjdk version "1.8.0_40-fastdebug"
OpenJDK Runtime Environment (build 1.8.0_40-fastdebug-b25)
OpenJDK 64-Bit Server VM (build 25.40-b25-fastdebug, mixed mode)

$ \time -pv java-1.8.0-openjdk-1.8.0.40-27.b25.fc22.x86_64/jdk8/build/jdk8.build/images/j2sdk-image/bin/javac -d . TestCryptoLevel.java 
        Command being timed: "java-1.8.0-openjdk-1.8.0.40-27.b25.fc22.x86_64/jdk8/build/jdk8.build/images/j2sdk-image/bin/javac -d . TestCryptoLevel.java"
        User time (seconds): 2.88
        System time (seconds): 0.11
        Percent of CPU this job got: 144%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 0:02.08
        Average shared text size (kbytes): 0
        Average unshared data size (kbytes): 0
        Average stack size (kbytes): 0
        Average total size (kbytes): 0
        Maximum resident set size (kbytes): 114240
        Average resident set size (kbytes): 0
        Major (requiring I/O) page faults: 10
        Minor (reclaiming a frame) page faults: 39638
        Voluntary context switches: 1230
        Involuntary context switches: 715
        Swaps: 0
        File system inputs: 12248
        File system outputs: 8
        Socket messages sent: 0
        Socket messages received: 0
        Signals delivered: 0
        Page size (bytes): 4096
        Exit status: 0

Comment 26 Omair Majid 2015-04-14 21:39:57 UTC
And on a normal ("release") build with "-fno-tree-vrp":

$ java-1.8.0-openjdk-1.8.0.40-27.b25.fc22.x86_64/jdk8/build/jdk8.build/images/j2sdk-image/bin/java -version
openjdk version "1.8.0_40"
OpenJDK Runtime Environment (build 1.8.0_40-b25)
OpenJDK 64-Bit Server VM (build 25.40-b25, mixed mode)

$ \time -pv java-1.8.0-openjdk-1.8.0.40-27.b25.fc22.x86_64/jdk8/build/jdk8.build/images/j2sdk-image/bin/javac -d . TestCryptoLevel.java 
        Command being timed: "java-1.8.0-openjdk-1.8.0.40-27.b25.fc22.x86_64/jdk8/build/jdk8.build/images/j2sdk-image/bin/javac -d . TestCryptoLevel.java"
        User time (seconds): 0.94
        System time (seconds): 0.10
        Percent of CPU this job got: 68%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 0:01.53
        Average shared text size (kbytes): 0
        Average unshared data size (kbytes): 0
        Average stack size (kbytes): 0
        Average total size (kbytes): 0
        Maximum resident set size (kbytes): 49412
        Average resident set size (kbytes): 0
        Major (requiring I/O) page faults: 93
        Minor (reclaiming a frame) page faults: 15943
        Voluntary context switches: 1440
        Involuntary context switches: 661
        Swaps: 0
        File system inputs: 44040
        File system outputs: 8
        Socket messages sent: 0
        Socket messages received: 0
        Signals delivered: 0
        Page size (bytes): 4096
        Exit status: 0

That's about 50 MB max memory usage and ~100 major page faults.

Comment 27 Severin Gehwolf 2015-04-15 08:32:32 UTC
I see Omair built it in rawhide and f22. Mikolaj, gil, could you please test if this build works for you?

https://koji.fedoraproject.org/koji/taskinfo?taskID=9480919

Meanwhile I'll look further into as to what goes wrong with tree-vrp turned on.

Comment 28 Severin Gehwolf 2015-04-15 10:27:03 UTC
With broken java-1.8.0-openjdk-1.8.0.40-26.b25.fc22.x86_64

$ rpm -qa | grep java-1.8.0-openjdk
java-1.8.0-openjdk-devel-1.8.0.40-26.b25.fc22.x86_64
java-1.8.0-openjdk-1.8.0.40-26.b25.fc22.x86_64
java-1.8.0-openjdk-headless-1.8.0.40-26.b25.fc22.x86_64

$ rm TestCryptoLevel.class 
$ \time -pv javac TestCryptoLevel.java
	Command being timed: "javac TestCryptoLevel.java"
	User time (seconds): 4.26
	System time (seconds): 1.44
	Percent of CPU this job got: 26%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 0:21.69
	Average shared text size (kbytes): 0
	Average unshared data size (kbytes): 0
	Average stack size (kbytes): 0
	Average total size (kbytes): 0
	Maximum resident set size (kbytes): 1868164
	Average resident set size (kbytes): 0
	Major (requiring I/O) page faults: 501
	Minor (reclaiming a frame) page faults: 658860
	Voluntary context switches: 3981
	Involuntary context switches: 1757
	Swaps: 0
	File system inputs: 73152
	File system outputs: 8
	Socket messages sent: 0
	Socket messages received: 0
	Signals delivered: 0
	Page size (bytes): 4096
	Exit status: 0

Reproducer from comment 0 (result: java process killed by OOM killer, build never succeeded for me).

$ \time -pv ../apache-maven-3.3.1/bin/mvn -V verify
Command terminated by signal 9
	Command being timed: "../apache-maven-3.3.1/bin/mvn -V verify"
	User time (seconds): 22.74
	System time (seconds): 26.87
	Percent of CPU this job got: 43%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 1:54.48
	Average shared text size (kbytes): 0
	Average unshared data size (kbytes): 0
	Average stack size (kbytes): 0
	Average total size (kbytes): 0
	Maximum resident set size (kbytes): 1876212
	Average resident set size (kbytes): 0
	Major (requiring I/O) page faults: 130254
	Minor (reclaiming a frame) page faults: 1501664
	Voluntary context switches: 153745
	Involuntary context switches: 36356
	Swaps: 0
	File system inputs: 4982744
	File system outputs: 15216
	Socket messages sent: 0
	Socket messages received: 0
	Signals delivered: 0
	Page size (bytes): 4096
	Exit status: 0


---------------------------------------------------------


With fixed java-1.8.0-openjdk-1.8.0.40-27.b25.fc22.x86_64

$ rpm -qa | grep java-1.8.0-openjdk
java-1.8.0-openjdk-1.8.0.40-27.b25.fc22.x86_64
java-1.8.0-openjdk-devel-1.8.0.40-27.b25.fc22.x86_64
java-1.8.0-openjdk-headless-1.8.0.40-27.b25.fc22.x86_64

$ rm TestCryptoLevel.class 
$ \time -pv javac TestCryptoLevel.java
	Command being timed: "javac TestCryptoLevel.java"
	User time (seconds): 0.60
	System time (seconds): 0.04
	Percent of CPU this job got: 150%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.43
	Average shared text size (kbytes): 0
	Average unshared data size (kbytes): 0
	Average stack size (kbytes): 0
	Average total size (kbytes): 0
	Maximum resident set size (kbytes): 60108
	Average resident set size (kbytes): 0
	Major (requiring I/O) page faults: 0
	Minor (reclaiming a frame) page faults: 16694
	Voluntary context switches: 652
	Involuntary context switches: 384
	Swaps: 0
	File system inputs: 2000
	File system outputs: 8
	Socket messages sent: 0
	Socket messages received: 0
	Signals delivered: 0
	Page size (bytes): 4096
	Exit status: 0

Reproducer from comment 0 (result: build passes).

$ \time -pv ../apache-maven-3.3.1/bin/mvn -V verify
[...]
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 02:03 min
[INFO] Finished at: 2015-04-15T12:23:18+02:00
[INFO] Final Memory: 64M/268M
[INFO] ------------------------------------------------------------------------
	Command being timed: "../apache-maven-3.3.1/bin/mvn -V verify"
	User time (seconds): 105.84
	System time (seconds): 6.00
	Percent of CPU this job got: 89%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 2:05.01
	Average shared text size (kbytes): 0
	Average unshared data size (kbytes): 0
	Average stack size (kbytes): 0
	Average total size (kbytes): 0
	Maximum resident set size (kbytes): 558700
	Average resident set size (kbytes): 0
	Major (requiring I/O) page faults: 22
	Minor (reclaiming a frame) page faults: 907339
	Voluntary context switches: 112797
	Involuntary context switches: 108481
	Swaps: 0
	File system inputs: 6528
	File system outputs: 163048
	Socket messages sent: 0
	Socket messages received: 0
	Signals delivered: 0
	Page size (bytes): 4096
	Exit status: 0

Comment 29 Severin Gehwolf 2015-04-16 15:14:37 UTC
It looks like type.o gets compiled wrong by GCC 5 with value range propagation for trees optimization turned on.

Comment 30 Severin Gehwolf 2015-04-17 15:06:56 UTC
Created attachment 1015629 [details]
Proposed patch for working around type.o miscompilation.

Comment 31 Severin Gehwolf 2015-04-17 15:08:39 UTC
Patch in comment 30 fixes the problem for me too. No need to set C(XX)FLAGS.

Comment 32 Severin Gehwolf 2015-04-24 12:04:29 UTC
I've filed gcc bug 1215127 since I've failed to find the offending snippet in that file. Hopefully gcc people have more luck.

Comment 33 Severin Gehwolf 2015-04-27 12:03:06 UTC
Created attachment 1019303 [details]
Proposed patch fixing signed integer overflow in type.cpp

This patch fixes hotspot to not rely on undefined behaviour. See https://bugzilla.redhat.com/show_bug.cgi?id=1215127#c5 With this patch applied, java-1.8.0-openjdk builds fine for me (release/fastdebug) and does not show the issue.

Comment 34 Severin Gehwolf 2015-04-27 14:22:03 UTC
Upstream review thread: http://mail.openjdk.java.net/pipermail/hotspot-compiler-dev/2015-April/017759.html

Comment 36 Fedora Update System 2015-04-30 08:21:18 UTC
java-1.8.0-openjdk-1.8.0.45-36.b13.fc22 has been submitted as an update for Fedora 22.
https://admin.fedoraproject.org/updates/java-1.8.0-openjdk-1.8.0.45-36.b13.fc22

Comment 37 Fedora Update System 2015-05-01 16:39:44 UTC
Package java-1.8.0-openjdk-1.8.0.45-36.b13.fc22:
* should fix your issue,
* was pushed to the Fedora 22 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing java-1.8.0-openjdk-1.8.0.45-36.b13.fc22'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2015-7301/java-1.8.0-openjdk-1.8.0.45-36.b13.fc22
then log in and leave karma (feedback).

Comment 38 Fedora Update System 2015-05-03 17:23:15 UTC
java-1.8.0-openjdk-1.8.0.45-36.b13.fc22 has been pushed to the Fedora 22 stable repository.  If problems still persist, please make note of it in this bug report.