Bug 1415137

Created attachment 1242721 [details]
hs log

java-1.8.0-openjdk-1.8.0.111-5.b16.fc25.x86_64
already have fixed java.security of security.useSystemPropertiesFile=false
So the fault might be aligned with it turned off, but enabled in code

Created attachment 1242727 [details]
working java.security file (no segfault)

Created attachment 1242728 [details]
non-working java.security file (segfault)

Created attachment 1242729 [details]
diff between working and non-working java.security

I did a `dnf upgrade` today, and I'm seeing this everywhere now. Is there a fix I can try, or is there a specific package that I can safely downgrade to workaround it?

Looking at the packages that got upgraded, and the conversation here, I'm assuming that it's the new nss* packages that triggered this for me. Does that sound right? Can those be safely downgraded?

I hit this, too. Downgrading the nss* packages solved it for the meantime.

(In reply to buddabrod from comment #7)
> I hit this, too. Downgrading the nss* packages solved it for the meantime.

Can you tell us which nss packages work and which do not? Perhaps some /var/log/dnf*.log file has that info?

I downgraded
nss nss-softokn nss-softokn-freebl nss-sysinit nss-tools nss-util
each from 3.28.1 to 3.27.0.

I did not test the individual packages, just downgraded all of them.

Here is a paste of the relevant snippet (I think)

https://nopaste.me/view/5cfe6b84

This appears to be the update that the nss packages came from: https://bodhi.fedoraproject.org/updates/FEDORA-2017-e42b513012

Downgrading everything from that update seems to be the most rational thing to do, and it is a working workaround for me.

Pasting content of paste from comment 10 here directly, since the paste might go away:

Jan 24 13:59:19 INFO Letzte Prüfung auf abgelaufene Metadaten: vor 0:34:02 am Tue Jan 24 13:25:16 2017.
Jan 24 13:59:19 DEBUG --> Abhängigkeitsauflösung wird gestartet
Jan 24 13:59:20 DEBUG ---> Paket nss.i686 3.28.1-1.3.fc25 wird zurückgesetzt
Jan 24 13:59:20 DEBUG ---> Paket nss.i686 3.27.0-1.1.fc25 wird ein Downgrade
Jan 24 13:59:20 DEBUG ---> Paket nss.x86_64 3.28.1-1.3.fc25 wird zurückgesetzt
Jan 24 13:59:20 DEBUG ---> Paket nss.x86_64 3.27.0-1.1.fc25 wird ein Downgrade
Jan 24 13:59:20 DEBUG ---> Paket nss-softokn.i686 3.28.1-1.0.fc25 wird zurückgesetzt
Jan 24 13:59:20 DEBUG ---> Paket nss-softokn.i686 3.27.0-1.0.fc25 wird ein Downgrade
Jan 24 13:59:20 DEBUG ---> Paket nss-softokn.x86_64 3.28.1-1.0.fc25 wird zurückgesetzt
Jan 24 13:59:20 DEBUG ---> Paket nss-softokn.x86_64 3.27.0-1.0.fc25 wird ein Downgrade
Jan 24 13:59:20 DEBUG ---> Paket nss-softokn-freebl.i686 3.28.1-1.0.fc25 wird zurückgesetzt
Jan 24 13:59:20 DEBUG ---> Paket nss-softokn-freebl.i686 3.27.0-1.0.fc25 wird ein Downgrade
Jan 24 13:59:20 DEBUG ---> Paket nss-softokn-freebl.x86_64 3.28.1-1.0.fc25 wird zurückgesetzt
Jan 24 13:59:20 DEBUG ---> Paket nss-softokn-freebl.x86_64 3.27.0-1.0.fc25 wird ein Downgrade
Jan 24 13:59:20 DEBUG ---> Paket nss-sysinit.x86_64 3.28.1-1.3.fc25 wird zurückgesetzt
Jan 24 13:59:20 DEBUG ---> Paket nss-sysinit.x86_64 3.27.0-1.1.fc25 wird ein Downgrade
Jan 24 13:59:20 DEBUG ---> Paket nss-tools.x86_64 3.28.1-1.3.fc25 wird zurückgesetzt
Jan 24 13:59:20 DEBUG ---> Paket nss-tools.x86_64 3.27.0-1.1.fc25 wird ein Downgrade
Jan 24 13:59:20 DEBUG ---> Paket nss-util.i686 3.28.1-1.0.fc25 wird zurückgesetzt
Jan 24 13:59:20 DEBUG ---> Paket nss-util.i686 3.27.0-1.0.fc25 wird ein Downgrade
Jan 24 13:59:20 DEBUG ---> Paket nss-util.x86_64 3.28.1-1.0.fc25 wird zurückgesetzt
Jan 24 13:59:20 DEBUG ---> Paket nss-util.x86_64 3.27.0-1.0.fc25 wird ein Downgrade
Jan 24 13:59:20 DEBUG --> Abhängigkeitsauflösung wurde abgeschlossen
Jan 24 13:59:20 DDEBUG timer: depsolve: 851 ms

I can reproduce the problem, but don't understand the cause yet.

It's sufficient to upgrade the
  nss-util + nss-softokn + nss-softokn-freebl
packages.

(The nss.rpm packages doesn't have an influence).

I discovered another bug while working on this.
Please make sure you NEVER downgrade ONLY nss-softokn. 
Always downgrade BOTH of nss-softokn and nss-softokn-freebl.

(When I accidently downgraded just one of them, which dnf allowed me to do without overrides, that broken dnf completely, by no longer being able to use https. We need to file another bug that ensures these packages are always used together, I guess, or find out what's causing that breakage.)

Back to this original bug report.

With the attached differences, and Hubert's detective skills, the following difference in the 
  /usr/lib/jvm/java-*/jre/lib/security/java.security
file triggers (or prevents the problem).

Broken:
jdk.tls.disabledAlgorithms=SSLv3, RC4, MD5withRSA, DH keySize < 768

Working:
jdk.tls.disabledAlgorithms=SSLv3, DH keySize < 768, EC, ECDHE, ECDH

This line works, too:
jdk.tls.disabledAlgorithms=SSLv3, RC4, MD5withRSA, DH keySize < 768, ECDHE      

Allowing ECDHE triggers the problem.

This breaks android dev tooling too. Based on buddabrod's comment here is a copy and paste workaround for now that downgrades nss related packages.

sudo dnf install nss-3.27.0-1.1.fc25.x86_64 nss-softokn-3.27.0-1.0.fc25.x86_64 nss-softokn-freebl-3.27.0-1.0.fc25.x86_64 nss-sysinit-3.27.0-1.1.fc25.x86_64 nss-tools-3.27.0-1.1.fc25.x86_64 nss-util-3.27.0-1.0.fc25.x86_64

(In reply to Kai Engert (:kaie) from comment #14)
> This line works, too:
> jdk.tls.disabledAlgorithms=SSLv3, RC4, MD5withRSA, DH keySize < 768, ECDHE  
> 
> 
> Allowing ECDHE triggers the problem.

That makes some sense as OpenJDK packages in Fedora gain EC support via nss.

It would be difficult to downgrade nss in fedora-updates.

If you can confirm that adding ECDHE to the disabled algorithms in java.security fixes the issue, would it be an option to publish an updated java-openjdk.rpm package, which disables ECDHE, to give us more time to investigate the issue?

This broke Eclipse and Davmail for me, downgrading nss fixed the issue.

 I can disable ECDHE by rpm patch, but isn't better to permanently downgrade nss? Aka to create upgrade with older, working version inside.

We just discussed this in our weekly meeting.

It seems that you are trying to enable EC in Java at the same time, as NSS adds an additional curve to NSS.

That new curve x25519 requires some new special code for handling it correctly. We suspect that the Java code is broken, and doesn't handle this new curve correctly, and that this situation is actually a bug in the OpenJDK package.

Also, downgrading NSS would cause other negative consequences. Firefox 51 (with new security fixes) was released today, and Firefox requires NSS 3.28.1

Given the complex situation, we'd like to ask you to please please please :-) submit the java package update, with EC* disabled, as a temporary fix, while we all try to better understand the situation.

In parallel, I will attempt to change the nss-softokn package, to disable this new curve, and test locally if it works around the issue. If that works, I will submit that as an update, too, just in case there's any other package could potentially be negatively affected by this new curve.

Is this suggestion acceptable, could you disable EC in a openjdk update?

http://pkgs.fedoraproject.org/cgit/rpms/java-1.8.0-openjdk.git/commit/?h=f24&id=dad68d0510a31d2b0365ca31a162632013f3b60b
and
http://pkgs.fedoraproject.org/cgit/rpms/java-1.8.0-openjdk.git/commit/?h=f25&id=19d6c38ce653ac7259300cb5a2f74698e5f56037

with
https://koji.fedoraproject.org/koji/taskinfo?taskID=17403125
and
https://koji.fedoraproject.org/koji/taskinfo?taskID=17403126

Intentionally skipped rawhide.

But whether the CPU release 1 or this "2" with disabled ECDHE will go to updates should be subject of further discussions

> Given the complex situation, we'd like to ask you to please please please
> :-) submit the java package update, with EC* disabled, as a temporary fix,
> while we all try to better understand the situation.
> 
...
> 
> Is this suggestion acceptable, could you disable EC in a openjdk update?

I lunched the builds before this comment. Is there anything else to disable except already excluded ECDHE ?

I don't know. Disabling ECDHE was sufficient to work around this specific bug. We don't know if anything else is broken. If you want to play it completely safe, you could disable all the new EC code. But it's up to you, we can try to disable ECDHE, only, since no other breakage is known yet.

(In reply to Kai Engert (:kaie) from comment #23)
> I don't know. Disabling ECDHE was sufficient to work around this specific
> bug. We don't know if anything else is broken. If you want to play it
> completely safe, you could disable all the new EC code. But it's up to you,
> we can try to disable ECDHE, only, since no other breakage is known yet.

ok, but only because you say so nicely;)

> Given the complex situation, we'd like to ask you to please please please
> :-) submit the java package update, with EC* disabled, as a temporary fix,

(In reply to Kai Engert (:kaie) from comment #17)
> It would be difficult to downgrade nss in fedora-updates.
> 

Just a result of "this update has to be pushed on monday" instead of waiting for some days of testing as suggested multiple times…

I investigated what I think is the same issue at bug #1411116, and found one change added by the Curve25519 code which might explain this issue (see my comments at bug #1411116 for the details): the ec_NewKey function (and others) now depends on a new pointSize field in the ecParams structure, and the Java code seems to create the ecParams by hand, thus missing the new field, which then contains garbage.

If my supposition is correct, just disabling the new curve in NSS wouldn't be enough to fix this issue, since the change affects all curves. The solution would be in the Java side, to correctly set the new field (and so it would have to depend on the new NSS which has that field).

Cesar, thanks a lot for your analysis, which sounds like a bug in the java openjdk code.

In a local experiment, my attempt to disable that curve in 3.28 nss-softokn indeed didn't fix the issue.

Jiri, has OpenJDK been rebuilt against the new NSS?

This issue with NSS 3.28 has also been brought up on Gentoo and rebuilding seems to fix it:

https://bugs.gentoo.org/show_bug.cgi?id=605430

The OpenJDK build has to statically link some parts of NSS because that's the only way they are exposed. Hence, when NSS is upgraded, OpenJDK needs to be re-built against it.

At present, the dependency in java-1.8.0-openjdk is >= %{NSS_BUILDTIME_NUMBER}. That works for preventing cases where OpenJDK is re-built against a newer NSS, but I really think it should be switched to '=' to prevent cases like this. The NSS upgrade would then be blocked until OpenJDK is updated and it would be more obvious what the bug is.

(In reply to Andrew John Hughes from comment #28)
> Jiri, has OpenJDK been rebuilt against the new NSS?
> 
> This issue with NSS 3.28 has also been brought up on Gentoo and rebuilding
> seems to fix it:
> 
> https://bugs.gentoo.org/show_bug.cgi?id=605430
> 
> The OpenJDK build has to statically link some parts of NSS because that's
> the only way they are exposed. Hence, when NSS is upgraded, OpenJDK needs to
> be re-built against it.
> 
> At present, the dependency in java-1.8.0-openjdk is >=
> %{NSS_BUILDTIME_NUMBER}. That works for preventing cases where OpenJDK is
> re-built against a newer NSS, but I really think it should be switched to
> '=' to prevent cases like this. The NSS upgrade would then be blocked until
> OpenJDK is updated and it would be more obvious what the bug is.

Does it fully fix the problem, or does it just make it not crash while still not working correctly for ECDH?

Apparently, the Java code does a calloc of the ECParams structure. With a java-1.8.0-openjdk compiled against an older NSS, the structure is 4 bytes shorter than it should be for NSS 3.28, and the field pointSize added by NSS 3.28 is an out-of-bounds read; once java-1.8.0-openjdk is recompiled with the headers for NSS 3.28, the structure should be allocated with the correct size, and the pointSize field should be always zero. Neither is the correct value for the pointSize field, and I have no idea how it would behave when it's zero.

Even then, IMO it's still worth it to recompile java-1.8.0-openjdk against the NSS 3.28 headers, just to prevent the out-of-bounds read.

(In reply to Cesar Eduardo Barros from comment #29)
> (In reply to Andrew John Hughes from comment #28)
> > Jiri, has OpenJDK been rebuilt against the new NSS?
> > 
> > This issue with NSS 3.28 has also been brought up on Gentoo and rebuilding
> > seems to fix it:
> > 
> > https://bugs.gentoo.org/show_bug.cgi?id=605430
> > 
> > The OpenJDK build has to statically link some parts of NSS because that's
> > the only way they are exposed. Hence, when NSS is upgraded, OpenJDK needs to
> > be re-built against it.
> > 
> > At present, the dependency in java-1.8.0-openjdk is >=
> > %{NSS_BUILDTIME_NUMBER}. That works for preventing cases where OpenJDK is
> > re-built against a newer NSS, but I really think it should be switched to
> > '=' to prevent cases like this. The NSS upgrade would then be blocked until
> > OpenJDK is updated and it would be more obvious what the bug is.
> 
> Does it fully fix the problem, or does it just make it not crash while still
> not working correctly for ECDH?
> 
> Apparently, the Java code does a calloc of the ECParams structure. With a
> java-1.8.0-openjdk compiled against an older NSS, the structure is 4 bytes
> shorter than it should be for NSS 3.28, and the field pointSize added by NSS
> 3.28 is an out-of-bounds read; once java-1.8.0-openjdk is recompiled with
> the headers for NSS 3.28, the structure should be allocated with the correct
> size, and the pointSize field should be always zero. Neither is the correct
> value for the pointSize field, and I have no idea how it would behave when
> it's zero.
> 
> Even then, IMO it's still worth it to recompile java-1.8.0-openjdk against
> the NSS 3.28 headers, just to prevent the out-of-bounds read.

Probably not from what you're saying, but it certainly won't work with mixing NSS 3.28 with whatever older version OpenJDK was compiled against. There's a security update due to trickle through anyway, so it should happen automatically with that. The build runs an ECDSA check, so it could even catch the failure there. If it does fail, then the right thing to do would be to disable it temporarily.

I'll look at a fix for the code itself when I have time. It's going to need to work with both versions of NSS. It's great to see NSS adopting ed25519. I do wonder if OpenJDK could also make use of it, but it might be difficult if it requires new fields, as that would require alterations to code that is part of the Java specification.

*** Bug 1411116 has been marked as a duplicate of this bug. ***

This bug affects pretty much all Gradle and Maven builds consuming artifacts from Maven Central via SSL. As others mentioned, reverting the NSS packages to the previous version is a workaround:

> sudo dnf downgrade nss nss-softokn nss-softokn-freebl nss-sysinit nss-tools nss-util

> At present, the dependency in java-1.8.0-openjdk is >=
> %{NSS_BUILDTIME_NUMBER}. That works for preventing cases where OpenJDK is
> re-built against a newer NSS, but I really think it should be switched to
> '=' to prevent cases like this. The NSS upgrade would then be blocked until
> OpenJDK is updated and it would be more obvious what the bug is.

That would be terrible lock without precedents I think.
Also it may kill many automated services dealng with openjdk  rpm.
NSS_BUILDTIME_NUMBER contans release. So that enfroces >=
If NSS_BUILDTIME_NUMBER would be onl version, it  *may* work... Something  like >= NSS_BUILDTIME_NUMBER && <= NSS_BUILDTIME_VERSIO.99999

Will try to find something.

I will post NSS versions used during builds in next comment

latest builds
java-1.8.0-openjdk-1.8.0.111-3.b16.fc25
          nss x86_64  3.27.0-1.1.fc25
java-1.8.0-openjdk-1.8.0.111-4.b16.fc25
          nss x86_64  3.27.0-1.2.fc25
java-1.8.0-openjdk-1.8.0.111-5.b16.fc25
          nss x86_64  3.27.0-1.3.fc25 
java-1.8.0-openjdk-1.8.0.121-1.b14.fc25 (current security update)
           nss x86_64  3.28.1-1.3.fc25 
java-1.8.0-openjdk-1.8.0.121-2.b14.fc25 (^ with disabled ECDHE)
          nss x86_64  3.28.1-1.3.fc25 

So if Your susipicions are correct, it is worthy to try java-1.8.0-openjdk-1.8.0.121-1.b14.fc25 first. If it passes, I will revert  https://bugzilla.redhat.com/show_bug.cgi?id=1415137#c21

(In reply to jiri vanek from comment #34)
> latest builds
[...]
> java-1.8.0-openjdk-1.8.0.121-1.b14.fc25 (current security update)
>            nss x86_64  3.28.1-1.3.fc25 
[...]
> So if Your susipicions are correct, it is worthy to try
> java-1.8.0-openjdk-1.8.0.121-1.b14.fc25 first. If it passes, I will revert 
> https://bugzilla.redhat.com/show_bug.cgi?id=1415137#c21

$ rpm -qa | grep java-1.8.0-openjdk
java-1.8.0-openjdk-1.8.0.121-1.b14.fc25.x86_64
java-1.8.0-openjdk-devel-1.8.0.121-1.b14.fc25.x86_64
java-1.8.0-openjdk-headless-1.8.0.121-1.b14.fc25.x86_64
$ rpm -q --requires java-1.8.0-openjdk-headless | grep nss
libnss3.so()(64bit)
libnss3.so(NSS_3.2)(64bit)
libnssutil3.so()(64bit)
libnssutil3.so(NSSUTIL_3.12)(64bit)
nss(x86-64) >= 3.28.1
nss-softokn(x86-64) >= 3.28.1

With that I don't get exceptions/coredumps using the reproducer from comment 0. I suggest to revert java-1.8.0-openjdk-1.8.0.121-2.b14.fc25. It's not needed.

Also from the build.log from java-1.8.0-openjdk-1.8.0.121-1.b14.fc25:

+ /builddir/build/BUILD/java-1.8.0-openjdk-1.8.0.121-1.b14.fc25.x86_64/openjdk/build/jdk8.build/images/j2sdk-image/bin/javac -d . /builddir/build/SOURCES/TestECDSA.java
++ sed 's|\.java||'
+++ basename /builddir/build/SOURCES/TestECDSA.java
++ echo TestECDSA.java
+ /builddir/build/BUILD/java-1.8.0-openjdk-1.8.0.121-1.b14.fc25.x86_64/openjdk/build/jdk8.build/images/j2sdk-image/bin/java TestECDSA
Signature: 3045022055e475550f489a02b651c5ef66b06d1e51134c923353a997d8cb4694a01413be022100fea86df4d0cb4fa5b0b12b90936e079ca1b4b2cba41d101a6dc0c2bd0134dcde
Test passed.

The test is in [1]. So the rebuild fixed the issue as far as I can tell.

How can we avoid this in future? Kai, would it be possible for you to loop us in when new nss versions get introduced? That would allow us to build OpenJDK against it, we'd be part of the nss update and this should no longer be an issue. Thoughts?

[1] http://pkgs.fedoraproject.org/cgit/rpms/java-1.8.0-openjdk.git/tree/TestECDSA.java?h=f25

(In reply to jiri vanek from comment #34)
> latest builds
> java-1.8.0-openjdk-1.8.0.111-3.b16.fc25
>           nss x86_64  3.27.0-1.1.fc25
> java-1.8.0-openjdk-1.8.0.111-4.b16.fc25
>           nss x86_64  3.27.0-1.2.fc25
> java-1.8.0-openjdk-1.8.0.111-5.b16.fc25
>           nss x86_64  3.27.0-1.3.fc25 
> java-1.8.0-openjdk-1.8.0.121-1.b14.fc25 (current security update)
>            nss x86_64  3.28.1-1.3.fc25 
> java-1.8.0-openjdk-1.8.0.121-2.b14.fc25 (^ with disabled ECDHE)
>           nss x86_64  3.28.1-1.3.fc25 
> 
> So if Your susipicions are correct, it is worthy to try
> java-1.8.0-openjdk-1.8.0.121-1.b14.fc25 first. If it passes, I will revert 
> https://bugzilla.redhat.com/show_bug.cgi?id=1415137#c21

I installed java-1.8.0-openjdk-1.8.0.121-1.b14.fc24.x86_64 from koji (note: fc24, not fc25). The reproducer in comment #0, Maven, and IntelliJ IDEA, now all work without crashing; Maven even downloaded something using https. I haven't verified that ECDH works, but it's already much better.

java-1.8.0-openjdk-1.8.0.121-1.b14.fc25 java-1.8.0-openjdk-1.8.0.121-1.b14.fc25 has been submitted as an update to Fedora 25. https://bodhi.fedoraproject.org/updates/FEDORA-2017-4076cf8494

java-1.8.0-openjdk-1.8.0.121-1.b14.fc24 has been submitted as an update to Fedora 24. https://bodhi.fedoraproject.org/updates/FEDORA-2017-2176221cd2

> With that I don't get exceptions/coredumps using the reproducer from comment
> 0. I suggest to revert java-1.8.0-openjdk-1.8.0.121-2.b14.fc25. It's not
> needed.
> 
> 
> [1]
> http://pkgs.fedoraproject.org/cgit/rpms/java-1.8.0-openjdk.git/tree/
> TestECDSA.java?h=f25


Reverted. Bumped release in rawhide, creating updates with CPU with 121-1 for f24+f25

*** Bug 1416121 has been marked as a duplicate of this bug. ***

java-1.8.0-openjdk-1.8.0.121-1.b14.fc24 has been pushed to the Fedora 24 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-2176221cd2

*** Bug 1416574 has been marked as a duplicate of this bug. ***

*** Bug 1416525 has been marked as a duplicate of this bug. ***

(In reply to jiri vanek from comment #33)
> > At present, the dependency in java-1.8.0-openjdk is >=
> > %{NSS_BUILDTIME_NUMBER}. That works for preventing cases where OpenJDK is
> > re-built against a newer NSS, but I really think it should be switched to
> > '=' to prevent cases like this. The NSS upgrade would then be blocked until
> > OpenJDK is updated and it would be more obvious what the bug is.
> 
> That would be terrible lock without precedents I think.
> Also it may kill many automated services dealng with openjdk  rpm.
> NSS_BUILDTIME_NUMBER contans release. So that enfroces >=
> If NSS_BUILDTIME_NUMBER would be onl version, it  *may* work... Something 
> like >= NSS_BUILDTIME_NUMBER && <= NSS_BUILDTIME_VERSIO.99999
> 
> Will try to find something.
> 
> I will post NSS versions used during builds in next comment

In addition, this will bound us to rebuild with every nss update. Although it sounds like a good idea from point of view of this bug, it is quite an effort to all nss maintainer, jdk maintainer, koji, bodhi and even end user.

Thoughts?

(In reply to jiri vanek from comment #44)
> (In reply to jiri vanek from comment #33)
> > > At present, the dependency in java-1.8.0-openjdk is >=
> > > %{NSS_BUILDTIME_NUMBER}. That works for preventing cases where OpenJDK is
> > > re-built against a newer NSS, but I really think it should be switched to
> > > '=' to prevent cases like this. The NSS upgrade would then be blocked until
> > > OpenJDK is updated and it would be more obvious what the bug is.
> > 
> > That would be terrible lock without precedents I think.
> > Also it may kill many automated services dealng with openjdk  rpm.
> > NSS_BUILDTIME_NUMBER contans release. So that enfroces >=
> > If NSS_BUILDTIME_NUMBER would be onl version, it  *may* work... Something 
> > like >= NSS_BUILDTIME_NUMBER && <= NSS_BUILDTIME_VERSIO.99999
> > 
> > Will try to find something.
> > 
> > I will post NSS versions used during builds in next comment
> 
> In addition, this will bound us to rebuild with every nss update. Although
> it sounds like a good idea from point of view of this bug, it is quite an
> effort to all nss maintainer, jdk maintainer, koji, bodhi and even end user.
> 
> Thoughts?

It would, but on the positive side, it avoids bugs like this and makes it clear to users what the problem is immediately; they won't be able to update to the new NSS because of the OpenJDK requirement.

How complex can dependencies be? Can you say something like e.g. >= 3.28.0 but < 3.29.0, so we'd only rebuild on minor number bumps, not micro?

ABI stability of internal structures in NSS is not guaranteed at all, they can change even because we went from 3.28.0-1 to 3.28.0-2, as a result of a backported patch to fix a security issue

(In reply to Hubert Kario from comment #47)
> ABI stability of internal structures in NSS is not guaranteed at all, they
> can change even because we went from 3.28.0-1 to 3.28.0-2, as a result of a
> backported patch to fix a security issue

Thanx. Tahts pretty killing the idea.

The correct solution is to no longer use private interface of NSS.

NSS has external interfaces, and makes compatibility and stability promises for them. You can find them in the .def files.

Instead of using freebl directly, you should go through the interfaces that nss-softokn exports.

We should research, how you can achieve your technical requirements with the supported exported interfaces, and in case something is missing, add what's necessary.

I would like to suggest to start a discussion on this mailing list, which is the primary public place for discussing NSS:
  https://lists.mozilla.org/listinfo/dev-tech-crypto

Could you please subscribe to the list, and send a message, describing what you need to do, how you currently do it, and ask if there is a way to to it using exported interfaces?


(The package dependency issue should be avoided. If you require an "equal" package version, you will prevent everyone from being able to upgrade to a newer Firefox, until you have been able to adjust the openjdk package. This would be unfortunate. For all the other packages that use the exported interfaces of NSS, only, the usual requirement of a minimum NSS version is sufficient.)

see also:
  https://bugzilla.mozilla.org/show_bug.cgi?id=1334108

*** Bug 1416561 has been marked as a duplicate of this bug. ***

(In reply to Mikolaj Izdebski from comment #3)
> Created attachment 1242727 [details]
> working java.security file (no segfault)

it seems to me that it works. Hope it lasts. Thank you very much Mikolaj and all the other great people helping me.

*** Bug 1417092 has been marked as a duplicate of this bug. ***

(In reply to Fedora Update System from comment #41)
> java-1.8.0-openjdk-1.8.0.121-1.b14.fc24 has been pushed to the Fedora 24
> testing repository.

Why only F24? I'm on F25 and also affected. I would have been more than happy to test the fix.

(In reply to Julien HENRY from comment #54)
> (In reply to Fedora Update System from comment #41)
> > java-1.8.0-openjdk-1.8.0.121-1.b14.fc24 has been pushed to the Fedora 24
> > testing repository.
> 
> Why only F24? I'm on F25 and also affected. I would have been more than
> happy to test the fix.

See comment 37. An update has been submitted, but it's currently stuck for reasons which are beyond me. I've filed this ticket to get this resolved:
https://pagure.io/fedora-infrastructure/issue/5726

For the time being, please download builds from koji for F25. Sorry for the inconvenience.

java-1.8.0-openjdk-1.8.0.121-1.b14.fc24 has been pushed to the Fedora 24 stable repository. If problems still persist, please make note of it in this bug report.

*** Bug 1417295 has been marked as a duplicate of this bug. ***

I can confirm that version 1.8.0.121-1.b14.fc25 fixes azureus. Thanks!

java-1.8.0-openjdk-1.8.0.121-1.b14.fc25 has been pushed to the Fedora 25 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-4076cf8494

*** Bug 1417350 has been marked as a duplicate of this bug. ***

*** Bug 1417317 has been marked as a duplicate of this bug. ***

java-1.8.0-openjdk-1.8.0.121-1.b14.fc25 has been pushed to the Fedora 25 stable repository. If problems still persist, please make note of it in this bug report.

*** Bug 1416173 has been marked as a duplicate of this bug. ***

*** Bug 1417544 has been marked as a duplicate of this bug. ***

This is a follow up to my comment 49 and comment 50.

You actually didn't make a mistake, the mistake was made by upstream NSS.

The structure ECParams, which the openjdk code uses, was extended in version NSS 3.28, but it shouldn't have been extended.

We're trying to get this change revert up stream, and release a fixed version of NSS, with the original change revert.

Bob Relyea, who works on the fix for upstream NSS, is asking for testing, and feedback if the intended patch fixes the issue.


I suggest we produce a scratch build of NSS 3.28.x that contains the upstream candidate fix, and test that with the older openjdk builds that were broken.

Would you be able to help with such testing?

(In reply to Kai Engert (:kaie) from comment #65)
> This is a follow up to my comment 49 and comment 50.
> 
> You actually didn't make a mistake, the mistake was made by upstream NSS.
> 

I never thought we did.

> The structure ECParams, which the openjdk code uses, was extended in version
> NSS 3.28, but it shouldn't have been extended.
> 
> We're trying to get this change revert up stream, and release a fixed
> version of NSS, with the original change revert.
> 
> Bob Relyea, who works on the fix for upstream NSS, is asking for testing,
> and feedback if the intended patch fixes the issue.
> 
> 
> I suggest we produce a scratch build of NSS 3.28.x that contains the
> upstream candidate fix, and test that with the older openjdk builds that
> were broken.
> 
> Would you be able to help with such testing?

Just rebuild java-1.8.0-openjdk against the new NSS update. We don't want the old OpenJDK builds, as they don't include the latest security fixes.

(In reply to Andrew John Hughes from comment #66)
> 
> Just rebuild java-1.8.0-openjdk against the new NSS update. We don't want
> the old OpenJDK builds, as they don't include the latest security fixes.


We're trying to ensure (and test), that the most recent NSS will restore binary compatibility with old builds, that's why I'd like to test with the older Java build (which was broken initially when used with NSS 3.28.x)

In the meantime I've done some tests.


I've locally built NSS with the candidate upstream NSS fix.

Then I've installed the java package that was initially reported in this bug, version java-1.8.0-openjdk-1.8.0.111-5.b16.fc25.x86_64
  https://koji.fedoraproject.org/koji/buildinfo?buildID=832422

The test case succeeds.


Then I've upgraded java to the latest released package for f25, version 1.8.0.121-1.b14.fc25

I've confirmed that, as expected, the test case fails when using the experimental fixed NSS version.


Then I create a local build of the most recent checked in java package, which is java-1.8.0-openjdk-1.8.0.121-3.b14

I've confirmed this build passes the test case.


We'll wait for NSS upstream to accept the fix and release it. Then we should rebuild java again, and use a Conflicts: statement, to declare it incompatible with any NSS that's older (older than the upcoming version number).

I'll coordinate with you by email, or in a separate bug, to get that executed. We should then submit a combined Fedora update, which contains both the NSS update and the new java build. After that, we should be back at binary compatibility between NSS versions.

build against fixed nss in rawhide: https://koji.fedoraproject.org/koji/buildinfo?buildID=861016

hopefully fixed build for f25: https://koji.fedoraproject.org/koji/buildinfo?buildID=862733

and freshly launched f24: https://koji.fedoraproject.org/koji/taskinfo?taskID=18105734

java-1.8.0-openjdk-1.8.0.121-8.b14.fc25 nss-3.28.3-1.0.fc25 nss-softokn-3.28.3-1.1.fc25 nss-util-3.28.3-1.0.fc25 has been submitted as an update to Fedora 25. https://bodhi.fedoraproject.org/updates/FEDORA-2017-f66b383735

java-1.8.0-openjdk-1.8.0.121-8.b14.fc25 nss-3.28.3-1.0.fc25 nss-softokn-3.28.3-1.1.fc25 nss-util-3.28.3-1.0.fc25 has been submitted as an update to Fedora 25. https://bodhi.fedoraproject.org/updates/FEDORA-2017-f66b383735

java-1.8.0-openjdk-1.8.0.121-8.b14.fc25, nss-3.28.3-1.0.fc25, nss-softokn-3.28.3-1.1.fc25, nss-util-3.28.3-1.0.fc25 has been pushed to the Fedora 25 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-f66b383735

nss-3.28.3-1.0.fc24 nss-softokn-3.28.3-1.1.fc24 nss-util-3.28.3-1.0.fc24 java-1.8.0-openjdk-1.8.0.121-8.b14.fc24 has been submitted as an update to Fedora 24. https://bodhi.fedoraproject.org/updates/FEDORA-2017-f22b7e0c19

java-1.8.0-openjdk-1.8.0.121-8.b14.fc24, nss-3.28.3-1.0.fc24, nss-softokn-3.28.3-1.1.fc24, nss-util-3.28.3-1.0.fc24 has been pushed to the Fedora 24 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-f22b7e0c19

java-1.8.0-openjdk-1.8.0.121-8.b14.fc24, nss-3.28.3-1.0.fc24, nss-softokn-3.28.3-1.1.fc24, nss-util-3.28.3-1.0.fc24 has been pushed to the Fedora 24 stable repository. If problems still persist, please make note of it in this bug report.

java-1.8.0-openjdk-1.8.0.121-8.b14.fc25, nss-3.28.3-1.0.fc25, nss-softokn-3.28.3-1.1.fc25, nss-util-3.28.3-1.0.fc25 has been pushed to the Fedora 25 stable repository. If problems still persist, please make note of it in this bug report.