Kai, I guess you may help here.

Is there a way to debug the logic which causes Firefox to reject the certificate?

The web server is "catcage.luxaviation-services.com" (a temp test machine for Tomcat experiments). The root CA certificate can from the page obtained by clicking on the "SSL certificates" link.

We are also experiencing this issue after upgrading to Firefox 50.  The site certificate was issued by our own internal CA and other browsers such as Chrome and IE have no issues accessing the site over SSL.  Firefox 49 loads the page with no errors.

ditto here; Firefox 49 worked fine, 50 fails with SSL_ERROR_BAD_CERT_DOMAIN.

I workaround the problem for now by generating the certificates with an additional DNS AlternateName matching the CN.

I'm able to access the site after adding a certificate exception however the site still shows as "insecure" in the address bar.  This only works if the site has HSTS disabled as well, firefox will not even *allow* you to add an exception for sites with HSTS enabled.

What info do we need?

Any update on this?  It's pretty annoying that we have to add an exception for every site that uses a cert issued by our own CA.

I'm investigating

The underlying code is tricky to debug, but I believe I already understand enough to make a good guess on the probable cause.

If a certificate has been issued after a certain date (*1), AND the root CA identified for that certificate is stored in the embedded trust module, then Firefox requires that the DNS name for the host is listed in the SubjectAltName extension of the certificate. In that case, Firefox no longer falls back to checking the common name of the certificate subject name.

Apparently that requirement originates in the Baseline Requirements defined by the CabForum, and Mozilla has started to enforce it. As an example see (*2)

It seems that the original intention of the Firefox code was to restrict this requirement to the CAs that are approved by Mozilla as part of their root CA program. However, to draw that conclusion, the Firefox code apparently checks if the CA is stored in a PKCS#11 module containing root CAs. That logic works when Firefox is built standalone and uses the original NSS module for CAs.

On Fedora, because we use the p11-kit-trust that allows to install additional CAs for the whole system, the Firefox logic (incorrectly) decides that CAs, that have been added to the local systemwide configuration, also belong into that category, and enforces the same requirements.

(This is why it works when you download an original Firefox build from Mozilla, that doesn't use Fedora's global trust store, and import your private CA into the trust store of the Firefox user profile.)

So, the immediate workaround is:

Update the certificates you use for your servers, and use certificates that have a subject alternative name extension.


If we believe that we really shouldn't enforce these requirements for private CAs (like the firefox programmers seem to have originally intended), then we'd have to patch the Firefox code for the Fedora builds, and disable the enforcement of this requirement. However, we'd have to disable it unconditionally, because at this time, we're unable to distinguish at runtime the CAs that originate from Mozilla's root CA program from those that have been added by a Fedora system administrator.


I'll bring this situation to the attention of the firefox developers.


*1 The date is apparently 2016-08-23, although I see an additional date
   in the firefox sources, one year earlier, 2015-08-23.
   I haven't yet debugged which of both dates is effective in
   which scenarios.

*2 https://bugzilla.mozilla.org/show_bug.cgi?id=1261919

I've reported the situation to Mozilla:
https://bugzilla.mozilla.org/show_bug.cgi?id=1324096

Just another context bug:
https://bugzilla.mozilla.org/show_bug.cgi?id=1245280

I've made an experimental/scratch build, that disables these additional restrictions (and therefore is less "secure" (or less strict) than Mozilla has intended), but which apparently works with the test sites:
https://koji.fedoraproject.org/koji/taskinfo?taskID=16919288

I'm running into the same issue at home. I have a personal installation of FreeIPA:

* private root CA in /etc/pki/ca-trust/source/ipa.p11-kit
* intermediate CA in FreeIPA / Dogtag PKI
* EE cert without a SAN extension

The server sends intermediate and EE cert in the TLS server hello. The hostname matches the subject CN of the EE cert. When I build my personal setup a couple of months ago, it worked fine with Firefox. An update about a month or two ago caused Firefox refuses the cert with error SSL_ERROR_BAD_CERT_DOMAIN.

I fear that this issue is going to affect users of FreeIPA. FreeIPA installs its private root CA in /etc/pki. FreeIPA just started to role out EE certs with a proper SAN extension a few versions ago (https://fedorahosted.org/freeipa/ticket/4970). Older deployments and old certs still lack SAN.

You should be able to update your EE cert by generating a new CSR using the same key as the current cert.  For example:

export sans="[SAN]\nsubjectAltName=DNS:example.com,DNS:www.example.com,DNS:www2.example.com\n"
cp /etc/pki/tls/openssl.cnf /root/ssl/
printf $sans >> /root/ssl/openssl.cnf

openssl req -new -sha256 -key site.example.key -reqexts SAN -config /root/ssl/openssl.cnf -subj '/C=US/ST=State/L=City/O=Example Corp/OU=IT/CN=server.example.com/emailAddress=it' -out site.csr

The CSR would then need to be signed by the IPA CA.

I've also noticed that this behavior in Firefox isn't consistent.  Some of our sites load fine over TLS and others will show a bad certificate warning as reported here.  The certs are all signed by the same CA as well.

(In reply to Christian Heimes from comment #13)
> I'm running into the same issue at home. I have a personal installation of
> FreeIPA:

[...]

> I fear that this issue is going to affect users of FreeIPA. FreeIPA installs
> its private root CA in /etc/pki. FreeIPA just started to role out EE certs
> with a proper SAN extension a few versions ago
> (https://fedorahosted.org/freeipa/ticket/4970). Older deployments and old
> certs still lack SAN.

Running

   ipa-getcert resubmit -d /etc/httpd/alias -n Server-Cert -D $( uname -n )

fixed the FreeIPA server for me in this situation.

(In reply to Michael Watters from comment #14)
> I've also noticed that this behavior in Firefox isn't consistent.  Some of
> our sites load fine over TLS and others will show a bad certificate warning
> as reported here.  The certs are all signed by the same CA as well.

Does the issue date explain it?
See *1 in comment 9.

We have made progress on the dependency bugs, and I hope that we can have a stable fix within the next 6 weeks.

In the meantime, to prevent people from suffering from this bug, I suggest to use a workaround patch, that temporarily disables these checks, by assuming that all CAs are private CAs. I'll attach a patch for that.

Created attachment 1248808 [details]
suggested temporary workaround for firefox 52

Added to firefox-51.0.1-7.fc24

openQA tests of F24 critpath updates are (I believe) running into this:

https://openqa.stg.fedoraproject.org/tests/73335

note the failed (red border) screens are Firefox complaining about the HTTPS connection to the IPA server web interface. Same test run on F25 works OK. This test runs on two hosts, one server, one client; the server deploys itself as a FreeIPA domain controller then waits, the client enrols into the domain using Cockpit, then restarts Firefox and attempts to test the FreeIPA web UI.

Can confirm that Jan's command from #c15 works around this issue for FreeIPA on F24.

I'd like to see this fixed and tested asap.

I'd like us to backport the upstream fix for this bug, which will also fix bug 1207335, in both F26 and F27 rawhide.

We already prepared the ca-certificates and p11-kit packages to provide the required base.

We need a backport of the following upstream bug fix for NSS:
  https://bugzilla.mozilla.org/show_bug.cgi?id=1334976

For correctness of the symbol export tables, we need to combine that with a another trivial symbol export from the upcoming release:
  https://bugzilla.mozilla.org/show_bug.cgi?id=1336487

(We have upstream approval that there won't be any other symbols with that version tag, so we can go ahead prior to upstream release.)

I made a combined patch for those two NSS bugs, which I'll attach.

Once we have that NSS build, we can create updated firefox builds, that strictly depend on the newer nss builds, and pick up the simple firefox code patch from
  https://bugzilla.mozilla.org/show_bug.cgi?id=1324096
and revert the workaround patch from this bug.

Created attachment 1258809 [details]
NSS backport

Created attachment 1259170 [details]
firefox patch, mozilla-1324096.patch

I'm removing the old "fixed in version" information, because that was only for a workaround, which disabled some checks.

Jan, Martin, all dependencies are provided,
we are ready to apply the "real" fix for Firefox, in Rawhide and F26.

The remaining steps for firefox.rpm are:

- please remove the workaround patch rhbz-1414535.patch

- please add the real fix, mozilla-1324096.patch (attached)

@@ -135,7 +135,7 @@ Patch224:        mozilla-1170092.patch
 Patch225:        mozilla-1005640-accept-lang.patch
 #ARM run-time patch
 Patch226:        rhbz-1354671.patch
-Patch227:        rhbz-1414535.patch
+Patch227:        mozilla-1324096.patch
 
 # Upstream patches
 Patch304:        mozilla-1253216.patch


To build, you depend on the backport from NSS 3.30 (upstream unreleased).

The backport has been provided in the following Fedora builds:

- nss-util-3.29.1-3.fc27
  https://koji.fedoraproject.org/koji/taskinfo?taskID=18137243

- nss-3.29.1-3.fc27
  https://koji.fedoraproject.org/koji/taskinfo?taskID=18137456

- nss-util-3.29.1-2.1.fc26
  https://koji.fedoraproject.org/koji/taskinfo?taskID=18137547

- nss-3.29.1-2.1.fc26
  https://koji.fedoraproject.org/koji/taskinfo?taskID=18137693


There are earlier 3.29.1 builds, without the backport, so it's necessary to require the build/release version, too.

I suggest the following, which works for both Rawhide and F26:


@@ -195,6 +195,11 @@ Requires:       nspr >= %{nspr_build_version}
 Requires:       nss >= %{nss_build_version}
 %endif
 
+# For early testing of rhbz#1400293 mozbz#1324096, temporarily require
+# a specific NSS version. Can be removed after we depend on NSS 3.30
+BuildRequires:  nss-devel >= 3.29.1-2.1
+Requires:       nss >= 3.29.1-2.1
+
 BuildRequires:  desktop-file-utils
 BuildRequires:  system-bookmarks
 %if %{?system_sqlite}


A Rawhide scratch build with above changes was successful:
  https://koji.fedoraproject.org/koji/taskinfo?taskID=18138818

I've tested that build, it works as expected.

Builds are here:
https://koji.fedoraproject.org/koji/buildinfo?buildID=863780
https://koji.fedoraproject.org/koji/buildinfo?buildID=863778

To everyone who has experienced this bug:

Could you please verify, that this bug is fixed, and both public and private CAs behave as expected, with the following packages?

- firefox-51.0.1-11.fc27 or firefox-51.0.1-11.fc26
- p11-kit-0.23.5-1.fc27  or p11-kit-0.23.5-1.fc26
- nss-3.29.1-3.fc27      or nss-3.29.1-2.1.fc26
- ca-certificates-2017.2.11-5.fc26

Hi,

(I had same problem: PKI FreeIPAv3 with certificate without SAN)

After upadate, i confirm that fix works, my environment:
Fedora release 24 (Twenty Four)
firefox-52.0-1.fc24.x86_64

Your patch disable SAN requirement on private CA isn't it?

(In reply to David Goudet from comment #29)
> 
> Your patch disable SAN requirement on private CA isn't it?

Yes.

Created attachment 1274533 [details]
patch for firefox.spec to enable the fix for F24+F25

We're ready to fix this bug correctly (and remove the workaround) for F24 and F25. I've tested a F25 scratch build using the backported patch, and it works as expected with the test cases.

I'm attaching a patch for the firefox package, to enable the backported patch from Firefox 54, remove the workaround patch, and add required package dependencies for F24/F25.

Jan/Martin, do you think you can pick up this fix for F24/f25 early, while we're still on FF53?

(If you want, you could pick up this fix for Thunderbird, too.)

I'll look into it also for the TB.

This message is a reminder that Fedora 24 is nearing its end of life.
Approximately 2 (two) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 24. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as EOL if it remains open with a Fedora  'version'
of '24'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version'
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not
able to fix it before Fedora 24 is end of life. If you would still like
to see this bug fixed and are able to reproduce it against a later version
of Fedora, you are encouraged  change the 'version' to a later Fedora
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's
lifetime, sometimes those efforts are overtaken by events. Often a
more recent Fedora release includes newer upstream software that fixes
bugs or makes them obsolete.

Fedora 24 changed to end-of-life (EOL) status on 2017-08-08. Fedora 24 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.