I can confirm that this issue is present in firefox-37.0.1-1.fc21.x86_64. I have tested on two separate Fedora installations with fresh profiles. With the build from firefox.com, it works, but with the build from the Fedora repository, it doesn't.

Kai, can you please look at it? Thanks!

Fedora 22 changed to end-of-life (EOL) status on 2016-07-19. Fedora 22 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.

Reproduced against Fedora 24, firefox-47.0-4.fc24.x86_64.

Hi,

I agree that HPKP still is not working correctly on Fedora 23/24. Too bad because it's a nice tool to improve security for your visitors.

How to reproduce:
=================
1) Open Firefox
2) Goto https://www.linqhost.nl
3) Prest CTRL-SHIFT-I
4) Goto Console
5) Enable server tab
6) Reload site

Console Result:
===============
Public-Key-Pins: The certificate used by the site was not issued by a certificate in the default root certificate store. To prevent accidental breakage, the specified header was ignored.

Workaround:
===========
Use the official Firefox binary coming from mozilla.org. This one DOES work correctly.

Some other observations:
========================
Fedora 23: Fedora Firefox 48 => NOT working
Fedora 24: Fedora Firefox 48 => NOT working
RHEL 7: Fedora Firefox 48 => Not working
Fedora 24: Official Firefox binarie 48 => Working
Centos 6: Firefox 45.3 => Working
Windows 10: Firefox 48 => working

Conclusion:
===========
It's something related to the Fedora Firefox build. As the official Firefox Binary does work. Maybe a bug in NSS build of Fedora?

Yes, seems to be related to NSS.

When Firefox built with in-tree nss, the HPKP extension test [1] is working. It's broken with our system nss.

Kai, is that something we can fix?

[1] https://projects.dm.id.lv/Public-Key-Pins_test

I hope my analysis from the initial post is already detailed enough. To reiterate, I think you have two choices here:

(1) Patch Firefox (and other browsers like Chromium) to trust P11-Kit's libnssckbi for verifying pinned keys.

(2) Patch P11-Kit's libnssckbi to look more like the stock NSS libnssckbi. This also obsoletes the existing patch 966424 to Firefox.

Maybe it would be best to implement (2) in P11-Kit upstream. Though I'm not sure whether replacing libnssckbi is really in scope of that project.

Thanks for the bug report and your analysis.
Unfortunately, the potential fix would have to be more complex than the suggestions from comment 7.

Firefox has decided to distinguish between root CAs that they ship as part of their code, and CAs that a user has configured.

(I believe we should attempt to follow the intention of the Firefox developers.)

These different sources of trusted CAs are easy to distinguish when using full Firefox binaries, by looking at the token name.

Unfortunately, for Fedora, looking at the token name doesn't entirely answer the question "system provided" or "user installed", since we have implemented our systemwide shared CA store.

With p11-kit-trust, we distinguish between 
- "Default Trust", which means those CAs have been installed at
    /usr/share/pki/ca-trust-source 
  which may be a ready only location on systems.
  This will contain the trust list from the ca-certificates package,
  but in addition, it might contain additional trust lists,
  installed as an RPM package as part of a deployment.

and

- "System Trust", which means those CAs have been installed at
    /etc/pki/ca-trust/source
  which is usually writable for the system administrator,
  and is the likely place for locally added CAs,
  that have been configured manually (without an RPM package).

(... and any CAs installed by the Firefox user will have a different token name, with both Mozilla.org binaries and Fedora binaries.)


If we wanted to follow Mozilla's intention, which tokens should we allow for HPKP?

Is Mozilla's intention to simply prevent pinning for user installed CAs?
Then the suggested patch would be sufficient.

Or, is Mozilla's intention to only allow pinning for CAs originally approved by Mozilla as part of the NSS releases?
Then the patch would be insufficient.


A solution, that perfectly matched the upstream behaviour, would implement an enhancement for the p11-kit package, that could detect which CAs have been provided by the ca-certificates package, and apply the libnssckbi token name "Builtin Object Token" for them, only.

This would require p11-kit to query the list of files inside the ca-certificates package, to be able to distinguish the token name from the names used for /usr and /etc sources.

I'm afraid we'll have to implement the perfect solution that I describe at the end of the previous comment.

After having had a chat with Mozilla developers of this feature, it became clear that we must correctly handle the scenario with MitM boxes and users with roaming devices.

As of today, in the Firefox default setting (non-strict), whenever the user visits a site, we find a trusted chain, but the root is a locally trusted CA (installed by the user into Firefox), then any earlier pin will be silently ignored.

The motivation is that people visit sites in the public web, receive a pin, then go back into their corporate environment, which might have set up a MitM box, which requires having installed a CA to allow connections through the MitM box without security errors.

If the pin was enforced for that scenario, too, then users couldn't connect to the sites from inside the allowed-MitM environment.


Now, let's assume for a moment we implemented a simpler solution, like treating locally admin-configured CAs as HPKP-allowed, as the patch suggests?

As a result, a user who received a pin on the public web, when travelling back to the internal MitM environment, would get a chain with the root CA from the MitM box. Because that MitM root is treated as HPKP-allowed, and therefore Firefox would decide that enforcing the pin is necessary. But the server key presented by the MitM box doesn't match the expectations, and the connection would fail.


I believe this scenario (which unfortunately seems to be common) demonstrates that we cannot use the simple patch as it has been suggested.

(In reply to Kai Engert (:kaie) from comment #9)
> After having had a chat with Mozilla developers of this feature, it became
> clear that we must correctly handle the scenario with MitM boxes and users
> with roaming devices.
> 
> As of today, in the Firefox default setting (non-strict), whenever the user
> visits a site, we find a trusted chain, but the root is a locally trusted CA
> (installed by the user into Firefox), then any earlier pin will be silently
> ignored.
> 
> The motivation is that people visit sites in the public web, receive a pin,
> then go back into their corporate environment, which might have set up a
> MitM box, which requires having installed a CA to allow connections through
> the MitM box without security errors.
> 
> If the pin was enforced for that scenario, too, then users couldn't connect
> to the sites from inside the allowed-MitM environment.

Since many users (or maybe all users outside of big corporate environments) are not interested in having their browser MitM-ed, will there be a config option to turn this down? Or will NSS distinguish between two different types of locally installed certificates? Just because I'm installing a CA to a computer doesn't mean I want this CA to be able to MitM all my traffic…

Yeah, I wouldn't want my devices being MitM'd, either, just because I've installed certs for VPN use and/or to connect to intranet services.

The HPKP pinning is implemented at the Firefox application level, not in the NSS library level.

I was told, there's a pref in Firefox, that controls if HPKP pinning is strict or non-strict. By default it's non-strict, and pins can be ignored if cert chains to a CA treated as "locally installed". If you never want your pins to be ignored, you apparently are able to configure that pref to "strict" (I don't know the name of the pref).

Note this comment describes the intended behavior of Firefox, based on my understnding, independent of the current state of Fedora.

After thinking about this problem more, here's a suggestion for a solution in Fedora:

- don't patch Firefox

- change the filenames used by the ca-certificates package.
  All files that contain CAs from the original Mozilla/NSS list,
  use a filename that starts with "mozilla-nss-".

- should the ca-certificate package ever ship any CA certificates as trusted,
  which aren't part of the original Mozilla/NSS list,
  the they will be installed in files, that start with a different prefix
     (not with "mozilla-nss-").
  (It's likely that we will never distribute such other CAs as part of the
   ca-certificates package.)

- after the above change to ca-certificates is ready, implement a change
  to p11-kit-trust

- in p11-kit-trust, implement the following rule:
  If the name of a loaded file starts with "mozilla-nss-",
  then use the token name "Builtin Object Token"
  for all CAs loaded from that file.
  (Any CAs that are loaded from files with other filenames will
   continue to use the different token names.)

With the above strategy, we could ensure that any CAs locally installed by an administrator (installed in files with names that don't start with "mozilla-nss-") will never be accepted for pinning.

I suggest that we assign this bug to the p11-kit component, for implementing a fix to p11-kit-trust.

I'll file a separate bug against the ca-certificates package, to get the filenames changed.

(In reply to Kai Engert (:kaie) from comment #12)
> I was told, there's a pref in Firefox, that controls if HPKP pinning is
> strict or non-strict. By default it's non-strict, and pins can be ignored if
> cert chains to a CA treated as "locally installed". If you never want your
> pins to be ignored, you apparently are able to configure that pref to
> "strict" (I don't know the name of the pref).

You probably mean security.cert_pinning.enforcement_level with these possible values:

    0. Pinning disabled
    1. Allow User MITM (pinning not enforced if the trust anchor is a user inserted CA, default)
    2. Strict. Pinning is always enforced.
    3. Enforce test mode.

Documentation: https://wiki.mozilla.org/SecurityEngineering/Public_Key_Pinning#How_to_use_pinning

I'd like to see this set to 1. I've reported this upstream at mozilla: https://bugzilla.mozilla.org/show_bug.cgi?id=1306833

With regard to not processing HPKP headers when a site's certificate was issued by what Firefox considers to be a non-built-in root, there's the boolean preference "security.cert_pinning.process_headers_from_non_builtin_roots" (I suppose you would have to set this to true as well as setting "security.cert_pinning.enforcement_level" to 2 to get what you want here).

Note also that the way Firefox determines if a certificate is a built-in root or not changed recently - see https://bugzilla.mozilla.org/show_bug.cgi?id=1272858 and https://hg.mozilla.org/mozilla-central/rev/50a6f462a97c (this landed on Firefox 50).

If this new token would contain only the CAs shipped by mozilla it may make sense to give it a more descriptive name for its contents (e.g., "Mozilla Trust").

commit comment from the change the David referred to:


use a name-agnostic method to find the built-in root PKCS#11 slot r=Cykesiopka

Previously this implementation would use the expected names of the built-in
module and slot to get a handle on them. This doesn't work on distributions that
use other names. The new implementation searches through the slots from the
default module list for one where PK11_HasRootCerts returns true (which
indicates that NSS considers that slot to contain the default built-in root
list).

Does p11-kit-trust return true for the PK11_HasRootCerts call?

If yes, is that really what we want?

Because, if it returns true, and the system has added local CAs that are used for enterprise MitM, then pins acquired externally will cause pin failures, which seems what Mozilla tried to avoid with pref value 1.

Surely this should be handled with a NSS-specific additional attribute (don't we already have a bunch of PKCS#11 objects for those anyway?) which indicates that a given cert should be treated this way? Rather than magically inferring it depending on where the cert came from?

I just noticed that upstream Firefox has changed their logic, how they detect if roots are on a builtin root module, or not.

Apparently that fix has been implemented in upstream Firefox 50.
https://bugzilla.mozilla.org/show_bug.cgi?id=1272858

Could someone of you having access to pinned sites please give an update on the current status on this with Firefox in Fedora?

Does pinning now work in Fedora?

... and ...

Because Firefox probably cannot distinguish between the public web CAs and locally installed CAs, that we now have the situation that connections are broken when users roam inside and outside of a corporate MITM environment?

Jan, do you have a response to comment 20? Thanks.

I can't test whether it works in Fedora now, but Firefox in Arch (with the patched libnssckbi) didn't behave differently since 50.0.

We're indeed in a situation where pins are applied even for locally installed roots, so browsing behind a MITMing proxy runs into validation errors: https://bugs.archlinux.org/task/52080

As far as I understand the patch, it can only result in *more* certs getting subject to pinning, not less. It should make the renaming patch we've been using obsolete, as far as a Firefox is concerned.

I'll double check this later today by using the upstream p11-kit-proxy.so, but I expect that nothing will change when I do. If so, Fedora is probably in the same situation now.

Removing the patch is a rather unattractive option because other code still looks at the names. I believe most of our users are better served by working pins.

I don't quite understand why we still have this magical decision based on where the CA comes from (be it by name, or other matching criteria).

Why isn't this just an additional attribute on the CA itself, akin to a 'purpose', for allowing it to pin certs that it validates?

NSS already stores enough additional objects in the PKCS#11 token for non-standard attributes associated with a cert; isn't that the sanest approach to this issue too?

David, you're right. And we've started to implement that approach.

For everyone interested, see the dependency list of bug 1400293. The fix we're working on should address both bugs.

Note to self:

The HPKP code in Firefox that checks for builtin roots is in 
nsSiteSecurityService::ProcessPKPHeader :

  bool isBuiltIn = false;
  mozilla::pkix::Result result = IsCertBuiltInRoot(rootNode->cert, isBuiltIn);
  if (result != mozilla::pkix::Success) {
    return NS_ERROR_FAILURE;
  }

  if (!isBuiltIn && !mProcessPKPHeadersFromNonBuiltInRoots) {
    if (aFailureResult) {
      *aFailureResult = nsISiteSecurityService::ERROR_ROOT_NOT_BUILT_IN;
    }
    return NS_ERROR_FAILURE;
  }

To everyone who has experienced this bug:

Could you please verify, that this bug is fixed, and HPKP behaves as expected, with the following packages?

- firefox-51.0.1-11.fc27 or firefox-51.0.1-11.fc26
- p11-kit-0.23.5-1.fc27  or p11-kit-0.23.5-1.fc26
- nss-3.29.1-3.fc27      or nss-3.29.1-2.1.fc26
- ca-certificates-2017.2.11-5.fc26

Actually, all the tests on https://projects.dm.id.lv/Public-Key-Pins_test including the more precise manual tests were run successfully on a F25 platform with these packages installed:

firefox-51.0.1-2.fc25.x86_64
p11-kit-0.23.2-2.fc24.x86_64
nss-3.28.3-1.0.fc25.x86_64
ca-certificates-2017.2.11-1.0.fc25.noarch

I tested using a fresh firefox profile.

(In reply to Christian Stadelmann from comment #27)
> Actually, all the tests on https://projects.dm.id.lv/Public-Key-Pins_test
> including the more precise manual tests were run successfully on a F25
> platform with these packages installed:

Christian, with the packages from comment 27, the positive results you got were misleading.

Because Firefox upstream had changed the way they decide if a root CA is public or private, with the root token we use in Fedora, things flipped to always incorrectly conclude "it's a public web root", and consequently, the behavior was changed to "HPKP always enabled", even if you were connecting to a private CA.

Unless you specifically tested edge cases, where you tested with an installed MITM gateway and with a private CA installed for that that MITM gateway, your test was incomplete.

Also, since package version firefox-51.0.1-7.fc24 (and fc25), we have temporarily disabled the feature altogether, to work around bug 1400293.

If you'd like to test the real fix, it's currently necessary to use developer snapshots of Fedora 26 or Rawhide, using the builds I mentioned in comment 26.

I'm hoping for some feedback, on both this bug and also bug 1400293, before we can merge it into stable F24 / F25.

Observations from a fresh F25 VM with firefox-52.0-6.fc25 and google-chrome-stable-57.0.2987.110-1.x86_64 (from Google's repo) -- pinning does *not* work as expected with either browser. Pinning works as expected with both browsers in Ubuntu 16.04.

Observed:

1) visit https://pinning-test.badssl.com/
2) no warning is presented

Chrome dev tools->Security states "Public-Key Pinning Bypassed
Public-key pinning was bypassed by a local root certificate."

Expected:

1) visit https://pinning-test.badssl.com/
2) warning is presented (either MOZILLA_PKIX_ERROR_KEY_PINNING_FAILURE [firefox] or NET::ERR_SSL_PINNED_KEY_NOT_IN_CERT_CHAIN [chrome])

Ah, just saw you wanted F26. I'll test with the Alpha once that's released.

(In reply to dw from comment #29)
> Expected:
> 
> 1) visit https://pinning-test.badssl.com/
> 2) warning is presented (either MOZILLA_PKIX_ERROR_KEY_PINNING_FAILURE
> [firefox] or NET::ERR_SSL_PINNED_KEY_NOT_IN_CERT_CHAIN [chrome])

I'm getting MOZILLA_PKIX_ERROR_KEY_PINNING_FAILURE with Firefox 52.0-6.fc25 and nss-3.29.3-1.0.1.with_tls1_3.fc25.x86_64 as described in devel mailing list: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/44O6EYI3GW7HSUU6YXLKIYGXZ5REM2KW/

Fedora 25 is NOT yet fixed.

I'm asking for your feedback by testing Fedora Rawhide or Fedora Branched F26.

I'd like to get confirmation that the fix is working, prior to backporting to the stable Fedora releases.

Fresh, dnf update-d F26 Alpha with:

- firefox-52.0-7.fc26
- p11-kit-0.23.5-1.fc26
- nss-3.29.3-1.3.fc26
- ca-certificates-2017.2.11-5.fc26
- google-chrome-stable-57.0.2987.133-1

tested with https://pinning-test.badssl.com

- Firefox gives me a MOZILLA_PKIX_ERROR_KEY_PINNING_FAILURE warning as expected
- Chrome still presents the site without warning

Chrome->Developer tools->Security still says 

Public-Key Pinning Bypassed
Public-key pinning was bypassed by a local root certificate.

dw, thank you for testing.

I understand your report as:
- Firefox works as expected
- Chrome is broken

My focus so far has been on Firefox, and I haven't looked at other browsers.

Chrome might require similar logic than it has been implemented for Firefox,
see https://bugzilla.mozilla.org/show_bug.cgi?id=1324096

I suggest to report a bug to the developers of Chrome.
If you, could you please add a reference here?

I reported it and linked them to this bug after doing my testing. I was told it was a duplicate of a previously reported bug with the original bug being https://bugs.chromium.org/p/chromium/issues/detail?id=707280 . However, I don't have permission to view the bug :/. I suggested they get in touch with people handling this bug so Fedora and Chrome aren't both playing musical chairs with their CAs and detection code.

This message is a reminder that Fedora 24 is nearing its end of life.
Approximately 2 (two) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 24. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as EOL if it remains open with a Fedora  'version'
of '24'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version'
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not
able to fix it before Fedora 24 is end of life. If you would still like
to see this bug fixed and are able to reproduce it against a later version
of Fedora, you are encouraged  change the 'version' to a later Fedora
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's
lifetime, sometimes those efforts are overtaken by events. Often a
more recent Fedora release includes newer upstream software that fixes
bugs or makes them obsolete.

Firefox has been fixed, so this bug can be closed.

Fedora 24 changed to end-of-life (EOL) status on 2017-08-08. Fedora 24 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.