Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.
Bug 1381671 - Fails to connect to Google, with legacy CAs disabled, or with ca-certificates version 2.10
Summary: Fails to connect to Google, with legacy CAs disabled, or with ca-certificates...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: empathy
Version: 25
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Debarshi Ray
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
: 1385539 (view as bug list)
Depends On:
Blocks: 1386616
TreeView+ depends on / blocked
 
Reported: 2016-10-04 17:08 UTC by Kai Engert (:kaie) (inactive account)
Modified: 2017-04-01 17:23 UTC (History)
17 users (show)

Fixed In Version: empathy-3.12.13-2.fc25 empathy-3.12.13-2.fc26
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1386616 (view as bug list)
Environment:
Last Closed: 2017-03-23 18:21:56 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)
A chain that fails to validate in telepathy-gabble/wocky (5.87 KB, text/plain)
2016-10-04 17:08 UTC, Kai Engert (:kaie) (inactive account)
no flags Details


Links
System ID Private Priority Status Summary Last Updated
GNOME Bugzilla 753260 0 None None None 2017-01-19 15:03:50 UTC
GNOME Bugzilla 780160 0 None None None 2017-03-16 18:33:31 UTC

Description Kai Engert (:kaie) (inactive account) 2016-10-04 17:08:17 UTC
Created attachment 1207293 [details]
A chain that fails to validate in telepathy-gabble/wocky

Jan reported that in Fedora 25, which disables the legacy CAs, the combination of empathy/gnome-online-accounts/telepathy-gabble/wocky fails to connect to Google Talk.

The chain reported by the server can be seen with the following command:
 openssl s_client -showcerts -tls1 -starttls xmpp -connect talk.l.google.com:5222

(I'll also attach the chain.)

Jan said, wocky implements its own certificate validation on top of GnuTLS.

The issue is, the server sends an intermediate that points to "Equifax Secure Certificate Authority". However, if the software ignored that topmost part of the chain, the software should be able to create a different chain, based on an earlier intermediate sent in the chain, which points to "GeoTrust Global CA".

I think we shouldn't change our plans, we should continue with the plan to disable all legacy CAs in Fedora 25.

How can we get the developers of this software to change their code? They shouldn't use their own implementation of certificate validation, but use one provided by the libraries (by GnuTLS, if that's what they use), which already is finding potentially shorter and working chains of trust.

Comment 1 Kai Engert (:kaie) (inactive account) 2016-10-04 17:14:04 UTC
Note there's even another issue lingering here.

The legacy CA this server is chaining to, has been kept as a legacy CA, because it's using a 1024-bit RSA key.

In the meantime, Mozilla has decided to completely remove this root for any purpose, because it's no longer included in published audit statements
(see number 4 in https://bugzilla.mozilla.org/show_bug.cgi?id=1288250 )

Because of that, I've intended to completely remove those, even remove them from the legacy CA list 2.10, for all supported versions of Fedora, as part of the most recent CA list update (published as part of NSS 3.27).

(I think I'll have to use a longer testing phase for the ca-certificates 2.10 update, to see how much it would break.)

Comment 2 Kai Engert (:kaie) (inactive account) 2016-10-04 17:27:57 UTC
Brian, how can we reach the developers that maintain the certificate validation inside this software? Do you have contacts?

Comment 3 Roman Kagan 2016-10-14 11:52:53 UTC
F24 is affected too.

With ca-certificates-2016.2.10-1.0.fc24.noarch arrived a couple of days ago telepathy started to complain about the certificate being self-signed (which is confusing because it pops up a dialog with gmail.com certificate signed by Google Internet Authority G2, and it takes quite an effort to figure out that in reality it can't verify Equifax' signature on GeoTrust's certificate).

On a quick check Google seems to include this certificate on other services, too: https://www.google.com/, imaps://imap.gmail.com/, etc.  Only telepathy-gabble broke with removal of Equifax certificates, though.

Comment 4 Kai Engert (:kaie) (inactive account) 2016-10-14 12:19:57 UTC
The problem is that the server uses a compatibility configuration, which requires client software to be smart, and be able to find an alternative chain of trust.

I have explained the background here:
https://rt.openssl.org/Ticket/Display.html?id=3621#txn-49999

It's the client software that is broken and incompatible with modern requirements.

The developers of the broken software should switch to better certificate verification code (and preferably not implement this part on their own, but rather use standard code from a SSL/TLS library).

Comment 5 Tom Prince 2016-10-18 23:31:27 UTC
*** Bug 1385539 has been marked as a duplicate of this bug. ***

Comment 6 Nikos Mavrogiannopoulos 2016-10-19 09:57:20 UTC
I'm not sure we have the right component. I see in the empathy debug window:
empathyTls-DEBUG: 10/19/2016 09:44:07.79547: perform_verification: Performing verification
empathyTls-DEBUG: 10/19/2016 09:44:07.79591: debug_certificate_chain: Certificate chain: length 3 status incomplete
empathyTls-DEBUG: 10/19/2016 09:44:07.79890: debug_certificate: Certificate: C=US, ST=California, L=Mountain View, O=Google Inc, CN=gmail.com
empathyTls-DEBUG: 10/19/2016 09:44:07.80071: debug_certificate: Certificate: C=US, O=Google Inc, CN=Google Internet Authority G2
empathyTls-DEBUG: 10/19/2016 09:44:07.80249: debug_certificate: Certificate: C=US, O=GeoTrust Inc., CN=GeoTrust Global CA
empathyTls-DEBUG: 10/19/2016 09:44:07.80549: perform_verification: Certificate verification gave result 0 with reason 6
empathyTls-DEBUG: 10/19/2016 09:44:07.80606: abort_verification: Verification error 6, aborting...


but I can find no trace of these debug messages at telepathy-gabble (which seems to not use the system certificates for verification -though its verification code is ok).

The issue seems to be in empathy itself perform_verification_cb() and perform_verification(), which attempts to create the trust chain.

Comment 7 Roman Kagan 2016-11-10 12:49:18 UTC
OK the problem is not in empathy but in gcr,
gcr/gcr-certificate-chain.c:perform_build_chain() which is the main worker in

...
  empathy_tls_verifier_verify_async
    gcr_certificate_chain_build_async
      thread_build_chain
        perform_build_chain


perform_build_chain() traverses the certificate chain passed in by the caller and looks up the certificate *itself* among pkcs11 anchors, and only looks up the issuer of the last certificate in the chain.

Apparently it should rather look up the issuer of the current certificate first, and, if found, stop the chain traversal.

Can someone with the rights please adjust the component?

Comment 8 Kai Engert (:kaie) (inactive account) 2016-11-10 12:57:53 UTC
Roman, thanks a lot for your research. Changing component.

Comment 9 Tomas Mraz 2016-11-10 13:08:34 UTC
However it is unclear whether gcr needs to perform its own certificate validation - why doesn't it delegate this job to gnutls?

Comment 10 Stef Walter 2016-11-10 13:20:33 UTC
> However it is unclear whether gcr needs to perform its own certificate validation - why doesn't it delegate this job to gnutls?

It certainly should. The certificate assembly and validation in GCR predates PKCS#11 support in GnuTLS. GCR and/or its callers should be migrated to use the latter now.

Comment 11 Jan Alexander Steffens 2016-11-10 13:22:29 UTC
Does this mean GCR should pick up a dependency on GnuTLS?

It already links to libp11-kit. Can p11-kit do this, too?

Comment 12 Stef Walter 2016-11-10 13:27:52 UTC
Nah, p11-kit isn't about TLS. GnuTLS uses p11-kit to load certificates and keys, but p11-kit doesn't know how they'll be chained, assembled, used or verified.

I spoke too hastily above though. glib-networking is the thing that has the multiple TLS backends and the GnuTLS dependency ... it also has custom certificate validation code that predates GnuTLS working with PKCS#11.

Gcr custom chain code should be removed and glib-networking's custom certificate chain code implemented using GnuTLS.

Comment 13 Kai Engert (:kaie) (inactive account) 2016-11-10 13:38:26 UTC
I think glib-networking was already changed to use the validation code from GnuTLS. If gcr already has a dependency on glib-networking (does it?), then maybe gcr could reuse the glib-networking code that uses GnuTLS?

Adding Dan, who has worked in this area.

Comment 14 Stef Walter 2016-11-10 13:48:12 UTC
If that's the case, that's great.

I wonder what callers actually use the Gcr code, if any. We may be able to just remove those APIs in favor of glib-networking. The Gcr API is marked as unstable, so with API removal ... along with some patches to known API callers ... I think we would be in good shape.

Comment 15 Dan Winship 2016-11-10 13:53:24 UTC
glib-networking uses gnutls's validation code by default, but currently still uses the broken internal code when using PKCS#11. See https://bugzilla.gnome.org/show_bug.cgi?id=753260#c9

Comment 16 jpg 2016-11-22 14:35:53 UTC
I'm hitting this issue... any workaround?

Comment 17 Roman Kagan 2017-01-19 11:11:23 UTC
Looks like this bug fell through the gaps between maintainers' responsibility zones...

Has it been decided what needs to be done?  Is anybody working on this?

Comment 18 Dan Winship 2017-01-19 13:21:39 UTC
Comment 15 links to the upstream bug that shows what needs to be done. So it's not fixed upstream yet, so there's nothing Fedora maintainers can do. (And it's not being worked on upstream, so there's no ETA, though if you want to try to dive in and tackle it, the upstream bug shows what needs to be done and I can at least help out with review/questions.)

Comment 19 Dan Winship 2017-03-01 15:58:09 UTC
Sorry, my previous comment is wrong; I should have re-read the whole bug. The main bug here is that empathy/gcr needs to be ported to use glib-networking to do its certificate validation. Once that is done, the bug here will be fixed for basically everyone. The upstream glib-networking bug is only relevant to people using PKCS#11, and only relevant *after* empathy/gcr is changed to actually use glib-networking's code.

Comment 20 Debarshi Ray 2017-03-15 20:21:41 UTC
(In reply to Stef Walter from comment #14)
> I wonder what callers actually use the Gcr code, if any. We may be able to
> just remove those APIs in favor of glib-networking. The Gcr API is marked as
> unstable, so with API removal ... along with some patches to known API
> callers ... I think we would be in good shape.

Yes, let's do that. In this particular case, we can teach empathy to just use the GIO / glib-networking API instead of Gcr.

Reassigning.

Comment 21 Debarshi Ray 2017-03-15 20:23:15 UTC
Reassigning to myself (again).

Comment 22 Debarshi Ray 2017-03-16 20:40:50 UTC
For those interested, patches at:
https://bugzilla.gnome.org/show_bug.cgi?id=780160

Comment 23 Fedora Update System 2017-03-21 17:06:57 UTC
empathy-3.12.13-2.fc25 has been submitted as an update to Fedora 25. https://bodhi.fedoraproject.org/updates/FEDORA-2017-bd15ca5490

Comment 24 Fedora Update System 2017-03-21 17:10:34 UTC
empathy-3.12.13-2.fc26 has been submitted as an update to Fedora 26. https://bodhi.fedoraproject.org/updates/FEDORA-2017-8840ec0204

Comment 25 Kai Engert (:kaie) (inactive account) 2017-03-21 17:31:07 UTC
Everyone affected by this bug, could you please test the package, and give us feedback? Thanks in advance

Comment 26 Fedora Update System 2017-03-22 00:53:24 UTC
empathy-3.12.13-2.fc25 has been pushed to the Fedora 25 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-bd15ca5490

Comment 27 Paolo Antinori 2017-03-22 11:28:23 UTC
Guys, how to test this?

when I run:

sudo dnf --enablerepo=updates-testing check-update  | grep empathy

I get no output on f25

Comment 28 Kai Engert (:kaie) (inactive account) 2017-03-22 12:55:07 UTC
Paolo, maybe it was too early, and the update had not yet reached the download servers. Can you try again? Or alternatively, if you don't want to wait, on the bodhi page, you can click download the build manually. Upper left, "Builds", circle/arrow icon, find rpm for your platform, download, install with rpm -U.

Comment 29 Fedora Update System 2017-03-22 15:27:23 UTC
empathy-3.12.13-2.fc26 has been pushed to the Fedora 26 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-8840ec0204

Comment 30 Paolo Antinori 2017-03-22 15:49:58 UTC
Tested and working fine for me on f25

sudo dnf install https://kojipkgs.fedoraproject.org//packages/empathy/3.12.13/2.fc25/x86_64/empathy-3.12.13-2.fc25.x86_64.rpm

Thank you!

Comment 31 Paolo Antinori 2017-03-22 16:03:33 UTC
Tested and working fine for me on f25

sudo dnf install https://kojipkgs.fedoraproject.org//packages/empathy/3.12.13/2.fc25/x86_64/empathy-3.12.13-2.fc25.x86_64.rpm

Thank you!

Comment 32 Fedora Update System 2017-03-23 18:21:56 UTC
empathy-3.12.13-2.fc25 has been pushed to the Fedora 25 stable repository. If problems still persist, please make note of it in this bug report.

Comment 33 Fedora Update System 2017-04-01 17:23:38 UTC
empathy-3.12.13-2.fc26 has been pushed to the Fedora 26 stable repository. If problems still persist, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.