Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.
Bug 2177834 - openvpn with certificate on yubikey fails
Summary: openvpn with certificate on yubikey fails
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: openvpn
Version: 38
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: David Sommerseth
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks: F38FinalBlocker F38FinalFreezeException
TreeView+ depends on / blocked
 
Reported: 2023-03-13 16:43 UTC by Florian Apolloner
Modified: 2023-03-18 00:18 UTC (History)
6 users (show)

Fixed In Version: openvpn-2.6.1-2.fc39 openvpn-2.6.1-2.fc38
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2023-03-17 07:35:57 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)
broken log (deleted)
2023-03-13 16:43 UTC, Florian Apolloner
no flags Details
working log (deleted)
2023-03-13 16:43 UTC, Florian Apolloner
no flags Details

Description Florian Apolloner 2023-03-13 16:43:16 UTC
Created attachment 1950273 [details]
broken log

Description of problem:

After upgrading to Fedora 38 my openvpn tunnel started failing. I am using a ECDSA certificate baked by a yubikey. The relevant configuration is (openvpn.conf):

pkcs11-id "pkcs11:model=PKCS%2315%20emulated;token=xxxx;manufacturer=piv_II;serial=xxxx;id=%01"
pkcs11-providers /usr/lib64/pkcs11/opensc-pkcs11.so
management 127.0.0.1 8888
management-query-passwords

Version-Release number of selected component (if applicable):
openvpn-2.6.0-2.fc38

I am attaching a debug log with a working attempt from fedora 37 and a broken one from fedora 38. Please note that the certificates are not the same but they are both baked by yubikeys.

In the log you can see that the broken configuration does "less" (compare with meld or another visual diff tool) and seems to return with an rv=1 indicating an error (?). Please let me know which other details I can provide.

Comment 1 Florian Apolloner 2023-03-13 16:43:40 UTC
Created attachment 1950274 [details]
working log

Comment 2 Florian Apolloner 2023-03-13 17:32:42 UTC
Correction: The above log doesn't actually show the error. Nothing in the client log really shows an error, I think I missinterpreted rv=1.

The server logs show this:
TLS Error: TLS handshake failed |
TLS Error: TLS object -> incoming plaintext read error |
TLS\_ERROR: BIO read tls\_read\_plaintext error |
OpenSSL: error:1417B07B:SSL routines:tls\_process\_cert\_verify:bad signature |
OpenSSL: error:0D07803A:asn1 encoding routines:asn1\_item\_embed\_d2i:nested asn1 error |
OpenSSL: error:0D0680A8:asn1 encoding routines:asn1\_check\_tlen:wrong tag |

So it seems like OpenVPN now generates something weirdly broken :(

Comment 3 Fedora Blocker Bugs Application 2023-03-13 17:36:16 UTC
Proposed as a Freeze Exception for 38-beta by Fedora user apollo13 using the blocker tracking app because:

 Users that require a yubikey for their OpenVPN privatekey require pkcs11/openvpn to be in a functional state.

Similar for older Fedora version: https://pagure.io/fedora-qa/blocker-review/issue/770

Comment 4 Kamil Páral 2023-03-13 17:57:40 UTC
I think the OP wanted to propose this as a Final blocker, correcting

Comment 5 David Sommerseth 2023-03-13 18:05:31 UTC
The configuration working on Fedora 37, is that running OpenVPN 2.5 or OpenVPN 2.6?

If it is running OpenVPN 2.5, can you please test with OpenVPN 2.6 using this Fedora Copr repository?
https://copr.fedorainfracloud.org/coprs/dsommers/openvpn-release-2.6/

This is needed to clarify if this is an issue in the PKCS#11 library stack or OpenVPN 2.6.

Comment 6 Florian Apolloner 2023-03-13 19:11:16 UTC
Fedora 37 is running OpenVPN 2.5. I did manually compile OpenVPN 2.5 on Fedora 38 and it works again. Do you need me to retry on Fedora 37 with 2.6 also or is it enough to know the Fedora 38 with Openvpn 2.5 works?

Comment 7 David Sommerseth 2023-03-13 19:25:46 UTC
It would be good if you could run a 2.6 test on Fedora 37 too.  I suspect it will fail, but with that confirmation we have a better grip on the failures.  Also the same log outputs would be good for comparison.  I will send this information to the upstream project once we have all details.

Comment 8 Florian Apolloner 2023-03-13 19:30:03 UTC
I can try Fedora 37 tomorrow. I managed to get it running with Fedora 38 and OpenVPN by using OpenVPN's external signing mechanism and manually implementing the required signature. What I noticed on the way when using the internal provider:
xkey_provider: In xkey_sign_dispatch: xkey_provider: external sign op returned ret = 1 siglen = 64

When I use my own external signature I get don't get 64 but 70-72. I get to 70-72 by using https://python-pkcs11.readthedocs.io/en/latest/api.html?highlight=get_objects#pkcs11.util.ec.encode_ecdsa_signature which encodes the raw 64 byte signature into DER encoded signature. Now I wonder if the xkey_provider should be doing this somewhere -- but at this point I am just guessing, this is a bit out of my league.

Comment 9 Florian Apolloner 2023-03-13 19:40:59 UTC
Details for the previous message (xkey_provider logs):

Working:

xkey_provider: In ec_keymgmt_name: entry
xkey_provider: In query_operation: op = 12
xkey_provider: In query_operation: op = 10
xkey_provider: In signature_newctx: entry
xkey_provider: In signature_digest_sign_init: mdname = <SHA256>
xkey_provider: In signature_set_ctx_params: entry
xkey_provider: In signature_freectx: entry
xkey_provider: In keydata_free: entry
xkey_provider: In keymgmt_get_params: entry
xkey_provider: In ec_keymgmt_name: entry
xkey_provider: In signature_newctx: entry
xkey_provider: In signature_digest_sign_init: mdname = <SHA2-256>
xkey_provider: In signature_set_ctx_params: entry
xkey_provider: In signature_digest_sign: entry
xkey_provider: In signature_digest_sign: entry
In xkey_management_sign with keytype = EC, op = DigestSign
xkey_management_sign: computing digest
In xkey_digest
xkey management_sign: requesting sig with algorithm <ECDSA>
xkey_provider: In xkey_sign_dispatch: xkey_provider: external sign op returned ret = 1 siglen = 70
xkey_provider: In signature_freectx: entry
xkey_provider: In keydata_free: entry

Broken:

xkey_provider: In ec_keymgmt_name: entry
xkey_provider: In query_operation: op = 12
xkey_provider: In query_operation: op = 10
xkey_provider: In signature_newctx: entry
xkey_provider: In signature_digest_sign_init: mdname = <SHA256>
xkey_provider: In signature_set_ctx_params: entry
xkey_provider: In signature_freectx: entry
xkey_provider: In keydata_free: entry
xkey_provider: In keymgmt_get_params: entry
xkey_provider: In ec_keymgmt_name: entry
xkey_provider: In signature_newctx: entry
xkey_provider: In signature_digest_sign_init: mdname = <SHA2-256>
xkey_provider: In signature_set_ctx_params: entry
xkey_provider: In signature_digest_sign: entry
xkey_provider: In signature_digest_sign: entry
xkey_pkcs11h_sign: computing digest
In xkey_digest
xkey_pkcs11h_sign: signing with EC key
xkey_provider: In xkey_sign_dispatch: xkey_provider: external sign op returned ret = 1 siglen = 64
xkey_provider: In signature_freectx: entry
xkey_provider: In keydata_free: entry

If you compare working with broken you see that the returned signature length is quite a bit different. I wonder if the new xkey_provider should be DER encoding (?) the results in xkey_pkcs11h_sign (my working copy uses xkey_management_sign to get the proper signature)

Comment 10 Florian Apolloner 2023-03-14 08:34:20 UTC
I can confirm that this problem also occurs on Fedora 37 with OpenVPN 2.6.0. The client logs are basically irrelevant, they don't really show any error. Only the server complains that the signature is wrong and that it cannot parse the ASN.1 structure.

Comment 11 David Sommerseth 2023-03-14 08:41:58 UTC
Thanks a lot!  I've had a brief discussion with upstream, and there is already a patch being reviewed which seems relevant: https://patchwork.openvpn.net/project/openvpn2/patch/20230311052403.1154030-1-selva.nair@gmail.com/

Comment 12 David Sommerseth 2023-03-14 09:01:19 UTC
I have a couple of Fedora Koji builds brewing now with the patch from the mailing list included.  Can you give those a spin, Florian?

Fedora 38: https://koji.fedoraproject.org/koji/taskinfo?taskID=98680872
     x86_64 packages: https://koji.fedoraproject.org/koji/taskinfo?taskID=98680943

Fedora 37: https://koji.fedoraproject.org/koji/taskinfo?taskID=98680878
     x86_64 packages: https://koji.fedoraproject.org/koji/taskinfo?taskID=98680991

Comment 13 Florian Apolloner 2023-03-14 09:48:58 UTC
Jupp, the Fedora 38 package works for me. Didn't test the 37 variant.

Comment 14 David Sommerseth 2023-03-14 09:50:44 UTC
Great!  Thanks for testing!  I'll report back.

Comment 15 Fedora Update System 2023-03-15 10:29:28 UTC
FEDORA-2023-dde2bc874f has been submitted as an update to Fedora 38. https://bodhi.fedoraproject.org/updates/FEDORA-2023-dde2bc874f

Comment 16 David Sommerseth 2023-03-15 13:38:28 UTC
The same fix has been pushed to the Fedora Copr repository for the OpenVPN 2.6 releases.  The upstream patch is included and will most likely be included in the next OpenVPN 2.6.2 release.

Comment 17 Fedora Update System 2023-03-17 04:03:38 UTC
FEDORA-2023-dde2bc874f has been pushed to the Fedora 38 testing repository.

You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2023-dde2bc874f

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 18 Fedora Update System 2023-03-17 07:32:30 UTC
FEDORA-2023-82ca19484b has been submitted as an update to Fedora 39. https://bodhi.fedoraproject.org/updates/FEDORA-2023-82ca19484b

Comment 19 Fedora Update System 2023-03-17 07:35:57 UTC
FEDORA-2023-82ca19484b has been pushed to the Fedora 39 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 20 Fedora Update System 2023-03-18 00:18:04 UTC
FEDORA-2023-dde2bc874f has been pushed to the Fedora 38 stable repository.
If problem still persists, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.