Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.
Bug 2093069 - openvpn 2.5.7 server won't start due to BF-CBC cipher option removed, but still referenced in systemd service
Summary: openvpn 2.5.7 server won't start due to BF-CBC cipher option removed, but sti...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: openvpn
Version: 36
Hardware: Unspecified
OS: Linux
unspecified
unspecified
Target Milestone: ---
Assignee: David Sommerseth
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2022-06-02 20:37 UTC by Sergio
Modified: 2023-04-21 22:10 UTC (History)
13 users (show)

Fixed In Version: openvpn-2.5.7-2.fc36
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2022-06-12 02:05:20 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)

Description Sergio 2022-06-02 20:37:27 UTC
Description of problem:
openvpn server won't start when executed as a systemctl service

Version-Release number of selected component (if applicable):
openvpn-2.5.7-1.fc36

How reproducible:
always

Steps to Reproduce:
1. update openvpn to 2.5.7-1.fc36
2. start it with "systemctl start openvpn-server"
3. check the status with "systemctl status openvpn-server"

Actual results:
the server fails to load

Expected results:
the server starts normally

Additional info:
/usr/lib/systemd/system/openvpn-server@.service has the following line:
ExecStart=/usr/sbin/openvpn --status %t/openvpn-server/status-%i.log --status-version 2 --suppress-timestamps --cipher AES-256-GCM --data-ciphers AES-256-GCM:AES-128-GCM:AES-256-CBC:AES-128-CBC:BF-CBC --config %i.conf
which is from:
https://src.fedoraproject.org/rpms/openvpn/blob/rawhide/f/0001-Change-the-default-cipher-to-AES-256-GCM-for-server-.patch

but due to BF-CBC not being recognized as a valid cipher in the updated openvpn the server fails to start.
If you manually remove the cipher in the systemd service, then run "systemctl daemon-reload" the server starts normally

Comment 1 Jonas Sell 2022-06-03 06:34:35 UTC
The problem seems not only related to the service file. Even when I'm starting the OpenVPN server without systemd through "openvpn $configfile" it still fails.
2022-06-03 08:25:24 us=711547 Cipher algorithm 'BF-CBC' not found
2022-06-03 08:25:24 us=711553 Cipher BF-CBC not supported
2022-06-03 08:25:24 us=711563 Exiting due to fatal error

Somewhere OpenVPN creates a "ciphername = 'BF-CBC'" entry when generating the config:

2022-06-03 08:25:24 us=705404   shared_secret_file = '[UNDEF]'
2022-06-03 08:25:24 us=705411   key_direction = not set
2022-06-03 08:25:24 us=705417   ciphername = 'BF-CBC'
2022-06-03 08:25:24 us=705423   ncp_enabled = ENABLED
2022-06-03 08:25:24 us=705430   ncp_ciphers = 'AES-256-GCM:AES-128-GCM:AES-256-CBC'

This seems to break everything.

Going back to 2.5.6 and the server starts normally.

Comment 2 Jonas Sell 2022-06-03 06:45:57 UTC
Adding

cipher AES-256-CBC
data-ciphers AES-256-GCM:AES-128-GCM:AES-256-CBC

to the config seems to fix the problem. OpenVPN is starting normal now here.

Comment 3 David Sommerseth 2022-06-03 07:37:47 UTC
So I'm a bit torn here.  I see two possibilities for the systemd unit file update.

a) add --providers legacy default

b) Remove BF-CBC


I consider a) a band-aid solution.  It re-enables BF-CBC.  But it doesn't help moving away from BF-CBC at all.  This "change default cipher" patch has been in use since July 2017, and was meant to provide a smooth transition away from BF-CBC by default.  If both client and server runs OpenVPN 2.4 and newer, this would enable upgrading to a AES-GCM automatically while being able to support OpenVPN 2.3 clients and older to connect without any issues.  Now, this was soon 5 years ago and OpenVPN 2.3 and older are no longer supported by the upstream project (that supported ended a year ago).  And it is just delaying the transition period away from insecure ciphers even further.  Eventually BF-CBC needs to go away, as OpenSSL is just going to make it more harder to use these insecure ciphers in the future.

Therefor I am strongly considering to just remove the BF-CBC reference already.  If there are anyone still connecting to outdated and unsupported OpenVPN 2.3 or older clients, they will need to upgrade their clients.


Thoughts?



In regards to the findings of Jonas Sell, that certainly is an unintended result of pulling in better OpenSSL 3 support into OpenVPN 2.5.7.  And removing BF-CBC is by far better than to add "providers legacy default" instead.  I'll discuss this within the upstream project.

Comment 4 David Sommerseth 2022-06-03 10:19:19 UTC
Upstream has sent a patch to the mailing list to be considered: 

I've put together a few quick test builds

F36: https://koji.fedoraproject.org/koji/taskinfo?taskID=87830798
Rawhide: https://koji.fedoraproject.org/koji/taskinfo?taskID=87830820

If some of you can do a quick test of those build to see if that helps, I can get a 2.5.7-2 prepared later this weekend.

Comment 5 David Sommerseth 2022-06-03 10:20:03 UTC
Upstream patch: https://patchwork.openvpn.net/patch/2504/

Comment 6 Ron Yorston 2022-06-03 19:57:53 UTC
This broke the OpenVPN client on my workstation yesterday. I managed to get it going again by adding 'cipher=AES-256-GCM' to the NetworkManager keyfile.

I held off updating my laptop until the test build became available:  with this build my VPN works without any changes in NetworkManager.

The warnings in the log about BF-CBC mention changing 'data-ciphers-fallback' or 'data-ciphers'. It's possible that wouldn't have helped in this case but I was unable to figure out how to get NetworkManager to accept these options. For for future reference, is there a way to do that?

Comment 7 Damian Wrobel 2022-06-04 10:18:58 UTC
(In reply to David Sommerseth from comment #4)
> F36: https://koji.fedoraproject.org/koji/taskinfo?taskID=87830798
> If some of you can do a quick test of those build to see if that helps, I
> can get a 2.5.7-2 prepared later this weekend.

I tested it on F36

$ rpm -q --changelog openvpn-2.5.7-1.fc36.x86_64 | head -n3
* Tue May 31 2022 David Sommerseth <davids> - 2.5.7-2
- Added additional upstream patch resolving BF-CBC issues (to be removed with 2.5.8)
  https://patchwork.openvpn.net/patch/2504/

and it didn't help to start openvpn-server:

Jun 04 12:13:02 localhost.localdomain systemd[1]: openvpn-server: Scheduled restart job, restart counter is at 5.
Jun 04 12:13:02 localhost.localdomain systemd[1]: Stopped openvpn-server - OpenVPN service for server.
Jun 04 12:13:02 localhost.localdomain systemd[1]: Starting openvpn-server - OpenVPN service for server...
Jun 04 12:13:02 localhost.localdomain openvpn[1687]: WARNING: Compression for receiving enabled. Compression has been used in the past to break encryption. Sent packets are not compressed unless "allow-compression yes" is also set.
Jun 04 12:13:02 localhost.localdomain openvpn[1687]: Unsupported cipher in --data-ciphers: BF-CBC
Jun 04 12:13:02 localhost.localdomain openvpn[1687]: Options error: NCP cipher list contains unsupported ciphers or is too long.
Jun 04 12:13:02 localhost.localdomain openvpn[1687]: Use --help for more information.
Jun 04 12:13:02 localhost.localdomain systemd[1]: openvpn-server: Main process exited, code=exited, status=1/FAILURE
Jun 04 12:13:02 localhost.localdomain systemd[1]: openvpn-server: Failed with result 'exit-code'.
Jun 04 12:13:02 localhost.localdomain systemd[1]: Failed to start openvpn-server - OpenVPN service for server.

what helps is to remove :BF-CBC from the service file:

# diff -u /tmp/openvpn-server@.service.orig /lib/systemd/system/openvpn-server@.service 
--- /tmp/openvpn-server@.service.orig	2022-06-03 12:21:11.000000000 +0200
+++ /lib/systemd/system/openvpn-server@.service	2022-06-04 12:13:49.841905776 +0200
@@ -10,7 +10,7 @@
 Type=notify
 PrivateTmp=true
 WorkingDirectory=/etc/openvpn/server
-ExecStart=/usr/sbin/openvpn --status %t/openvpn-server/status-%i.log --status-version 2 --suppress-timestamps --cipher AES-256-GCM --data-ciphers AES-256-GCM:AES-128-GCM:AES-256-CBC:AES-128-CBC:BF-CBC --config %i.conf
+ExecStart=/usr/sbin/openvpn --status %t/openvpn-server/status-%i.log --status-version 2 --suppress-timestamps --cipher AES-256-GCM --data-ciphers AES-256-GCM:AES-128-GCM:AES-256-CBC:AES-128-CBC --config %i.conf
 CapabilityBoundingSet=CAP_IPC_LOCK CAP_NET_ADMIN CAP_NET_BIND_SERVICE CAP_NET_RAW CAP_SETGID CAP_SETUID CAP_SYS_CHROOT CAP_DAC_OVERRIDE CAP_AUDIT_WRITE
 LimitNPROC=10
 DeviceAllow=/dev/null rw

Comment 8 Jasper Siepkes 2022-06-06 09:32:54 UTC
> This broke the OpenVPN client on my workstation yesterday.

Same here. Tried configuring various ciphers (including the ones I think should be correct) but that didn't fix the issue.

Ironically, doing a "dnf downgrade openvpn" also doesn't solve the problem. Should not that I don't think I have ever successfully managed to rollback a botched updated on a YUM / DNF based distro. No NetworkManager packages were updated in my case. Downgrading openssl and openssl-libs also wasn't enough to revert to a working system.

Comment 9 Łukasz Posadowski 2022-06-09 07:29:23 UTC
Hi.

I have the same problem. I downgraded OpenVPN to penvpn-2.5.6-1.fc36.x86_64 and it is working OK and added *openvpn* to my excludes in dnf right away.

Thanks for everyone trying to solve this, it is somewhat crucial. :) I'll try to reconfigure it with ciphers mentioned by Jonas Sell, but would prefer upstream patch included.

Comment 10 Chris Siebenmann 2022-06-10 18:01:16 UTC
I am another person who cannot establish an OpenVPN client connection in Fedora 36 with the current openvpn RPM.
A failed connection (with this version) logs:

Jun 10 13:56:05 fedora36 nm-openvpn[2583]: --cipher is not set. Previous OpenVPN version defaulted to BF-CBC as fallback when cipher negotiation failed in this case. If you need this fallback please add '--data-ciphers-fallback BF-CBC' to your configuration and/or add BF-CBC to --data-ciphers.
Jun 10 13:56:05 fedora36 nm-openvpn[2583]: OpenVPN 2.5.7 x86_64-redhat-linux-gnu [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [PKCS11] [MH/PKTINFO] [AEAD] built on May 31 2022
Jun 10 13:56:05 fedora36 nm-openvpn[2583]: library versions: OpenSSL 3.0.3 3 May 2022, LZO 2.10
Jun 10 13:56:05 fedora36 nm-openvpn[2583]: WARNING: No server certificate verification method has been enabled.  See http://openvpn.net/howto.html#mitm for more info.
Jun 10 13:56:05 fedora36 nm-openvpn[2583]: NOTE: the current --script-security setting may allow this configuration to call user-defined scripts
Jun 10 13:56:05 fedora36 nm-openvpn[2583]: Cipher BF-CBC not supported
Jun 10 13:56:05 fedora36 nm-openvpn[2583]: Exiting due to fatal error

A successful connection with openvpn-2.5.6-1.fc36.x86_64 logs:
Jun 10 13:59:03 fedora36 nm-openvpn[2685]: --cipher is not set. Previous OpenVPN version defaulted to BF-CBC as fallback when cipher negotiation failed in this case. If you need this fallback please add '--data-ciphers-fallback BF-CBC' to your configuration and/or add BF-CBC to --data-ciphers.
Jun 10 13:59:03 fedora36 nm-openvpn[2685]: OpenVPN 2.5.6 x86_64-redhat-linux-gnu [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [PKCS11] [MH/PKTINFO] [AEAD] built on Mar 16 2022
Jun 10 13:59:03 fedora36 nm-openvpn[2685]: library versions: OpenSSL 3.0.3 3 May 2022, LZO 2.10
Jun 10 13:59:03 fedora36 nm-openvpn[2685]: WARNING: No server certificate verification method has been enabled.  See http://openvpn.net/howto.html#mitm for more info.
Jun 10 13:59:03 fedora36 nm-openvpn[2685]: NOTE: the current --script-security setting may allow this configuration to call user-defined scripts
Jun 10 13:59:03 fedora36 nm-openvpn[2685]: TCP/UDP: Preserving recently used remote address: [AF_INET]128.100.3.115:1194
Jun 10 13:59:03 fedora36 nm-openvpn[2685]: UDP link local: (not bound)
Jun 10 13:59:03 fedora36 nm-openvpn[2685]: UDP link remote: [AF_INET]128.100.3.115:1194
Jun 10 13:59:03 fedora36 nm-openvpn[2685]: NOTE: UID/GID downgrade will be delayed because of --client, --pull, or --up-delay
Jun 10 13:59:03 fedora36 nm-openvpn[2685]: [ovenpan.cs.toronto.edu] Peer Connection Initiated with [AF_INET]128.100.3.115:1194
Jun 10 13:59:03 fedora36 nm-openvpn[2685]: TUN/TAP device tun0 opened

Comment 11 Fedora Update System 2022-06-10 20:07:52 UTC
FEDORA-2022-8ca0f56650 has been submitted as an update to Fedora 36. https://bodhi.fedoraproject.org/updates/FEDORA-2022-8ca0f56650

Comment 12 David Sommerseth 2022-06-10 20:13:28 UTC
The update I've pushed out via Fedora Bodhi now (FEDORA-2022-8ca0f56650) carries an updated upstream patch for better handling deprecated and unavailable ciphers. This patch is now included officially for a coming OpenVPN 2.5.8 release coming later.

In addition this update also removes BF-CBC from the --data-ciphers list in the openvpn-server@.service unit file.

The change to the unit file will result in OpenVPN server running on Fedora 36 and newer will no longer support BF-CBC out-of-the-box.  This will now require an explicit configuration change to re-enable this.  I concluded this was the best approach, since OpenSSL 3.0 by default does no longer support this cipher without re-enabling legacy ciphers.

Please test this update and give the update positive karma on successful testing.  Since the 2.5.7-1 release caused so much issues, I decided to require more +1s to label it as stable.

Comment 13 Fedora Update System 2022-06-11 01:31:42 UTC
FEDORA-2022-8ca0f56650 has been pushed to the Fedora 36 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2022-8ca0f56650`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2022-8ca0f56650

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 14 Fedora Update System 2022-06-12 02:05:20 UTC
FEDORA-2022-8ca0f56650 has been pushed to the Fedora 36 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 15 Łukasz Posadowski 2022-06-15 08:33:09 UTC
It is still is not working

# systemctl status openvpn-server@myvpn
● openvpn-server - OpenVPN service for myvpn
     Loaded: loaded (/etc/systemd/system/openvpn-server; enabled; vendor preset: disabled)
     Active: activating (auto-restart) (Result: exit-code) since Wed 2022-06-15 10:28:04 CEST; 1s ago
       Docs: man:openvpn(8)
             https://community.openvpn.net/openvpn/wiki/Openvpn24ManPage
             https://community.openvpn.net/openvpn/wiki/HOWTO
    Process: 2977305 ExecStart=/usr/sbin/openvpn --status /run/openvpn-server/status-myvpn.log --status-version 2 --cipher AES-256-GCM --data-ciphers AES-256-GCM:AES-128-GCM:AES-256-CBC:AES-128-CBC:BF-CBC --config myvpn.conf (code=exited, status=1/FAILURE)
   Main PID: 2977305 (code=exited, status=1/FAILURE)
        CPU: 16ms


After downgrading to version 2.5.6 it is working fine

# systemctl status openvpn-server@myvpn
● openvpn-server - OpenVPN service for myvpn
     Loaded: loaded (/etc/systemd/system/openvpn-server; enabled; vendor preset: disabled)
     Active: active (running) since Wed 2022-06-15 10:30:36 CEST; 10s ago
       Docs: man:openvpn(8)
             https://community.openvpn.net/openvpn/wiki/Openvpn24ManPage
             https://community.openvpn.net/openvpn/wiki/HOWTO
   Main PID: 2978737 (openvpn)
     Status: "Initialization Sequence Completed"
      Tasks: 1 (limit: 2274)
     Memory: 1.8M
        CPU: 23ms
     CGroup: /system.slice/system-openvpn\x2dserver.slice/openvpn-server
             └─ 2978737 /usr/sbin/openvpn --status /run/openvpn-server/status-myvpn.log --status-version 2 --cipher AES-25>

cze 15 10:30:36 vps627971.ovh.net systemd[1]: Starting openvpn-server - OpenVPN service for myvpn...
cze 15 10:30:36 vps627971.ovh.net systemd[1]: Started openvpn-server - OpenVPN service for myvpn.


Do anyone still have an issue, or it's only me?

thanks

Comment 16 Łukasz Posadowski 2022-06-15 08:36:08 UTC
Adding

cipher AES-256-CBC
data-ciphers AES-256-GCM:AES-128-GCM:AES-256-CBC

indeed solves the issue.

Comment 17 David Sommerseth 2022-06-20 08:17:22 UTC
(In reply to Łukasz Posadowski from comment #15)
> It is still is not working
> 
> # systemctl status openvpn-server@myvpn
> ● openvpn-server - OpenVPN service for myvpn
>      Loaded: loaded (/etc/systemd/system/openvpn-server;
                       ^^^^^^^^^^^^^^^^^^^

> 
> Do anyone still have an issue, or it's only me?

This is most likely just you.  You seem to use your own openvpn-server@.service unit file, not the one packaged in the RPM:

$ rpm -ql openvpn | grep openvpn-server@
/usr/lib/systemd/system/openvpn-server@.service

Comment 18 Łukasz Posadowski 2022-06-20 08:54:47 UTC
(In reply to David Sommerseth from comment #17)
> (In reply to Łukasz Posadowski from comment #15)
> > It is still is not working
> > 
> > # systemctl status openvpn-server@myvpn
> > ● openvpn-server - OpenVPN service for myvpn
> >      Loaded: loaded (/etc/systemd/system/openvpn-server;
>                        ^^^^^^^^^^^^^^^^^^^
> 
> > 
> > Do anyone still have an issue, or it's only me?
> 
> This is most likely just you.  You seem to use your own
> openvpn-server@.service unit file, not the one packaged in the RPM:
> 
> $ rpm -ql openvpn | grep openvpn-server@
> /usr/lib/systemd/system/openvpn-server@.service

Ok, thank You.

Comment 19 Václav Nováček 2023-04-19 07:56:07 UTC
Here we are again... openvpn-2.6.2-1.fc38.x86_64
BF-CBC present within /usr/lib/systemd/system/openvpn-server@.service again

Comment 20 Damian Wrobel 2023-04-19 09:46:54 UTC
It seems to be re-introduced by [1]:

 + +ExecStart=@sbindir@/openvpn --status %t/openvpn-server/status-%i.log --status-version 2 --suppress-timestamps --cipher AES-256-GCM --data-ciphers AES-256-GCM:AES-128-GCM:AES-256-CBC:AES-128-CBC:BF-CBC --config %i.conf


[1] https://src.fedoraproject.org/rpms/openvpn/c/ce20bc1bb840ae21084181c2a0d833901191b47a?branch=rawhide

Comment 21 David Sommerseth 2023-04-21 22:10:29 UTC
This should be opened as a new ticket, referencing the openvpn-2.6.2.  I missed this ticket in my mailbox, as it said 2.5.7 in the subject field and the latest 2.5 release is openvpn-2.5.9.

Anyhow, I just pushed out a new update for Fedora 38 + Rawhide with openvpn-2.6.3.  This release also fixes this issue.
https://bodhi.fedoraproject.org/updates/FEDORA-2023-0097fde807


Note You need to log in before you can comment on or make changes to this bug.