Bug 1435036

An unexpected error occured creating the scratch build: please report this issue to the-new-hotness issue tracker at https://github.com/fedora-infra/the-new-hotness/issues

dsommers's openvpn-2.4.1-1.fc27 completed http://koji.fedoraproject.org/koji/buildinfo?buildID=871788

openvpn-2.4.1-1.fc25 has been submitted as an update to Fedora 25. https://bodhi.fedoraproject.org/updates/FEDORA-2017-51ae70f9a0

openvpn-2.4.1-1.fc26 has been submitted as an update to Fedora 26. https://bodhi.fedoraproject.org/updates/FEDORA-2017-f41671c546

openvpn-2.4.1-1.fc26 has been pushed to the Fedora 26 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-f41671c546

openvpn-2.4.1-1.el6 has been submitted as an update to Fedora EPEL 6. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2017-3da5d4f25f

I thought a major upgrade like this would be contrary to the EPEL policy.

"All updates should strive to avoid situations that require manual intervention to keep the package functioning after update."

Upgrading from 2.3 to 2.4 requires configuration changes which may be disruptive.

openvpn-2.4.1-1.fc25 has been pushed to the Fedora 25 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-51ae70f9a0

(In reply to Mauricio Teixeira from comment #7)
> I thought a major upgrade like this would be contrary to the EPEL policy.
> 
> "All updates should strive to avoid situations that require manual
> intervention to keep the package functioning after update."
> 
> Upgrading from 2.3 to 2.4 requires configuration changes which may be
> disruptive.

Absolutely a good point.  But there are some really important features (like AEAD/GCM, data channel cipher negotiations and --tls-crypt) in v2.4 which will improve the security of VPN tunnels which are too intrusive to backport.  This cannot simply be ignored.

So I am really conflicted here, as the main issue with this upgrade is that a feature which got _improved_ (--tls-remote to --verify-x509-name) in v2.3.0 and later _deprecated_ in the v2.3.1 release 4 years ago (Mar 27, 2013) is such a blocker.  Should VPN tunnel security enhancements be ignored and leave users to build v2.4 themselves because of an old and long deprecated feature?

For EPEL-6, this might be acceptable as RHEL-6 goes EOL in about 3.5 years and RHEL-7 is newer and more up-to-date.  But I find staying on v2.3 really unacceptable for RHEL/EPEL-7 which will be supported for another 7 years - which happens to be the newest available RHEL release.

As an alternative I can look at re-introducing --tls-remote for the EPEL-6 and EPEL-7 builds only.  Otherwise the option would be to introduce a new openvpn24 package and have some alternatives wrapping around it; I am not convinced it's worth the effort in the long run, especially not when considering EPEL-7 and that this effort is handled "in-between tasks" by non-RH employees.

(In reply to David Sommerseth from comment #9)
> As an alternative I can look at re-introducing --tls-remote for the EPEL-6
> and EPEL-7 builds only.  Otherwise the option would be to introduce a new
> openvpn24 package and have some alternatives wrapping around it; I am not
> convinced it's worth the effort in the long run, especially not when
> considering EPEL-7 and that this effort is handled "in-between tasks" by
> non-RH employees.

I think this would be a very good solution indeed.

(In reply to Mauricio Teixeira from comment #10)
> (In reply to David Sommerseth from comment #9)
> > As an alternative I can look at re-introducing --tls-remote for the EPEL-6
> > and EPEL-7 builds only.  Otherwise the option would be to introduce a new
> > openvpn24 package and have some alternatives wrapping around it; I am not
> > convinced it's worth the effort in the long run, especially not when
> > considering EPEL-7 and that this effort is handled "in-between tasks" by
> > non-RH employees.
> 
> I think this would be a very good solution indeed.

https://bodhi.fedoraproject.org/updates/openvpn-2.4.1-2.el6

This release re-introduces --tls-remote, but will complain in the logs about its usage.  Man page and Changes.rst have been updated accordingly, noticing this being a Fedora EPEL specific feature, as well as the openvpn --version string.

openvpn-2.4.1-1.el7 has been submitted as an update to Fedora EPEL 7. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2017-5c642f8063

The openvpn-2.4.1-1.el7 is built against OpenSSL with the same --tls-remote patch used in openvpn-2.4.1-2.el6.

openvpn-2.4.1-1.fc25 has been pushed to the Fedora 25 stable repository. If problems still persist, please make note of it in this bug report.

Missing path for /var/run/openvpn in tmpfiles config:

# cat /usr/lib/tmpfiles.d/openvpn.conf
d /run/openvpn-client 0710 root root -
d /run/openvpn-server 0710 root root -

So openvpn@.service can't start because of it 
ExecStart=/usr/sbin/openvpn --daemon --writepid /var/run/openvpn/%i.pid --cd /etc/openvpn/ --config %i.conf

Options error: --writepid fails with '/var/run/openvpn/server.pid': No such file or directory

openvpn-2.4.1-2.el6 has been pushed to the Fedora EPEL 6 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2017-3da5d4f25f

openvpn-2.4.1-1.el7 has been pushed to the Fedora EPEL 7 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2017-5c642f8063

dsommers's openvpn-2.4.1-2.fc27 completed http://koji.fedoraproject.org/koji/buildinfo?buildID=872620

(In reply to nucleo from comment #15)
> Missing path for /var/run/openvpn in tmpfiles config:
> 
> # cat /usr/lib/tmpfiles.d/openvpn.conf
> d /run/openvpn-client 0710 root root -
> d /run/openvpn-server 0710 root root -
> 
> So openvpn@.service can't start because of it 
> ExecStart=/usr/sbin/openvpn --daemon --writepid /var/run/openvpn/%i.pid --cd
> /etc/openvpn/ --config %i.conf
> 
> Options error: --writepid fails with '/var/run/openvpn/server.pid': No such
> file or directory

I will actually claim that the unit file is faulty here.  I did not review Fedora specific unit file, but rather want to encourage you to try using either openvpn-server@.service or openvpn-client@.service.  I will add a new README.systemd file to document this better as well, see here for the current one in Rawhide:
http://pkgs.fedoraproject.org/cgit/rpms/openvpn.git/tree/README.systemd

I will however update the unit file for F26 and F25.  There is no reason to use --daemon and --writepid with systemd; systemd seems to do the Type=simple (systemd default) far better.

As already said in the Bodhi update, I think the update to 2.4 in F25 is in violation of the Updates Policy, as it changes user experience.

For example, --proto tcp/upd, which formerly only used IPv4 now uses IPv6 by default, causing trouble. This requires reconfiguration.

To make things worse, there's no corresponding change in Network Manager that would allow me to use the new --proto tcp4/udp4 options.

This change actually puts OpenVPN in line with the (In reply to Thomas Moschny from comment #20)
> As already said in the Bodhi update, I think the update to 2.4 in F25 is in
> violation of the Updates Policy, as it changes user experience.
> 
> For example, --proto tcp/upd, which formerly only used IPv4 now uses IPv6 by
> default, causing trouble. This requires reconfiguration.

I have to disagree again.  OpenVPN v2.4 is actually fixing a noticeable misbehaviour in the IPv6 implementation.  Prior to v2.4, OpenVPN did not follow the RFC regarding IPv6 behaviour.  This has now been fixed and OpenVPN behaves according to RFC 6724.  See the examples in section 10.2 and 10.3 if you do not believe me.
<https://tools.ietf.org/html/rfc6724#section-10.2>.  This has also been a long awaited feature among many OpenVPN users.  If you wonder why this have not happened to the OpenVPN 2.3 releases, it is related to the changes required to resolve this issue was too intrusive.

Further, all since v2.3 (released Jan 7, 2013) OpenVPN have been capable of listening to and allow clients to connect on both IPv4 and IPv6 addresses simultaneously.  If a server have both public IPv4 and IPv6 addresses and chooses to listen to only IPv4 you cannot complain that an connection fails when the hostname resolves to both addresses.  This is the same situation for all kind of network based applications.  This is a misconfiguration of the server and must therefore be resolved properly there.

The workaround until the server configuration is fixed properly is to *either* use the IPv4 address directly in the OpenVPN configuration.  Or you can add a hostname to /etc/hosts which lists the IPv4 address of the server.

A proper configuration change on the server side is to *either* ensure --local is not listed in the configuration (which allows OpenVPN to bind to both IPv4 and IPv6 sockets simultaneously) and to ensure the firewall allows such connections.  Or to ensure the server's hostname does not resolve to any IPv6 addresses.

We simply cannot block or hold back an update just because the update actually makes the application behave according to official specifications.

To reply to Comment 15 and 19 regarding the changes to the unit files and the tmpfiles config:

This is the type of change that sometimes gives Fedora package updates a black eye to end-users.  I had three production openvpn clients break due to this update upon reboot because the old openvpn unit file still reference the now non-existent PID directory.

Might I suggest that the tmpfile openvpn.conf read something like this, instead:

d /run/openvpn-client 0710 root root -
d /run/openvpn-server 0710 root root -
D /run/openvpn 0710 root openvpn -

This is just the merging of both the new and old tmpfile configs, which now allows for the original openvpn unit file to continue to function and the user can opt to migrate to the new unit config (openvpn-server and openvpn-client) at their leisure.

Note that I'm not arguing against the new method in any way.  I'm just not fond of how this change has broken production systems, forcing the user to either adapt to the new system and change their openvpn config paths, or manually fix the tmpfile to ensure that the PID directory exists.

I see no reason to not allow for a transition to the new unit files and allow for all three PID destinations (perhaps at least until F26, when we expect some breakage).  Otherwise, why even keep the original openvpn@ unit file still around?

(In reply to T.J. Rowe from comment #22)
> I see no reason to not allow for a transition to the new unit files and
> allow for all three PID destinations (perhaps at least until F26, when we
> expect some breakage).  Otherwise, why even keep the original openvpn@ unit
> file still around?

First, have a look at bz #1435831.

Secondly.  By fixing the unit file (a scratch build [0] is put out there, which is under testing as we speak), the need for --daemon, --writepid and PIDFile= is no longer present.  OpenVPN v2.4 will use sd_notify() and talk directly to systemd when started with Type=notify.

[0] https://koji.fedoraproject.org/koji/taskinfo?taskID=18582382

I simply reject adding back erroneous hacks when a more correct solution is in the pipe.  Insisting on using PIDFile= more shows lack of understanding of how the systemd integration in OpenVPN have evolved since the last major release 4 years ago.

(In reply to David Sommerseth from comment #23)
> (In reply to T.J. Rowe from comment #22)
> > I see no reason to not allow for a transition to the new unit files and
> > allow for all three PID destinations (perhaps at least until F26, when we
> > expect some breakage).  Otherwise, why even keep the original openvpn@ unit
> > file still around?
> 
> First, have a look at bz #1435831.
> 
> Secondly.  By fixing the unit file (a scratch build [0] is put out there,
> which is under testing as we speak), the need for --daemon, --writepid and
> PIDFile= is no longer present.  OpenVPN v2.4 will use sd_notify() and talk
> directly to systemd when started with Type=notify.
> 
> [0] https://koji.fedoraproject.org/koji/taskinfo?taskID=18582382
> 
> I simply reject adding back erroneous hacks when a more correct solution is
> in the pipe.  Insisting on using PIDFile= more shows lack of understanding
> of how the systemd integration in OpenVPN have evolved since the last major
> release 4 years ago.

Excellent!  I'm glad to hear that this has already been addressed for the old unit file (which I realized is moving to legacy in favor of the new separate unit files).

Again, I make no argument against the new system, but as was noted in bz #1435831, the mid-cycle breakage was the only major concern.  (Plus, I also realize that users are only affected *if* they reboot between when the initial openvpn update was applied and when the fixed legacy unit file is applied.)

In any case, I'll pass this info along to let the affected folks know to switch to the new client/server units ASAP, prior to either reboot or the next Fedora release.  Thanks to all for your time on this!

(In reply to T.J. Rowe from comment #24)
> Again, I make no argument against the new system, but as was noted in bz
> #1435831, the mid-cycle breakage was the only major concern. 

I am truly sorry for this.  The update to v2.4 isn't really that dramatic (except of removal of --tls-remote and a few other things - see Changes.rst [0] for more info).  So I honestly didn't expect all these challenges to the systemd unit files to suddenly appear.  But it does unfortunately show the sad state this package was in, when moving towards default expectations of the upstream projects causes all these unexpected havocs.  It was not in my wildest dreams the expected scenario.  But I rather fix things and move forwards than to revert when such issues are not too dramatic.

[0] https://github.com/OpenVPN/openvpn/blob/master/Changes.rst  (this will be packaged in the openvpn package as well)

> In any case, I'll pass this info along to let the affected folks know to switch 
> to the new client/server units ASAP, prior to either reboot or the next Fedora 
> release.  Thanks to all for your time on this!

Thank you too!  And moving towards the new scheme will hopefully work far better and provide an even more secure default setup for you.

openvpn-2.4.1-2.fc26 has been submitted as an update to Fedora 26. https://bodhi.fedoraproject.org/updates/FEDORA-2017-c07a04251a

openvpn-2.4.1-2.fc25 has been submitted as an update to Fedora 25. https://bodhi.fedoraproject.org/updates/FEDORA-2017-0ffb8246fa

openvpn-2.4.1-2.fc26 has been pushed to the Fedora 26 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-c07a04251a

openvpn-2.4.1-2.fc25 has been pushed to the Fedora 25 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-0ffb8246fa

dsommers's openvpn-2.4.1-3.fc27 completed http://koji.fedoraproject.org/koji/buildinfo?buildID=873924

openvpn-2.4.1-3.el6 has been submitted as an update to Fedora EPEL 6. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2017-3da5d4f25f

openvpn-2.4.1-3.fc26 has been submitted as an update to Fedora 26. https://bodhi.fedoraproject.org/updates/FEDORA-2017-c07a04251a

(In reply to David Sommerseth from comment #19)
> (In reply to nucleo from comment #15)
> > Missing path for /var/run/openvpn in tmpfiles config:
> > 
> > # cat /usr/lib/tmpfiles.d/openvpn.conf
> > d /run/openvpn-client 0710 root root -
> > d /run/openvpn-server 0710 root root -
> > 
> > So openvpn@.service can't start because of it 
> > ExecStart=/usr/sbin/openvpn --daemon --writepid /var/run/openvpn/%i.pid --cd
> > /etc/openvpn/ --config %i.conf
> > 
> > Options error: --writepid fails with '/var/run/openvpn/server.pid': No such
> > file or directory
> 
> I will actually claim that the unit file is faulty here.  I did not review
> Fedora specific unit file, but rather want to encourage you to try using
> either openvpn-server@.service or openvpn-client@.service.  I will add a new
> README.systemd file to document this better as well, see here for the
> current one in Rawhide:
> http://pkgs.fedoraproject.org/cgit/rpms/openvpn.git/tree/README.systemd
> 
> I will however update the unit file for F26 and F25.  There is no reason to
> use --daemon and --writepid with systemd; systemd seems to do the
> Type=simple (systemd default) far better.

hit this error, opnvpn refuse to start because it can not write pid file

well not sure if this is only the unit file that is faulty here....

why does the rpm still claim for the missing path ????
# rpm -qf /var/run/openvpn
openvpn-2.4.1-1.fc25.x86_64


I hit this error on two servers after upgrade.
did 
mkdir /var/run/openvpn

You could still fix the unit if you want, but still the update should not be disruptive, and you should be sure to keep the old path /var/run/openvpn for manually created units. Again I second a regular update should not break existing behaviour.

(In reply to Jean-Philippe Pialasse from comment #33)
> why does the rpm still claim for the missing path ????
> # rpm -qf /var/run/openvpn
> openvpn-2.4.1-1.fc25.x86_64

Because this is fixed here:
https://bodhi.fedoraproject.org/updates/openvpn-2.4.1-2.fc25

openvpn-2.4.1-2.fc25 was requested for stable updates 5 hours ago.

confirms this fixs the issue,openvpn-2.4.1-2.fc25 from updates-testing
legacy openvpn configuration directly in /etc/openvpn launch with the unit start
 the /var/run/openvpn does not exist
if created is not claimed anymore by openvpn rpm

In 2.4-2.fc25 /run/openvpn (which is the really proper directory path - /var/run is a symlink to /run) is no longer needed with openvpn@.service.

As of this update PID files are not used any more. The OpenVPN process is started by systemd with Type=notifiy instead of Type=fork in the unit file.  When OpenVPN detects this, it uses sd_notify() under the hood to talk directly with systemd - which gives systemd far better measures to control this process.

So while all this breakage is indeed sad and truly not wanted at all, it just shows how much the OpenVPN packaging had drifted away from the possibilities OpenVPN software provides today.  I'm just trying to pull the packaging into a far better state; which unfortunately broke some pieces along the way.  I am sorry about that.

openvpn-2.4.1-3.fc26 has been pushed to the Fedora 26 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-c07a04251a

openvpn-2.4.1-2.fc25 has been pushed to the Fedora 25 stable repository. If problems still persist, please make note of it in this bug report.

openvpn-2.4.1-3.el6 has been pushed to the Fedora EPEL 6 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2017-3da5d4f25f

openvpn-2.4.1-1.fc26 has been pushed to the Fedora 26 stable repository. If problems still persist, please make note of it in this bug report.

openvpn-2.4.1-2.el7 has been submitted as an update to Fedora EPEL 7. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2017-5c642f8063

openvpn-2.4.1-3.fc26 has been pushed to the Fedora 26 stable repository. If problems still persist, please make note of it in this bug report.

openvpn-2.4.1-2.el7 has been pushed to the Fedora EPEL 7 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2017-5c642f8063

openvpn-2.4.1-2.el7 has been pushed to the Fedora EPEL 7 stable repository. If problems still persist, please make note of it in this bug report.

openvpn-2.4.1-3.el6 has been pushed to the Fedora EPEL 6 stable repository. If problems still persist, please make note of it in this bug report.