Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.

Bug 1377312

Summary: Certificate renewal automation
Product: [Fedora] Fedora Reporter: Didier <d.bz-redhat>
Component: certbotAssignee: James Hogarth <james.hogarth>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 26CC: bmw, dominik, goeran, ilmostro7, itamar, james.hogarth, nb, nick, olchansk, rbu, tadej.j
Target Milestone: ---   
Target Release: ---   
Hardware: All   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-03-22 18:21:32 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1289778, 1385167    
Bug Blocks:    

Description Didier 2016-09-19 12:25:37 UTC
Description of problem:

As the LetsEncrypt issued certificates have a life time of 3 months, automated certificate renewal is of paramount importance.
Currently, the certbot package is missing a (cron- or systemd-based) automation mechanism.

The EFF (https://certbot.eff.org/all-instructions/#centos-rhel-7-apache) suggests running the cron/systemd job twice a day.



Version-Release number of selected component (if applicable):

certbot-0.8.1



Additional info:

https://wiki.archlinux.org/index.php/Let%E2%80%99s_Encrypt#Automatic_renewal
https://mjanja.ch/2016/07/using-systemd-timers-to-renew-lets-encrypt-certificates/

Comment 1 ilmostro7 2016-09-26 09:22:13 UTC
Excellent point! Though, as you linked to RHEL/CentOS instructions, I'd like to point out that the `certbot` package is provided by EPEL.  Therefore, I'm not sure how/where this should be reported.

Comment 2 Didier 2016-09-26 10:03:47 UTC
IIRC, the instructions are mostly generic/identical for all distributions, hence it applies to Fedora too ...

Comment 3 James Hogarth 2016-09-26 10:24:10 UTC
The report is fine and on our radar.

This will be picked up in a future update

Comment 4 Göran Uddeborg 2016-10-14 20:08:58 UTC
May I also bring your attention to that the instructions asks us to randomize the minute the renewal is done, presumably to avoid overloading the servers.  The default anacrontab has a random delay which would do this for a cron.daily job, or it could be a systemd timer unit with AccuracySec and/or RandomizedDelaySec.

Comment 5 Dominik 'Rathann' Mierzejewski 2017-01-12 10:30:26 UTC
Current SELinux policy prevents certificate modifications from cron, so bug 1385167 must be fixed first.

Comment 6 Fedora End Of Life 2017-02-28 10:20:00 UTC
This bug appears to have been reported against 'rawhide' during the Fedora 26 development cycle.
Changing version to '26'.

Comment 7 James Hogarth 2017-03-06 12:56:58 UTC
Hi Dominik,

I'm planning to address this in the near future.

The main blocker is to get the correct certificate type applied to files in /etc/letsencrypt/(live|archive)/ 

I've opened this PR against the base policy to get it applied:

https://github.com/fedora-selinux/selinux-policy/pull/194

After the current 0.12.0 update goes out the next will include a systemd timer to handle this.

I'll update this when the package is built and ready to test.

Comment 8 Dominik 'Rathann' Mierzejewski 2017-03-06 13:21:58 UTC
Hi, James.
Excellent news! Thank you for working on this.

Comment 9 Fedora Update System 2017-03-06 14:56:59 UTC
certbot-0.12.0-3.fc24 python-acme-0.12.0-3.fc24 python-certbot-apache-0.12.0-1.fc24 has been submitted as an update to Fedora 24. https://bodhi.fedoraproject.org/updates/FEDORA-2017-0b35be64b3

Comment 10 Fedora Update System 2017-03-06 14:59:07 UTC
certbot-0.12.0-3.fc25 python-acme-0.12.0-3.fc25 python-certbot-apache-0.12.0-1.fc25 has been submitted as an update to Fedora 25. https://bodhi.fedoraproject.org/updates/FEDORA-2017-e06b5ed81c

Comment 11 Fedora Update System 2017-03-06 15:00:00 UTC
certbot-0.12.0-3.el7 python-acme-0.12.0-2.el7 python-certbot-apache-0.12.0-1.el7 has been submitted as an update to Fedora EPEL 7. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2017-7036057408

Comment 12 Fedora Update System 2017-03-06 15:00:13 UTC
certbot-0.12.0-3.el7 python-acme-0.12.0-2.el7 python-certbot-apache-0.12.0-1.el7 has been submitted as an update to Fedora EPEL 7. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2017-7036057408

Comment 13 James Hogarth 2017-03-06 15:07:24 UTC
Okay so we can shortcut this a little.

The certbot package now has a dependency on semanage to correctly label the certificates, this gets run in %post

When the base policy is updated I'll remove the %post and requires.

Meanwhile I tested this on my server and systemctl start certbot-renew.service correctly generated my webroot based certificates and a post hook restarting httpd worked.

If you could test the systems unit in your environments as well that would be great.

Comment 14 Brad Warren 2017-03-06 22:28:17 UTC
Great! I've been meaning to ask something like this be added for a while. Taking a look at https://src.fedoraproject.org/cgit/rpms/certbot.git/tree/, I appreciate you randomizing when the timer fires. I'd like to see a few things changed though.

First, you'll probably want to include `-q/--quiet` on the command line. By default Certbot prints a ton of information about what it's doing. Adding `-q/--quiet` suppresses this and only prints any errors if they occur.

Second, I think you should significantly increase how often Certbot runs. Certbot by default only attempts to renew your certs 30 days before they expire. If Let's Encrypt happens to be down when you try to renew, your certs may expire before Certbot is run again. Additionally, my knowledge of systemd unit files is limited, but because monthly is defined as 30.44 days (according to https://www.freedesktop.org/software/systemd/man/systemd.time.html#Parsing%20Time%20Spans) and you randomize it over a week, I believe it's possible Certbot runs right before your certificate reaches 30 days from expiration and not run again until after it expires.

The recommendation we give everyone is to run `certbot renew` twice a day. This the approach taken in the systemd timers in the Debian. If there is nothing to renew, the command should be a very lightweight operation; it reads the notAfter date of your certs, compares it to the time, and exits). In addition to solving the potential problems above, it should also prevent outages if Let's Encrypt has to revoke your cert for some reason and it allows us to do things like optionally handle OCSP fetching for the user in the future (see https://github.com/certbot/certbot/issues/956).

Comment 15 James Hogarth 2017-03-07 00:42:25 UTC
Thanks for taking a look and providing feedback :)

Since the stdout goes to the journal I'm going to err on the side of being verbose in the output by default, at least for now, to help people keep an eye on what's going on.

The way this unit is set up they can always add --quiet to CERTBOT_ARGS in /etc/sysconfig/certbot if they want to reduce the verbosity... Perhaps I should update the Fedora readme and comments in the config file to that extent.

Happy to reduce the default timer though, I'll update the build tomorrow with a daily check with an accuracy of a few hours.

Comment 16 Fedora Update System 2017-03-07 01:50:19 UTC
certbot-0.12.0-3.fc25, python-acme-0.12.0-3.fc25, python-certbot-apache-0.12.0-1.fc25 has been pushed to the Fedora 25 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-e06b5ed81c

Comment 17 Fedora Update System 2017-03-07 05:53:28 UTC
certbot-0.12.0-3.el7, python-acme-0.12.0-2.el7, python-certbot-apache-0.12.0-1.el7 has been pushed to the Fedora EPEL 7 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2017-7036057408

Comment 18 Fedora Update System 2017-03-07 10:00:46 UTC
certbot-0.12.0-4.fc25 python-acme-0.12.0-3.fc25 python-certbot-apache-0.12.0-1.fc25 has been submitted as an update to Fedora 25. https://bodhi.fedoraproject.org/updates/FEDORA-2017-e06b5ed81c

Comment 19 Fedora Update System 2017-03-07 10:01:15 UTC
certbot-0.12.0-4.el7 python-acme-0.12.0-2.el7 python-certbot-apache-0.12.0-1.el7 has been submitted as an update to Fedora EPEL 7. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2017-7036057408

Comment 20 James Hogarth 2017-03-07 16:20:56 UTC
ok that's built and waiting to go out ... I'm not going to do any further edits yet otherwise 0.12.0 will never make it to stable ;)

Comment 21 Fedora Update System 2017-03-07 16:51:40 UTC
certbot-0.12.0-4.fc26 has been submitted as an update to Fedora 26. https://bodhi.fedoraproject.org/updates/FEDORA-2017-c51c77f333

Comment 22 Fedora Update System 2017-03-07 17:53:00 UTC
certbot-0.12.0-3.fc24, python-acme-0.12.0-3.fc24, python-certbot-apache-0.12.0-1.fc24 has been pushed to the Fedora 24 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-0b35be64b3

Comment 23 Fedora Update System 2017-03-07 19:11:39 UTC
certbot-0.12.0-4.fc24 python-acme-0.12.0-3.fc24 python-certbot-apache-0.12.0-1.fc24 has been submitted as an update to Fedora 24. https://bodhi.fedoraproject.org/updates/FEDORA-2017-0b35be64b3

Comment 24 Fedora Update System 2017-03-08 14:52:51 UTC
certbot-0.12.0-4.fc24, python-acme-0.12.0-3.fc24, python-certbot-apache-0.12.0-1.fc24 has been pushed to the Fedora 24 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-0b35be64b3

Comment 25 Fedora Update System 2017-03-08 14:53:58 UTC
certbot-0.12.0-4.fc25, python-acme-0.12.0-3.fc25, python-certbot-apache-0.12.0-1.fc25 has been pushed to the Fedora 25 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-e06b5ed81c

Comment 26 Fedora Update System 2017-03-08 16:17:11 UTC
certbot-0.12.0-4.el7, python-acme-0.12.0-2.el7, python-certbot-apache-0.12.0-1.el7 has been pushed to the Fedora EPEL 7 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2017-7036057408

Comment 27 Fedora Update System 2017-03-10 23:20:07 UTC
certbot-0.12.0-4.fc26 has been pushed to the Fedora 26 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-c51c77f333

Comment 28 Fedora Update System 2017-03-16 21:17:55 UTC
certbot-0.12.0-4.fc24, python-acme-0.12.0-3.fc24, python-certbot-apache-0.12.0-1.fc24 has been pushed to the Fedora 24 stable repository. If problems still persist, please make note of it in this bug report.

Comment 29 Fedora Update System 2017-03-16 22:21:00 UTC
certbot-0.12.0-4.fc25, python-acme-0.12.0-3.fc25, python-certbot-apache-0.12.0-1.fc25 has been pushed to the Fedora 25 stable repository. If problems still persist, please make note of it in this bug report.

Comment 30 Fedora Update System 2017-03-23 20:18:19 UTC
certbot-0.12.0-4.el7, python-acme-0.12.0-2.el7, python-certbot-apache-0.12.0-1.el7 has been pushed to the Fedora EPEL 7 stable repository. If problems still persist, please make note of it in this bug report.

Comment 31 Göran Uddeborg 2017-03-23 21:29:34 UTC
It seems you forgot to update the documentation when you shortened the timer interval.  :-)  The README.fedora file still says it runs monthly, with a week of fudge.

Comment 32 Fedora Update System 2017-04-01 16:50:57 UTC
certbot-0.12.0-4.fc26 has been pushed to the Fedora 26 stable repository. If problems still persist, please make note of it in this bug report.

Comment 33 Konstantin Olchanski 2017-04-02 15:28:15 UTC
(In reply to Fedora Update System from comment #30)
> certbot-0.12.0-4.el7, python-acme-0.12.0-2.el7,
> python-certbot-apache-0.12.0-1.el7 has been pushed to the Fedora EPEL 7
> stable repository. If problems still persist, please make note of it in this
> bug report.

On CentOS7, certbot-renew timer runs, but does not renew the certificates,
looks like it runs the wrong certbot command. The correct command is "certbot renew",
and what is all this --port-hook and --renew-hook stuff and who is supposed to expand all these $PRE_HOOK
variables.

[root@titan00 ~]# systemctl status certbot-renew -l
● certbot-renew.service - This service automatically renews any certbot certificates found
   Loaded: loaded (/usr/lib/systemd/system/certbot-renew.service; static; vendor preset: disabled)
   Active: failed (Result: exit-code) since Sun 2017-04-02 00:00:01 PDT; 8h ago
  Process: 20545 ExecStart=/usr/bin/certbot renew --pre-hook $PRE_HOOK --post-hook $POST_HOOK --renew-hook $RENEW_HOOK $CERTBOT_ARGS (code=exited, status=2)
 Main PID: 20545 (code=exited, status=2)

Apr 02 00:00:01 titan00.triumf.ca certbot[20545]: usage:
Apr 02 00:00:01 titan00.triumf.ca certbot[20545]: certbot [SUBCOMMAND] [options] [-d DOMAIN] [-d DOMAIN] ...
Apr 02 00:00:01 titan00.triumf.ca certbot[20545]: Certbot can obtain and install HTTPS/TLS/SSL certificates.  By default,
Apr 02 00:00:01 titan00.triumf.ca certbot[20545]: it will attempt to use a webserver both for obtaining and installing the
Apr 02 00:00:01 titan00.triumf.ca certbot[20545]: cert.
Apr 02 00:00:01 titan00.triumf.ca certbot[20545]: certbot: error: argument --pre-hook: expected one argument
[root@titan00 ~]# 
[root@titan00 ~]# rpm -q certbot
certbot-0.12.0-4.el7.noarch
[root@titan00 ~]# 
[root@titan00 ~]# more /etc/redhat-release 
CentOS Linux release 7.3.1611 (Core) 
[root@titan00 ~]# 

K.O.