Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.
Bug 1172841 - Service start returns success even when service fails to start
Summary: Service start returns success even when service fails to start
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora EPEL
Classification: Fedora
Component: redis
Version: epel7
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Nathan Scott
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2014-12-10 21:31 UTC by Bill Nottingham
Modified: 2018-08-04 19:41 UTC (History)
6 users (show)

Fixed In Version: redis-3.2.11-1.fc27 redis-3.2.11-1.fc26 redis-4.0.2-2.fc25 redis-3.2.12-1.el7
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-11-28 04:11:01 UTC
Type: Bug
Embargoed:

Attachments (Terms of Use)

Description Bill Nottingham 2014-12-10 21:31:30 UTC 
Description of problem:

[root@localhost ~]# systemctl start redis.service ; echo $? 
0
[root@localhost ~]# systemctl status redis.service -n 50
redis.service - Redis persistent key-value database
   Loaded: loaded (/usr/lib/systemd/system/redis.service; disabled)
   Active: failed (Result: exit-code) since Wed 2014-12-10 21:28:37 UTC; 7s ago
  Process: 32584 ExecStop=/usr/bin/redis-shutdown (code=exited, status=1/FAILURE)
  Process: 32583 ExecStart=/usr/bin/redis-server /etc/redis.conf --daemonize no (code=exited, status=1/FAILURE)
 Main PID: 32583 (code=exited, status=1/FAILURE)

Dec 10 21:28:37 localhost.localdomain systemd[1]: Starting Redis persistent key-value database...
Dec 10 21:28:37 localhost.localdomain systemd[1]: Started Redis persistent key-value database.
Dec 10 21:28:37 localhost.localdomain redis-server[32583]: [32583] 10 Dec 21:28:37.775 # Fatal error, can't open config file '/etc/redis.conf'
Dec 10 21:28:37 localhost.localdomain systemd[1]: redis.service: main process exited, code=exited, status=1/FAILURE
Dec 10 21:28:37 localhost.localdomain redis-shutdown[32584]: + REDIS_CLI=/usr/bin/redis-cli
Dec 10 21:28:37 localhost.localdomain redis-shutdown[32584]: + SERVICE_NAME=
Dec 10 21:28:37 localhost.localdomain redis-shutdown[32584]: + '[' -z '' ']'
Dec 10 21:28:37 localhost.localdomain redis-shutdown[32584]: + SERVICE_NAME=redis
Dec 10 21:28:37 localhost.localdomain redis-shutdown[32584]: + CONFIG_FILE=/etc/redis.conf
Dec 10 21:28:37 localhost.localdomain redis-shutdown[32584]: ++ awk '/^[[:blank:]]*port/ { print $2 }' /etc/redis.conf
Dec 10 21:28:37 localhost.localdomain redis-shutdown[32584]: awk: fatal: cannot open file `/etc/redis.conf' for reading (Permission denied)
Dec 10 21:28:37 localhost.localdomain redis-shutdown[32584]: + PORT=
Dec 10 21:28:37 localhost.localdomain redis-shutdown[32584]: + '[' redis = redis ']'
Dec 10 21:28:37 localhost.localdomain redis-shutdown[32584]: + PORT=6279
Dec 10 21:28:37 localhost.localdomain redis-shutdown[32584]: + /usr/bin/redis-cli -p 6279 shutdown
Dec 10 21:28:37 localhost.localdomain redis-shutdown[32584]: Could not connect to Redis at 127.0.0.1:6279: Connection refused
Dec 10 21:28:37 localhost.localdomain systemd[1]: redis.service: control process exited, code=exited status=1
Dec 10 21:28:37 localhost.localdomain systemd[1]: Unit redis.service entered failed state.

If the service fails to start in this manner, the 'start' command should return an error.

Version-Release number of selected component (if applicable):

redis-2.8.14-2.el7.x86_64

How reproducible:

100%

Steps to Reproduce:
1. chown root.root /etc/redis.conf
2. chmod 640 /etc/redis.conf
3. systemctl start redis.service

Actual results:

returns 0, service fails.

Expected results:

returns 1, service fails.
Comment 1 Flavio Percoco 2014-12-11 08:28:59 UTC 
Thanks for the report, Bill.

Assigning to @hguemar
Comment 2 hguemar 2014-12-11 12:56:07 UTC 
ack.
Comment 3 Nathan Scott 2017-08-28 04:22:59 UTC 
I've looked into this one, understand the root cause now and have a fix.  The underlying problem is that systemd starts the service successfully (this is a Type=Simple service - the default - not Type=forking, or another service type).  

So, systemd starts the process and as far as *it* can tell, it does so successfully even though the process fails very soon after starting (its not an immediate failure from systemd's POV).

This phenomena is described here:
https://serverfault.com/questions/751030/systemd-ignores-return-code-while-starting-service

There is also a possible solution recommended there - to use Type=forking and a script (or helper program) that can verify more carefully that the daemon started and ensure the return code is more meaningful.

Not described there however, is another solution which I think suits better here, and that is the use of a Type=notify service, since current Redis supports this natively.  This is enabled by changing service type to "notify" in the service file, and changing the line "supervised no" to "supervised systemd" in the Redis configuration file).

Here's how Bill's failure scenario (unavailable redis.conf) pans out now ...

# mv /etc/redis.conf /etc/redis.conf.saved
# systemctl start redis
Job for redis.service failed because the control process exited with error code.
See "systemctl  status redis.service" and "journalctl  -xe" for details.
# echo $?
1
# mv /etc/redis.conf.saved /etc/redis.conf
# systemctl start redis
# echo $?
0
# 


I'll continue testing, but assuming all goes well I'll include this change in the next Redis build.
Comment 4 Remi Collet 2017-09-05 06:55:35 UTC 
> This is enabled by changing service type to "notify" in the service file,
> and changing the line "supervised no" to "supervised systemd" in the Redis configuration file).

Please don't rely on configuration file (this will be broken on update, config files are not updated)

Instead, use option in unit file

Type=Notify
ExecStart=/usr/bin/redis-server /etc/redis.conf --daemonize no --supervised systemd

From a quick test, seems to work as expected.
Comment 5 Nathan Scott 2017-09-05 06:58:52 UTC 
Ah yes - thanks Remi - I'll make that change before I push it.

cheers.
Comment 6 Remi Collet 2017-09-05 08:01:58 UTC 
Addition notice: supervised implies daemonize=no, so perhaps this option could be dropped

in src/server.c (checked in 4.0.1)

    int background = server.daemonize && !server.supervised;
    if (background) darmonize();
Comment 7 Nathan Scott 2017-09-05 21:59:49 UTC 
Same code is there in the 3.x series - thanks Remi, I'll make this change as well.
Comment 8 Fedora Update System 2017-09-27 01:02:58 UTC 
redis-3.2.11-1.fc27 has been submitted as an update to Fedora 27. https://bodhi.fedoraproject.org/updates/FEDORA-2017-6d0c7eb24e
Comment 9 Fedora Update System 2017-09-27 01:04:41 UTC 
redis-3.2.11-1.fc25 has been submitted as an update to Fedora 25. https://bodhi.fedoraproject.org/updates/FEDORA-2017-a2c6d143e8
Comment 10 Fedora Update System 2017-09-27 01:05:13 UTC 
redis-3.2.11-1.el7 has been submitted as an update to Fedora EPEL 7. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2017-2735f42785
Comment 11 Fedora Update System 2017-09-27 01:11:36 UTC 
redis-3.2.11-1.fc26 has been submitted as an update to Fedora 26. https://bodhi.fedoraproject.org/updates/FEDORA-2017-165ee8a3f6
Comment 12 Fedora Update System 2017-09-27 15:53:40 UTC 
redis-3.2.11-1.fc27 has been pushed to the Fedora 27 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-6d0c7eb24e
Comment 13 Fedora Update System 2017-09-29 00:20:41 UTC 
redis-3.2.11-1.el7 has been pushed to the Fedora EPEL 7 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2017-2735f42785
Comment 14 Fedora Update System 2017-09-29 00:54:02 UTC 
redis-3.2.11-1.fc26 has been pushed to the Fedora 26 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-165ee8a3f6
Comment 15 Fedora Update System 2017-09-29 01:50:44 UTC 
redis-3.2.11-1.fc25 has been pushed to the Fedora 25 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-a2c6d143e8
Comment 16 Fedora Update System 2017-10-01 18:18:33 UTC 
redis-3.2.11-1.fc27 has been pushed to the Fedora 27 stable repository. If problems still persist, please make note of it in this bug report.
Comment 17 Fedora Update System 2017-10-06 15:20:31 UTC 
redis-3.2.11-1.fc26 has been pushed to the Fedora 26 stable repository. If problems still persist, please make note of it in this bug report.
Comment 18 Fedora Update System 2017-11-17 06:50:55 UTC 
redis-4.0.2-2.fc25 has been submitted as an update to Fedora 25. https://bodhi.fedoraproject.org/updates/FEDORA-2017-343c274808
Comment 19 Fedora Update System 2017-11-17 23:35:35 UTC 
redis-4.0.2-2.fc25 has been pushed to the Fedora 25 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-343c274808
Comment 20 Fedora Update System 2017-11-28 04:11:01 UTC 
redis-4.0.2-2.fc25 has been pushed to the Fedora 25 stable repository. If problems still persist, please make note of it in this bug report.
Comment 21 Fedora Update System 2018-07-20 07:49:00 UTC 
redis-3.2.12-1.el7 has been submitted as an update to Fedora EPEL 7. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2018-8de40d24ca
Comment 22 Fedora Update System 2018-07-20 17:19:45 UTC 
redis-3.2.12-1.el7 has been pushed to the Fedora EPEL 7 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2018-8de40d24ca
Comment 23 Fedora Update System 2018-08-04 19:41:29 UTC 
redis-3.2.12-1.el7 has been pushed to the Fedora EPEL 7 stable repository. If problems still persist, please make note of it in this bug report.
Note You need to log in before you can comment on or make changes to this bug.