Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.
Bug 1504814
| Summary: | /etc/init.d/nagios stop hangs forever | ||
|---|---|---|---|
| Product: | [Fedora] Fedora EPEL | Reporter: | Dimitri Maziuk <dmaziuk> |
| Component: | nagios | Assignee: | Stephen John Smoogen <smooge> |
| Status: | CLOSED ERRATA | QA Contact: | Fedora Extras Quality Assurance <extras-qa> |
| Severity: | high | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | el6 | CC: | affix, athmanem, b.heden, herrold, jose.p.oliveira.oss, lemenkov, linux, nb, ondrejj, peter.meier, shawn.starr, smooge, smooge, s, swilkerson |
| Target Milestone: | --- | ||
| Target Release: | --- | ||
| Hardware: | x86_64 | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | nagios-4.3.4-7.el6 | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2017-12-10 23:28:03 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
|
Description
Dimitri Maziuk
2017-10-20 17:05:06 UTC
... it *looks* like the cure is worse ... I have not been able to replicate this on my el6 systems. Could you let me know if you have selinux enabled, and if you do if you have the nagios-selinux rpm installed? After that I would edit the file and comment out the /dev/null part and see if it saying something. We can go from there. No, no selinux here. There is nagiosgraph and a whole lot of custom monitoring scripts (some of which seems to have b0rked on this update) but nothing obvious to explain this. Without the redirect: ``` [root@hereford init.d]# ./nagios stop Stopping nagios:. done. 0+0 records in 0+0 records out 0 bytes (0 B) copied, 7.2405e-05 s, 0.0 kB/s ``` and it's hanging: ``` [root@hereford ~]# ps auxw | grep nagios root 5875 0.0 0.0 106512 1668 pts/2 S+ 18:07 0:00 /bin/sh ./nagios stop root 5887 0.0 0.0 106512 860 pts/2 S+ 18:07 0:00 /bin/sh ./nagios stop root 5929 0.0 0.0 103324 824 pts/0 S+ 18:10 0:00 grep nagios ``` I do have two more notes after looking: 1. does this look right: ``` (dd if=$NagiosCommandFile~ count=0 2>/dev/null & echo -n "" >$NagiosCommandFile~) ``` -- I mean the single ampersand before echo. 2. Stopping nagios should probably print "done" after it's removed command file and cleaned up etc. I tried this: dd if=/var/spool/nagios/cmd/nagios.cmd~ count=0 of=/dev/null just to make sure it's not hanging trying to write to the tty, and it's still sitting there. So it looks to me like it's waiting on the pipe -- no idea why PPS: sending `dd` a USR1: ``` dd: opening `/var/spool/nagios/cmd/nagios.cmd~': Interrupted system call + rm -f /var/spool/nagios/cmd/nagios.cmd /var/spool/nagios/cmd/nagios.cmd~ + rm -f /var/log/nagios/status.dat /var/run/nagios/nagios.pid /var/lock/subsys/nagios ``` Ok the & is correct because it is putting that process in as a background process. The echo "" > $NagiosCommandFile~ should then cause the background task and any other items attached to the pipe to fail and close. Could you see if the following script also locks? #!/bin/bash rm -f /tmp/test.pipe mkfifo /tmp/test.pipe ls -1 /tmp > /tmp/test.pipe & mv /tmp/test.pipe /tmp/test.pipe~ ## The () make this a subshell. The & is intentional as we want to bg this. ( dd if=/tmp/test.pipe~ count=0 2>/dev/null & echo -n "" > /tmp/test.pipe~ ) jobs rm -f /tmp/test.pipe~ ### end of script OK from the dd that says something is locking the sub dd process from being able to read the pipe file but the only thing I can think of would be selinux or a filesystem error. What kind of filesystem is /var/spool on this box and could you do a ls -lZ /var/spool/nagios/cmd/nagios.cmd* when it is running so I can see what the file is setup as? Thanks. ``` # ls -lZ /var/spool/nagios/cmd/nagios.cmd* prw-rw---- nagios nagios ? /var/spool/nagios/cmd/nagios.cmd ``` It's an ext4 on spinning rust with like 4 out 200GB used. The script hangs about 15% of the time. Can't say I see a rhyme or reason to it. ``` [root@hereford ~]# ./testpipe.sh [1]+ Running ls -1 /tmp > /tmp/test.pipe & [root@hereford ~]# ./testpipe.sh [1]+ Running ls -1 /tmp > /tmp/test.pipe & [root@hereford ~]# ./testpipe.sh [1]+ Running ls -1 /tmp > /tmp/test.pipe & [root@hereford ~]# ./testpipe.sh [1]+ Running ls -1 /tmp > /tmp/test.pipe & [root@hereford ~]# ./testpipe.sh [1]+ Running ls -1 /tmp > /tmp/test.pipe & [root@hereford ~]# ./testpipe.sh [1]+ Running ls -1 /tmp > /tmp/test.pipe & [root@hereford ~]# ./testpipe.sh [1]+ Running ls -1 /tmp > /tmp/test.pipe & [root@hereford ~]# ./testpipe.sh [1]+ Running ls -1 /tmp > /tmp/test.pipe & [root@hereford ~]# ./testpipe.sh [1]+ Running ls -1 /tmp > /tmp/test.pipe & [root@hereford ~]# ./testpipe.sh [1]+ Running ls -1 /tmp > /tmp/test.pipe & [root@hereford ~]# ./testpipe.sh [1]+ Running ls -1 /tmp > /tmp/test.pipe & [root@hereford ~]# ./testpipe.sh [1]+ Running ls -1 /tmp > /tmp/test.pipe & [root@hereford ~]# ./testpipe.sh [1]+ Running ls -1 /tmp > /tmp/test.pipe & [root@hereford ~]# ./testpipe.sh [1]+ Running ls -1 /tmp > /tmp/test.pipe & [root@hereford ~]# ./testpipe.sh [1]+ Running ls -1 /tmp > /tmp/test.pipe & [root@hereford ~]# ./testpipe.sh [1]+ Running ls -1 /tmp > /tmp/test.pipe & [root@hereford ~]# ./testpipe.sh [1]+ Running ls -1 /tmp > /tmp/test.pipe & [root@hereford ~]# ./testpipe.sh ^C./testpipe.sh: line 8: /tmp/test.pipe~: Interrupted system call [root@hereford ~]# ./testpipe.sh [1]+ Running ls -1 /tmp > /tmp/test.pipe & [root@hereford ~]# ./testpipe.sh [1]+ Running ls -1 /tmp > /tmp/test.pipe & [root@hereford ~]# ./testpipe.sh ^C./testpipe.sh: line 8: /tmp/test.pipe~: Interrupted system call [root@hereford ~]# ./testpipe.sh ^C./testpipe.sh: line 8: /tmp/test.pipe~: Interrupted system call ``` OK that is weird. I haven't been able to replicate yet, but I fear that nagios is going to hang whether or not the line is in the services because something is holding on the named pipe and isn't allowing an interrupt. I am going to get some input from more experienced engineers on what should be done here. This is an old dual-core pentium D with 4GB RAM, but about the only other thing that's running there is salt-minion. There's maybe a GB of RAM used and an occasional 0.2% iowait or sys load... can't really say the hardware is strained. Other than that it's a stock centos 6.latest x86_64, it's been running nagios happily for years until this update, nagiosgraph is about the only non-stock thing on it that I can think of. The consensus I have gotten so far is that it working for me is a fluke and I will raise it with upstream to remove this from their code. https://github.com/NagiosEnterprises/nagioscore/issues/443 I am commenting this out and will put in an updated build. LOL It is a weird one though, I managed to do a couple of /etc/init.d/nagios stops without the hang, too, while playing with it. I'm observing/affected by the same behavior. Interesting enough: It worked fine, once I restarted the server. I now added the fix from upstream, as I regularly restart nagios due to config changes, I assume I might see it again quite soon, if it still happens. OK I got a probable fix for this. The problem is that the echo also needs to be backgrounded so that the () goes completely into the background. I have tried it on my test boxes and it doesn't hang. But I was not able to trigger it very often. A -5 with this will appear in epel-testing soon. if accessing through ssh, adding -t sometimes helps as does, when application, adding a < /dev/null to keep it from hanging up looking for stdin ssh would probably be the most common: restart after reconfig, but you could fire it from e.g. salt state or something too... nagios-4.3.4-5.el6 has been submitted as an update to Fedora EPEL 6. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2017-723d26389e nagios-4.3.4-5.el6 has been pushed to the Fedora EPEL 6 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2017-723d26389e nagios-4.3.4-6.el6 has been submitted as an update to Fedora EPEL 6. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2017-ecb67df0a6 nagios-4.3.4-7.el6 has been pushed to the Fedora EPEL 6 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2017-ecb67df0a6 nagios-4.3.4-7.el6 has been pushed to the Fedora EPEL 6 stable repository. If problems still persist, please make note of it in this bug report. |