1083003 – Nagios SIGSEGV on (internal to nagios) scheduled log rotate if livestatus module is loaded and a downtime is set

Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.

Bug 1083003 - Nagios SIGSEGV on (internal to nagios) scheduled log rotate if livestatus module is loaded and a downtime is set

Summary: Nagios SIGSEGV on (internal to nagios) scheduled log rotate if livestatus mod...

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Fedora EPEL
Classification:	Fedora
Component:	nagios
Sub Component:
Version:	el6
Hardware:	Unspecified
OS:	Linux
Priority:	unspecified
Severity:	unspecified
Target Milestone:	---
Assignee:	Scott Wilkerson
QA Contact:	Fedora Extras Quality Assurance
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2014-04-01 10:00 UTC by Mike Battersby
Modified:	2017-07-23 02:22 UTC (History)
CC List:	13 users (show)
Fixed In Version:	nagios-4.0.8-1.el7 nagios-4.3.2-5.el6
Doc Type:	Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed:	2017-07-23 02:22:17 UTC
Type:	Bug
Embargoed:

Attachments	(Terms of Use)
Patch for downtime.h to work with EPEL nagios. (577 bytes, patch) 2015-03-19 23:43 UTC, Mike Battersby	no flags	Details \| Diff
View All

Description Mike Battersby 2014-04-01 10:00:15 UTC

Description of problem:

Nagios dies with SIGSEGV when it does its internally scheduled log rotation (by default configured to be midnight daily) if the livestatus.o broker module from the check-mk-livestatus package is loaded and at least one host has scheduled downtime.

Version-Release number of selected component (if applicable):

check-mk-livestatus-1.2.2p2-2.el6.i686

How reproducible:

Run nagios with livestatus loaded, schedule a host in downtime, wait for log rotation.

Steps to Reproduce:

1. install nagios (yum install nagios nagios-plugins-all)

2. install check-mk-livestatus (yum install check-mk-livestatus)

3. in /etc/nagios/nagios.cf add "broker_module=/usr/lib/check_mk/livestatus.o /var/spool/nagios/cmd/live"

4. in /etc/nagios/nagios.cf change "log_rotation_method=d" to "log_rotation_method=h" (to see the crash on the hour instead of midnight)

5. start nagios (service nagios start)

6. schedule localhost (it comes pre-configured in the nagios RPM) in downtime

START=`date +%s`; END=`expr $START + 8640000`;
echo "[$START] SCHEDULE_HOST_DOWNTIME;localhost;$START;$END;1;0;8640000;root;test" >> /var/spool/nagios/cmd/nagios.cmd

You can see this reflected in the log (/var/log/nagios/nagios.log) and the nagios status file (/var/log/nagios/status.dat) with a "hostdowntime" section.

7. wait for an hour changeover (e.g., 08:59 -> 09:00).


Actual results:

Nagios process dies with SIGSEGV (logged in /var/log/nagios/nagios.log)
E.g., "[1396011600] Caught SIGSEGV, shutting down..."

Expected results:

Nagios process still running.

Additional info:

This may be (is probably?) a known but unpatched bug.  See:
http://lists.mathias-kettner.de/pipermail/checkmk-en/2013-December/011087.html

Comment 1 Andrea Veri 2014-04-01 15:18:09 UTC

As you may be aware of we don't package the Open Monitoring Distribution (OMD) at all on the EPEL / Fedora repositories. OMD is a bundle of monitoring softwares (Nagios, Icinga, Check-mk itself, mod_python for distributions that have switched to mod_wsgi already until check-mk gets ported to mod_wsgi for WATO to work as expected (and without mod_python)) and the patch you correctly pointed out is related to a bug in Nagios and not in check-mk itself.

I'm therefore changing the product from check-mk to nagios so that it'll be hopefully addressed.

Comment 2 Mike Battersby 2014-04-03 00:26:47 UTC

Great, thanks for that.

I just hope this doesn't fall into one of those "it's their product" vs "it only happens with their module" blackholes.

Comment 3 Mike Battersby 2014-04-14 04:56:27 UTC

More information:

http://support.nagios.com/forum/viewtopic.php?f=7&t=26168&p=92551&hilit=livestatus&sid=f071033f0fb075074980de555068d38a#p92551

According to the Nagios support web site:

> The issue was that at some point in the 3.x branch the internal nagios downtime
> structure changed.
> 
> Livestatus is compiled against the old format, hence the problem.

Maybe this really is a livestatus bug which can be patched in the livestatus RPM?

Comment 4 Mike Battersby 2014-04-14 05:02:19 UTC

A further comment (http://support.nagios.com/forum/viewtopic.php?f=16&t=24172&start=20 but requires login):

> Tue Dec 24, 2013 6:09 am
> Down graded mk-livestatus to 1.2.0p3 after trying both (1.2.2p3 and 1.2.4b3)
> unsucessfully. Log rotation successful even with downtime configurations.

Comment 5 Andrea Veri 2014-04-14 12:03:50 UTC

Mike, can you test if the bug is still there with the latest check-mk release being 1.2.4p1 available on epel-testing?

Comment 6 Mike Battersby 2014-04-16 06:03:17 UTC

Crash bug still exists in check-mk-1.2.4p2-1.el6 from epel-testing.

Using with nagios-3.5.1-1.el6.

Comment 7 Mike Battersby 2015-03-19 23:43:21 UTC

Created attachment 1004252 [details]
Patch for downtime.h to work with EPEL nagios.

Attached patch to stop coredumping with downtime.  I expect nobody at Red Hat cares but another customer might come across this and find it useful.  We have been running with this for some time now.

Problem with livestatus is that Nagios Core has changed the downtime struct in the course of Nagios 3, and check-mk includes and is compiled against an old version.  The check-mk authors don't update it because their Nagios fork OMD uses the old struct.

You also need some changes like this to the check-mk.spec:

Patch1:         02-Downtime-Struct-Update.patch

%prep
%setup -q -n check_mk-%{version}
tar xf agents.tar.gz
tar xf web.tar.gz && rm -f web.tar.gz
%patch0 -p0
tar zcf web.tar.gz htdocs plugins
gunzip livestatus.tar.gz
tar xf livestatus.tar
%patch1 -p0
tar rf livestatus.tar nagios/downtime.h
gzip livestatus.tar.gz

Comment 8 Benji Wakely 2015-04-01 22:27:55 UTC

Mike's patch tested / works for us with the following package combination:
nagios / epel 
check-mk / epel
check-mk-livestatus repackaged from epel package with Mike's patch.

Thanks Mike!

Andrea, respectfully suggest that this would be better treated as a check-mk-livestatus issue to be repackaged when it enters EPEL, to provide compatibility with the packaged nagios distribution - currently they're incompatible.

- nagios core have changed to the new struct way of doing things on purpose, I would guess they're unlikely to revert.
- check-mk author also have no reason to change.

Only a suggestion - it's working for us now here,
but it seems that this is a forced manual step for every user who wants to use these packages together...

--Benji

Comment 9 Andrea Veri 2015-04-08 13:39:10 UTC

Thanks Mike for the patch. I'm currently preparing the package for the 1.2.6 release and will ship the patch you kindly provided.

Comment 10 Mike Battersby 2015-04-09 02:19:50 UTC

Wonderful, thank you very much.

Comment 11 Fedora Update System 2015-04-13 14:31:29 UTC

check-mk-1.2.6p1-1.fc22 has been submitted as an update for Fedora 22.
https://admin.fedoraproject.org/updates/check-mk-1.2.6p1-1.fc22

Comment 12 Fedora Update System 2015-04-13 14:31:35 UTC

check-mk-1.2.6p1-1.el6 has been submitted as an update for Fedora EPEL 6.
https://admin.fedoraproject.org/updates/check-mk-1.2.6p1-1.el6

Comment 13 Fedora Update System 2015-04-13 14:31:41 UTC

check-mk-1.2.6p1-1.fc21 has been submitted as an update for Fedora 21.
https://admin.fedoraproject.org/updates/check-mk-1.2.6p1-1.fc21

Comment 14 Fedora Update System 2015-04-13 14:32:50 UTC

check-mk-1.2.6p1-1.fc20 has been submitted as an update for Fedora 20.
https://admin.fedoraproject.org/updates/check-mk-1.2.6p1-1.fc20

Comment 15 Fedora Update System 2015-04-13 14:34:00 UTC

check-mk-1.2.6p1-1.el5 has been submitted as an update for Fedora EPEL 5.
https://admin.fedoraproject.org/updates/check-mk-1.2.6p1-1.el5

Comment 16 Fedora Update System 2015-04-13 14:34:07 UTC

check-mk-1.2.6p1-1.el7 has been submitted as an update for Fedora EPEL 7.
https://admin.fedoraproject.org/updates/check-mk-1.2.6p1-1.el7

Comment 17 Ira W. Snyder 2015-04-15 15:24:13 UTC

(In reply to Andrea Veri from comment #9)
> Thanks Mike for the patch. I'm currently preparing the package for the 1.2.6
> release and will ship the patch you kindly provided.

Hi Andrea,

I tried the package from Comment #16 on RHEL7 (already using check-mk-1.2.4p5-2.el7.x86_64 from EPEL). In contrast to the older package in EPEL, the new one requires mod_python, which isn't available on EL7.

I don't see anything in the package that actually requires mod_python to run. I think that the dependency on mod_python should be removed, at least on EL7.

Thanks! I look forward to being able to use downtimes again!

Comment 18 Fedora Update System 2015-04-17 13:07:44 UTC

check-mk-1.2.6p1-2.el7 has been submitted as an update for Fedora EPEL 7.
https://admin.fedoraproject.org/updates/check-mk-1.2.6p1-2.el7

Comment 19 Fedora Update System 2015-04-21 18:37:24 UTC

check-mk-1.2.6p1-1.fc22 has been pushed to the Fedora 22 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 20 Fedora Update System 2015-04-26 12:44:29 UTC

check-mk-1.2.6p1-1.fc21 has been pushed to the Fedora 21 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 21 Fedora Update System 2015-04-26 12:50:29 UTC

check-mk-1.2.6p1-1.fc20 has been pushed to the Fedora 20 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 22 Fedora Update System 2015-04-29 19:17:52 UTC

check-mk-1.2.6p1-1.el5 has been pushed to the Fedora EPEL 5 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 23 Fedora Update System 2015-04-29 19:18:24 UTC

check-mk-1.2.6p1-1.el6 has been pushed to the Fedora EPEL 6 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 24 Fedora Update System 2015-05-03 23:12:39 UTC

check-mk-1.2.6p1-2.el7 has been pushed to the Fedora EPEL 7 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 25 Fedora Update System 2015-09-20 14:39:17 UTC

nagios-4.0.8-1.fc21 has been submitted as an update to Fedora 21. https://bodhi.fedoraproject.org/updates/FEDORA-2015-16238

Comment 26 Fedora Update System 2015-09-20 14:39:18 UTC

nagios-4.0.8-1.el7 has been submitted as an update to Fedora EPEL 7. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2015-8155

Comment 27 Fedora Update System 2015-09-20 14:40:15 UTC

nagios-4.0.8-1.fc22 has been submitted as an update to Fedora 22. https://bodhi.fedoraproject.org/updates/FEDORA-2015-16239

Comment 28 Fedora Update System 2015-09-20 14:40:17 UTC

nagios-4.0.8-1.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2015-16240

Comment 29 Fedora Update System 2015-09-20 14:41:05 UTC

nagios-4.0.8-1.el6 has been submitted as an update to Fedora EPEL 6. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2015-8156

Comment 30 Fedora Update System 2015-09-20 16:49:28 UTC

nagios-4.0.8-1.fc22 has been pushed to the Fedora 22 testing repository. If problems still persist, please make note of it in this bug report.
If you want to test the update, you can install it with
$ su -c 'dnf --enablerepo=updates-testing update nagios'
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2015-16239

Comment 31 Fedora Update System 2015-09-20 16:51:29 UTC

nagios-4.0.8-1.fc21 has been pushed to the Fedora 21 testing repository. If problems still persist, please make note of it in this bug report.
If you want to test the update, you can install it with
$ su -c 'dnf --enablerepo=updates-testing update nagios'
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2015-16238

Comment 32 Fedora Update System 2015-09-20 17:19:54 UTC

nagios-4.0.8-1.fc23 has been pushed to the Fedora 23 testing repository. If problems still persist, please make note of it in this bug report.
If you want to test the update, you can install it with
$ su -c 'dnf --enablerepo=updates-testing update nagios'
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2015-16240

Comment 33 Fedora Update System 2015-09-21 02:47:13 UTC

nagios-4.0.8-1.el6 has been pushed to the Fedora EPEL 6 testing repository. If problems still persist, please make note of it in this bug report.
If you want to test the update, you can install it with
$ su -c 'dnf --enablerepo=updates-testing update nagios'
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2015-8156

Comment 34 Fedora Update System 2015-09-21 02:50:43 UTC

nagios-4.0.8-1.el7 has been pushed to the Fedora EPEL 7 testing repository. If problems still persist, please make note of it in this bug report.
If you want to test the update, you can install it with
$ su -c 'dnf --enablerepo=updates-testing update nagios'
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2015-8155

Comment 35 Fedora Update System 2015-12-05 20:31:00 UTC

nagios-4.0.8-1.el7 has been pushed to the Fedora EPEL 7 stable repository. If problems still persist, please make note of it in this bug report.

Comment 36 Fedora Update System 2017-02-09 17:28:43 UTC

nagios-4.2.4-3.el6 has been submitted as an update to Fedora EPEL 6. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2017-270f6f4375

Comment 37 Fedora Update System 2017-02-10 22:41:01 UTC

nagios-4.2.4-4.el6 has been submitted as an update to Fedora EPEL 6. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2017-acd2c2af0d

Comment 38 Fedora Update System 2017-02-11 16:18:47 UTC

nagios-4.2.4-4.el6 has been pushed to the Fedora EPEL 6 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2017-acd2c2af0d

Comment 39 Fedora Update System 2017-05-11 19:16:59 UTC

nagios-4.3.2-1.el6 has been submitted as an update to Fedora EPEL 6. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2017-1f2571d162

Comment 40 Fedora Update System 2017-05-12 19:51:52 UTC

nagios-4.3.2-1.el6 has been pushed to the Fedora EPEL 6 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2017-1f2571d162

Comment 41 Fedora Update System 2017-06-14 19:00:41 UTC

nagios-4.3.2-3.el6 has been submitted as an update to Fedora EPEL 6. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2017-16880697fe

Comment 42 Fedora Update System 2017-06-15 09:47:23 UTC

nagios-4.3.2-3.el6 has been pushed to the Fedora EPEL 6 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2017-16880697fe

Comment 43 Fedora Update System 2017-06-28 19:28:37 UTC

nagios-4.3.2-5.el6 has been submitted as an update to Fedora EPEL 6. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2017-328a23d1ed

Comment 44 Fedora Update System 2017-06-30 01:17:11 UTC

nagios-4.3.2-5.el6 has been pushed to the Fedora EPEL 6 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2017-328a23d1ed

Comment 45 Fedora Update System 2017-07-23 02:22:17 UTC

nagios-4.3.2-5.el6 has been pushed to the Fedora EPEL 6 stable repository. If problems still persist, please make note of it in this bug report.

Note You need to log in before you can comment on or make changes to this bug.