Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.

Bug 1179980

Summary: Too many journal files for journalctl to handle
Product: [Fedora] Fedora Reporter: Norman Gaywood <ngaywood>
Component: systemdAssignee: systemd-maint
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 21CC: johannbg, jsynacek, lnykryn, msekleta, s, systemd-maint, vpavlin, zbyszek
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: systemd-216-16.fc21 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-01-21 23:02:50 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Norman Gaywood 2015-01-08 00:47:59 UTC
Description of problem:

I have many journal files, due to many users using the system and each creating their own journal:

ls /var/log/journal/e3f1ec33551971df031878511bf6c160/ | wc -l 
1126

If ulimit -n is not set sufficiently above 1024, journalctl silently fails to read many journal files and hangs without tailing logs when I use a "journalctl -f".

If ulimit is set high enough, I get informed of what journal files are skipped:

ulimit -n
200000

journalctl -n
...[clip 100 lines]...
Too many open journal files, not adding /var/log/journal/e3f1ec33551971df031878511bf6c160/user-41277871.journal.
Too many open journal files, not adding /var/log/journal/e3f1ec33551971df031878511bf6c160/user-55024.
Error was encountered while opening journal files: Too many references: cannot splice


Version-Release number of selected component (if applicable):

systemd-216-13.fc21.x86_64

How reproducible:

Problem has been in journalctl for some time, only just got to report it.

Comment 1 Lennart Poettering 2015-01-08 02:23:57 UTC
I now changed git upstream to bump RLIMIT_NOFILE to 16K if possible (effectively this is limited to 4K though for normal journalctl invocations since the default hard limit for user code defaults to 4K on fedora).

http://cgit.freedesktop.org/systemd/systemd/commit/?id=de45d726034f33afdb0a185e62fc61bf10a0acd2

This is not a full fix though. We need to come up with a strategy what to do when we hit the limit. In the worst case we need to fail gracefully. This could mean warning the user that files are not shown and ensuring that the oldest files are the ones missing, not the newest ones...

Comment 2 Norman Gaywood 2015-01-08 03:26:18 UTC
Thanks for the quick attention!

Not that I know the code, but do you also need to change the hard-coded constant in sd-journal.c

./sd-journal.c:#define JOURNAL_FILES_MAX 1024

Comment 3 Zbigniew Jędrzejewski-Szmek 2015-01-09 14:32:38 UTC
I removed the check now in commit b914ea8d379b446c4c9fac4ba181771676ef38cd.

Comment 4 Fedora Update System 2015-01-19 19:23:28 UTC
systemd-216-16.fc21 has been submitted as an update for Fedora 21.
https://admin.fedoraproject.org/updates/systemd-216-16.fc21

Comment 5 Fedora Update System 2015-01-20 21:02:31 UTC
Package systemd-216-16.fc21:
* should fix your issue,
* was pushed to the Fedora 21 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing systemd-216-16.fc21'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2015-0916/systemd-216-16.fc21
then log in and leave karma (feedback).

Comment 6 Fedora Update System 2015-01-21 23:02:50 UTC
systemd-216-16.fc21 has been pushed to the Fedora 21 stable repository.  If problems still persist, please make note of it in this bug report.