Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.
Bug 2100549 - Fail2ban stops processing journal log events after an imjournal rotation
Summary: Fail2ban stops processing journal log events after an imjournal rotation
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora EPEL
Classification: Fedora
Component: fail2ban
Version: epel9
Hardware: x86_64
OS: Linux
unspecified
medium
Target Milestone: ---
Assignee: Orion Poplawski
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
: 2095722 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2022-06-23 16:29 UTC by Kevin
Modified: 2023-04-10 00:42 UTC (History)
13 users (show)

Fixed In Version: fail2ban-1.0.2-3.el9
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2023-04-10 00:42:26 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker FC-508 0 None None None 2022-07-12 14:46:37 UTC

Description Kevin 2022-06-23 16:29:18 UTC
Description of problem:

I was testing AlmaLinux 9 on Linode and found that fail2ban would stop processing sshd log events after an imjournal rotation.  It would not process them again until I restarted fail2ban or rebooted.  Until the imjournal rotation, fail2ban does work correctly.  Forcing a rotation with "journalctl --rotate" stops fail2ban from processing the journal events and is an easier way to test than waiting for the system to do it.

I was able to reproduce this on CentOS Stream 9 and also with AlmaLinux 9 from the official ISO in a VMware Fusion VM.  Interestingly, it works as expected on Fedora 36 Server.

I enabled debug on systemd and fail2ban which did not help.  I then used setroubleshoot and found that it was prohibiting python3.9 from watching the journal directory /run/log/journal.  Once I created and applied the SELinux policy as suggested by the setroubleshoot output, it worked correctly.

Here is the policy suggestion from setroubleshoot:

 *****  Plugin catchall (100. confidence) suggests   ************************>
   
   If you believe that python3.9 should be allowed watch access on the ffffffff>
   Then you should report this as a bug.
   You can generate a local policy module to allow this access.
   Do
   allow this access for now by executing:
   # ausearch -c 'f2b/f.sshd' --raw | audit2allow -M my-f2bfsshd
   # semodule -X 300 -i my-f2bfsshd.pp


Steps to Reproduce:

1. Start with a fresh installation.
2. Install epel-release, update packages, and install fail2ban.
3. Configure a basic sshd jail.
4. Observe that it correctly reports sshd anomalous events.
5. Use the command "journalctl --rotate" to force an imjournal rotation.
6. Verify that fail2ban will no longer process sshd events, even though it should.

Comment 1 Zdenek Pytela 2022-06-24 19:31:31 UTC
Kevin,

Please attach the output of

  # ausearch -i -m avc,user_avc,selinux_err,user_selinux_err -ts today

Switching the component as fail2ban ships its own policy.

Comment 2 Kevin 2022-06-24 20:15:35 UTC
Here is the output:

----
type=PROCTITLE msg=audit(06/24/2022 16:11:28.590:966) : proctitle=/usr/bin/python3 -s /usr/bin/fail2ban-server -xf start 
type=SYSCALL msg=audit(06/24/2022 16:11:28.590:966) : arch=x86_64 syscall=inotify_add_watch success=no exit=EACCES(Permission denied) a0=0x8 a1=0x7fa62bffd990 a2=0x1000386 a3=0x9 items=0 ppid=1 pid=13256 auid=unset uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=(none) ses=unset comm=f2b/f.sshd exe=/usr/bin/python3.9 subj=system_u:system_r:fail2ban_t:s0 key=(null) 
type=AVC msg=audit(06/24/2022 16:11:28.590:966) : avc:  denied  { watch } for  pid=13256 comm=f2b/f.sshd path=/run/log/journal dev="tmpfs" ino=58 scontext=system_u:system_r:fail2ban_t:s0 tcontext=system_u:object_r:syslogd_var_run_t:s0 tclass=dir permissive=0 
----
type=PROCTITLE msg=audit(06/24/2022 16:11:28.590:967) : proctitle=/usr/bin/python3 -s /usr/bin/fail2ban-server -xf start 
type=SYSCALL msg=audit(06/24/2022 16:11:28.590:967) : arch=x86_64 syscall=inotify_add_watch success=no exit=EACCES(Permission denied) a0=0x8 a1=0x7fa62bffd910 a2=0x1002fc6 a3=0xc items=0 ppid=1 pid=13256 auid=unset uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=(none) ses=unset comm=f2b/f.sshd exe=/usr/bin/python3.9 subj=system_u:system_r:fail2ban_t:s0 key=(null) 
type=AVC msg=audit(06/24/2022 16:11:28.590:967) : avc:  denied  { watch } for  pid=13256 comm=f2b/f.sshd path=/run/log/journal/5b5b1df497254580be34dc64b37be4e5 dev="tmpfs" ino=454 scontext=system_u:system_r:fail2ban_t:s0 tcontext=system_u:object_r:syslogd_var_run_t:s0 tclass=dir permissive=0 
----
type=PROCTITLE msg=audit(06/24/2022 16:11:28.591:968) : proctitle=/usr/bin/python3 -s /usr/bin/fail2ban-server -xf start 
type=SYSCALL msg=audit(06/24/2022 16:11:28.591:968) : arch=x86_64 syscall=inotify_add_watch success=no exit=EACCES(Permission denied) a0=0x8 a1=0x7fa62bffd910 a2=0x1002fc6 a3=0xc items=0 ppid=1 pid=13256 auid=unset uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=(none) ses=unset comm=f2b/f.sshd exe=/usr/bin/python3.9 subj=system_u:system_r:fail2ban_t:s0 key=(null) 
type=AVC msg=audit(06/24/2022 16:11:28.591:968) : avc:  denied  { watch } for  pid=13256 comm=f2b/f.sshd path=/run/log/journal/ffffffffffffffffffffffffffffffff dev="tmpfs" ino=59 scontext=system_u:system_r:fail2ban_t:s0 tcontext=system_u:object_r:syslogd_var_run_t:s0 tclass=dir permissive=0 
----
type=PROCTITLE msg=audit(06/24/2022 16:11:32.800:1063) : proctitle=/usr/bin/python3 -s /usr/bin/fail2ban-server -xf start 
type=SYSCALL msg=audit(06/24/2022 16:11:32.800:1063) : arch=x86_64 syscall=inotify_add_watch success=no exit=EACCES(Permission denied) a0=0x8 a1=0x7f75051df990 a2=0x1000386 a3=0x9 items=0 ppid=1 pid=13512 auid=unset uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=(none) ses=unset comm=f2b/f.sshd exe=/usr/bin/python3.9 subj=system_u:system_r:fail2ban_t:s0 key=(null) 
type=AVC msg=audit(06/24/2022 16:11:32.800:1063) : avc:  denied  { watch } for  pid=13512 comm=f2b/f.sshd path=/run/log/journal dev="tmpfs" ino=58 scontext=system_u:system_r:fail2ban_t:s0 tcontext=system_u:object_r:syslogd_var_run_t:s0 tclass=dir permissive=0 
----
type=PROCTITLE msg=audit(06/24/2022 16:11:32.801:1064) : proctitle=/usr/bin/python3 -s /usr/bin/fail2ban-server -xf start 
type=SYSCALL msg=audit(06/24/2022 16:11:32.801:1064) : arch=x86_64 syscall=inotify_add_watch success=no exit=EACCES(Permission denied) a0=0x8 a1=0x7f75051df910 a2=0x1002fc6 a3=0xc items=0 ppid=1 pid=13512 auid=unset uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=(none) ses=unset comm=f2b/f.sshd exe=/usr/bin/python3.9 subj=system_u:system_r:fail2ban_t:s0 key=(null) 
type=AVC msg=audit(06/24/2022 16:11:32.801:1064) : avc:  denied  { watch } for  pid=13512 comm=f2b/f.sshd path=/run/log/journal/5b5b1df497254580be34dc64b37be4e5 dev="tmpfs" ino=454 scontext=system_u:system_r:fail2ban_t:s0 tcontext=system_u:object_r:syslogd_var_run_t:s0 tclass=dir permissive=0 
----
type=PROCTITLE msg=audit(06/24/2022 16:11:32.801:1065) : proctitle=/usr/bin/python3 -s /usr/bin/fail2ban-server -xf start 
type=SYSCALL msg=audit(06/24/2022 16:11:32.801:1065) : arch=x86_64 syscall=inotify_add_watch success=no exit=EACCES(Permission denied) a0=0x8 a1=0x7f75051df910 a2=0x1002fc6 a3=0xc items=0 ppid=1 pid=13512 auid=unset uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=(none) ses=unset comm=f2b/f.sshd exe=/usr/bin/python3.9 subj=system_u:system_r:fail2ban_t:s0 key=(null) 
type=AVC msg=audit(06/24/2022 16:11:32.801:1065) : avc:  denied  { watch } for  pid=13512 comm=f2b/f.sshd path=/run/log/journal/ffffffffffffffffffffffffffffffff dev="tmpfs" ino=59 scontext=system_u:system_r:fail2ban_t:s0 tcontext=system_u:object_r:syslogd_var_run_t:s0 tclass=dir permissive=0

Comment 3 Richard Shaw 2022-06-25 13:55:18 UTC
This looks similar to something Orion fixed in rawhide but I'm not sure if it's made it into an EPEL build...

https://src.fedoraproject.org/rpms/fail2ban/c/ec52ec24716b4d6e820431dbe7b33aceb20112d0?branch=rawhide

Comment 4 Kevin 2022-06-25 18:53:49 UTC
That looks similar but the problem was resolved by allowing watch on syslogd_var_run_t:dir.  Fail2ban uses the systemd journal and not files in /var/log.

Here is the content of my-f2bfsshd.te that resolved the issue for me:

module my-f2bfsshd 1.0;

require {
        type syslogd_var_run_t;
        type fail2ban_t;
        class dir watch;
}

#============= fail2ban_t ==============
allow fail2ban_t syslogd_var_run_t:dir watch;

Comment 5 Samuel Sieb 2022-06-26 21:47:25 UTC
The systemd journal files are in /var/log/journal.  Is it different in RHEL?

Comment 6 Samuel Sieb 2022-06-26 21:50:48 UTC
I do have a /var/run/log/journal directory on my F36 system, but it's empty and unowned, so I have no idea where it came from.  I'm guessing it's a leftover from some previous release.

Comment 7 Kevin 2022-06-27 03:31:41 UTC
From the man page for systemd-journald:

       The journal service stores log data either persistently below
       /var/log/journal or in a volatile way below /run/log/journal/ (in the
       latter case it is lost at reboot). By default, log data is stored
       persistently if /var/log/journal/ exists during boot, with an implicit
       fallback to volatile storage otherwise. Use Storage= in
       journald.conf(5) to configure where log data is placed, independently
       of the existence of /var/log/journal/.

On a fresh install of AlmaLinux 9 (or RHEL 9) it will perform volatile logging as mentioned above.  In addition, rsyslog monitors the journal and persistently logs some of the data to various log files in /var/log.

Samuel, I assume you have an older or perhaps a modified configuration.

Comment 8 Samuel Sieb 2022-06-27 04:58:46 UTC
I think you misunderstood my comment.  My logs are in the standard /var/log/journal.  I just realized while writing this why there is an empty /var/run/log/journal directory.  /var/run is symlinked to /run.

But I was also being misled by the selinux type which says "syslogd_**var_run**_t" and missed that the indicated directory is actually "/run/log" and now that I discovered the symlink, that makes more sense.

Comment 9 Dan Tucny 2022-11-03 14:42:05 UTC
Kevin, do you still see this problem? If you just install fail2ban, I expect you do.

If so, does installing the additional fail2ban-selinux package resolve it for you? I believe it would, but, I wasn't able to find anything that states that it's required for fail2ban to actually work.

I'd suspect that the most common use case for fail2ban would be brute force protection for SSH, it's a bit of a problem that it will silently fail to do that without installing an additional package that you wouldn't know about without looking for it.

I think it makes sense to make fail2ban-selinux a dependency to fail2ban.

Comment 10 Richard Shaw 2022-11-03 21:14:35 UTC
Currently the selinux package is only installed on Fedora. I admit I don't have time to keep up with the EL stuff, so it looks like it is needed on EL9?

I can update:
%if 0%{?fedora}
Requires: (%{name}-selinux if selinux-policy-%{selinuxtype})
%endif

To:
%if 0%{?fedora} || 0%{?rhel} >= 9
Requires: (%{name}-selinux if selinux-policy-%{selinuxtype})
%endif

Comment 11 Kevin 2022-11-04 03:04:42 UTC
Dan, I did a fresh install of AlmaLinux 9, updated all the packages, and only installed fail2ban.  With just that package it did seem to exhibit the same problem.  I installed fail2ban-selinux and it appears to fix it.  I tried to look at the the policy in that package.  While it has a lot of rules in there, it did seem to include watch on syslogd_var_run_t.  It looks like installing fail2ban-selinux does fix the issue.

Since SELinux is enabled by default, it would make sense to require fail2ban-selinux when installing fail2ban.

Comment 12 Richard Shaw 2022-11-04 12:19:27 UTC
I added EL 9 to the conditional for requiring the selinux subpackage.

If it's OK I'd like to close this as "next release". I'm still working on the iptables vs nftables problem and would prefer to wait until I have a fix for that before doing builds.

Comment 13 Kevin 2022-11-09 16:08:45 UTC
I'm ok with waiting on it but I have already worked around it.

Comment 14 Orion Poplawski 2023-03-30 00:06:49 UTC
*** Bug 2095722 has been marked as a duplicate of this bug. ***

Comment 15 Fedora Update System 2023-04-01 14:44:01 UTC
FEDORA-EPEL-2023-07bf30a1f1 has been submitted as an update to Fedora EPEL 9. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2023-07bf30a1f1

Comment 16 Fedora Update System 2023-04-02 02:50:58 UTC
FEDORA-EPEL-2023-07bf30a1f1 has been pushed to the Fedora EPEL 9 testing repository.

You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2023-07bf30a1f1

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 17 Fedora Update System 2023-04-10 00:42:26 UTC
FEDORA-EPEL-2023-07bf30a1f1 has been pushed to the Fedora EPEL 9 stable repository.
If problem still persists, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.