1533760 – fail2ban fails to create ipset rules

Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.

Bug 1533760 - fail2ban fails to create ipset rules

Summary: fail2ban fails to create ipset rules

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Fedora
Classification:	Fedora
Component:	fail2ban
Sub Component:
Version:	27
Hardware:	All
OS:	Linux
Priority:	unspecified
Severity:	high
Target Milestone:	---
Assignee:	Orion Poplawski
QA Contact:	Fedora Extras Quality Assurance
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2018-01-12 07:00 UTC by Hui Li
Modified:	2018-12-27 17:09 UTC (History)
CC List:	6 users (show)
Fixed In Version:	fail2ban-0.10.2-1.fc27 fail2ban-0.10.2-1.fc28
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2018-04-06 15:01:50 UTC
Type:	Bug
Embargoed:
Dependent Products:

Attachments	(Terms of Use)

Description Hui Li 2018-01-12 07:00:43 UTC

Description of problem:
fail2ban package is failed to ban hosts for ssh service due to failing to executing ipset command.

Version-Release number of selected component (if applicable):
fail2ban-server-0.10.0-1.fc27.noarch
fail2ban-firewalld-0.10.0-1.fc27.noarch
fail2ban-sendmail-0.10.0-1.fc27.noarch
fail2ban-0.10.0-1.fc27.noarch


How reproducible:
Always

Steps to Reproduce:
1. add local.conf under /etc/fail2ban/jail.d. 
[DEFAULT]
bantime = 604800
sender = root@localhost
destemail = root
action = %(action_)s

[sshd]
enabled = true
port = all
protocol = tcp
filter = sshd

2. change the blocktype in iptables-common.conf and firewallcmd-common.conf to DROP as the original just blocks ICMP requests.

blocktype = DROP

3. restart fail2ban.service via systemctl.

Actual results:

In /var/log/fail2ban.log, following error messages throws out:
2018-01-12 01:40:18,160 fail2ban.actions        [626]: NOTICE  [sshd] Ban 212.164.53.17
2018-01-12 01:40:18,161 fail2ban.action         [626]: DEBUG   ipset create f2b-sshd hash:ip timeout 604800
firewall-cmd --direct --add-rule ipv4 filter INPUT 0 -p tcp -m multiport --dports all -m set --match-set f2b-sshd src -j DROP
2018-01-12 01:40:23,110 fail2ban.utils          [626]: Level 39 ffff94b6af10 -- exec: ipset create f2b-sshd hash:ip timeout 604800
firewall-cmd --direct --add-rule ipv4 filter INPUT 0 -p tcp -m multiport --dports all -m set --match-set f2b-sshd src -j DROP
2018-01-12 01:40:23,112 fail2ban.utils          [626]: ERROR   ffff94b6af10 -- stderr: 'ipset v6.32: Set cannot be created: set with the same name already exists'
2018-01-12 01:40:23,113 fail2ban.utils          [626]: ERROR   ffff94b6af10 -- stderr: '\x1b[91mError: COMMAND_FAILED\x1b[00m'
2018-01-12 01:40:23,114 fail2ban.utils          [626]: ERROR   ffff94b6af10 -- returned 13
2018-01-12 01:40:23,115 fail2ban.actions        [626]: ERROR   Failed to execute ban jail 'sshd' action 'firewallcmd-ipset' info 'ActionInfo({'ip': 212.164.53.17, 'family': 'inet4', 'ip-rev': '17.53.164.212.', 'ip-host': 'b-internet.212.164.53.17.nsk.rt.ru', 'fid': 212.164.53.17, 'failures': 8, 'time': 1515487826.247413, 'matches': '2018-01-09T03:50:08.896426rpiserver sshd[19448]: Failed password for root from 212.164.53.17 port 36734 ssh2\n2018-01-09T03:50:12.068387rpiserver sshd[19448]: Failed password for root from 212.164.53.17 port 36734 ssh2\n2018-01-09T03:50:14.351841rpiserver sshd[19448]: Failed password for root from 212.164.53.17 port 36734 ssh2\n2018-01-09T03:50:16.822032rpiserver sshd[19448]: Failed password for root from 212.164.53.17 port 36734 ssh2\n2018-01-09T03:50:23.866125rpiserver sshd[19448]: Failed password for root from 212.164.53.17 port 36734 ssh2\n2018-01-09T03:50:26.246369rpiserver sshd[19448]: Failed password for root from 212.164.53.17 port 36734 ssh2\n2018-01-09T03:50:26.247413rpiserver sshd[19448]: error: maximum authentication attempts exceeded for root from 212.164.53.17 port 36734 ssh2 [preauth]', 'restored': 0, 'F-*': {'matches': ['2018-01-09T03:50:08.896426rpiserver sshd[19448]: Failed password for root from 212.164.53.17 port 36734 ssh2', '2018-01-09T03:50:12.068387rpiserver sshd[19448]: Failed password for root from 212.164.53.17 port 36734 ssh2', '2018-01-09T03:50:14.351841rpiserver sshd[19448]: Failed password for root from 212.164.53.17 port 36734 ssh2', '2018-01-09T03:50:16.822032rpiserver sshd[19448]: Failed password for root from 212.164.53.17 port 36734 ssh2', '2018-01-09T03:50:23.866125rpiserver sshd[19448]: Failed password for root from 212.164.53.17 port 36734 ssh2', '2018-01-09T03:50:26.246369rpiserver sshd[19448]: Failed password for root from 212.164.53.17 port 36734 ssh2', '2018-01-09T03:50:26.247413rpiserver sshd[19448]: error: maximum authentication attempts exceeded for root from 212.164.53.17 port 36734 ssh2 [preauth]'], 'failures': 8, 'mlfid': 'rpiserver sshd[19448]: ', 'user': '', 'ip4': '212.164.53.17'}, 'ipmatches': '2018-01-09T03:50:08.896426rpiserver sshd[19448]: Failed password for root from 212.164.53.17 port 36734 ssh2\n2018-01-09T03:50:12.068387rpiserver sshd[19448]: Failed password for root from 212.164.53.17 port 36734 ssh2\n2018-01-09T03:50:14.351841rpiserver sshd[19448]: Failed password for root from 212.164.53.17 port 36734 ssh2\n2018-01-09T03:50:16.822032rpiserver sshd[19448]: Failed password for root from 212.164.53.17 port 36734 ssh2\n2018-01-09T03:50:23.866125rpiserver sshd[19448]: Failed password for root from 212.164.53.17 port 36734 ssh2\n2018-01-09T03:50:26.246369rpiserver sshd[19448]: Failed password for root from 212.164.53.17 port 36734 ssh2\n2018-01-09T03:50:26.247413rpiserver sshd[19448]: error: maximum authentication attempts exceeded for root from 212.164.53.17 port 36734 ssh2 [preauth]', 'ipjailmatches': '2018-01-09T03:50:08.896426rpiserver sshd[19448]: Failed password for root from 212.164.53.17 port 36734 ssh2\n2018-01-09T03:50:12.068387rpiserver sshd[19448]: Failed password for root from 212.164.53.17 port 36734 ssh2\n2018-01-09T03:50:14.351841rpiserver sshd[19448]: Failed password for root from 212.164.53.17 port 36734 ssh2\n2018-01-09T03:50:16.822032rpiserver sshd[19448]: Failed password for root from 212.164.53.17 port 36734 ssh2\n2018-01-09T03:50:23.866125rpiserver sshd[19448]: Failed password for root from 212.164.53.17 port 36734 ssh2\n2018-01-09T03:50:26.246369rpiserver sshd[19448]: Failed password for root from 212.164.53.17 port 36734 ssh2\n2018-01-09T03:50:26.247413rpiserver sshd[19448]: error: maximum authentication attempts exceeded for root from 212.164.53.17 port 36734 ssh2 [preauth]', 'ipfailures': 8, 'ipjailfailures': 8, 'fq-hostname': 'rpiserver', 'sh-hostname': 'rpiserver'})': Error starting action Jail('sshd')/firewallcmd-ipset
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/fail2ban/server/actions.py", line 404, in __checkBan
    action.ban(aInfo)
  File "/usr/lib/python3.6/site-packages/fail2ban/server/action.py", line 423, in ban
    self.start(family)
  File "/usr/lib/python3.6/site-packages/fail2ban/server/action.py", line 405, in start
    return self._executeOperation('<actionstart>', 'starting', family=family)
  File "/usr/lib/python3.6/site-packages/fail2ban/server/action.py", line 374, in _executeOperation
    raise RuntimeError("Error %s action %s/%s" % (operation, self._jail, self._name,))
RuntimeError: Error starting action Jail('sshd')/firewallcmd-ipset

firewalld log reports:

Jan 12 01:40:17 rpiserver firewalld[586]: WARNING: '/usr/sbin/iptables-restore --wait=2 -n' failed:
Jan 12 01:40:17 rpiserver firewalld[586]: ERROR: COMMAND_FAILED
Jan 12 01:40:22 rpiserver firewalld[586]: WARNING: '/usr/sbin/iptables-restore --wait=2 -n' failed:
Jan 12 01:40:22 rpiserver firewalld[586]: ERROR: COMMAND_FAILED

"ipset --list" shows:

Name: f2b-sshd
Type: hash:ip
Revision: 4
Header: family inet hashsize 1024 maxelem 65536 timeout 604800
Size in memory: 88
References: 0
Number of entries: 0
Members:


Expected results:

IPs should be banned and no such errors.

Additional info:

Comment 1 Richard Shaw 2018-01-28 14:06:23 UTC

I'm seeing this on my x86_64 box too...


I found:

https://github.com/fail2ban/fail2ban/issues/1994


Which may or may not be relevant but looking at Fedora SCM there only seems to be an older patch which doesn't include everything in:

https://github.com/fail2ban/fail2ban/commit/309a1cb337604e03f764bf50839bdd3cb8280757

Comment 2 Richard Shaw 2018-02-11 13:24:05 UTC

Reverting to iptables-multiport works around the problem in the short term.

Comment 3 Richard Shaw 2018-03-28 12:49:41 UTC

Any plan to address this? I tried looking upstream and found a patch that differed from the one you applied and gave up.

Comment 4 Fedora Update System 2018-03-28 20:25:50 UTC

fail2ban-0.10.2-1.fc28 has been submitted as an update to Fedora 28. https://bodhi.fedoraproject.org/updates/FEDORA-2018-5eaf74dad4

Comment 5 Fedora Update System 2018-03-28 20:26:00 UTC

fail2ban-0.10.2-1.fc27 has been submitted as an update to Fedora 27. https://bodhi.fedoraproject.org/updates/FEDORA-2018-37f01b2610

Comment 6 Fedora Update System 2018-03-29 00:44:51 UTC

fail2ban-0.10.2-1.fc27 has been pushed to the Fedora 27 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2018-37f01b2610

Comment 7 Fedora Update System 2018-03-29 13:56:56 UTC

fail2ban-0.10.2-1.fc28 has been pushed to the Fedora 28 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2018-5eaf74dad4

Comment 8 Fedora Update System 2018-04-06 15:01:50 UTC

fail2ban-0.10.2-1.fc27 has been pushed to the Fedora 27 stable repository. If problems still persist, please make note of it in this bug report.

Comment 9 Fedora Update System 2018-05-22 15:05:57 UTC

fail2ban-0.10.2-1.fc28 has been pushed to the Fedora 28 stable repository. If problems still persist, please make note of it in this bug report.

Comment 10 bence 2018-08-24 09:33:47 UTC

Just upgraded to fail2ban-0.10.3.1-2.fc27 the problem still persists:

On start with action set to firewalld-ipset, it complains about the ip set cannot be created, although if I givet the commant it is created. The INPUT_direct rules are missing too. 
On stop it is in vica-versa the ipset is not flushed and not deleted.

journal:
aug 24 11:00:47 XXXXXXXX firewalld[1115]: WARNING: '/usr/sbin/iptables-restore --wait=2 -n' failed:
aug 24 11:00:47 XXXXXXXX firewalld[1115]: ERROR: COMMAND_FAILED
aug 24 11:00:47 XXXXXXXX fail2ban-server[20102]: fail2ban.utils [20102]: Level 39 7f4fe4026030 -- exec: ipset create f2b-sshd hash:ip timeout 7200
                                                        firewall-cmd --direct --add-rule ipv4 filter INPUT_direct 0 -p tcp -m multiport --dports ssh -m set --match-set f2b-sshd src -j DROP
aug 24 11:00:47 XXXXXXXX fail2ban-server[20102]: fail2ban.utils [20102]: ERROR 7f4fe4026030 -- stderr: '\x1b[91mError: COMMAND_FAILED\x1b[00m'
aug 24 11:00:47 XXXXXXXX fail2ban-server[20102]: fail2ban.utils [20102]: ERROR 7f4fe4026030 -- returned 13
aug 24 11:00:47 XXXXXXXX sshd[20096]: Failed password for root from 198.244.101.169 port 55894 ssh2
aug 24 11:00:47 XXXXXXXX fail2ban-server[20102]: fail2ban.actions [20102]: ERROR Failed to execute ban jail 'sshd' action

Comment 11 Richard Shaw 2018-12-26 16:00:59 UTC

This does not appear to be "fixed"...

I'm getting this in my journal:

Dec 26 08:46:45 firewalld[1114]: ERROR: '/usr/sbin/iptables-restore -w -n' failed: iptables-restore v1.8.0 (legacy): Set fail2ban->
                                                    
                                                    Error occurred at line: 2
                                                    Try `iptables-restore -h' or 'iptables-restore --help' for more information.
Dec 26 08:46:45 firewalld[1114]: ERROR: COMMAND_FAILED: Direct: '/usr/sbin/iptables-restore -w -n' failed: iptables-restore v1.8.0>
                                                    
                                                    Error occurred at line: 2
                                                    Try `iptables-restore -h' or 'iptables-restore --help' for more information.


I have narrowed this down to a change in the fail2ban config with ipset. The set name was changed from "fail2ban-<jail>" to "f2b-<jail>" but even when I fix this in /etc/firewalld/direct.xml it still fails because the ipset "f2b-sshd" is not always created.

I've run "ipset list" multiple times while trying to troubleshoot this and I have only seen the set "f2b-sshd" once.

My /etc/fail2ban/jail.d/sshd.local:

# cat sshd.local 
[DEFAULT]
bantime = 3600

[sshd]
enabled = true

Comment 12 Orion Poplawski 2018-12-26 23:34:06 UTC

Is fail2ban being restarted after firewalld is?  Need more fail2ban log output as well showing the stderr output of the commands that fail.

Comment 13 Richard Shaw 2018-12-27 00:02:36 UTC

I have a thread of just me replying to myself as I figured things out in the devel list detailing some of it.

From what I can tell fail2ban doesn't always call "ipset create" when starting. My current assumption is that perhaps it doesn't create the set until it has an IP to add to it? But it does always call the firewalld --direct command which specifies the ipset set name which it chokes on because it doesn't exist.

Comment 14 Richard Shaw 2018-12-27 00:04:08 UTC

There is a problem that needs to be fixed in firewalld though.. It looks for the set to be named fail2ban-<jail> but at some point fail2ban changed the prefix f2b-<jail>.

Comment 15 Orion Poplawski 2018-12-27 17:09:26 UTC

Also, I highly recommend filing issues upstream as well.

Note You need to log in before you can comment on or make changes to this bug.