IP blocked in firewall won't unblock

[jndawson]

We have a customer that's been automatically FTPing a webcam file (that shows the local surf conditions) from IP 1.2.3.4 to 10.11.12.13 every 5 minutes for years.

Suddenly, they started getting this: Error message: "connection failed check username and password"

We use cPHulk, CSF/LFD, CXS, and mod_sec for security.

The affected IP is whitelisted in cPHulk and csf/lfd, and has been for quite some time.

/var/log/messages shows the IP is blocked:

Apr 21 12:01:09 cp2 kernel: Firewall: *TCP_IN Blocked* IN=ens192 OUT= MAC=00:50:56:95:a7:f8:54:1e:56:72:64:00:08:00 SRC=1.2.3.4 DST=10.11.12.13 LEN=48 TOS=0x00 PREC=0x00 TTL=115 ID=31457 DF PROTO=TCP SPT=52594 DPT=21 WINDOW=8192 RES=0x00 SYN URGP=0

/var/log/lfd.log shows no entries.
cPHulk history log shows nothing.
mod_sec logs show nothing.
iptables -L |grep 1.2.3.4 shows nothing
iptables -S |grep 1.2.3.4 shows nothing
iptables-save|grep 1.2.3.4 shows nothing
grepping all logs for customer account being blocked shows no login failures

Using CSF to clear all blocks:
ConfigServer Security & Firewall - csf v14.18
Unblock 1.2.3.4, trying permanent blocks...
csf: 1.2.3.4 not found in csf.deny
...Done.
Unblock 1.2.3.4, trying temporary blocks...
csf: 21.2.3.4 not found in temporary bans
...Done.

How do we unblock that IP so our customer can continue to provide their service?

 

[cPRex]

Hey there! Remember, cPanel doesn't provide or distribute the CSF software.

If you grep for their IP address in all of /etc/csf, do you get any results? If not, do you have cPHulk enabled on the machine and somehow the IP has ended up there?

 

[jndawson]

Hey there! Remember, cPanel doesn't provide or distribute the CSF software.

We're quite aware of that.

If you grep for their IP address in all of /etc/csf, do you get any results? If not, do you have cPHulk enabled on the machine and somehow the IP has ended up there?

What else besides the info I already posted in the original post that you need to answer your question?

 

[mtindor]

We have a customer that's been automatically FTPing a webcam file (that shows the local surf conditions) from IP 1.2.3.4 to 10.11.12.13 every 5 minutes for years.

Suddenly, they started getting this: Error message: "connection failed check username and password"

We use cPHulk, CSF/LFD, CXS, and mod_sec for security.

The affected IP is whitelisted in cPHulk and csf/lfd, and has been for quite some time.

/var/log/messages shows the IP is blocked:

Apr 21 12:01:09 cp2 kernel: Firewall: *TCP_IN Blocked* IN=ens192 OUT= MAC=00:50:56:95:a7:f8:54:1e:56:72:64:00:08:00 SRC=1.2.3.4 DST=10.11.12.13 LEN=48 TOS=0x00 PREC=0x00 TTL=115 ID=31457 DF PROTO=TCP SPT=52594 DPT=21 WINDOW=8192 RES=0x00 SYN URGP=0

/var/log/lfd.log shows no entries.
cPHulk history log shows nothing.
mod_sec logs show nothing.
iptables -L |grep 1.2.3.4 shows nothing
iptables -S |grep 1.2.3.4 shows nothing
iptables-save|grep 1.2.3.4 shows nothing
grepping all logs for customer account being blocked shows no login failures

Using CSF to clear all blocks:
ConfigServer Security & Firewall - csf v14.18
Unblock 1.2.3.4, trying permanent blocks...
csf: 1.2.3.4 not found in csf.deny
...Done.
Unblock 1.2.3.4, trying temporary blocks...
csf: 21.2.3.4 not found in temporary bans
...Done.

How do we unblock that IP so our customer can continue to provide their service?

Grep all of your lfd.logs, since they typically rotate.

zgrep 1.2.3.4 /var/log/lfd.log*

And it may be that you, the system, or some other admin may have blocked a /24 or greater that included 1.2.3.4 in it. And if that's the case, you won't see 1.2.3.4 in the log. Maybe grep the log for '1\.2\.3\.' such as:

zgrep '1\.2\.3\.' /var/log/lfd.log*

Of course, if the IP has been blocked for a month or more, you may not have any logs showing that it has been blocked. And you'll also need to check CPHulk for 1.2.3.* (maybe a 1.2.3.0/24 or something)

And do grep your IPtables for '1\.2\.3\.' as well. And if your system also uses IPSET, then don't forget:

ipset list|grep '1\.2\.3\.'

 

[cPRex]

That is why I also mentioned grepping the entire directory, just to see if the IP was anywhere odd.

 

[jndawson]

Grep all of your lfd.logs, since they typically rotate.

zgrep 1.2.3.4 /var/log/lfd.log*

And it may be that you, the system, or some other admin may have blocked a /24 or greater that included 1.2.3.4 in it. And if that's the case, you won't see 1.2.3.4 in the log. Maybe grep the log for '1\.2\.3\.' such as:

zgrep '1\.2\.3\.' /var/log/lfd.log*

Of course, if the IP has been blocked for a month or more, you may not have any logs showing that it has been blocked. And you'll also need to check CPHulk for 1.2.3.* (maybe a 1.2.3.0/24 or something)

And do grep your IPtables for '1\.2\.3\.' as well. And if your system also uses IPSET, then don't forget:

ipset list|grep '1\.2\.3\.'

Guess I wasn't clear - ALL logs and other files related to csf/lfd, cPHulk, cxs, mod_sec were examined for any reference to the affected IP, and we started investigating about 10 hours after the IP was blocked when alerted by the customer.

Also didn't mention that we don't use firewalld, which is what most of the Internet said was the problem because:

Apr 21 12:01:09 cp2 kernel: Firewall: *TCP_IN Blocked* IN=ens192 OUT= MAC=00:50:56:95:a7:f8:54:1e:56:72:64:00:08:00 SRC=1.2.3.4 DST=10.11.12.13 LEN=48 TOS=0x00 PREC=0x00 TTL=115 ID=31457 DF PROTO=TCP SPT=52594 DPT=21 WINDOW=8192 RES=0x00 SYN URGP=0

Weirdly, the block suddenly cleared itself with no further action from us.