CSF Firewall: *TCP_OUT Blocked

[chirpy]

Most likely you're seeing the ACK FIN because the connection state has dropped out of the table and so doesn't have an established connection anymore. Since it is an ACK FIN it doesn't matter as it's the end of the TCP connection anyway.

You will definitely see more blocks with that option enabled - that's the whole point of it. Just because you're seeing log lines, doesn't mean that there's any problem, it's only if you know that people cannot connect in, which appears to be what jols is seeing.

 

[kernow]

At this stage, I would asume buggy iptables .

On three different Linus OSs ?? i think thats unlikely .......... though anything is possible.
We only get an hourly email report form one server, so at first we believed the problem just happened there, but checking /var/log/messages shows the problem occurs on all three servers that have csf inctalled.
Chirpy, i think this problem is more wide spread than you think, and if only a few have reported the errors its probably because most haven't checked their logs lately ???

 

[chirpy]

I'm confused - did you actually read my reply? Those types of block are perfectly normal for the reason I explained.

When I was talking about buggy iptables, I was referring to jols issue which isn't the same as the one you're posting about.

 

[kernow]

How about the ones with ACK RST and ACK PSH ????

 

[kernow]

I'm confused - did you actually read my reply?

I was making the other post while you were posting, so i didn't see it till after i posted.
Thanks for your reply.

 

[chirpy]

It's the same explaination, i.e. the connection state table entries have expired for the given TCP connection. You see them with csf because it configures iptables to log them, the likes of APF don't log them but still block them, so you won't even know they're happening all the time. It's the whole point of SPI, i.e. to check the state of each packet and to discard ones that are not in sequence when a connection isn't in an ESTABLISHED state.

 

[chirpy]

I was making the other post while you were posting, so i didn't see it till after i posted.
Thanks for your reply.

Ah iptables is a very interesting kernel module and a brief read of the netfilters mailing list can get you spinning in circles.

But the information on how stateful packet inspection is out there. One thing it shows is that it isn't and cannot ever be "perfect" since you're relying on what is essentially a kernel hack to inspect each and every packet and maintain a database of those connections which by its nature has to be kept small and therefore connections, at some point, have to be considered stale and they're dropped from the connection database. When this happens it's possible for stray packets to still be working their way through on the server and on the network. That's why you see packets being dropped from what appear to be valid connections, but what in fact iptables tidying up after the TCP connection has long gone.

You can actually see this type of underlying issue by looking at the output of:

netstat -autpn

If you look in the State column you'll see many connections in TIME_WAIT or FIN_WAIT2. These are stale connections that have either been closed by both parties, or only by one. In the case of the latter, the OS keeps them alive just incase stray packets are found relating to the connection eventhough it's considered gone. This is so the OS will accept the packets and then discard them. What you're often seeing is iptables picking them up and blocking them because they don't relate to any connection that it knows about.

That's a long winded way to describe what I've found and how I understand it, but might give you an idea of what is happening. With DROP_PF_LOGGING enabled you're seeing even more of these packets. These are packets that are considered INVALID by iptables (i.e. that have no valid connection) or in the case of the more complex ones, are packets that have invalid data in the TCP header. These are packets the firewall is blocking for good reason, they're often used by port scanners and DOS attackers to bring your server down to its knees.

Having said all that, it could also be seen as something to investigate further if they're excessive. I don't know what excessive would be, but there may be some underlying issue that's affecting the ability of your server to close down or keep track of the connection state of TCP connections. That's where the role of the kernel, the NIC, then NIC drivers, your datacenters network, etc, come into focus.

There's also the issue of kernel hacks that many poke into /etc/sysctl.conf that can play havoc with the network protcols and I've seen more often than not cause serious network degradation on servers when implemented.

 

[kernow]

Chirpy,
Thanks for taking the time to explain it in a way i can mostly understand.
I feel we can relax now ............ well a bit anyway !!

 

[jols]

At this stage, I would asume buggy iptables as csf is now running on thousands of servers and I've only had these two reports of any problems with the rules in this manner. It's never going to suite everyone, and if it doesn't work for you, then using an alternative is always going to be a good idea, it would be easy to dial back the rules and make them less stringent, but that would make it less secure for everyone and defeat the main purpose of an SPI firewall.

Chirpy, sorry but I think you've missed the point. And why for gosh sake would you be left with just "assuming" about this!?!?! Read on:

TO REITERATE ---> Our customer's blocked IP's, nor the subnets of their IP's were ending up in iptables, and they were not being put in the csf.deny file, and there was no record of the blocking event being recorded in any logs. BUT CSF was for sure behind the issue!!!! This was never a question of making CSF less stringent. We had a sure-fire, REPEATABLE, TESTABLE anomaly going on here with CSF. We gave you root access, gave you a 1-2-3 step-by-step method of testing this on one of our servers, made multiple requests that you check it out, we offered to pay you a fee, all of this to absolutely no avail.

I understand that we all get busy, but our theory here was that this issue with CSF may have been even more important for you than for us as we seemed to have uncover a real crippling bug in CSF that, by the way, others are now reporting (see further up this thread).

NOTE - If it was the result of "buggy iptables" why then would there be no problem with APF/BFD? Same iptables, different firewall, now the legitimate IPs that were formerly (any quite mysteriously) blocked by CSF have instantly become unblocked. Also by the way, flushing iptables had no effect, only turning off, or completely uninstalling CSF worked to resolve the issue!

CSF was great, it suited us very well. But when CSF was WITHOUT-A-DOUBT randomly (and permanently) blocking our customer's IPs, WITHOUT putting them in the csf.deny file, with NO record of the blocking event in the logs, etc. then what good is it?

By the way, this same thing was occurring across multiple servers where we had CSF installed. And as you may recall, you recently upgraded the kernels for us on every one of these.

 

[chirpy]

Chirpy, sorry but I think you've missed the point. And why for gosh sake would you be left with just "assuming" about this!?!?!

I think you're missing the point. csf is just a script that adds rules to iptables. csf doesn't block anything whatsoever, iptables does. If iptables is blocking something then you need to look at the rules that might be blocking things for you. You are the only person that has reported any such problem as you describe, no-one else has.

The other people that have mentioned issues are those that have misunderstood the reasons why iptables may be blocking connections.

The reason why it may work for you with APF is the way in which it configures iptables, since no two firewall configuration scripts are the same. If that's suitable for your needs, then that's great.

We have csf working without any issues at all on thousands of servers and that is great for them, it shows it works in by far the vast majority of cases. I can't and don't support iptables itself and if some configuration that csf makes with iptables doesn't work for you, then by all means use APF. That's the benefit of having the choice of free iptables configuration scripts.

 

[chirpy]

Chirpy, sorry but I think you've missed the point. And why for gosh sake would you be left with just "assuming" about this!?!?!

I think you're missing the point. csf is just a script that adds rules to iptables. csf doesn't block anything whatsoever, iptables does. If iptables is blocking something then you need to look at the rules that might be blocking things for you. You are the only person that has reported any such problem as you describe, no-one else has.

The other people that have mentioned issues are those that have misunderstood the reasons why iptables may be blocking connections.

The reason why it may work for you with APF is the way in which it configures iptables, since no two firewall configuration scripts are the same. If that's suitable for your needs, then that's great.

We have csf working without any issues at all on thousands of servers and that is great for them, it shows it works in by far the vast majority of cases. I can't and don't support iptables itself and if some configuration that csf makes with iptables doesn't work for you, then by all means use APF.

That's the benefit of having the choice of free iptables configuration scripts, if one works for you and another doesn't then go ahead and use that instead.

In fact, it's the very reason that I found that the way APF configured iptables that didn't work on such a variety of servers that I installed it on, that I wrote csf for those circumstances. So far, the experience with csf has shown it to be as suitable, if not more so, on cPanel servers as APF.

 

[mctDarren]

I think you're missing the point. csf is just a script that adds rules to iptables. csf doesn't block anything whatsoever, iptables does.

This is what I was waiting for someone to say (since this really isn't my conversation other than the fact that I use CSF on our boxes). When Chirpy (and myself in my subscribe post) talk about "buggy iptables" we actually mean it's buggy the way that CSF and iptables interact. After all, CSF should be simply adding rules to iptables - not blocking traffic on it's own. The firewall in the title is a misnomer, really. So CSF is adding a rule based on the fact that the packet is destined for a lost connection. ??

The part I can't understand is when Jols says he has no rule in iptables showing the IP block. If it truly is CSF adding the block to iptables, where is that in iptables --list ? Weird.

 

[jols]

The part I can't understand is when Jols says he has no rule in iptables showing the IP block. If it truly is CSF adding the block to iptables, where is that in iptables --list ? Weird.

Yes, bingo, this is the part that really threw us for a loop as well.

Here's the sequence of events that I agree is very strange and prompted us to open the ticket at configservers about this:

1 -- Customer's IP is blocked.

2 -- We check csf.deny, it's not there.

3 -- We do iptables --list, it does not show up (nor a subnet of that IP).

4 -- We put the ip in the csf.allow file, then restart csf.

5 -- The customer's IP is now UN-blocked.

AND, we found this to be occuring with 3 to 4 servers that were similarly configured with CSF, etc.


We are in the process of replacing CSF with APF everywhere, but I am still VERY open for some kind of an explanation about all of this.

Also adding that I think CSF is an outstanding security package. We just need to get to the bottom of the above descirbed issue and why this was occurring.
And, sorry for the frustration I may have posted earlier. Chirpy's customer service is typically outstanding as well.

 

[chirpy]

Another thing to try that has come to mind is emptying out DROP_NOLOG and restarting csf. If connections are coming in to those ports (especially on ephemeral ports) that are blocked, then won't get logged by the nature and purpose of that setting.

Another abvious thing is to make sure you haven't listed any IP addresses in /etc/csf/csf.sips and that if you have mutiple ethernet devices that they're listed appropriately.

 

[mctDarren]

3 -- We do iptables --list, it does not show up (nor a subnet of that IP).
4 -- We put the ip in the csf.allow file, then restart csf.
5 -- The customer's IP is now UN-blocked.

I would be curious to know if there is an ALLOW rule added after you put the IP in csf.allow and then restart. That would explain why traffic is coming through again and would only indicate to me that CSF is adding an allow rule, not blocking the traffic in the first place.

In one of your previous posts, you said that stopping CSF allowed traffic through again, which is curious to me - as CSF in not a firewall in the true sense that it stops/allows traffic. It assigns rules to iptables and iptables handles blocking. That's the part that doesn't make sense to me.

 

[jols]

I would be curious to know if there is an ALLOW rule added after you put the IP in csf.allow and then restart. That would explain why traffic is coming through again and would only indicate to me that CSF is adding an allow rule, not blocking the traffic in the first place.

We did not check iptables --list after putting the IP in the csf.allow. I only noted that access was again available after doing this.

In one of your previous posts, you said that stopping CSF allowed traffic through again,

Yes, true.

which is curious to me - as CSF in not a firewall in the true sense that it stops/allows traffic. It assigns rules to iptables and iptables handles blocking. That's the part that doesn't make sense to me.

Stopping CSF also flushes the rules from iptables. Which by the way is neither here nor there because the DENY rule never made it to iptables to begin with regarding the handfull of examples that we had of mysteriously blocked IPs.

I think this whole thing boils down to some alternative blocking method that CSF uses to determine which IPs should be blocked in some kind of pre-processing fashion, i.e. outside of the usual method involving the csf.deny list. In a recent post Chirpy mentioned disabling another facility, which is probably such a method. I only wish we knew about this before.

Chirpy, is there anyway to get a list of all the methods that CSF uses to block where there is no corresponding rules inserted into iptables?

 

[chirpy]

Chirpy, is there anyway to get a list of all the methods that CSF uses to block where there is no corresponding rules inserted into iptables?

As I mentioned, csf doesn't block anything at all, it configures iptables to do that. All the blocks are iptables rules, so you can see them all with:

iptables -L -n

There's no more or less than that.

 

[jsnape]

Same thing here

I noticed the same thing. Certain IP's are unreachable unless specifically allowed. The also don't show up in iptables using the list command.

It was happening on a new server but I took that IP and tried to trace out to it from other servers in other data centers, and whole other bandwidth providers including a server running cpanel on a sprint dsl line and its the same thing.

From Atlanta Server (installed long time)

BEFORE adding to allow list:

[email protected] [/usr/local/apache/conf]# traceroute 66.244.235.1
traceroute to 66.244.235.1 (66.244.235.1), 30 hops max, 38 byte packets
traceroute: sendto: Operation not permitted
1 traceroute: wrote 66.244.235.1 38 chars, ret=-1

AFTER manualy adding the IP

[email protected] [/usr/local/apache/conf]# traceroute 66.244.235.1
traceroute to 66.244.235.1 (66.244.235.1), 30 hops max, 38 byte packets
1 209.51.153.185 (209.51.153.185) 0.551 ms 0.527 ms 0.461 ms
2 209.51.137.97 (209.51.137.97) 0.301 ms 0.346 ms 0.381 ms
3 209.51.149.110 (209.51.149.110) 0.418 ms 0.726 ms 0.376 ms
4 bpr2-ge-2-3-0.atlantapaix.savvis.net (208.173.60.217) 0.492 ms 0.555 ms 0.505 ms
5 dcr2-so-4-3-0.Atlanta.savvis.net (208.172.75.237) 0.578 ms 0.627 ms 0.581 ms
6 POS2-0.BR2.ATL5.ALTER.NET (204.255.168.105) 1.102 ms 1.209 ms 1.062 ms
7 0.so-2-3-0.XL1.ATL5.ALTER.NET (152.63.82.198) 1.192 ms 1.351 ms 1.341 ms
8 0.so-5-0-0.XL1.CHI13.ALTER.NET (152.63.64.18) 30.590 ms 30.562 ms 30.475 ms
9 POS4-0.GW2.CHI13.ALTER.NET (152.63.71.213) 30.492 ms 30.371 ms 30.354 ms
10 BigPipe.customer.alter.net (63.84.148.130) 30.037 ms 30.077 ms 29.996 ms
11 rd2ec-ge0-0-0.il.shawcable.net (66.163.65.18) 29.894 ms 29.950 ms 29.903 ms
12 rc2ch-pos4-0.il.shawcable.net (66.163.65.13) 29.978 ms 30.066 ms 30.075 ms
13 rc2nr-pos10-0-0.wp.shawcable.net (66.163.77.93) 48.135 ms
<snipped>

********
********

Sacramento Server CentOS 2.6.9-42.ELsmp (has csf installed a long time)
Before:
[email protected] [/]# traceroute 66.244.235.1
traceroute to 66.244.235.1 (66.244.235.1), 30 hops max, 38 byte packets
traceroute: sendto: Operation not permitted
1 traceroute: wrote 66.244.235.1 38 chars, ret=-1

After manualy allowing:

[email protected] [/]# traceroute 66.244.235.1
traceroute to 66.244.235.1 (66.244.235.1), 30 hops max, 38 byte packets
1 38.102.33.1 (38.102.33.1) 0.555 ms 0.521 ms 0.483 ms
2 g3-0-185-core01.smf01.atlas.cogentco.com (38.112.242.165) 0.612 ms 0.683 ms 0.487 ms
3 g3-0-3504.core01.smf01.atlas.cogentco.com (38.112.38.225) 0.610 ms 0.725 ms 0.362 ms
4 p4-0.core02.sfo01.atlas.cogentco.com (154.54.1.253) 3.487 ms 3.085 ms 3.371 ms
5 t3-3.mpd01.sfo01.atlas.cogentco.com (154.54.3.118) 3.237 ms 3.557 ms 3.242 ms
6 t2-2.mpd01.sjc01.atlas.cogentco.com (154.54.1.26) 4.336 ms 4.564 ms 4.478 ms
MPLS Label=164 CoS=5 TTL=1 S=0
7 v3499.mpd01.sjc03.atlas.cogentco.com (154.54.6.238) 5.219 ms 4.951 ms 4.859 ms
8 shaw.sjc03.atlas.cogentco.com (154.54.13.86) 67.871 ms 67.701 ms 67.676 ms
9 rc1sj-ge3-0-0.cl.shawcable.net (66.163.67.9) 68.007 ms 67.674 ms 67.724 ms
10 rc2wt-pos0-0-0.wa.shawcable.net (66.163.76.65) 71.521 ms 71.426 ms


Same thing on other servers in South Bend and Ohio locations.

All running CentOS 4.4 except Atlanta (CentOS 3.8, 2.4.21-47.0.1.ELsmp) and tried it on another box in Atlanta running Fedora with 2.4.x kernel with the same result


Once a day or every other day we get a ticket about it from someone, look into it and the IP isn't blocked, add the IP and the problem is solved. I had always assumed their IP was previously blocked and was somehow getting cleared from the blocked list in csf, but not transferring that "clearing" to iptables.

Never checked the same IP from several servers spread around the U.S. But have checked the list of IPtables rules in the past and couldn't see the IP and assumed it was some caching issue. I doubt that the IP in the example has connected to (and been blacklisted) from many different Linux servers of mine, in many locations, including the one on the dsl connection that basicly serves on site.

 

[jols]

NOTE: This may not have anything to do with your particular issue, but ---> We discovered just recently that using the traceroute to determine a blocked IP is an invalid method for testing, that is, if you have not configured CSF for allowing traceroutes to begin with. See the csf config file for more about this.

 

[jsnape]

Thanks for the info. It seems like I used to ping the IP, but that stopped working even if the IP wasn't blocked, so I switched to using traceroute. Now its the other way around. I CAN ping out the IP's from all the servers, but can only traceroute to the IP if I add it to the allow list - and still leave traceroute disabled. That sounds odd too - maybe I'll go through and enable traceroute on all of them and see if the once a day, or every other day tickets go away.

The person with that IP is definitely blocked at the server, but I can ping out to his IP from the server.