fkatzenb

Well-Known Member
Sep 23, 2009
216
0
66
Lusby, Maryland, United States
cPanel Access Level
Root Administrator
cPanel Community,
I am writing a munin plugin to look at CSF entries. One of things I am trying to do is to read the csf.allow and csf.deny entries and determine the number IP/CIDR entries are loaded. My regex is not working correctly. The regex into the CIDR variable is not pulling in the number. I also have another problem is that this feature is only supported in what I think is perl 5.12 and beyond, so while I can test it out on my local machine with 5.18, it will not work on my cPanel server which is 5.8.8.

Suggestions?
Thanks,
Frank

Code:
#open ALLOW, "/etc/csf/csf.allow";
open ALLOW, "csf.allow.txt";

my $allow_value = 0;

while (my $line=<ALLOW>)
{
	next if $line =~ m/^\s*(?:#.*)?$/;
  	$line =~ m#^\s*(?<ipaddress>(?:(([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])\.){3}(25[0-5]|2[0-4][0-9]|1[0-9]{2}|[1-9][0-9]|[0-9])(\/|\s)))(?<CIDR>(3[0-2]|[1-2][0-9]|[0-9]))?#;
	my ($ipaddress,$cidr) = ($+{ipaddress},$+{cidr});
	print $ipaddress,$cidr;;
	if (defined $cidr)
	{
		$allow_value += 2**(32 - $cidr);
	}
	else
	{
		$allow_value += 1;
	}
}

close(ALLOW);


#
# csf.allow example, but ignore the first # as that does not exist
# 204.11.102.34 # Manually allowed - Fri Mar  4 05:41:19 2011
# 99.63.50.19/24 # Manually allowed - Thu Mar 10 00:07:47 2011
#
- - - Updated - - -

Just to provide more about my project, this is what I plan to have this plugin do:

Code:
# Configuration 1: CSF_status
#  - Shows # of IP entries in the following log files
#    - /etc/csf/csf.allow 
#      - use CIDR format to turn all entires into actual number of IPs allowed.  IP address is first in the line
#      - file has # for comments at the beginning of the line
#    - /etc/csf/csf.deny
#      - use CIDR format to turn all entires into actual number of IPs denied.  IP address is first in the line
#      - file has # for comments at the beginning of the line
#  - Output
#    - Title "CSF Status: Loaded IP Addresses For IPTABLES"
#    - CSF_Allow.label "# of IP Addresses Allowed"
#    - CSF_Allow.value xx (where xx is equal to the number of IP addresses allowed)
#    - CSF_Deny.label "# of IP Addresses Denied"
#    - CSF_Deny.value  yy (where yy is equal to the number of IP addresses denied)
# Configuration 2: CSF_denyCountry
#  - Shows # of IP entries for each Country
#    - /etc/csf/csf.deny
#      - use CIDR format to turn all entires into actual number of IPs denied.  IP address is first in the line
#      - file has # for comments at the beginning of the line
#    - Country is encapsulated in "...(AA/Abcdefg/FQDN)..." where AA is the country code and Abcdefg is the Long Country Name
#    - Output
#      - Title "CSF Status: Denied IP Addresses by Country of Origin"
#      - CSF_Deny_AA.label Abcdefg (where AA is the country code and Abcdefg is the long country name)
#      - CSF_Deny_AA.value xx (where AA is the country code and xx is the number of IP CIDR entries for that country)
# Configuration 3: CSF_denyType
#  - Shows reason for denial
#    - /etc/csf/csf.deny
#      - use CIDR format to turn all entires into actual number of IPs denied.  IP address is first in the line
#      - file has # for comments at the beginning of the line
#    - Detected failure has four examples:
#      - "...# lfd: (type)..." where type equals the service that was attacked and detected
#      - "...# lfd: *type*..." where type equals a non service detection
#      - "...# lfd: aa.bb.cc.dd (AA/Abcdefg/FQDN), xx distributed_type..." where distrubted type equals the service and type of attack
#      - "...# type..." where type is actual a manual entry because lfd: is not listed.  
#    - Output
#      - Title "CSF Status: Denied IP Addresses by Detected Attack"
#      - CSF_Deny_type.label type (where type is qual to the value of type where type is equal the above examples of type, distributed_type, or manual)
#      - CSF_Deny_type.value xx (where type is qual to the value of type where type is equal the above examples of type, distributed_type, or manual.  xx is the number of CIDR entires)
 

KostonConsulting

Well-Known Member
Verifed Vendor
Jun 17, 2010
255
1
68
San Francisco, CA
cPanel Access Level
Root Administrator
If you're on Perl before 5.10, there are not named capture buffers for regular expressions. Instead, you'll need to use $1, $2, $3, etc. Those special variables are defined by parentheses groupings within your regex. i.e. /(\s+)(abc)(d(e|f))/ would have $1,$2, and $3 available for the 3 sets of parenthesis (the sub-parentheses don't count).
 

fkatzenb

Well-Known Member
Sep 23, 2009
216
0
66
Lusby, Maryland, United States
cPanel Access Level
Root Administrator
If you're on Perl before 5.10, there are not named capture buffers for regular expressions. Instead, you'll need to use $1, $2, $3, etc. Those special variables are defined by parentheses groupings within your regex. i.e. /(\s+)(abc)(d(e|f))/ would have $1,$2, and $3 available for the 3 sets of parenthesis (the sub-parentheses don't count).
Thank you for the great help. You launched me well on my way. Currently my code now successfully counts all IPs in the allow and deny.

Code:
my $DisplayVer=0.1;

use strict;
use warnings;

#"/etc/csf/csf.allow"
#"/etc/csf/csf.deny"

my $allow = "csf.allow.txt";
my $deny = "csf.deny.txt";

print getIPcount($allow),"\n";
print getIPcount($deny),"\n";


sub getIPcount
{
	my $value = 0;
	my ($filename) = @_;
	open FILE, $filename;
	
	while (my $line=<FILE>)
	{
		next if $line =~ m/^\s*(?:#.*)?$/;

		my ($ipaddress) = ($line =~ m#^\s*(((25[0-5]|2[0-4][0-9]|1[0-9]{2}|[1-9][0-9]|[0-9])\.){3}(25[0-5]|2[0-4][0-9]|1[0-9]{2}|[1-9][0-9]|[0-9]))#);
		my ($cidr)      = ($line =~ m#\/(3[0-2]|[1-2][0-9]|[0-9])\s#);
		
		if (defined $cidr)
		{
			$value += 2**(32 - $cidr);
			print "cdr?\n";
		}
		else
		{
			$value += 1;
		}
	}

	close(FILE);
	return $value;
}
On to my next step.
 
Last edited:

fkatzenb

Well-Known Member
Sep 23, 2009
216
0
66
Lusby, Maryland, United States
cPanel Access Level
Root Administrator
So I finished my alpha version and it worked like a champ (screenshots attached).
Munin -- xltweb.com -- one.xltweb.com -- CSF deny country.pngMunin -- xltweb.com -- one.xltweb.com -- CSF allow reseller.pngMunin -- xltweb.com -- one.xltweb.com -- CSF deny reseller.pngMunin -- xltweb.com -- one.xltweb.com -- CSF deny service.png

One of the things that was missing is Port Scan entries for CSF_deny_service.pl call. I changed my regex from
Code:
\((\b\w*)\)\s(\b\w*\s\w*\s\w*\slogin|\b\w*\s\w*\slogin|\b\w*)
to
Code:
\s\W((\b\w*\s\w*)|(\b\w*))\W\s((\w*\s\w*\s\w*\slogin)|(\w*\s\w*\slogin)|(\w*))
That is when I began to see issues where everything in the brackets immediately following \W\s was no longer detected as my $part2 at line 107 went undefined and my one if statement requires it to be defined to create the label at line 115.

Any help would be great. Sorry if some of it is doing things the hard way, this is my first attempt at perl by myself.

V/R,
Frank
Code:
#!/usr/bin/perl -w
#
# Munin plugin for CSF: ConfigServer Firewall
# This plugin graphs your csf.allow and csf.deny data
#
#---------------------
# Examples
# Create a symbolic link to CSF_<allow|deny>_<reseller|country|service>
#       ln -s /usr/share/munin/plugins/CSF_ /etc/munin/plugins/CSF_allow_reseller
#           graph 
#       ln -s /usr/share/munin/plugins/CSF_ /etc/munin/plugins/CSF_deny_reseller
#           graph 
#       ln -s /usr/share/munin/plugins/CSF_ /etc/munin/plugins/CSF_deny_country
#           graph 
#       ln -s /usr/share/munin/plugins/CSF_ /etc/munin/plugins/CSF_deny_service
#           graph 
#
#---------------------
# Log
# Revision 0.1  2013/10/28  F. Katzenberger
# - First version tested reading csf.deny and countying IP addresses w/ CIDR support
# Revision 0.2  2013/10/31  F. Katzenberger
# - Added/developed regex for detecting resellers, countries, and services
# Revision 0.3  2013/11/02 F. Katzenberger
# - developed the param structure for the munin config call
# Revision 0.4  2013/11/03  F. Katzenberger
# - alpha testing release
#
my $DisplayVer=0.4;
#---------------------
# Known Issues
# 1. when a country or service no longer has an entry in csf.deny, it will be removed by munin
#    from the graph regardless of any datapoints still on the graph.
# 2. does not yet support port scan and cluster member entries
# 3. distributed attacks are not detected and included in the deny_service in most cases
# 4. the SMTP service label is cut short, regex is not showing the forth word.  need to consider 
#    a search and replace form SMTP AUTH to smtpauth which is the name for distributed attacks against SMTP
# 5. add a graph that shows deny by distributed on accounts.
# 6. no ipv6 support yet
# 7. does not yet track other lfd features such as excessive resources and ignore or temporary

use strict;
use warnings;
use Data::Dumper;

# determine what is to be reported on and hwo to display it from the basefile name
my ($report,$type) = ($0 =~ m#_(\b[A-Z]|[a-z]*)_(\b[A-Z]|[a-z]*)#);

# load up the allow/deny file
# my $filename = "/etc/csf/csf.".$report;
my $filename = "csf.".$report.".txt";

# display preferences based on the report and type being passed into this script
my %param = (
	allow => {
		label => 'Reported As: ',
		title => "IP Allowed",
		vtitle => 'addresses',
		graph_args => '--base 1000 -l 0',
		warning => '',
		critical => '',
		info_tag => "",
	},
	deny => {
		label => 'Reported As: ',
		title => "IP Denied",
		vtitle => 'addresses',
		graph_args => '--base 1000 -l 0',
		warning => '',
		critical => '',
		info_tag => "",
	},
	reseller => {
		lookfor => '\bReseller\s(\b\w*)(\s)',
		draw1 => 'AREA',
		draw2 => 'STACK',
	},
	country => {
		lookfor => '\s\((\w*)\/((\w*)|(\w*\s\w*))\/',
		draw1 => 'AREA',
		draw2 => 'STACK',
	},
	service => {
		lookfor => '\s\W((\b\w*\s\w*)|(\b\w*))\W\s((\w*\s\w*\s\w*\slogin)|(\w*\s\w*\slogin)|(\w*))',
		draw1 => 'AREA',
		draw2 => 'STACK',
	},
);

#Munin Config Options
if ($ARGV[0] and $ARGV[0] eq "config"){
	print "graph_title $param{$report}->{title} by $type\n";
	print "graph_vtitle $param{$report}->{vtitle}\n";
	print "graph_args $param{$report}->{graph_args}\n";
	print "graph_scale yes\n";
	print "graph_category configserver\n";
	print "graph_info This graph shows how many IP addresses in $filename are $report and filtered by $type\n";

	open FILE, $filename;
	
	my @list;
	
	# determine all possible line items of data to be displayed
	my $index = 0;
	while (my $line = <FILE>) {
		next if $line =~ m/^\s*(?:#.*)?$/;
		my ($part1,$part2) = ($line =~ m#$param{$type}->{lookfor}#);
		if ($type eq "reseller") {
			$part2 = $part1;
		}
		if ($type eq "service" and defined $part1 and $part1 eq "Port Scan") {
			$part1 = "port_scan";
			$part2 = "Port Scan Detected";
		}
		if (defined $part1 and defined $part2) {
			$list[$index] = $part1.",".$part2;
			$index += 1;
		} 		
	}
		
	close(FILE);
	
	# reduce those line items for data labels to unique values
	$index = 0;
	my %comb;
	@comb{@list} = ();
	my @uniq = sort keys %comb;
	
	# generate the print statements for each unique label names and how to draw the data
	for my $item (@uniq){
		my ($again1,$again2) = ($item =~ m#(\b\w*)\,((\w*\s\w*\s\w*)|(\w*\s\w*)|(\w*))#);
		print $again1.".label ".$again2."\n";
		if ($index == 0) {
			print $again1.".draw $param{$type}->{draw1}\n";	
		} else {
			print $again1.".draw $param{$type}->{draw2}\n";	
		}
		$index += 1;
	}
	exit 0;
}


open FILE, $filename;

my @list;

# determine all possible line items of data to find values for	
my $index = 0;
while (my $line = <FILE>) {
	next if $line =~ m/^\s*(?:#.*)?$/;
	my ($part1,$part2) = ($line =~ m#$param{$type}->{lookfor}#);
	if (defined $part1) {
		$list[$index] = $part1;
		$index += 1;
	} 		
}
close(FILE);

# reduce those line items for data labels to unique values
my %comb;
@comb{@list} = ();
my @uniq = sort keys %comb;

# for each unique value, 
for my $item (@uniq){
	open FILE, $filename;
	my $value = 0;
	my $index = 0;
	while (my $line = <FILE>) {
		next if $line =~ m/^\s*(?:#.*)?$/;
		
		# find the instances in the file and parse out the IP address
		my ($ipaddress) = ($line =~ m#^\s*(((25[0-5]|2[0-4][0-9]|1[0-9]{2}|[1-9][0-9]|[0-9])\.){3}(25[0-5]|2[0-4][0-9]|1[0-9]{2}|[1-9][0-9]|[0-9]))#);
		my ($cidr)      = ($line =~ m#\/(3[0-2]|[1-2][0-9]|[0-9])\s#);
		my ($again1,$again2)    = ($line =~ m#$param{$type}->{lookfor}#);

		if (defined $again1 and $again1 eq $item) {
			if (defined $cidr) {
				$value += 2**(32 - $cidr);
			} else {
				$value += 1;
			}
		}
	}
	# create the print statement for the actual values
	if ($item eq "Port Scan") {
			$item = "port_scan";
		}
	print $item.".value ".$value."\n";
	close(FILE);
}
Here is some data:
Code:
###############################################################################
# Copyright 2006-2011, Way to the Web Limited
# URL: http://www.configserver.com
# Email: [email protected]
###############################################################################
# The following IP addresses will be blocked in iptables
# One IP address per line
# CIDR addressing allowed with a quaded IP (e.g. 192.168.254.0/24)
# Only list IP addresses, not domain names (they will be ignored)
#
# Note: If you add the text "do not delete" to the comments of an entry then
# DENY_IP_LIMIT will ignore those entries and not remove them
#
# Advanced port+ip filtering allowed with the following format
# tcp/udp|in/out|s/d=port|s/d=ip
#
# See readme.txt for more information regarding advanced port filtering
#
175.42.82.185 # lfd: (ftpd) Failed FTP login from 175.42.82.185 (CN/China/-): 10 in the last 300 secs - Wed Sep 11 09:32:21 2013
14.222.41.125 # lfd: (smtpauth) Failed SMTP AUTH login from 14.222.41.125 (CN/China/-): 10 in the last 300 secs - Wed Sep 11 13:39:37 2013
187.23.95.249 # lfd: (ftpd) Failed FTP login from 187.23.95.249 (BR/Brazil/bb175ff9.virtua.com.br): 10 in the last 300 secs - Wed Sep 11 22:40:42 2013
220.113.246.65 # lfd: (ftpd) Failed FTP login from 220.113.246.65 (CN/China/-): 10 in the last 300 secs - Wed Sep 11 23:08:00 2013
217.39.146.163 # lfd: (smtpauth) Failed SMTP AUTH login from 217.39.146.163 (GB/United Kingdom/host217-39-146-163.in-addr.btopenworld.com): 10 in the last 300 secs - Thu Sep 12 01:58:32 2013
163.125.230.244 # lfd: (ftpd) Failed FTP login from 163.125.230.244 (CN/China/-): 10 in the last 300 secs - Thu Sep 12 03:28:46 2013
209.21.92.21 # lfd: *Port Scan* detected from 209.21.92.21 (US/United States/-). 16 hits in the last 105 seconds - Thu Sep 12 08:42:14 2013
220.113.241.16 # lfd: (ftpd) Failed FTP login from 220.113.241.16 (CN/China/-): 10 in the last 300 secs - Thu Sep 12 11:18:07 2013
184.22.252.114 # lfd: (pop3d) Failed POP3 login from 184.22.252.114 (US/United States/184-22-252-114.static.hostnoc.net): 10 in the last 300 secs - Thu Sep 12 11:35:38 2013
174.21.233.139 # lfd: *Port Scan* detected from 174.21.233.139 (US/United States/174-21-233-139.tukw.qwest.net). 16 hits in the last 210 seconds - Thu Sep 12 14:14:17 2013
85.112.1.153 # lfd: (pop3d) Failed POP3 login from 85.112.1.153 (ES/Spain/-): 10 in the last 300 secs - Thu Sep 12 18:54:00 2013
50.80.232.125 # lfd: (smtpauth) Failed SMTP AUTH login from 50.80.232.125 (US/United States/50-80-232-125.client.mchsi.com): 10 in the last 300 secs - Thu Sep 12 23:00:27 2013
54.225.82.140 # lfd: (ftpd) Failed FTP login from 54.225.82.140 (US/United States/ns1.safe-net.eu): 10 in the last 300 secs - Fri Sep 13 04:02:38 2013
124.42.255.213 # lfd: (ftpd) Failed FTP login from 124.42.255.213 (CN/China/-): 10 in the last 300 secs - Fri Sep 13 11:00:10 2013
216.245.217.198 # lfd: (ftpd) Failed FTP login from 216.245.217.198 (US/United States/shoppingservers.crabdance.com): 10 in the last 300 secs - Fri Sep 13 11:57:12 2013
208.177.76.15 # lfd: *Port Scan* detected from 208.177.76.15 (US/United States/208.177.76.15.ptr.us.xo.net). 16 hits in the last 110 seconds - Fri Sep 13 13:57:34 2013
58.22.10.93 # lfd: *Port Scan* detected from 58.22.10.93 (CN/China/-). 16 hits in the last 250 seconds - Fri Sep 13 14:24:56 2013
213.229.68.54 # lfd: *Port Scan* detected from 213.229.68.54 (GB/United Kingdom/213-229-68-54.static.as29550.net). 16 hits in the last 115 seconds - Fri Sep 13 17:12:48 2013
112.65.211.238 # lfd: (smtpauth) Failed SMTP AUTH login from 112.65.211.238 (CN/China/-): 10 in the last 300 secs - Fri Sep 13 17:47:00 2013
108.222.151.41 # lfd: *Port Scan* detected from 108.222.151.41 (US/United States/108-222-151-41.lightspeed.irvnca.sbcglobal.net). 16 hits in the last 135 seconds - Fri Sep 13 20:53:19 2013
112.95.176.206 # lfd: (ftpd) Failed FTP login from 112.95.176.206 (CN/China/-): 10 in the last 300 secs - Sat Sep 14 04:31:06 2013
86.108.97.237 # lfd: *Port Scan* detected from 86.108.97.237 (JO/Jordan/86.108.x.237.go.com.jo). 16 hits in the last 165 seconds - Sat Sep 14 05:23:03 2013
192.74.237.220 # lfd: *Port Scan* detected from 192.74.237.220 (US/United States/-). 16 hits in the last 135 seconds - Sat Sep 14 05:32:35 2013
163.125.237.241 # lfd: (ftpd) Failed FTP login from 163.125.237.241 (CN/China/-): 10 in the last 300 secs - Sat Sep 14 05:57:30 2013
199.15.233.134 # lfd: *Port Scan* detected from 199.15.233.134 (US/United States/-). 16 hits in the last 110 seconds - Sat Sep 14 07:32:16 2013
5.133.27.66 # lfd: *Port Scan* detected from 5.133.27.66 (PS/Palestinian Territory/-). 16 hits in the last 255 seconds - Sat Sep 14 08:54:45 2013
188.159.114.56 # lfd: *Port Scan* detected from 188.159.114.56 (IR/Iran, Islamic Republic of/-). 16 hits in the last 40 seconds - Sat Sep 14 11:56:20 2013
120.37.237.40 # lfd: (ftpd) Failed FTP login from 120.37.237.40 (CN/China/-): 10 in the last 300 secs - Sat Sep 14 13:13:59 2013
120.37.237.239 # lfd: (ftpd) Failed FTP login from 120.37.237.239 (CN/China/-): 10 in the last 300 secs - Sat Sep 14 13:14:26 2013
120.37.237.183 # lfd: (ftpd) Failed FTP login from 120.37.237.183 (CN/China/-): 10 in the last 300 secs - Sat Sep 14 13:14:46 2013
213.123.201.4 # lfd: (smtpauth) Failed SMTP AUTH login from 213.123.201.4 (GB/United Kingdom/host213-123-201-4.in-addr.btopenworld.com): 10 in the last 300 secs - Sun Sep 15 07:55:55 2013
112.111.185.226 # lfd: *Port Scan* detected from 112.111.185.226 (CN/China/-). 16 hits in the last 190 seconds - Sun Sep 15 11:32:00 2013
177.32.150.201 # lfd: (pop3d) Failed POP3 login from 177.32.150.201 (BR/Brazil/b12096c9.virtua.com.br): 10 in the last 300 secs - Sun Sep 15 13:36:36 2013
Debuggex Visual Tool https://www.debuggex.com/
Debuggex- Online visual regex tester. JavaScript, Python, and PCRE..png
 
Last edited:

KostonConsulting

Well-Known Member
Verifed Vendor
Jun 17, 2010
255
1
68
San Francisco, CA
cPanel Access Level
Root Administrator
Well I solved it. It was my judicial use of parenthesis. I was using them like I do math functions.
Glad to hear you got this sorted and the plugin is looking good. Maybe I'm missing it but is there somewhere in Munin that you can see failed attempts per IP?