The Community Forums

Interact with an entire community of cPanel & WHM users!
  1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Perl Assistance

Discussion in 'cPanel Developers' started by fkatzenb, Oct 30, 2013.

  1. fkatzenb

    fkatzenb Well-Known Member

    Joined:
    Sep 23, 2009
    Messages:
    216
    Likes Received:
    0
    Trophy Points:
    16
    Location:
    Lusby, Maryland, United States
    cPanel Access Level:
    Root Administrator
    cPanel Community,
    I am writing a munin plugin to look at CSF entries. One of things I am trying to do is to read the csf.allow and csf.deny entries and determine the number IP/CIDR entries are loaded. My regex is not working correctly. The regex into the CIDR variable is not pulling in the number. I also have another problem is that this feature is only supported in what I think is perl 5.12 and beyond, so while I can test it out on my local machine with 5.18, it will not work on my cPanel server which is 5.8.8.

    Suggestions?
    Thanks,
    Frank

    Code:
    
    #open ALLOW, "/etc/csf/csf.allow";
    open ALLOW, "csf.allow.txt";
    
    my $allow_value = 0;
    
    while (my $line=<ALLOW>)
    {
    	next if $line =~ m/^\s*(?:#.*)?$/;
      	$line =~ m#^\s*(?<ipaddress>(?:(([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])\.){3}(25[0-5]|2[0-4][0-9]|1[0-9]{2}|[1-9][0-9]|[0-9])(\/|\s)))(?<CIDR>(3[0-2]|[1-2][0-9]|[0-9]))?#;
    	my ($ipaddress,$cidr) = ($+{ipaddress},$+{cidr});
    	print $ipaddress,$cidr;;
    	if (defined $cidr)
    	{
    		$allow_value += 2**(32 - $cidr);
    	}
    	else
    	{
    		$allow_value += 1;
    	}
    }
    
    close(ALLOW);
    
    
    #
    # csf.allow example, but ignore the first # as that does not exist
    # 204.11.102.34 # Manually allowed - Fri Mar  4 05:41:19 2011
    # 99.63.50.19/24 # Manually allowed - Thu Mar 10 00:07:47 2011
    #
    
    - - - Updated - - -

    Just to provide more about my project, this is what I plan to have this plugin do:

    Code:
    # Configuration 1: CSF_status
    #  - Shows # of IP entries in the following log files
    #    - /etc/csf/csf.allow 
    #      - use CIDR format to turn all entires into actual number of IPs allowed.  IP address is first in the line
    #      - file has # for comments at the beginning of the line
    #    - /etc/csf/csf.deny
    #      - use CIDR format to turn all entires into actual number of IPs denied.  IP address is first in the line
    #      - file has # for comments at the beginning of the line
    #  - Output
    #    - Title "CSF Status: Loaded IP Addresses For IPTABLES"
    #    - CSF_Allow.label "# of IP Addresses Allowed"
    #    - CSF_Allow.value xx (where xx is equal to the number of IP addresses allowed)
    #    - CSF_Deny.label "# of IP Addresses Denied"
    #    - CSF_Deny.value  yy (where yy is equal to the number of IP addresses denied)
    # Configuration 2: CSF_denyCountry
    #  - Shows # of IP entries for each Country
    #    - /etc/csf/csf.deny
    #      - use CIDR format to turn all entires into actual number of IPs denied.  IP address is first in the line
    #      - file has # for comments at the beginning of the line
    #    - Country is encapsulated in "...(AA/Abcdefg/FQDN)..." where AA is the country code and Abcdefg is the Long Country Name
    #    - Output
    #      - Title "CSF Status: Denied IP Addresses by Country of Origin"
    #      - CSF_Deny_AA.label Abcdefg (where AA is the country code and Abcdefg is the long country name)
    #      - CSF_Deny_AA.value xx (where AA is the country code and xx is the number of IP CIDR entries for that country)
    # Configuration 3: CSF_denyType
    #  - Shows reason for denial
    #    - /etc/csf/csf.deny
    #      - use CIDR format to turn all entires into actual number of IPs denied.  IP address is first in the line
    #      - file has # for comments at the beginning of the line
    #    - Detected failure has four examples:
    #      - "...# lfd: (type)..." where type equals the service that was attacked and detected
    #      - "...# lfd: *type*..." where type equals a non service detection
    #      - "...# lfd: aa.bb.cc.dd (AA/Abcdefg/FQDN), xx distributed_type..." where distrubted type equals the service and type of attack
    #      - "...# type..." where type is actual a manual entry because lfd: is not listed.  
    #    - Output
    #      - Title "CSF Status: Denied IP Addresses by Detected Attack"
    #      - CSF_Deny_type.label type (where type is qual to the value of type where type is equal the above examples of type, distributed_type, or manual)
    #      - CSF_Deny_type.value xx (where type is qual to the value of type where type is equal the above examples of type, distributed_type, or manual.  xx is the number of CIDR entires)
    
     
  2. KostonConsulting

    KostonConsulting Well-Known Member

    Joined:
    Jun 17, 2010
    Messages:
    255
    Likes Received:
    1
    Trophy Points:
    18
    Location:
    San Francisco, CA
    cPanel Access Level:
    Root Administrator
    If you're on Perl before 5.10, there are not named capture buffers for regular expressions. Instead, you'll need to use $1, $2, $3, etc. Those special variables are defined by parentheses groupings within your regex. i.e. /(\s+)(abc)(d(e|f))/ would have $1,$2, and $3 available for the 3 sets of parenthesis (the sub-parentheses don't count).
     
  3. fkatzenb

    fkatzenb Well-Known Member

    Joined:
    Sep 23, 2009
    Messages:
    216
    Likes Received:
    0
    Trophy Points:
    16
    Location:
    Lusby, Maryland, United States
    cPanel Access Level:
    Root Administrator
    Thank you for the great help. You launched me well on my way. Currently my code now successfully counts all IPs in the allow and deny.

    Code:
    my $DisplayVer=0.1;
    
    use strict;
    use warnings;
    
    #"/etc/csf/csf.allow"
    #"/etc/csf/csf.deny"
    
    my $allow = "csf.allow.txt";
    my $deny = "csf.deny.txt";
    
    print getIPcount($allow),"\n";
    print getIPcount($deny),"\n";
    
    
    sub getIPcount
    {
    	my $value = 0;
    	my ($filename) = @_;
    	open FILE, $filename;
    	
    	while (my $line=<FILE>)
    	{
    		next if $line =~ m/^\s*(?:#.*)?$/;
    
    		my ($ipaddress) = ($line =~ m#^\s*(((25[0-5]|2[0-4][0-9]|1[0-9]{2}|[1-9][0-9]|[0-9])\.){3}(25[0-5]|2[0-4][0-9]|1[0-9]{2}|[1-9][0-9]|[0-9]))#);
    		my ($cidr)      = ($line =~ m#\/(3[0-2]|[1-2][0-9]|[0-9])\s#);
    		
    		if (defined $cidr)
    		{
    			$value += 2**(32 - $cidr);
    			print "cdr?\n";
    		}
    		else
    		{
    			$value += 1;
    		}
    	}
    
    	close(FILE);
    	return $value;
    }
    
    On to my next step.
     
    #3 fkatzenb, Oct 31, 2013
    Last edited: Oct 31, 2013
  4. fkatzenb

    fkatzenb Well-Known Member

    Joined:
    Sep 23, 2009
    Messages:
    216
    Likes Received:
    0
    Trophy Points:
    16
    Location:
    Lusby, Maryland, United States
    cPanel Access Level:
    Root Administrator
    So I finished my alpha version and it worked like a champ (screenshots attached).
    Munin -- xltweb.com -- one.xltweb.com -- CSF deny country.png Munin -- xltweb.com -- one.xltweb.com -- CSF allow reseller.png Munin -- xltweb.com -- one.xltweb.com -- CSF deny reseller.png Munin -- xltweb.com -- one.xltweb.com -- CSF deny service.png

    One of the things that was missing is Port Scan entries for CSF_deny_service.pl call. I changed my regex from
    Code:
    \((\b\w*)\)\s(\b\w*\s\w*\s\w*\slogin|\b\w*\s\w*\slogin|\b\w*)
    to
    Code:
    \s\W((\b\w*\s\w*)|(\b\w*))\W\s((\w*\s\w*\s\w*\slogin)|(\w*\s\w*\slogin)|(\w*))
    That is when I began to see issues where everything in the brackets immediately following \W\s was no longer detected as my $part2 at line 107 went undefined and my one if statement requires it to be defined to create the label at line 115.

    Any help would be great. Sorry if some of it is doing things the hard way, this is my first attempt at perl by myself.

    V/R,
    Frank
    Code:
    #!/usr/bin/perl -w
    #
    # Munin plugin for CSF: ConfigServer Firewall
    # This plugin graphs your csf.allow and csf.deny data
    #
    #---------------------
    # Examples
    # Create a symbolic link to CSF_<allow|deny>_<reseller|country|service>
    #       ln -s /usr/share/munin/plugins/CSF_ /etc/munin/plugins/CSF_allow_reseller
    #           graph 
    #       ln -s /usr/share/munin/plugins/CSF_ /etc/munin/plugins/CSF_deny_reseller
    #           graph 
    #       ln -s /usr/share/munin/plugins/CSF_ /etc/munin/plugins/CSF_deny_country
    #           graph 
    #       ln -s /usr/share/munin/plugins/CSF_ /etc/munin/plugins/CSF_deny_service
    #           graph 
    #
    #---------------------
    # Log
    # Revision 0.1  2013/10/28  F. Katzenberger
    # - First version tested reading csf.deny and countying IP addresses w/ CIDR support
    # Revision 0.2  2013/10/31  F. Katzenberger
    # - Added/developed regex for detecting resellers, countries, and services
    # Revision 0.3  2013/11/02 F. Katzenberger
    # - developed the param structure for the munin config call
    # Revision 0.4  2013/11/03  F. Katzenberger
    # - alpha testing release
    #
    my $DisplayVer=0.4;
    #---------------------
    # Known Issues
    # 1. when a country or service no longer has an entry in csf.deny, it will be removed by munin
    #    from the graph regardless of any datapoints still on the graph.
    # 2. does not yet support port scan and cluster member entries
    # 3. distributed attacks are not detected and included in the deny_service in most cases
    # 4. the SMTP service label is cut short, regex is not showing the forth word.  need to consider 
    #    a search and replace form SMTP AUTH to smtpauth which is the name for distributed attacks against SMTP
    # 5. add a graph that shows deny by distributed on accounts.
    # 6. no ipv6 support yet
    # 7. does not yet track other lfd features such as excessive resources and ignore or temporary
    
    use strict;
    use warnings;
    use Data::Dumper;
    
    # determine what is to be reported on and hwo to display it from the basefile name
    my ($report,$type) = ($0 =~ m#_(\b[A-Z]|[a-z]*)_(\b[A-Z]|[a-z]*)#);
    
    # load up the allow/deny file
    # my $filename = "/etc/csf/csf.".$report;
    my $filename = "csf.".$report.".txt";
    
    # display preferences based on the report and type being passed into this script
    my %param = (
    	allow => {
    		label => 'Reported As: ',
    		title => "IP Allowed",
    		vtitle => 'addresses',
    		graph_args => '--base 1000 -l 0',
    		warning => '',
    		critical => '',
    		info_tag => "",
    	},
    	deny => {
    		label => 'Reported As: ',
    		title => "IP Denied",
    		vtitle => 'addresses',
    		graph_args => '--base 1000 -l 0',
    		warning => '',
    		critical => '',
    		info_tag => "",
    	},
    	reseller => {
    		lookfor => '\bReseller\s(\b\w*)(\s)',
    		draw1 => 'AREA',
    		draw2 => 'STACK',
    	},
    	country => {
    		lookfor => '\s\((\w*)\/((\w*)|(\w*\s\w*))\/',
    		draw1 => 'AREA',
    		draw2 => 'STACK',
    	},
    	service => {
    		lookfor => '\s\W((\b\w*\s\w*)|(\b\w*))\W\s((\w*\s\w*\s\w*\slogin)|(\w*\s\w*\slogin)|(\w*))',
    		draw1 => 'AREA',
    		draw2 => 'STACK',
    	},
    );
    
    #Munin Config Options
    if ($ARGV[0] and $ARGV[0] eq "config"){
    	print "graph_title $param{$report}->{title} by $type\n";
    	print "graph_vtitle $param{$report}->{vtitle}\n";
    	print "graph_args $param{$report}->{graph_args}\n";
    	print "graph_scale yes\n";
    	print "graph_category configserver\n";
    	print "graph_info This graph shows how many IP addresses in $filename are $report and filtered by $type\n";
    
    	open FILE, $filename;
    	
    	my @list;
    	
    	# determine all possible line items of data to be displayed
    	my $index = 0;
    	while (my $line = <FILE>) {
    		next if $line =~ m/^\s*(?:#.*)?$/;
    		my ($part1,$part2) = ($line =~ m#$param{$type}->{lookfor}#);
    		if ($type eq "reseller") {
    			$part2 = $part1;
    		}
    		if ($type eq "service" and defined $part1 and $part1 eq "Port Scan") {
    			$part1 = "port_scan";
    			$part2 = "Port Scan Detected";
    		}
    		if (defined $part1 and defined $part2) {
    			$list[$index] = $part1.",".$part2;
    			$index += 1;
    		} 		
    	}
    		
    	close(FILE);
    	
    	# reduce those line items for data labels to unique values
    	$index = 0;
    	my %comb;
    	@comb{@list} = ();
    	my @uniq = sort keys %comb;
    	
    	# generate the print statements for each unique label names and how to draw the data
    	for my $item (@uniq){
    		my ($again1,$again2) = ($item =~ m#(\b\w*)\,((\w*\s\w*\s\w*)|(\w*\s\w*)|(\w*))#);
    		print $again1.".label ".$again2."\n";
    		if ($index == 0) {
    			print $again1.".draw $param{$type}->{draw1}\n";	
    		} else {
    			print $again1.".draw $param{$type}->{draw2}\n";	
    		}
    		$index += 1;
    	}
    	exit 0;
    }
    
    
    open FILE, $filename;
    
    my @list;
    
    # determine all possible line items of data to find values for	
    my $index = 0;
    while (my $line = <FILE>) {
    	next if $line =~ m/^\s*(?:#.*)?$/;
    	my ($part1,$part2) = ($line =~ m#$param{$type}->{lookfor}#);
    	if (defined $part1) {
    		$list[$index] = $part1;
    		$index += 1;
    	} 		
    }
    close(FILE);
    
    # reduce those line items for data labels to unique values
    my %comb;
    @comb{@list} = ();
    my @uniq = sort keys %comb;
    
    # for each unique value, 
    for my $item (@uniq){
    	open FILE, $filename;
    	my $value = 0;
    	my $index = 0;
    	while (my $line = <FILE>) {
    		next if $line =~ m/^\s*(?:#.*)?$/;
    		
    		# find the instances in the file and parse out the IP address
    		my ($ipaddress) = ($line =~ m#^\s*(((25[0-5]|2[0-4][0-9]|1[0-9]{2}|[1-9][0-9]|[0-9])\.){3}(25[0-5]|2[0-4][0-9]|1[0-9]{2}|[1-9][0-9]|[0-9]))#);
    		my ($cidr)      = ($line =~ m#\/(3[0-2]|[1-2][0-9]|[0-9])\s#);
    		my ($again1,$again2)    = ($line =~ m#$param{$type}->{lookfor}#);
    
    		if (defined $again1 and $again1 eq $item) {
    			if (defined $cidr) {
    				$value += 2**(32 - $cidr);
    			} else {
    				$value += 1;
    			}
    		}
    	}
    	# create the print statement for the actual values
    	if ($item eq "Port Scan") {
    			$item = "port_scan";
    		}
    	print $item.".value ".$value."\n";
    	close(FILE);
    }
    
    Here is some data:
    Code:
    ###############################################################################
    # Copyright 2006-2011, Way to the Web Limited
    # URL: http://www.configserver.com
    # Email: sales@waytotheweb.com
    ###############################################################################
    # The following IP addresses will be blocked in iptables
    # One IP address per line
    # CIDR addressing allowed with a quaded IP (e.g. 192.168.254.0/24)
    # Only list IP addresses, not domain names (they will be ignored)
    #
    # Note: If you add the text "do not delete" to the comments of an entry then
    # DENY_IP_LIMIT will ignore those entries and not remove them
    #
    # Advanced port+ip filtering allowed with the following format
    # tcp/udp|in/out|s/d=port|s/d=ip
    #
    # See readme.txt for more information regarding advanced port filtering
    #
    175.42.82.185 # lfd: (ftpd) Failed FTP login from 175.42.82.185 (CN/China/-): 10 in the last 300 secs - Wed Sep 11 09:32:21 2013
    14.222.41.125 # lfd: (smtpauth) Failed SMTP AUTH login from 14.222.41.125 (CN/China/-): 10 in the last 300 secs - Wed Sep 11 13:39:37 2013
    187.23.95.249 # lfd: (ftpd) Failed FTP login from 187.23.95.249 (BR/Brazil/bb175ff9.virtua.com.br): 10 in the last 300 secs - Wed Sep 11 22:40:42 2013
    220.113.246.65 # lfd: (ftpd) Failed FTP login from 220.113.246.65 (CN/China/-): 10 in the last 300 secs - Wed Sep 11 23:08:00 2013
    217.39.146.163 # lfd: (smtpauth) Failed SMTP AUTH login from 217.39.146.163 (GB/United Kingdom/host217-39-146-163.in-addr.btopenworld.com): 10 in the last 300 secs - Thu Sep 12 01:58:32 2013
    163.125.230.244 # lfd: (ftpd) Failed FTP login from 163.125.230.244 (CN/China/-): 10 in the last 300 secs - Thu Sep 12 03:28:46 2013
    209.21.92.21 # lfd: *Port Scan* detected from 209.21.92.21 (US/United States/-). 16 hits in the last 105 seconds - Thu Sep 12 08:42:14 2013
    220.113.241.16 # lfd: (ftpd) Failed FTP login from 220.113.241.16 (CN/China/-): 10 in the last 300 secs - Thu Sep 12 11:18:07 2013
    184.22.252.114 # lfd: (pop3d) Failed POP3 login from 184.22.252.114 (US/United States/184-22-252-114.static.hostnoc.net): 10 in the last 300 secs - Thu Sep 12 11:35:38 2013
    174.21.233.139 # lfd: *Port Scan* detected from 174.21.233.139 (US/United States/174-21-233-139.tukw.qwest.net). 16 hits in the last 210 seconds - Thu Sep 12 14:14:17 2013
    85.112.1.153 # lfd: (pop3d) Failed POP3 login from 85.112.1.153 (ES/Spain/-): 10 in the last 300 secs - Thu Sep 12 18:54:00 2013
    50.80.232.125 # lfd: (smtpauth) Failed SMTP AUTH login from 50.80.232.125 (US/United States/50-80-232-125.client.mchsi.com): 10 in the last 300 secs - Thu Sep 12 23:00:27 2013
    54.225.82.140 # lfd: (ftpd) Failed FTP login from 54.225.82.140 (US/United States/ns1.safe-net.eu): 10 in the last 300 secs - Fri Sep 13 04:02:38 2013
    124.42.255.213 # lfd: (ftpd) Failed FTP login from 124.42.255.213 (CN/China/-): 10 in the last 300 secs - Fri Sep 13 11:00:10 2013
    216.245.217.198 # lfd: (ftpd) Failed FTP login from 216.245.217.198 (US/United States/shoppingservers.crabdance.com): 10 in the last 300 secs - Fri Sep 13 11:57:12 2013
    208.177.76.15 # lfd: *Port Scan* detected from 208.177.76.15 (US/United States/208.177.76.15.ptr.us.xo.net). 16 hits in the last 110 seconds - Fri Sep 13 13:57:34 2013
    58.22.10.93 # lfd: *Port Scan* detected from 58.22.10.93 (CN/China/-). 16 hits in the last 250 seconds - Fri Sep 13 14:24:56 2013
    213.229.68.54 # lfd: *Port Scan* detected from 213.229.68.54 (GB/United Kingdom/213-229-68-54.static.as29550.net). 16 hits in the last 115 seconds - Fri Sep 13 17:12:48 2013
    112.65.211.238 # lfd: (smtpauth) Failed SMTP AUTH login from 112.65.211.238 (CN/China/-): 10 in the last 300 secs - Fri Sep 13 17:47:00 2013
    108.222.151.41 # lfd: *Port Scan* detected from 108.222.151.41 (US/United States/108-222-151-41.lightspeed.irvnca.sbcglobal.net). 16 hits in the last 135 seconds - Fri Sep 13 20:53:19 2013
    112.95.176.206 # lfd: (ftpd) Failed FTP login from 112.95.176.206 (CN/China/-): 10 in the last 300 secs - Sat Sep 14 04:31:06 2013
    86.108.97.237 # lfd: *Port Scan* detected from 86.108.97.237 (JO/Jordan/86.108.x.237.go.com.jo). 16 hits in the last 165 seconds - Sat Sep 14 05:23:03 2013
    192.74.237.220 # lfd: *Port Scan* detected from 192.74.237.220 (US/United States/-). 16 hits in the last 135 seconds - Sat Sep 14 05:32:35 2013
    163.125.237.241 # lfd: (ftpd) Failed FTP login from 163.125.237.241 (CN/China/-): 10 in the last 300 secs - Sat Sep 14 05:57:30 2013
    199.15.233.134 # lfd: *Port Scan* detected from 199.15.233.134 (US/United States/-). 16 hits in the last 110 seconds - Sat Sep 14 07:32:16 2013
    5.133.27.66 # lfd: *Port Scan* detected from 5.133.27.66 (PS/Palestinian Territory/-). 16 hits in the last 255 seconds - Sat Sep 14 08:54:45 2013
    188.159.114.56 # lfd: *Port Scan* detected from 188.159.114.56 (IR/Iran, Islamic Republic of/-). 16 hits in the last 40 seconds - Sat Sep 14 11:56:20 2013
    120.37.237.40 # lfd: (ftpd) Failed FTP login from 120.37.237.40 (CN/China/-): 10 in the last 300 secs - Sat Sep 14 13:13:59 2013
    120.37.237.239 # lfd: (ftpd) Failed FTP login from 120.37.237.239 (CN/China/-): 10 in the last 300 secs - Sat Sep 14 13:14:26 2013
    120.37.237.183 # lfd: (ftpd) Failed FTP login from 120.37.237.183 (CN/China/-): 10 in the last 300 secs - Sat Sep 14 13:14:46 2013
    213.123.201.4 # lfd: (smtpauth) Failed SMTP AUTH login from 213.123.201.4 (GB/United Kingdom/host213-123-201-4.in-addr.btopenworld.com): 10 in the last 300 secs - Sun Sep 15 07:55:55 2013
    112.111.185.226 # lfd: *Port Scan* detected from 112.111.185.226 (CN/China/-). 16 hits in the last 190 seconds - Sun Sep 15 11:32:00 2013
    177.32.150.201 # lfd: (pop3d) Failed POP3 login from 177.32.150.201 (BR/Brazil/b12096c9.virtua.com.br): 10 in the last 300 secs - Sun Sep 15 13:36:36 2013
    
    Debuggex Visual Tool https://www.debuggex.com/
    Debuggex- Online visual regex tester. JavaScript, Python, and PCRE..png
     
    #4 fkatzenb, Nov 7, 2013
    Last edited: Nov 7, 2013
  5. fkatzenb

    fkatzenb Well-Known Member

    Joined:
    Sep 23, 2009
    Messages:
    216
    Likes Received:
    0
    Trophy Points:
    16
    Location:
    Lusby, Maryland, United States
    cPanel Access Level:
    Root Administrator
    Well I solved it. It was my judicial use of parenthesis. I was using them like I do math functions.
     
  6. KostonConsulting

    KostonConsulting Well-Known Member

    Joined:
    Jun 17, 2010
    Messages:
    255
    Likes Received:
    1
    Trophy Points:
    18
    Location:
    San Francisco, CA
    cPanel Access Level:
    Root Administrator
    Glad to hear you got this sorted and the plugin is looking good. Maybe I'm missing it but is there somewhere in Munin that you can see failed attempts per IP?
     
Loading...

Share This Page