Persistent bot/brute-force attack, iptables not working

David_spm

Well-Known Member
May 28, 2017
57
0
6
Thailand
cPanel Access Level
Root Administrator
Over the weekend nearly on sites on one server went very slow, I eventually found that Apache was bogged down with too many wait requests, finally found a Russian ip (using netstat) that had hundreds of requests coming in so I added it to iptables, saved iptables, restarted along with apache and all was fine.

Having the same problems again now in the last few hrs but its coming from the same ip, why isnt iptables blocking it? I tried adding it again, doesnt work (the culprit ip is definitely saved there).

After looking at Apache status in WHM dashboard I could pinpoint the exact attack location, it was to an xmlrpc.php file on one site and hundreds of POST requests were being sent there in a brute-force attempt. So next I denied access to that file in the sites .htaccess, that didnt work, attack continued. Finally I just renamed the file, the attack stopped and everything was fine...for about 30 mins and now its come back, the exact same ip and I can see the exact same file being hit.

Any advice on how to handle this would be appreciated.

p.s forgot to mention we already have CpanelHulk running
 

Jcats

Well-Known Member
PartnerNOC
May 25, 2011
806
156
168
New Jersey
cPanel Access Level
DataCenter Provider
Please post your iptables rules or at least the one you added to block the IP.

Is the site being hit using Cloudflare? Its possible you are blocking the IP properly but the actual IP iptables see's is a Cloudflare IP and not what Apache sees assuming you are using mod_cloudflare.

Simply adding:

Code:
deny from 1.1.1.1
in the .htaccess file of the site getting hit should block the IP EVEN IF its using Cloudflare since Apache see's the correct IP.

You should block access to xmlrpc.php (except from wordpress.com(jetpack mainly)) globally to negate these attacks completely as they are very common, if needed you can allow access to them per site via .htaccess.

In WHM > Apache Configuration > Include Editor > Pre VirtualHost Include

Code:
<FilesMatch "^(xmlrpc\.php)">
Order Deny,Allow
# Whitelist Jetpack/ Automattic CIDR IP Address Blocks
Allow from 192.0.64.0/18
Allow from 209.15.0.0/16
Allow from 66.155.0.0/17
Allow from wordpress.com
Allow from 45.56.2.139
Deny from all
</FilesMatch>
So next I denied access to that file in the sites .htacces
What rules did you put in .htaccess? When looking at the access logs, does it show a 200 code being thrown or 404? If a 404 this means its still triggering WordPress's dynamic content which would be the 404 page why you are still seeing load from the attack in which case, you can add:

Code:
ErrorDocument 403 default
right after the .htaccess rules that blocks access to the xmlrpc.php and it should stop the usage all together and properly throw that 403 which consumes 0 resources.

You should also look into mod_evasive, this will help deter brute force attacks as well but it can also cause false positives with the default config that is used when you install it via EasyApache as its a little to aggressive.
 

David_spm

Well-Known Member
May 28, 2017
57
0
6
Thailand
cPanel Access Level
Root Administrator
thanks for the detailed answer, not using CloudFlare. I just did a basic deny all rule to that php file in .htaccess, I will try your rule now. However I forgot to mention I installed mod_security the other day but hadn't configured it further and just enabled rules v3 and httpd doesn't seem to be getting hit quite so much now BUT after checking with netstat I can still see that ip hitting the sever just not quite as bad now (about 60 requests or so)...

edit: seems to have dropped off entirely now and can't see any hits from that ip, Im guessing it was the mod_sec rule that fixed it in the end? Im still confused why the iptables rule didnt work though.
 
Last edited:

David_spm

Well-Known Member
May 28, 2017
57
0
6
Thailand
cPanel Access Level
Root Administrator
its back, and the same ip and targeting the same file on the same site, why cant it be blocked?

edit: nevermind I think I know why now, In my iptables-config file I didnt have it set to use the saved rules on a reboot and I rebooted the server myself and also it reboots automatically after the nightly backup is run so the blocked ip rule was gone. At least I think thats the issue anyway!
 
Last edited:

cPanelLauren

Product Owner
Staff member
Nov 14, 2017
13,297
1,259
313
Houston
Hi @David_spm


That does sound like it would cause the issue you're presenting. Please let us know if the issue persists once you rectify the iptables configuration.


Thanks!
 

David_spm

Well-Known Member
May 28, 2017
57
0
6
Thailand
cPanel Access Level
Root Administrator
Hi @David_spm


That does sound like it would cause the issue you're presenting. Please let us know if the issue persists once you rectify the iptables configuration.


Thanks!
yes and no, Im getting spammed at xmlrpc.php for another domain on the same site, different ip though, this isnt as bad as before and Im not seeing any overload in Apache.

Im going to ban this ip and also look at writing deny changes for xmlrpc to the .htaccess for all the domains on the server, if you have any other tips do let me know, thanks
 

Jcats

Well-Known Member
PartnerNOC
May 25, 2011
806
156
168
New Jersey
cPanel Access Level
DataCenter Provider
Its pretty rare blocking IP's does anything outside of maybe stopping the attack for a bit but to many infected hosts out there, you'll never block them all. You should check my previous post, hopefully I didn't hit you with too much info at once but it does contain a way of blocking access to xmlrpc.php server wide.
 

David_spm

Well-Known Member
May 28, 2017
57
0
6
Thailand
cPanel Access Level
Root Administrator
thanks
Its pretty rare blocking IP's does anything outside of maybe stopping the attack for a bit but to many infected hosts out there, you'll never block them all. You should check my previous post, hopefully I didn't hit you with too much info at once but it does contain a way of blocking access to xmlrpc.php server wide.
thanks yes I read all that and Ive implemented that .htaccess deny on a couple of sites now and will add it to the rest later. I will look at using CloudFlare on some sites too, I dont have the budget to use it on all my domains tbh (unless I can use the free tier for multiple sites which I dont think you can)
 

David_spm

Well-Known Member
May 28, 2017
57
0
6
Thailand
cPanel Access Level
Root Administrator
You may want to consider installing CSF Firewall (it’s FREE!) - it makes blocking IP’s fast and easy. You can also configure it to automatically block IP’s based on Mod_security rule-hits.
EDIT: sorry ignore the below, tried again and seems to be working now

Ive read about that actually and been meaning to look into, just tried installing it via this tutorial - Removed - though and run into these errors after installation has complete and running
Code:
perl csftest.pl
Code:
[email protected] [/usr/local/csf/bin]# perl csftest.pl
Testing ip_tables/iptable_filter...OK
Testing ipt_LOG...OK
Testing ipt_multiport/xt_multiport...OK
Testing ipt_REJECT...FAILED [FATAL Error: Another app is currently holding the xtables lock. Perhaps you want to use the -w option?] - Required for csf to function
Testing ipt_state/xt_state...FAILED [FATAL Error: Another app is currently holding the xtables lock. Perhaps you want to use the -w option?] - Required for csf to function
Testing ipt_limit/xt_limit...FAILED [FATAL Error: Another app is currently holding the xtables lock. Perhaps you want to use the -w option?] - Required for csf to function
Testing ipt_recent...OK
Testing xt_connlimit...OK
Testing ipt_owner/xt_owner...OK
Testing iptable_nat/ipt_REDIRECT...OK
Testing iptable_nat/ipt_DNAT...OK

RESULT: csf will not function on this server due to FATAL errors from missing modules [3]
 
Last edited by a moderator:

David_spm

Well-Known Member
May 28, 2017
57
0
6
Thailand
cPanel Access Level
Root Administrator
OK so unfortunately things got a lot worse later.

After setting up CSF I also setup CloudFlare on a couple of sites, everything was fine and I checked in on the sever and some sites several times later on.

Then about 12hrs ago the entire server went down and all sites offline and I couldnt even ssh into the site. Ive had to contact the company we rent the server off to see if they can reboot it but still not heard back.

Does anyone have any ideas what could have caused this? A more intensified DDOS attack or perhaps a mis-configured CSF?

We have automated backups to AWS S3 of all the site databases at midnight US time and I can see that those completed just fine in our AWS account. I think that a server reboot is initiated as part of that so perhaps some vital configs were reset then?

I have scanned the server ip with nmap also and I can see that ports 25 and 80 are open but all 998 others are filtered.

Does anyone have any suggestions while I wait for our hosting company to get back to me?
 

cPanelLauren

Product Owner
Staff member
Nov 14, 2017
13,297
1,259
313
Houston
Hello,

If all the ports are being filtered it sounds as if you have a misconfiguration of the firewall. Once you get the server back up I would suggest disabling CSF temporarily to determine if it's causing the issues. If so you'll need to relook at the configuration to ensure it's correct.


Thanks!
 

David_spm

Well-Known Member
May 28, 2017
57
0
6
Thailand
cPanel Access Level
Root Administrator
Thanks Lauren, I managed to get back into the server yesterday and this is where Im at now:

I could see in Munin logs and graphs that everything went down around 6am EST, I couldnt see any major spike in resource usage before then though (but the Firewall data in network does show a sharp spike then. )

In CSF logs it looks like right around that time there was an error and CSF shutdown, from what I can tell from the CSF logs there were a lot of brute force attempts that were getting killed/blocked and this was going on every minute for hrs:

Code:
Jun 14 06:20:05 server lfd[12677]: *User Processing* PID:10998 Kill:0 User:mysite_A VM:555(MB) EXE:/opt/cpanel/ea-php70/root/usr/sbin/php-fpm CMDhp-fpm: pool mysite_A_com
Jun 14 06:20:05 server lfd[12677]: *User Processing* PID:11297 Kill:0 User:mysite_B VM:556(MB) EXE:/opt/cpanel/ea-php70/root/usr/sbin/php-fpm CMDhp-fpm: pool mysite_B_net
Jun 14 06:20:24 server lfd[13375]: (sshd) Failed SSH login from some.ip.here (CA/Canada/ns510220.ip-.net): 5 in the last 3600 secs - *Blocked in csf* [LF_SSHD]
Jun 14 06:20:28 server lfd[1556]: *Error* You have an unresolved error when starting csf. You need to restart csf successfully before restarting lfd (see /etc/csf/csf.error). *lfd stopped*, at line 1117
Jun 14 06:20:28 server lfd[1556]: daemon stopped
what does this MB data mean? VM:555(MB)

Currently CSF is not enabled.

Ive also just noticed that again the same ip is spamming one site and again it seems that iptables is not saving my rule where I blocked that ip, I dont understand why this rule keeps getting removed and also why the deny in .htaccess to stop access to the xmlrpc.php file isnt working.

This is a copy of my iptables-config file, as you can see I have it setup so that saves rules should work in the event of a shutdown or reboot

pastebin.com/Lie0MYgD

I should also add that right after I managed to get back into the server I found Apache was down and threw this error when I tried to restart it:

Code:
No space left on device: AH00023: Couldn't create the mpm-accept mutex (28)No space left on device: could not create accept mutex
pastebin.com/Lie0MYgD

Ive never seen this error before and was confused by it as there was plenty of disk space but after some Googling found this article:

- Removed -

I followed that and managed to restart Apache and everything seems to be back to normal now but Ive never heard of semaphores before in relation to Apache or Linux, how can I go about finding what caused this issue?
 
Last edited by a moderator:

cPanelLauren

Product Owner
Staff member
Nov 14, 2017
13,297
1,259
313
Houston
In CSF logs it looks like right around that time there was an error and CSF shutdown, from what I can tell from the CSF logs there were a lot of brute force attempts that were getting killed/blocked and this was going on every minute for hrs:
Based on that output it seems like there was a couple of issues:

This shows LFD killing php-fpm processes from this user for high processing:
Code:
Jun 14 06:20:05 server lfd[12677]: *User Processing* PID:10998 Kill:0 User:mysite_A VM:555(MB) EXE:/opt/cpanel/ea-php70/root/usr/sbin/php-fpm CMDhp-fpm: pool mysite_A_com
Jun 14 06:20:05 server lfd[12677]: *User Processing* PID:11297 Kill:0 User:mysite_B VM:556(MB) EXE:/opt/cpanel/ea-php70/root/usr/sbin/php-fpm CMDhp-fpm: pool mysite_B_net
This shows a failed SSH login which subsequently was blocked:
Code:
Jun 14 06:20:24 server lfd[13375]: (sshd) Failed SSH login from some.ip.here (CA/Canada/ns510220.ip-.net): 5 in the last 3600 secs - *Blocked in csf* [LF_SSHD]
This indicates that you potentially restarted LFD which failed due to an error with CSF
Code:
Jun 14 06:20:28 server lfd[1556]: *Error* You have an unresolved error when starting csf. You need to restart csf successfully before restarting lfd (see /etc/csf/csf.error). *lfd stopped*, at line 1117
Jun 14 06:20:28 server lfd[1556]: daemon stopped
what does this MB data mean? VM:555(MB)
MB stands for MegaBytes

Currently CSF is not enabled.
Is this due to the error or did you stop it purposely?

Ive also just noticed that again the same ip is spamming one site and again it seems that iptables is not saving my rule where I blocked that ip, I dont understand why this rule keeps getting removed and also why the deny in .htaccess to stop access to the xmlrpc.php file isnt working.
When you stop CSF and disable the configuration the iptables rules are flushed which may be why the rule isn't working.


I followed that and managed to restart Apache and everything seems to be back to normal now but Ive never heard of semaphores before in relation to Apache or Linux, how can I go about finding what caused this issue?
I'm glad you were able to resolve the semaphore issue. I found a very nice explanation regarding the purpose of semaphores on serverfault:
semaphores are IPC (interprocess communications structures). Semaphores like all IPC are used to allow different processes to communicate with each other.

They are basically counters that are created, accessed and destroyed using special system calls, such as sempost(3), semwait(3), semget(2) and semop(2). See sem_overview(7) on a linux system for a brief description.

The definition of communicate here is pretty primitive. "Communicate" for semaphores means reading, incrementing or decrementing a counter via the system/library calls mentioned above.

The special thing about semaphores apart from the fact that they are is that only one process at a time can perform an operation on them, and the semaphore operations are guaranteed atomic, that is to say you can't get into a race condition over a semaphore as the kernel will not swap out a process that is performing a semaphore operation.

The other special thing is that they are created in shared memory which allows multiple processes to access them.

How they manifest/created is that programs create them using semget(2). E.g. apache creates sempahores when it runs.

ipcs -l will tell you about the system's ipc resources.

You can manipulate some system semaphore and ipc related limits with sysctls. Try sysctl kernel.sem to view the sempahore related settings via sysctl. If you want to persist any sysctl changes you try put them into /etc/sysctl.conf.
 

David_spm

Well-Known Member
May 28, 2017
57
0
6
Thailand
cPanel Access Level
Root Administrator
Currently CSF is not enabled.

Ive left it like this since it crashed as it caused the entire server to go down which I dont want to happen again.

This shows LFD killing php-fpm processes from this user for high processing:

what I showed you was a very small sample from hundreds of attempts like that with many sites over many hours, is that normal or do I have a bigger problem?

I think I may have the issue under control for now after putting CloudFlare in front of a few of the sites that were getting targeted.

However there is still one ip which I have added to DROP to my iptable rules many times now but for some reason these rules keep vanishing later on and the ip comes back and its trying to spma /xmlrpc.php again on one site only this time it seems to be in a very small amount due to CF doing its job I guess?

So I guess something else is flushing my iptables?
 

cPanelLauren

Product Owner
Staff member
Nov 14, 2017
13,297
1,259
313
Houston
HI @David_spm

what I showed you was a very small sample from hundreds of attempts like that with many sites over many hours, is that normal or do I have a bigger problem?
I think you have a bigger problem, attacks like this will cause high processing for the user and they don't stop.

I think I may have the issue under control for now after putting CloudFlare in front of a few of the sites that were getting targeted.
CloudFlare should help but you may want to look at further ways to prevent the issue from occurring.

However there is still one ip which I have added to DROP to my iptable rules many times now but for some reason these rules keep vanishing later on and the ip comes back and its trying to spma /xmlrpc.php again on one site only this time it seems to be in a very small amount due to CF doing its job I guess?
I can't say why the rule isn't being kept, it could be an issue with the rule itself, when you add it and then run
Code:
iptables --list
Do you see it present?

Another issue with blocking per IP is that xmlrpc.php attacks typically originate from multiple IP addresses, blocking one just disallows one of hundred/thousands of IP's that are being used. There are a ton of articles online for suggestions on how to stop these attacks similar to what @keat63 provided.