Please whitelist cPanel in your adblocker so that you’re able to see our version release promotions, thanks!

The Community Forums

Interact with an entire community of cPanel & WHM users!

Persistent bot/brute-force attack, iptables not working

Discussion in 'Security' started by David_spm, Jun 12, 2018.

  1. David_spm

    David_spm Active Member

    Joined:
    May 28, 2017
    Messages:
    42
    Likes Received:
    0
    Trophy Points:
    6
    Location:
    Thailand
    cPanel Access Level:
    Root Administrator
    Over the weekend nearly on sites on one server went very slow, I eventually found that Apache was bogged down with too many wait requests, finally found a Russian ip (using netstat) that had hundreds of requests coming in so I added it to iptables, saved iptables, restarted along with apache and all was fine.

    Having the same problems again now in the last few hrs but its coming from the same ip, why isnt iptables blocking it? I tried adding it again, doesnt work (the culprit ip is definitely saved there).

    After looking at Apache status in WHM dashboard I could pinpoint the exact attack location, it was to an xmlrpc.php file on one site and hundreds of POST requests were being sent there in a brute-force attempt. So next I denied access to that file in the sites .htaccess, that didnt work, attack continued. Finally I just renamed the file, the attack stopped and everything was fine...for about 30 mins and now its come back, the exact same ip and I can see the exact same file being hit.

    Any advice on how to handle this would be appreciated.

    p.s forgot to mention we already have CpanelHulk running
     
  2. Jcats

    Jcats Well-Known Member

    Joined:
    May 25, 2011
    Messages:
    720
    Likes Received:
    123
    Trophy Points:
    168
    Location:
    New Jersey
    cPanel Access Level:
    DataCenter Provider
    Please post your iptables rules or at least the one you added to block the IP.

    Is the site being hit using Cloudflare? Its possible you are blocking the IP properly but the actual IP iptables see's is a Cloudflare IP and not what Apache sees assuming you are using mod_cloudflare.

    Simply adding:

    Code:
    deny from 1.1.1.1
    in the .htaccess file of the site getting hit should block the IP EVEN IF its using Cloudflare since Apache see's the correct IP.

    You should block access to xmlrpc.php (except from wordpress.com(jetpack mainly)) globally to negate these attacks completely as they are very common, if needed you can allow access to them per site via .htaccess.

    In WHM > Apache Configuration > Include Editor > Pre VirtualHost Include

    Code:
    <FilesMatch "^(xmlrpc\.php)">
    Order Deny,Allow
    # Whitelist Jetpack/ Automattic CIDR IP Address Blocks
    Allow from 192.0.64.0/18
    Allow from 209.15.0.0/16
    Allow from 66.155.0.0/17
    Allow from wordpress.com
    Allow from 45.56.2.139
    Deny from all
    </FilesMatch>
    What rules did you put in .htaccess? When looking at the access logs, does it show a 200 code being thrown or 404? If a 404 this means its still triggering WordPress's dynamic content which would be the 404 page why you are still seeing load from the attack in which case, you can add:

    Code:
    ErrorDocument 403 default
    right after the .htaccess rules that blocks access to the xmlrpc.php and it should stop the usage all together and properly throw that 403 which consumes 0 resources.

    You should also look into mod_evasive, this will help deter brute force attacks as well but it can also cause false positives with the default config that is used when you install it via EasyApache as its a little to aggressive.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  3. David_spm

    David_spm Active Member

    Joined:
    May 28, 2017
    Messages:
    42
    Likes Received:
    0
    Trophy Points:
    6
    Location:
    Thailand
    cPanel Access Level:
    Root Administrator
    thanks for the detailed answer, not using CloudFlare. I just did a basic deny all rule to that php file in .htaccess, I will try your rule now. However I forgot to mention I installed mod_security the other day but hadn't configured it further and just enabled rules v3 and httpd doesn't seem to be getting hit quite so much now BUT after checking with netstat I can still see that ip hitting the sever just not quite as bad now (about 60 requests or so)...

    edit: seems to have dropped off entirely now and can't see any hits from that ip, Im guessing it was the mod_sec rule that fixed it in the end? Im still confused why the iptables rule didnt work though.
     
    #3 David_spm, Jun 13, 2018
    Last edited: Jun 13, 2018
  4. David_spm

    David_spm Active Member

    Joined:
    May 28, 2017
    Messages:
    42
    Likes Received:
    0
    Trophy Points:
    6
    Location:
    Thailand
    cPanel Access Level:
    Root Administrator
    its back, and the same ip and targeting the same file on the same site, why cant it be blocked?

    edit: nevermind I think I know why now, In my iptables-config file I didnt have it set to use the saved rules on a reboot and I rebooted the server myself and also it reboots automatically after the nightly backup is run so the blocked ip rule was gone. At least I think thats the issue anyway!
     
    #4 David_spm, Jun 13, 2018
    Last edited: Jun 13, 2018
  5. cPanelLauren

    cPanelLauren Forums Analyst II
    Staff Member

    Joined:
    Nov 14, 2017
    Messages:
    3,052
    Likes Received:
    213
    Trophy Points:
    173
    Location:
    Houston
    cPanel Access Level:
    DataCenter Provider
    Hi @David_spm


    That does sound like it would cause the issue you're presenting. Please let us know if the issue persists once you rectify the iptables configuration.


    Thanks!
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  6. David_spm

    David_spm Active Member

    Joined:
    May 28, 2017
    Messages:
    42
    Likes Received:
    0
    Trophy Points:
    6
    Location:
    Thailand
    cPanel Access Level:
    Root Administrator
    yes and no, Im getting spammed at xmlrpc.php for another domain on the same site, different ip though, this isnt as bad as before and Im not seeing any overload in Apache.

    Im going to ban this ip and also look at writing deny changes for xmlrpc to the .htaccess for all the domains on the server, if you have any other tips do let me know, thanks
     
  7. Jcats

    Jcats Well-Known Member

    Joined:
    May 25, 2011
    Messages:
    720
    Likes Received:
    123
    Trophy Points:
    168
    Location:
    New Jersey
    cPanel Access Level:
    DataCenter Provider
    Its pretty rare blocking IP's does anything outside of maybe stopping the attack for a bit but to many infected hosts out there, you'll never block them all. You should check my previous post, hopefully I didn't hit you with too much info at once but it does contain a way of blocking access to xmlrpc.php server wide.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  8. David_spm

    David_spm Active Member

    Joined:
    May 28, 2017
    Messages:
    42
    Likes Received:
    0
    Trophy Points:
    6
    Location:
    Thailand
    cPanel Access Level:
    Root Administrator
    thanks
    thanks yes I read all that and Ive implemented that .htaccess deny on a couple of sites now and will add it to the rest later. I will look at using CloudFlare on some sites too, I dont have the budget to use it on all my domains tbh (unless I can use the free tier for multiple sites which I dont think you can)
     
  9. Tearabite

    Tearabite Well-Known Member

    Joined:
    Nov 28, 2010
    Messages:
    76
    Likes Received:
    11
    Trophy Points:
    58
    Location:
    Southern California
    cPanel Access Level:
    Root Administrator
    You may want to consider installing CSF Firewall (it’s FREE!) - it makes blocking IP’s fast and easy. You can also configure it to automatically block IP’s based on Mod_security rule-hits.
     
  10. David_spm

    David_spm Active Member

    Joined:
    May 28, 2017
    Messages:
    42
    Likes Received:
    0
    Trophy Points:
    6
    Location:
    Thailand
    cPanel Access Level:
    Root Administrator
    EDIT: sorry ignore the below, tried again and seems to be working now

    Ive read about that actually and been meaning to look into, just tried installing it via this tutorial - Removed - though and run into these errors after installation has complete and running
    Code:
    perl csftest.pl 
    Code:
    root@server [/usr/local/csf/bin]# perl csftest.pl
    Testing ip_tables/iptable_filter...OK
    Testing ipt_LOG...OK
    Testing ipt_multiport/xt_multiport...OK
    Testing ipt_REJECT...FAILED [FATAL Error: Another app is currently holding the xtables lock. Perhaps you want to use the -w option?] - Required for csf to function
    Testing ipt_state/xt_state...FAILED [FATAL Error: Another app is currently holding the xtables lock. Perhaps you want to use the -w option?] - Required for csf to function
    Testing ipt_limit/xt_limit...FAILED [FATAL Error: Another app is currently holding the xtables lock. Perhaps you want to use the -w option?] - Required for csf to function
    Testing ipt_recent...OK
    Testing xt_connlimit...OK
    Testing ipt_owner/xt_owner...OK
    Testing iptable_nat/ipt_REDIRECT...OK
    Testing iptable_nat/ipt_DNAT...OK
    
    RESULT: csf will not function on this server due to FATAL errors from missing modules [3]
    
     
    #10 David_spm, Jun 13, 2018
    Last edited by a moderator: Jun 16, 2018
  11. David_spm

    David_spm Active Member

    Joined:
    May 28, 2017
    Messages:
    42
    Likes Received:
    0
    Trophy Points:
    6
    Location:
    Thailand
    cPanel Access Level:
    Root Administrator
    OK so unfortunately things got a lot worse later.

    After setting up CSF I also setup CloudFlare on a couple of sites, everything was fine and I checked in on the sever and some sites several times later on.

    Then about 12hrs ago the entire server went down and all sites offline and I couldnt even ssh into the site. Ive had to contact the company we rent the server off to see if they can reboot it but still not heard back.

    Does anyone have any ideas what could have caused this? A more intensified DDOS attack or perhaps a mis-configured CSF?

    We have automated backups to AWS S3 of all the site databases at midnight US time and I can see that those completed just fine in our AWS account. I think that a server reboot is initiated as part of that so perhaps some vital configs were reset then?

    I have scanned the server ip with nmap also and I can see that ports 25 and 80 are open but all 998 others are filtered.

    Does anyone have any suggestions while I wait for our hosting company to get back to me?
     
  12. cPanelLauren

    cPanelLauren Forums Analyst II
    Staff Member

    Joined:
    Nov 14, 2017
    Messages:
    3,052
    Likes Received:
    213
    Trophy Points:
    173
    Location:
    Houston
    cPanel Access Level:
    DataCenter Provider
    Hello,

    If all the ports are being filtered it sounds as if you have a misconfiguration of the firewall. Once you get the server back up I would suggest disabling CSF temporarily to determine if it's causing the issues. If so you'll need to relook at the configuration to ensure it's correct.


    Thanks!
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  13. David_spm

    David_spm Active Member

    Joined:
    May 28, 2017
    Messages:
    42
    Likes Received:
    0
    Trophy Points:
    6
    Location:
    Thailand
    cPanel Access Level:
    Root Administrator
    Thanks Lauren, I managed to get back into the server yesterday and this is where Im at now:

    I could see in Munin logs and graphs that everything went down around 6am EST, I couldnt see any major spike in resource usage before then though (but the Firewall data in network does show a sharp spike then. )

    In CSF logs it looks like right around that time there was an error and CSF shutdown, from what I can tell from the CSF logs there were a lot of brute force attempts that were getting killed/blocked and this was going on every minute for hrs:

    Code:
    Jun 14 06:20:05 server lfd[12677]: *User Processing* PID:10998 Kill:0 User:mysite_A VM:555(MB) EXE:/opt/cpanel/ea-php70/root/usr/sbin/php-fpm CMDhp-fpm: pool mysite_A_com
    Jun 14 06:20:05 server lfd[12677]: *User Processing* PID:11297 Kill:0 User:mysite_B VM:556(MB) EXE:/opt/cpanel/ea-php70/root/usr/sbin/php-fpm CMDhp-fpm: pool mysite_B_net
    Jun 14 06:20:24 server lfd[13375]: (sshd) Failed SSH login from some.ip.here (CA/Canada/ns510220.ip-.net): 5 in the last 3600 secs - *Blocked in csf* [LF_SSHD]
    Jun 14 06:20:28 server lfd[1556]: *Error* You have an unresolved error when starting csf. You need to restart csf successfully before restarting lfd (see /etc/csf/csf.error). *lfd stopped*, at line 1117
    Jun 14 06:20:28 server lfd[1556]: daemon stopped
    
    what does this MB data mean? VM:555(MB)

    Currently CSF is not enabled.

    Ive also just noticed that again the same ip is spamming one site and again it seems that iptables is not saving my rule where I blocked that ip, I dont understand why this rule keeps getting removed and also why the deny in .htaccess to stop access to the xmlrpc.php file isnt working.

    This is a copy of my iptables-config file, as you can see I have it setup so that saves rules should work in the event of a shutdown or reboot

    pastebin.com/Lie0MYgD

    I should also add that right after I managed to get back into the server I found Apache was down and threw this error when I tried to restart it:

    Code:
    No space left on device: AH00023: Couldn't create the mpm-accept mutex (28)No space left on device: could not create accept mutex
    pastebin.com/Lie0MYgD

    Ive never seen this error before and was confused by it as there was plenty of disk space but after some Googling found this article:

    - Removed -

    I followed that and managed to restart Apache and everything seems to be back to normal now but Ive never heard of semaphores before in relation to Apache or Linux, how can I go about finding what caused this issue?
     
    #13 David_spm, Jun 16, 2018
    Last edited by a moderator: Jun 16, 2018
  14. cPanelLauren

    cPanelLauren Forums Analyst II
    Staff Member

    Joined:
    Nov 14, 2017
    Messages:
    3,052
    Likes Received:
    213
    Trophy Points:
    173
    Location:
    Houston
    cPanel Access Level:
    DataCenter Provider
    Based on that output it seems like there was a couple of issues:

    This shows LFD killing php-fpm processes from this user for high processing:
    Code:
    Jun 14 06:20:05 server lfd[12677]: *User Processing* PID:10998 Kill:0 User:mysite_A VM:555(MB) EXE:/opt/cpanel/ea-php70/root/usr/sbin/php-fpm CMDhp-fpm: pool mysite_A_com
    Jun 14 06:20:05 server lfd[12677]: *User Processing* PID:11297 Kill:0 User:mysite_B VM:556(MB) EXE:/opt/cpanel/ea-php70/root/usr/sbin/php-fpm CMDhp-fpm: pool mysite_B_net
    This shows a failed SSH login which subsequently was blocked:
    Code:
    Jun 14 06:20:24 server lfd[13375]: (sshd) Failed SSH login from some.ip.here (CA/Canada/ns510220.ip-.net): 5 in the last 3600 secs - *Blocked in csf* [LF_SSHD]
    
    This indicates that you potentially restarted LFD which failed due to an error with CSF
    Code:
    Jun 14 06:20:28 server lfd[1556]: *Error* You have an unresolved error when starting csf. You need to restart csf successfully before restarting lfd (see /etc/csf/csf.error). *lfd stopped*, at line 1117
    Jun 14 06:20:28 server lfd[1556]: daemon stopped
    MB stands for MegaBytes

    Is this due to the error or did you stop it purposely?

    When you stop CSF and disable the configuration the iptables rules are flushed which may be why the rule isn't working.


    I'm glad you were able to resolve the semaphore issue. I found a very nice explanation regarding the purpose of semaphores on serverfault:
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  15. David_spm

    David_spm Active Member

    Joined:
    May 28, 2017
    Messages:
    42
    Likes Received:
    0
    Trophy Points:
    6
    Location:
    Thailand
    cPanel Access Level:
    Root Administrator
    Currently CSF is not enabled.

    Ive left it like this since it crashed as it caused the entire server to go down which I dont want to happen again.

    This shows LFD killing php-fpm processes from this user for high processing:

    what I showed you was a very small sample from hundreds of attempts like that with many sites over many hours, is that normal or do I have a bigger problem?

    I think I may have the issue under control for now after putting CloudFlare in front of a few of the sites that were getting targeted.

    However there is still one ip which I have added to DROP to my iptable rules many times now but for some reason these rules keep vanishing later on and the ip comes back and its trying to spma /xmlrpc.php again on one site only this time it seems to be in a very small amount due to CF doing its job I guess?

    So I guess something else is flushing my iptables?
     
  16. keat63

    keat63 Well-Known Member

    Joined:
    Nov 20, 2014
    Messages:
    1,013
    Likes Received:
    45
    Trophy Points:
    28
    cPanel Access Level:
    Root Administrator
  17. cPanelLauren

    cPanelLauren Forums Analyst II
    Staff Member

    Joined:
    Nov 14, 2017
    Messages:
    3,052
    Likes Received:
    213
    Trophy Points:
    173
    Location:
    Houston
    cPanel Access Level:
    DataCenter Provider
    HI @David_spm

    I think you have a bigger problem, attacks like this will cause high processing for the user and they don't stop.

    CloudFlare should help but you may want to look at further ways to prevent the issue from occurring.

    I can't say why the rule isn't being kept, it could be an issue with the rule itself, when you add it and then run
    Code:
    iptables --list
    Do you see it present?

    Another issue with blocking per IP is that xmlrpc.php attacks typically originate from multiple IP addresses, blocking one just disallows one of hundred/thousands of IP's that are being used. There are a ton of articles online for suggestions on how to stop these attacks similar to what @keat63 provided.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
Loading...

Share This Page

  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice