Mass attack on multiple wordpress sites hundred of IP addresses used

[WorkinOnIt]

Greetings.

I would like to seek advice on the best way to mitigate a rather severe attack that brought down our server last night for a period of 30 minutes due to a mass DDOS / vulnerability scan of dozens of wordpress sites simultaneously.

I've not seen an attack of this nature - it was not particularly complex or clever, but a mass of IP addresses (hundreds) used all at once were "probing" for specific wordpress plugin files (I assume these must be notified vulnerable software exploits) on dozens of sites on one server virtually simultaneously. Having gone through all of the sites and logs, none of our sites actually have any of the wordpress plugins installed, so I guess this co-ordinated bot attack must just be doing a wide sweep to see where it can get lucky with finding one of these installations....

I do have a copy of the Apache Server Status log that shows the attack, but I don't really want to assist the attackers by publishing the vulnerabilities they were probing for. However, for sample here are just a few of the hundreds:

GET /wp-content/plugins/simple-download-button-shortcode/simple
GET /wp-content/plugins/robotcpa/f.php?l=../../../wp-config.php
GET /wp-content/plugins/wp-swimteam/include/user/download.php?f
GET /wp-content/plugins/advanced-custom-fields/core/actions/exp
GET /wp-content/plugins/gracemedia-media-player/templates/files
GET /wp-content/plugins/wp-swimteam/include/user/download.php?f
GET /wp-content/plugins/bookx/includes/bookx_export.php?file=..

So what to do:

I have the usual security set up: using CSF firewall, cphulk, modsecurity etc.... but none of these effectively stopped this.

I guess what would be good would be a script or mod security rule that would block an IP after x number of 404's ? Would anyone happen to know one of those ?

Thanks

 

[GoMummyHosting]

Hello,

As you said the attack brought down your server in that case, you must enable hardware firewall with the help of DC to mitigate the attack. CSF/Cphulk/Modsec are the software firewalls which can protect you up to certain level. By optimizing apache/csf will help you too just tweak following parameters accordingly.
~~~~~~~~~~~~~~~
In Apache:
MaxClients 150
Timeout 300
Minimum Spare Servers
Maximum Spare Servers
Max Request Workers
~~~~~~~~~~~~~~~
In CSF:
TESTING = "0"
SMTP_BLOCK = "0"
SYNFLOOD = "0"
SYNFLOOD_RATE = "100/s"
SYNFLOOD_BURST = "150"
CONNLIMIT = ""
PORTFLOOD = ""
CC_DENY = ""
LF_SCRIPT_ALERT = "0"
LF_DIRWATCH_DISABLE = "0"
CT_LIMIT = "0"
PT_ALL_USERS = "0"
~~~~~~~~~~~~~~~

 

[quietFinn]

There is an option in CSF, LF_APACHE_404, might be worth checking.

 

[rackaid]

I see these scans often.

The log snippet you provided did not show the error code. Did these sites send true 404 errors?

I've seen some WP plugins cause issues with 404's. They route the request into WP, which depending on the plugins, can cause performance issues.

How were server loads during the attack?
Did you check the Apache error log for notices about hitting MaxRequestWorkers?

In some cases, simply providing more workers to Apache will mitigate the attack.

Mod QoS
I currently use use mod_qos to deal with these attacks when upstream solutions are not available. Mod_QoS has various rate-limiting rules that can be powerful when dealing with large botnet attacks, especially if they are hitting multiple sites at the same time.

I used to have a blog post on this but I think we removed it as it was outdated.

The option QS_LocRequestLimitMatch can be powerful when you see the same URLs on multiple sites. This option limits concurrent connections to the specified location on a per-IP basis. This is over the entire server, not just a single site. So for example:

QS_LocRequestLimitMatch /xmlrpc.php 2

Would permit just 2 request/sec to xmlrpc.php. I have used these an similar rules to mitigate large bot nets.

The trick is identifying the right rules for the type of attack. QS_LocRequestPerSecLimitMatch is another good rule that limits on rate rather than number of connections.


More info on mod_qos

 

[cPanelLauren]

mod_qos is a really good suggestion for this issue, it's on of the modules biggest assets and it's pretty configurable.

mod_qos may be used to determine which requests should be served and which shouldn't in order to avoid resource oversubscription. The module collects different attributes such as the request URL, HTTP request and response headers, the IP source address, country codes, the HTTP response code, history data (based on user session and source IP address), the number of concurrent requests to the server (total or requests having similar attributes), the number of concurrent TCP connections (total or from a single source IP), and so forth.

The LF_APACHE_404 suggestion is also a good one to implement - the full description on that can be found here:

# This option will keep track of the number of "File does not exist" errors in
# HTACCESS_LOG. If the number of hits is more than LF_APACHE_404 in LF_INTERVAL
# seconds then the IP address will be blocked
#
# Care should be used with this option as it could generate many
# false-positives, especially Search Bots (use csf.rignore to ignore such bots)
# so only use this option if you know you are under this type of attack
#
# A sensible setting for this would be quite high, perhaps 200
#
# To disable set to "0"
LF_APACHE_404 = "0"

mod_security can facilitate blocking some attacks like this but in order for mod_security to facilitate blocking this kind of issue you'd need to add some custom rulesets. It primarily is just DOS protection that it's able to provide as well - if it's a distributed attack it won't be able to be as effective.

ModEvasive might also be an option , you can read about mod_evasive here: Apache Module: Evasive | cPanel & WHM Documentation

 

[ffeingol]

mod_qos is a really good suggestion for this issue, it's on of the modules biggest assets and it's pretty configurable.

Sorry for digging up an old post, but I'm a bit confused. @cPanelLauren your recommending mod_qos, but as far as I can see, it's not supported in EasyApache???