Server hammered with HTTP requests

[adeyjones]

Hi guys

Have got a server that hosts around 60 websites. The average load has been absolutely hammered today with HTTP requests but they all seem to be for one particular site, I have been checking the raw apache log throughout the day and blocking the various IP addresses but each time I block one, a new one appears. I trace them and they are from all different locations too, starting with USA, then Sweden, Netherlands etc..

Here is a snippet from the apache log for this 1 site, and the log currently contains 70,000+ lines:
84.17.46.229 - - [10/May/2022:19:33:13 +0000] "POST /page/3/ HTTP/1.1" 200 15246 "-" "Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.1.6) Gecko/20091201 Firefox/3.5.6 GTB5"
84.17.46.229 - - [10/May/2022:19:33:13 +0000] "POST /page/3/ HTTP/1.1" 200 15249 "-" "Opera/9.80 (Windows NT 6.1; U; en) Presto/2.7.62 Version/11.01"
84.17.46.229 - - [10/May/2022:19:33:13 +0000] "POST /page/3/ HTTP/1.1" 200 15248 "-" "Mozilla/4.8 [en] (Windows NT 5.1; U)"
84.17.46.229 - - [10/May/2022:19:33:13 +0000] "POST /page/3/ HTTP/1.1" 200 15265 "-" "Opera/9.80 (Windows NT 5.2; U; en) Presto/2.2.15 Version/10.10"
84.17.46.229 - - [10/May/2022:19:33:13 +0000] "POST /page/3/ HTTP/1.1" 200 15248 "-" "SonyEricssonK610i/R1CB Browser/NetFront/3.3 Profile/MIDP-2.0 Configuration/CLDC-1.1"
84.17.46.229 - - [10/May/2022:19:33:13 +0000] "POST /page/3/ HTTP/1.1" 200 15250 "-" "Mozilla/5.0 (Macintosh; U; PPC Mac OS X; en) AppleWebKit/125.2 (KHTML, like Gecko) Safari/125.8"
84.17.46.229 - - [10/May/2022:19:33:14 +0000] "POST /page/3/ HTTP/1.1" 200 15283 "-" "Mozilla/5.0 (Linux; U; Android 1.5; en-us; T-Mobile G1 Build/CRB43) AppleWebKit/528.5 (KHTML, like Gecko) Version/3.1.2 Mobile Safari 525.20.1"
84.17.46.229 - - [10/May/2022:19:33:14 +0000] "POST /page/3/ HTTP/1.1" 200 15249 "-" "Mozilla/5.0 (compatible; Konqueror/3.5; Linux 2.6.30-7.dmz.1-liquorix-686; X11) KHTML/3.5.10 (like Gecko) (Debian package 4:3.5.10.dfsg.1-1 b1)"
84.17.46.229 - - [10/May/2022:19:33:14 +0000] "POST /page/3/ HTTP/1.1" 200 15255 "-" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_2; en-us) AppleWebKit/531.21.8 (KHTML, like Gecko) Version/4.0.4 Safari/531.21.10"
84.17.46.229 - - [10/May/2022:19:33:14 +0000] "POST /page/3/ HTTP/1.1" 200 167 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.2; en-US) AppleWebKit/533.17.8 (KHTML, like Gecko) Version/5.0.1 Safari/533.17.8"
84.17.46.229 - - [10/May/2022:19:33:14 +0000] "POST /page/3/ HTTP/1.1" 200 15231 "-" "msnbot/1.0 ( Bing Webmaster Tools)"
84.17.46.229 - - [10/May/2022:19:33:15 +0000] "POST /page/3/ HTTP/1.1" 200 15249 "-" "Mozilla/5.0 (X11; U; Linux x86_64; us; rv:1.9.1.19) Gecko/20110430 shadowfox/7.0 (like Firefox/7.0"
84.17.46.229 - - [10/May/2022:19:33:19 +0000] "POST /page/3/ HTTP/1.1" 200 - "-" "Googlebot-Video/1.0"
84.17.46.229 - - [10/May/2022:19:33:17 +0000] "POST /page/3/ HTTP/1.1" 200 15248 "-" "BlackBerry8330/4.3.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 VendorID/105"
84.17.46.229 - - [10/May/2022:19:33:19 +0000] "POST /page/3/ HTTP/1.1" 200 - "-" "DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; Googlebot-Mobile/2.1; What Is Googlebot | Google Search Central | Documentation | Google Developers)"
84.17.46.229 - - [10/May/2022:19:33:18 +0000] "POST /page/3/ HTTP/1.1" 200 15265 "-" "Mozilla/5.0 (Linux; U; Android 2.0.1; de-de; Milestone Build/SHOLS_U2_01.14.0) AppleWebKit/530.17 (KHTML, like Gecko) Version/4.0 Mobile Safari/530.17"
84.17.46.229 - - [10/May/2022:19:33:18 +0000] "POST /page/3/ HTTP/1.1" 200 15247 "-" "SonyEricssonK310iv/R4DA Browser/NetFront/3.3 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Link/6.3.1.13.0"
84.17.46.229 - - [10/May/2022:19:33:18 +0000] "POST /page/3/ HTTP/1.1" 200 15264 "-" "MOTORIZR-Z8/46.00.00 Mozilla/4.0 (compatible; MSIE 6.0; Symbian OS; 356) Opera 8.65 [it] UP.Link/6.3.0.0.0"
84.17.46.229 - - [10/May/2022:19:33:18 +0000] "POST /page/3/ HTTP/1.1" 200 15248 "-" "SonyEricssonK550i/R1JD Browser/NetFront/3.3 Profile/MIDP-2.0 Configuration/CLDC-1.1"
84.17.46.229 - - [10/May/2022:19:33:19 +0000] "POST /page/3/ HTTP/1.1" 200 15228 "-" "Nokia6630/1.0 (2.39.15) SymbianOS/8.0 Series60/2.6 Profile/MIDP-2.0 Configuration/CLDC-1.1"
84.17.46.229 - - [10/May/2022:19:33:19 +0000] "POST /page/3/ HTTP/1.1" 200 15270 "-" "Mozilla/5.0 (X11; U; FreeBSD i386; de-CH; rv:1.9.2.8) Gecko/20100729 Firefox/3.6.8"
84.17.46.229 - - [10/May/2022:19:33:20 +0000] "POST /page/3/ HTTP/1.1" 200 167 "-" "Mozilla/5.0 (compatible; Konqueror/3.5; Linux 2.6.30-7.dmz.1-liquorix-686; X11) KHTML/3.5.10 (like Gecko) (Debian package 4:3.5.10.dfsg.1-1 b1)"
84.17.46.229 - - [10/May/2022:19:33:19 +0000] "POST /page/3/ HTTP/1.1" 200 15248 "-" "Gaisbot/3.0 ([email protected]; http://gais.cs.ccu.edu.tw/robot.php)"
84.17.46.229 - - [10/May/2022:19:33:19 +0000] "POST /page/3/ HTTP/1.1" 200 15251 "-" "Mozilla/5.0 (Linux; U; Android 1.5; fr-fr; GT-I5700 Build/CUPCAKE) AppleWebKit/528.5 (KHTML, like Gecko) Version/3.1.2 Mobile Safari/525.20.1"
84.17.46.229 - - [10/May/2022:19:33:19 +0000] "POST /page/3/ HTTP/1.1" 200 15248 "-" "everyfeed-spider/2.0 (Everyfeed.com is For Sale | BrandBucket)"
84.17.46.229 - - [10/May/2022:19:33:20 +0000] "POST /page/3/ HTTP/1.1" 200 15281 "-" "Wget/1.9.1"
84.17.46.229 - - [10/May/2022:19:33:20 +0000] "POST /page/3/ HTTP/1.1" 200 15264 "-" "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.2) Gecko/20090803 Ubuntu/9.04 (jaunty) Shiretoko/3.5.2"
84.17.46.229 - - [10/May/2022:19:33:20 +0000] "POST /page/3/ HTTP/1.1" 200 15257 "-" "WDG_Validator/1.6.2"
84.17.46.229 - - [10/May/2022:19:33:20 +0000] "POST /page/3/ HTTP/1.1" 200 15258 "-" "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.2; WOW64; Trident/5.0)"
84.17.46.229 - - [10/May/2022:19:33:20 +0000] "POST /page/3/ HTTP/1.1" 200 15246 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:2.0.1) Gecko/20100101 Firefox/4.0.1"
84.17.46.229 - - [10/May/2022:19:33:21 +0000] "POST /page/3/ HTTP/1.1" 200 167 "-" "Mozilla/5.0 (Linux; U; Android 2.0; en-us; Milestone Build/ SHOLS_U2_01.03.1) AppleWebKit/530.17 (KHTML, like Gecko) Version/4.0 Mobile Safari/530.17"
84.17.46.229 - - [10/May/2022:19:33:20 +0000] "POST /page/3/ HTTP/1.1" 200 15249 "-" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2.14) Gecko/20110218 AlexaToolbar/alxf-2.0 Firefox/3.6.14"
84.17.46.229 - - [10/May/2022:19:33:21 +0000] "POST /page/3/ HTTP/1.1" 200 15282 "-" "Mozilla/5.0 (X11; U; Linux i686; it; rv:1.9.2.3) Gecko/20100406 Firefox/3.6.3 (Swiftfox)"
84.17.46.229 - - [10/May/2022:19:33:21 +0000] "POST /page/3/ HTTP/1.1" 200 15268 "-" "Mozilla/5.0 (Linux; U; Android 1.5; fr-fr; GT-I5700 Build/CUPCAKE) AppleWebKit/528.5 (KHTML, like Gecko) Version/3.1.2 Mobile Safari/525.20.1"
84.17.46.229 - - [10/May/2022:19:33:21 +0000] "POST /page/3/ HTTP/1.1" 200 15267 "-" "Mozilla/5.0 (X11; Linux i686; rv:5.0) Gecko/20100101 Firefox/5.0"
84.17.46.229 - - [10/May/2022:19:33:21 +0000] "POST /page/3/ HTTP/1.1" 200 15248 "-" "SonyEricssonK610i/R1CB Browser/NetFront/3.3 Profile/MIDP-2.0 Configuration/CLDC-1.1"
84.17.46.229 - - [10/May/2022:19:33:22 +0000] "POST /page/3/ HTTP/1.1" 200 15229 "-" "Mozilla/5.0 (Android; Linux armv7l; rv:10.0.1) Gecko/20100101 Firefox/10.0.1 Fennec/10.0.1"
84.17.46.229 - - [10/May/2022:19:33:22 +0000] "POST /page/3/ HTTP/1.1" 200 167 "-" "Mozilla/5.0 (iPad; U; CPU OS 3_2 like Mac OS X; en-us) AppleWebKit/531.21.10 (KHTML, like Gecko) Version/4.0.4 Mobile/7B334b Safari/531.21.10"
84.17.46.229 - - [10/May/2022:19:33:22 +0000] "POST /page/3/ HTTP/1.1" 200 15229 "-" "Mozilla/5.0 (Windows; U; Windows NT 6.0; en-GB; rv:1.9.0.11) Gecko/2009060215 Firefox/3.0.11 (.NET CLR 3.5.30729)"
84.17.46.229 - - [10/May/2022:19:33:22 +0000] "POST /page/3/ HTTP/1.1" 200 15266 "-" "Microsoft URL Control - 6.00.8862"
84.17.46.229 - - [10/May/2022:19:33:22 +0000] "POST /page/3/ HTTP/1.1" 200 15289 "-" "Mozilla/5.0 (Linux; U; Android 1.5; de-de; HTC Magic Build/PLAT-RC33) AppleWebKit/528.5 (KHTML, like Gecko) Version/3.1.2 Mobile Safari/525.20.1 FirePHP/0.3"
84.17.46.229 - - [10/May/2022:19:33:22 +0000] "POST /page/3/ HTTP/1.1" 200 15246 "-" "Mozilla/5.0 (X11; U; Linux armv61; en-US; rv:1.9.1b2pre) Gecko/20081015 Fennec/1.0a1"
84.17.46.229 - - [10/May/2022:19:33:22 +0000] "POST /page/3/ HTTP/1.1" 200 15249 "-" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2.14) Gecko/20110218 AlexaToolbar/alxf-2.0 Firefox/3.6.14"
84.17.46.229 - - [10/May/2022:19:33:22 +0000] "POST /page/3/ HTTP/1.1" 200 15263 "-" "Opera/9.80 (Windows NT 6.1; U; es-ES) Presto/2.9.181 Version/12.00"
84.17.46.229 - - [10/May/2022:19:33:22 +0000] "POST /page/3/ HTTP/1.1" 200 15226 "-" "Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; Win 9x 4.90)"
84.17.46.229 - - [10/May/2022:19:33:22 +0000] "POST /page/3/ HTTP/1.1" 200 15232 "-" "Mozilla/4.0 (compatible; Linux 2.6.22) NetFront/3.4 Kindle/2.0 (screen 600x800)"
84.17.46.229 - - [10/May/2022:19:33:22 +0000] "POST /page/3/ HTTP/1.1" 200 15265 "-" "Mozilla/5.0 (Linux; U; Android 1.1; en-gb; dream) AppleWebKit/525.10 (KHTML, like Gecko) Version/3.0.4 Mobile Safari/523.12.2"
84.17.46.229 - - [10/May/2022:19:33:23 +0000] "POST /page/3/ HTTP/1.1" 200 15227 "-" "Mozilla/5.0 (X11; U; Linux armv61; en-US; rv:1.9.1b2pre) Gecko/20081015 Fennec/1.0a1"
84.17.46.229 - - [10/May/2022:19:33:23 +0000] "POST /page/3/ HTTP/1.1" 200 15271 "-" "BlackBerry8330/4.3.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 VendorID/105"
84.17.46.229 - - [10/May/2022:19:33:22 +0000] "POST /page/3/ HTTP/1.1" 200 15248 "-" "WebZIP/3.5 (WebZIP & WinMHT Offline Browser - Download websites, copy complete web pages. Browse offline, save web sites.)"
84.17.46.229 - - [10/May/2022:19:33:22 +0000] "POST /page/3/ HTTP/1.1" 200 15248 "-" "Mozilla/2.02E (Win95; U)"

Any idea how I can prevent this apart from manually sitting here tracing and blocking IP's?

Thanks

 

[cPRex]

Hey there! That sounds like a classic DoS attack. If you want to try and fix it on the server-side, I'd recommend the Apache Evasive module:

Apache Module: Evasive | cPanel & WHM Documentation

This module provides DoS, DDoS, and brute force attack protection.

docs.cpanel.net

If you try that and you're still having issues, it would be best to reach out to your hosting provider or datacenter to see if they have any external DoS mitigation services so your machine doesn't have to handle the traffic at all.

 

[adeyjones]

Thanks @cPRex - i'll try the Evasive. Unfortunately I don't have any other support, it's an AWS EC2 instance so i'm on my own with this one.

 

[adeyjones]

I have had the Evasive module installed since this thread, unfortunately today this seems to have re-occured, the load on the server as I type is currently "102.46 89.32 64.66".

I have run netstat -tn 2>/dev/null | grep ":80" which I read somewhere but this doesn't really help identify anything.

 

[adeyjones]

Have just amended the config for the mod evasive to change site count from 100 to 50, restarted apache, load has gone down for now but will monitor to see if this is only due to the restart or if it is a more long term solution.

 

[adeyjones]

Unfortunately the issue is still there, load is going up as I type, attached it from process manager - any further ideas?

 

[cPRex]

With so many request happening you'll want to reach out to your hosting provider or datacenter and see if they can provide an external solution to help with this traffic. With the load so high, your server isn't going to be able to handle that traffic.

 

[adeyjones]

@cPRex - Thanks, unfortuately I don't have a hosting provider, this is an EC2 instance by AWS so i'm on my own. I have been advised to install mod_dumpio so that I can see the headers from the POST requests, but where do I change the config of it to change the log level and where do I also see the logs?
Thanks

 

[cPRex]

cPanel does have that module available as part of EasyApache so you can install that through WHM >> EasyApache 4. The various configuration options are listed here:

mod_dumpio - Apache HTTP Server Version 2.4

Those values can be added directly to the Apache configuration temporarily while you perform your work, or in the include system if you plan to keep them around for a while. In general, these should also be enabled for a short time as they can create a very large amount of data.

I'm not sure that's the best way to handle this issue though - we already know there's too much traffic reaching your machine, and that your machine is struggling to handle it. Making it do additional logging is just going to increase that strain. Have you checked out the details here? Mitigation techniques - AWS Best Practices for DDoS Resiliency

 

[adeyjones]

Thanks again - I have looked at AWS Shield from your link and it states that it is automatically enabled with all EC2 instances so I already have this although the dashboard has identified 0 events in the last year.