Apache Bug (Build 11.38.x)? (get /robots.txt)

easyswiss

Active Member
PartnerNOC
Apr 19, 2011
40
0
56
We have a problem on 2 servers since ~july/august 2013, they have every day a lot of processes like this here:

945950 1/43/7079 G 1.80 68687 0 2.7 0.50 175.53 66.249.66.104 HOSTNAME:4 GET /robots.txt HTTP/1.1
872984 1/9/36597 G 0.19 11952 0 2.7 0.19 1167.38 66.249.66.36 HOSTNAME:4 GET /robots.txt HTTP/1.1

This results in heavy load (this processes take 100% cpu usage -> as nobody).

Is there a solution?
Add this IP's to the host access access control have no effect.
Only this "get /robots.txt processes need so much cpu.
 

cPanelMichael

Administrator
Staff member
Apr 11, 2011
47,913
2,201
363
Hello :)

Do you have a third-party firewall application such as CSF installed on your system? If not, it may be a good idea to install a firewall so you can block IP addresses from accessing your server. You may also want to install Mod_Security to help with certain access attempts.

Thank you.
 

cPanelJamyn

Social Engineer
Staff member
Jan 29, 2009
105
2
143