How to find which account is crawled by Bots?

[Logesh K]

My server was on High load and high outgoing bandwidth. Upon checking with the iftop tool I found lot of IP's with hostname of the Bots - semrush, ahreafs, baidu were crawling the websites on my server. But I am unable to find out which account/domain was crawled by these bots.

Can someone help me in this?

Thanks

 

[cPanelLauren]

Hi @Logesh K

The domain access logs should give you this data, they can be found at /etc/apache2/domlogs/ for each domain.

 

[Logesh K]

Hi @Logesh K

The domain access logs should give you this data, they can be found at /etc/apache2/domlogs/ for each domain.

Thanks for the reply. But I don't find it easy to check the numerous files manually to find out the domain. Is there any way to find realtime data on this? Because once my server freezed due to ahrefs and semrush bots, while I was watching the bandwidth data using iftop.

 

[Henry Carter]

Hello Logesh,

You can hit the following command on terminal as a root, it will show up the top utilizing domain access logs. cd /var/log/apache2/domlogs && ll -S | head You can view Bandwidth utilization through WHM >> Home >> Account Information >> View Bandwidth Usage, hit "Xfer(Best Fit)" on the top to sort the usage list. Use "iotop" command, it shows a table of current I/O usage by processes or threads.