Investigating high server loads

shoptalk

Member
Nov 29, 2012
8
0
1
cPanel Access Level
Root Administrator
Hi

I run around 80 clients' sites on a VM, 2 x 2.4GHz processors and 2Gb RAM. The sites are relatively low traffic, probably no more than 10k page views between them per day. They all run up-to-date versions of WordPress, I keep any plug-ins up to date, running caching plug-ins, and generally (I think) keep the sites well maintained.

I have recently, over the last 5-6 weeks, seen unexpected spikes in the server loads. Normally, loads would be well under 1.0 but every now and again I see them spiking to 100+. This slows or brings down the server.

I often see spikes at around 3am - one suggested explanation for this is that stats are being run on multiple VMs at this time, which is creating a peak. We've changed the time that my stats get run and will monitor this.

However, I also see spikes at other, less predictable times. Yesterday, server loads peaked at 200 - seemingly out of nowhere. This was late afternoon, there wasn't a lot of activity that I could see. Looking at the server load logs leading up to the peak, loads were consistently at less than 1.0, then suddenly 200...

There are two accounts on the server that I don't manage. I've removed these accounts to eliminate the possibility that they are creating the loads. I've been through each account, checked the individual error logs, and fixed any php errors that I found.

My question is how can I go about investigating the cause of these spikes? My knowledge of servers is pretty limited - I'm a designer with some programming knowledge.

What I feel I want to be able to do is track down the account that might have caused the loads to spike to 200 - is this possible to do or is there a better approach?

Grateful for any advice.
 

shoptalk

Member
Nov 29, 2012
8
0
1
cPanel Access Level
Root Administrator
Hi

Thanks - yes, I've looked at the Daily Process Log and I'm not entirely sure how best to interpret it. Where I look at the averages for the different users, I don't see anything that strikes me as excessive. The top average for a user is 2.39% CPU, 1.06% MEM, 0.1 MySQL Processes. After that, root is 1.78% | 8.49% | 2.0, then the users are less than 1% CPU and less than 1% MEM. So as far as I can tell, that's okay?

If I look at top processes, it's different. I see approximately 20 processes at between 50% and 90% CPU. These processes belong to 12 different users.

There are probably another 20 processes at between 20% and 50%, belonging to 10-12 users.

I don't know if this is high or not - the averages don't seem high, but the top processes do...

Any advice on this would be appreciated.
 

Infopro

Well-Known Member
May 20, 2003
17,076
524
613
Pennsylvania
cPanel Access Level
Root Administrator
Twitter
Normally, loads would be well under 1.0 but every now and again I see them spiking to 100+. This slows or brings down the server.
What day was this? I don't need to know, you do. Then you might look here on this page, on that day and hopefully see the account shown in red. There's a clue.

Do you have CSF installed?
ConfigServer Security & Firewall
 

shoptalk

Member
Nov 29, 2012
8
0
1
cPanel Access Level
Root Administrator
Thank you.

So yesterday afternoon, there was one of these spikes. The processes I quoted in my last post were from yesterday - but I didn't see anything highlighted in red.

I do have CSF installed.
 

Infopro

Well-Known Member
May 20, 2003
17,076
524
613
Pennsylvania
cPanel Access Level
Root Administrator
Twitter
Go to CSF, find Firewall Security Level button on main page of CSF. Click the button here for High. These settings can also be found in the CSF configuration section for tweaking them individually, later of course.

Is email from CSF being received normally?
 

Infopro

Well-Known Member
May 20, 2003
17,076
524
613
Pennsylvania
cPanel Access Level
Root Administrator
Twitter
Are you getting any email from the server?

I think the options speak for themselves really. Tighten security and you'll be alerted more often when an account trips up that security.

You might want to read thru the readme for CSF:
http://www.configserver.com/free/csf/readme.txt
Or go thru your CSF configuration section line by line and read the notes there.
 

shoptalk

Member
Nov 29, 2012
8
0
1
cPanel Access Level
Root Administrator
Okay, I'll give the readme a read.

I do get email from the server - just not from CSF that I'm aware of (maybe I'm just not aware it's coming from CSF)

If I click the 'Check Server Security' button, I get a lot of red and a score of 97/133. I think that's something for me to go back to my provider with.

If I understand you correctly, you're suggesting that the server load spikes are a security issue rather than server or account config? Would that be right?
 

Infopro

Well-Known Member
May 20, 2003
17,076
524
613
Pennsylvania
cPanel Access Level
Root Administrator
Twitter
My question is how can I go about investigating the cause of these spikes? My knowledge of servers is pretty limited - I'm a designer with some programming knowledge.

What I feel I want to be able to do is track down the account that might have caused the loads to spike to 200 - is this possible to do or is there a better approach?

CSF should be helpful to you here. But you'll need to make sure you're getting the emails.