inalto

Member
Mar 27, 2004
22
0
151
Hello,

here is my history of high load on a server.

I want to share this experience to see if I can improve something.

my configuration

Amd64 3000 (2GHz) 1GB ram.
3992.74 BogoMIPS
WHM 10.8.0 cPanel 10.9.0-R57
CentOS 4.4 i686 - WHM X v3.1.0

httpd with:

LoadModule rewrite_module libexec/mod_rewrite.so
LoadModule expires_module libexec/mod_expires.so
LoadModule bwlimited_module libexec/mod_bwlimited.so
LoadModule bytes_log_module libexec/mod_log_bytes.so
LoadModule auth_passthrough_module libexec/mod_auth_passthrough.so
LoadModule security_module libexec/mod_security.so

only few sites 3-4, using joomla.

I had high load issues, related to apache.
Is not a bandwith problem.
I have read almost everything related on the forum :)

The situation:

top - 14:55:02 up 10:06, 1 user, load average: 17.01, 19.10, 12.93
Tasks: 113 total, 25 running, 88 sleeping, 0 stopped, 0 zombie
Cpu(s): 99.0% us, 1.0% sy, 0.0% ni, 0.0% id, 0.0% wa, 0.0% hi, 0.0% si
Mem: 970536k total, 572108k used, 398428k free, 4316k buffers
Swap: 2048248k total, 331308k used, 1716940k free, 35036k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
19470 nobody 16 0 50124 35m 1132 R 35.7 3.8 11:19.91 httpd
19467 nobody 16 0 43536 37m 1504 R 16.0 4.0 11:50.81 httpd
4564 nobody 16 0 40928 35m 1108 S 7.5 3.8 0:26.76 httpd
19468 nobody 16 0 44504 35m 1164 R 6.9 3.7 13:49.75 httpd
5060 nobody 17 0 39572 34m 1088 R 3.6 3.7 0:01.98 httpd
19475 nobody 17 0 53816 36m 1504 R 3.3 3.8 11:50.24 httpd
19473 nobody 17 0 52016 35m 1504 R 2.9 3.8 12:21.12 httpd
19482 nobody 17 0 44872 34m 1164 R 2.9 3.6 12:19.19 httpd
32387 nobody 17 0 42848 36m 1116 R 2.9 3.9 2:45.81 httpd
5312 nobody 17 0 33908 29m 956 R 2.9 3.1 0:00.19 httpd
5313 nobody 17 0 33772 28m 956 R 2.9 3.1 0:00.19 httpd
5315 nobody 17 0 33772 29m 956 R 2.9 3.1 0:00.19 httpd
5316 nobody 17 0 33368 28m 956 R 2.6 3.0 0:00.17 httpd
4783 nobody 18 0 39812 35m 1084 R 1.6 3.7 0:02.71 httpd
4593 nobody 17 0 42260 36m 1096 R 1.3 3.9 0:14.11 httpd
5311 nobody 18 0 34724 29m 960 R 1.3 3.2 0:00.21 httpd
19483 nobody 16 0 43352 35m 1504 R 1.0 3.8 11:48.33 httpd
4563 nobody 18 0 40564 35m 1096 R 0.3 3.8 0:05.16 httpd

netstat -plan|grep :80| grep -v TIME_WAIT | wc -l

gives:
30

I have checked with netstat and does not seems bad traffic, only regular users..

1) I have updated the kernel to the latest version.
2) I have done checks to see if is a swap problem but the problem is not swap related.

vmstat 3 60

gives:

procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
16 0 144020 197688 28324 103816 21 28 39 90 36 164 55 1 44 0
16 0 144020 196216 28356 103816 0 0 0 127 1020 228 98 2 0 0
17 0 144020 184632 28356 103820 0 0 0 0 1020 168 99 1 0 0

3) I have changed my httpd.conf

Timeout 300
KeepAlive On
MaxKeepAliveRequests 100
KeepAliveTimeout 12
MinSpareServers 10
MaxSpareServers 20
StartServers 2
MaxClients 150
MaxRequestsPerChild 30

4) I have partially solved the problem removing some minor mod_security rules and the load average
drop down to 2.+

mod security version 1.9.1

and max cpu% per httpd PID drop down to maximum 19%

with netstat -plan|grep :80| grep -v TIME_WAIT | wc -l

24

WOW, seems I have too many mod_security rules...

5) I cut my domlogs files that have grown up to 300Mb and I get a load average to 1.9+...

My actual situation is, the server is working but load average is too high!
Damn it's an amd64 2ghz 1Gbram with 4 sites!

6) I recompile apache, nothing change.

Now things are working just because there are only few sites...

But in fact the main question is How can I evaluate that situation?
I dont have much experience on server loads, and maybe some more experienced user could help me.