sluggishness on dual xeon?

[Secret Agent]

I have this server:

Dual Xeon 3.4GHz HT
4GB Memory
100MBit Uplink
2x300GB SCSI 10k Disks

Software
Apache 1.33
PHP 4.4.0
MySQL 4.1.13
cpanel 10.8x Current

APF
BFD
LSM
SIM
LES
mod dosevasive
mod throttle
zend optimizer
eaccelerator
nessus

Server Load:
CPU: always less than 0.25
Memory: always less than 20%
Disk: 10% usage
tmp: 2% usage

Problem: acting very sluggish loading the only two sites on the server.
1. Vbulletin forum (no more than 10-15 concurrent users max) about 300 unique hits per day
2. Business site (simple flash / php) about 300 unique hits per day

This was not the case until about two weeks ago. I don't know what could cause it but here are some configurations. Please let me know what could use the sites to act abnormally like the following

1. no page found
2. slow page load
3. normal page load (becomming almost rare).

I tried: Avant browser, IE 6.x (completely cleared cache/cookies and all). I'm on a cable LAN modem (extremely fast).

/etc/my.cnf

[mysqld]
datadir=/var/lib/mysql
skip-locking
skip-innodb
max_connections = 500
key_buffer = 128M
myisam_sort_buffer_size = 64M
join_buffer_size = 1M
read_buffer_size = 1M
sort_buffer_size = 2M
table_cache = 1800
thread_cache_size = 384
wait_timeout = 900
connect_timeout = 10
tmp_table_size = 256M
read_rnd_buffer_size = 524288
bulk_insert_buffer_size = 8M
max_allowed_packet = 64M
max_connect_errors = 10
query_cache_limit = 3M
query_cache_size = 256M
query_cache_type = 1
query_prealloc_size = 16384
query_alloc_block_size = 16384
old-passwords

[mysql.server]
user=mysql
basedir=/var/lib

[mysqld_safe]
err-log=/var/log/mysqld.log
pid-file=/var/lib/mysql/mysql.pid
open_files_limit=8192

[mysqldump]
quick
max_allowed_packet=32M

[mysql]
no-auto-rehash
#safe-updates

[isamchk]
key_buffer=64M
sort_buffer=64M
read_buffer=16M
write_buffer=16M

[myisamchk]
key_buffer=256M
sort_buffer=256M
read_buffer=128M
write_buffer=128M

[mysqlhotcopy]
interactive-timeout

php.ini (Zend / eaccelerator)

zend_extension="/ea/eaccelerator-0.9.3/modules/eaccelerator.so"
eaccelerator.shm_size="1024"
eaccelerator.cache_dir="/tmp/eaccelerator"
eaccelerator.enable="1"
eaccelerator.optimizer="1"
eaccelerator.check_mtime="1"
eaccelerator.debug="0"
eaccelerator.filter=""
eaccelerator.shm_max="0"
eaccelerator.shm_ttl="0"
eaccelerator.shm_prune_period="0"
eaccelerator.shm_only="0"
eaccelerator.compress="1"
eaccelerator.compress_level="5"

[Zend]
zend_optimizer.optimization_level=5
zend_extension_manager.optimizer=/usr/local/Zend/lib/Optimizer-2.5.10
zend_extension_manager.optimizer_ts=/usr/local/Zend/lib/Optimizer_TS-2.5.10
zend_optimizer.version=2.5.10a
zend_extension=/usr/local/Zend/lib/ZendExtensionManager.so
zend_extension_ts=/usr/local/Zend/lib/ZendExtensionManager_TS.so

max_execution_time = 30 ; Maximum execution time of each script, in seconds
memory_limit = 8M ; Maximum amount of memory a script may consume (8MB)

httpd.conf

Timeout 300
KeepAlive On
MaxKeepAliveRequests 120
KeepAliveTimeout 15
MinSpareServers 15
MaxSpareServers 20
StartServers 10
MaxClients 150
MaxRequestsPerChild 1000
#Listen 3000
#Listen 12.34.56.78:80

/etc/resolv.conf
confirmed with data center correct

TOP

17647 root      17   0  5780 1348   908 R     0.9  0.0   0:00   3 top
    1 root      16   0  2100  572   488 S     0.0  0.0   0:10   0 init
    2 root      RT   0     0    0     0 SW    0.0  0.0   0:07   0 migration/0
    3 root      34  19     0    0     0 SWN   0.0  0.0   0:02   0 ksoftirqd/0
    4 root      RT   0     0    0     0 SW    0.0  0.0   0:05   1 migration/1
    5 root      34  19     0    0     0 SWN   0.0  0.0   0:02   1 ksoftirqd/1
    6 root      RT   0     0    0     0 SW    0.0  0.0   0:09   2 migration/2
    7 root      34  19     0    0     0 SWN   0.0  0.0   0:09   2 ksoftirqd/2
    8 root      RT   0     0    0     0 SW    0.0  0.0   0:09   3 migration/3
    9 root      34  19     0    0     0 SWN   0.0  0.0   0:03   3 ksoftirqd/3
   10 root       5 -10     0    0     0 SW<   0.0  0.0   0:00   0 events/0
   11 root       5 -10     0    0     0 SW<   0.0  0.0   0:00   1 events/1
   12 root       5 -10     0    0     0 SW<   0.0  0.0   0:00   2 events/2
   13 root       5 -10     0    0     0 SW<   0.0  0.0   0:00   3 events/3
   14 root      12 -10     0    0     0 SW<   0.0  0.0   0:00   3 khelper
   15 root      15 -10     0    0     0 SW<   0.0  0.0   0:00   0 kacpid
   38 root       5 -10     0    0     0 SW<   0.0  0.0   0:00   0 kblockd/0
   39 root       5 -10     0    0     0 SW<   0.0  0.0   0:00   1 kblockd/1
   40 root       5 -10     0    0     0 SW<   0.0  0.0   0:00   2 kblockd/2
   41 root       5 -10     0    0     0 SW<   0.0  0.0   0:00   3 kblockd/3
   42 root      15   0     0    0     0 SW    0.0  0.0   0:00   3 khubd
   54 root      14 -10     0    0     0 SW<   0.0  0.0   0:00   0 aio/0
   55 root      14 -10     0    0     0 SW<   0.0  0.0   0:00   1 aio/1

Any help is appreciated.

 

[jackie46]

Have you checked the DNS records of those two sites to make sure that they are correct?
Have you checked the error_log to see if php is spewing any errors from those two sites?
Is your NOC experiencing issues with slow and slugglish behaviour
Have you installed mod_security with a big ruleset? This slows the server Iv found.
Have you made changes to /etc/syscntrl.conf to optimize the settings in there?
Your old-passwords should be old_passwords in my.cnf by the way.
Have you optimized or tried to run a repair on the the db for these two sites?
Have you checked their domlogs for issues?
Is the box being hammered DOS style?
What version of php are you running?
Have you looked at MySQL? ps aux shows? Have you restarted MySQL?

It could be a million things. You said its only 2 sites. Not all of them? Sounds like a MySQL issue to me.

 

[Secret Agent]

DNS is perfectly fine and untouched. PHP as I already mentioned is 4.4.0

I am not getting DDOS'd and I installed mod_security via WHM > addon modules without customizing anything.

Also, old-passwords has always been old-passwords as mentioned endless times in the forums regarding mysql.

I did not do a repair as they are working fine anyway. As for optimizing, vbulletin has its own and the other is tiny db.

 

[Secret Agent]

Another thing I noticed is SSL is much slower, especially than usual. Any ideas of what could cause that to happen? The logs are rotateted often (ssl / domain).

 

[anup123]

See if the following can be of any help:

http://www.webperf.org/breakdown.html

Anup

 

[bamasbest]

As an adder, just 'cause your box might not be under attack, your NOC/LAN can certainly be getting hammered.

What is the max capacity that your DC/NOC is capable of handling/contracted for? What percentage of that capacity do they run at?

The core networks for my DC utilizes multiple OC-12 & OC-48 SONET rings, CISCO 12012 Backbone Routers with GigE connects to multiple bonded/reduntant providers and extensive peering agreements. With some (estimated) 50,000 + servers online, I have yet to see them spike (under normal circumstances) over 35% capacity.

 

[rpmws]

download and install www.pingplotter.com and run a ping / tracert graph to your site for a few days and then do the same thing to other servers on that same network and see if you can find a problem at the network you are on.

 

[panayot]

To find out if it is your server, or the datacenter, login in ssh and start the lynx browser:

lynx www.yoursite.com

if its you, then maybe recompile apache and php (leaving out all php modules you dont need).

 

[Logan]

I recommend a cache, like eAccelerator. Here are some very simply copy & paste for shell lines: http://www.vbulletin.com/forum/showthread.php?t=75878

You'll notice an immense speed increase.

 

[jackie46]

I recommend a cache, like eAccelerator. Here are some very simply copy & paste for shell lines: http://www.vbulletin.com/forum/showthread.php?t=75878

You'll notice an immense speed increase.

On a dual pentium running 4 gigs of ram and 2 scsi drives? He shouldnt have to with those stats.

 

[slmsoft]

please try to raise maxclients to 512 in httpd.conf (or more)

 

[btrieve]

How about a traceroute to the host in question?

As well as an IP address for others to run traceroutes and test for "sluggishness."

 

[rpmws]

How about a traceroute to the host in question?

As well as an IP address for others to run traceroutes and test for "sluggishness."

thats' why i suggested www.pingplotter.com on a win32 box on his end. He can run it and graph every hop in 1 second intervals over a 36 hour period if he wants. This will nail down exactally what points if any are causing latency.