server load...please help me trace

[Secret Agent]

I can't seem to resolve this issue with a single server I have.

Dual Xeon 2.4GHz HT
2GB ECC Memory
cpanel 10.2x
Apache 1.33x
PHP 4.3.11
MySQL 4.1

PS AUX (portion of high usage only)

mysql    19570  0.1  9.9 298700 205824 ?     S    May25   1:32 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --pid-file=/var/lib/mysql/server.mydomain.com.pid --skip-lock
mysql    19573  0.1  9.9 298700 205824 ?     S    May25   1:31 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --pid-file=/var/lib/mysql/server.mydomain.com.pid --skip-lock
mysql    19595  0.1  9.9 298700 205824 ?     S    May25   1:52 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --pid-file=/var/lib/mysql/server.mydomain.com.pid --skip-lock
mysql    19596  0.1  9.9 298700 205824 ?     S    May25   1:32 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --pid-file=/var/lib/mysql/server.mydomain.com.pid --skip-lock
mysql    12379  0.1  9.9 298700 205824 ?     S    May25   1:06 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --pid-file=/var/lib/mysql/server.mydomain.com.pid --skip-lock
mysql    12386  0.1  9.9 298700 205824 ?     S    May25   1:09 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --pid-file=/var/lib/mysql/server.mydomain.com.pid --skip-lock
mysql    12403  0.1  9.9 298700 205824 ?     S    May25   1:24 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --pid-file=/var/lib/mysql/server.mydomain.com.pid --skip-lock
mysql    12422  0.1  9.9 298700 205824 ?     S    May25   1:04 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --pid-file=/var/lib/mysql/server.mydomain.com.pid --skip-lock
mysql    12427  0.1  9.9 298700 205824 ?     S    May25   1:06 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --pid-file=/var/lib/mysql/server.mydomain.com.pid --skip-lock
mysql    12430  0.1  9.9 298700 205824 ?     S    May25   1:10 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --pid-file=/var/lib/mysql/server.mydomain.com.pid --skip-lock
mysql    12431  0.1  9.9 298700 205824 ?     S    May25   1:11 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --pid-file=/var/lib/mysql/server.mydomain.com.pid --skip-lock
root     22307  0.0  0.3 21308 8004 ?        S    May25   0:00 /usr/sbin/clamd
mailnull 22313  0.0  0.0  7752 1692 ?        S    May25   0:00 /usr/sbin/exim -bd -q60m
mailnull 22318  0.0  0.0  8460 1624 ?        S    May25   0:00 /usr/sbin/exim -tls-on-connect -bd -oX 465
root     22328  0.1  1.0 24824 22824 ?       S    May25   0:00 /usr/bin/spamd -d --allowed-ips=127.0.0.1 --pidfile=/var/run/spamd.pid --max-children=2
root     22349  0.0  0.0  3404 1580 ?        S    May25   0:00 antirelayd
infinity 27760  0.0  0.0   384   12 ?        TN   May24   0:00 /usr/local/cpanel/3rdparty/bin/english/webalizer -N 10 -D /home/infinity/tmp/webalizer/dns_cache.db -R 250 -p -n sttdesign.com -o /home
mysql    27411  0.1  9.9 298700 205824 ?     S    May24   3:25 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --pid-file=/var/lib/mysql/server.mydomain.com.pid --skip-lock
mysql    27863  0.1  9.9 298700 205824 ?     S    May24   3:43 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --pid-file=/var/lib/mysql/server.mydomain.com.pid --skip-lock
mysql    27867  0.1  9.9 298700 205824 ?     S    May24   2:35 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --pid-file=/var/lib/mysql/server.mydomain.com.pid --skip-lock
mysql    27868  0.1  9.9 298700 205824 ?     S    May24   3:33 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --pid-file=/var/lib/mysql/server.mydomain.com.pid --skip-lock
mysql    27921  0.1  9.9 298700 205824 ?     S    May24   4:00 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --pid-file=/var/lib/mysql/server.mydomain.com.pid --skip-lock
mysql    27929  0.1  9.9 298700 205824 ?     S    May24   3:36 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --pid-file=/var/lib/mysql/server.mydomain.com.pid --skip-lock
mysql    27930  0.1  9.9 298700 205824 ?     S    May24   3:44 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --pid-file=/var/lib/mysql/server.mydomain.com.pid --skip-lock
mysql    27933  0.1  9.9 298700 205824 ?     S    May24   3:37 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --pid-file=/var/lib/mysql/server.mydomain.com.pid --skip-lock
mysql    27934  0.1  9.9 298700 205824 ?     S    May24   3:56 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --pid-file=/var/lib/mysql/server.mydomain.com.pid --skip-lock
mysql    27935  0.1  9.9 298700 205824 ?     S    May24   3:46 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --pid-file=/var/lib/mysql/server.mydomain.com.pid --skip-lock
mysql    27936  0.1  9.9 298700 205824 ?     S    May24   4:02 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --pid-file=/var/lib/mysql/server.mydomain.com.pid --skip-lock
mysql    27937  0.1  9.9 298700 205824 ?     S    May24   3:19 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --pid-file=/var/lib/mysql/server.mydomain.com.pid --skip-lock
mysql    27938  0.1  9.9 298700 205824 ?     S    May24   3:32 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --pid-file=/var/lib/mysql/server.mydomain.com.pid --skip-lock
mysql    27939  0.1  9.9 298700 205824 ?     S    May24   3:56 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --pid-file=/var/lib/mysql/server.mydomain.com.pid --skip-lock
mysql    27942  0.1  9.9 298700 205824 ?     S    May24   3:40 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --pid-file=/var/lib/mysql/server.mydomain.com.pid --skip-lock
agruner   8556  0.0  0.0  2528 1128 ?        S    May24   0:01 imapd
  22380  0.1  9.9 298700 205824 ?     S    May12  26:49 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --pid-file=/var/lib/mysql/server.mydomain.com.pid --skip-lock
mysql    23995  0.1  9.9 298700 205824 ?     S    May12  26:16 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --pid-file=/var/lib/mysql/server.mydomain.com.pid --skip-lock
mysql      371  0.1  9.9 298700 205824 ?     S    May14  20:11 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --pid-file=/var/lib/mysql/server.mydomain.com.pid --skip-lock
mysql      372  0.1  9.9 298700 205824 ?     S    May14  24:34 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --pid-file=/var/lib/mysql/server.mydomain.com.pid --skip-lock
mysql      379  0.1  9.9 298700 205824 ?     S    May14  24:35 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --pid-file=/var/lib/mysql/server.mydomain.com.pid --skip-lock
root     10969  0.0  0.0  1828  680 ?        S    May15   0:00 CROND

TOP:

  805 root      25   0     0    0    0 R 99.9  0.0   5494:04 kjournald
22354 basesoft  16   0 27064  24m 2644 S  4.0  1.2   0:05.62 spamd
 5098 mailnull  16   0  8688 3656 2540 S  2.3  0.2   0:00.07 exim
 5109 mailnull  16   0     0    0    0 Z  1.7  0.0   0:00.05 exim <defunct>
 4794 root      16   0  2472 1012  724 R  0.7  0.0   0:00.57 top
32327 nobody    15   0 46008  34m 4708 S  0.3  1.7   0:00.72 httpd

How do I resolve this mysql skip locking and kjournald problem?

 

[Secret Agent]

This PID won't even die

killall kjournald
killall -9 kjournald
kill 805


Nothing will kill it

 

[theuruguayan]

looks like your mysql is going crazy, probably needs optimization.

can you paste your /etc/my.cnf ?

carlos

 

[Secret Agent]

The main thing is the kjournald causing serious overload and PID won't die out. I know its kernel related but what should I do? Some on linux IRC are telling me reboot but dont' kill pid which makes no sense at all.

[mysqld]
port = 3306
skip-locking
skip-innodb
query_cache_limit=1M
query_cache_size=32M
query_cache_type=1
max_connections=500
interactive_timeout=100
wait_timeout=100
connect_timeout=10
thread_cache_size=32
key_buffer_size=128M
join_buffer_size=1M
max_allowed_packet=4M
table_cache=2048
#record_buffer=1M <- replaced with read_buffer_size
sort_buffer_size=2M
read_buffer_size=1M
max_connect_errors=10
# Try number of CPU's*2 for thread_concurrency
thread_concurrency=4
myisam_sort_buffer_size=32M
#log-bin
server-id=1

[mysqld_safe]
open_files_limit = 8192
err-log=/var/log/mysqld.log
pid-file=/var/lib/mysql/mysql.pid

[mysqldump]
quick
max_allowed_packet = 16M

[mysql]
no-auto-rehash
# Remove the next comment character if you are not familiar with SQL
#safe-updates

[isamchk]
key_buffer_size = 64M
sort_buffer_size = 64M
read_buffer_size = 4M
write_buffer_size = 4M

[myisamchk]
key_buffer_size = 64M
sort_buffer_size = 64M
read_buffer_size = 4M
write_buffer_size = 4M

[mysqlhotcopy]
interactive-timeout

 

[theuruguayan]

is part of kernel, and you are not going to be avail to kill it. So a reboot is the way to go.

if i was you i add more cache to my my.cnf settings, so it reduce the load (query/thread).

It for sure needs some optimization. Which os you running?

 

[Secret Agent]

Fedora Core 2

 

[anup123]

kjournald: On the ext3 filesystem there are a few different modes of journaling Any idea which mode is your system using (journal, ordered, writeback)?

httpd: Do you have mod_perl enabled in your httpd?
Persistent connections if being used results in so many mysql processes. Normal. I have 50 at any instant.

Anup

 

[Secret Agent]

mod_perl not found / enabled in httpd.conf

root@server [~]# mount
/dev/hda2 on / type ext3 (rw,usrquota)
none on /proc type proc (rw)
none on /sys type sysfs (rw)
none on /dev/pts type devpts (rw,gid=5,mode=620)
usbfs on /proc/bus/usb type usbfs (rw)
/dev/hda1 on /boot type ext3 (rw)
none on /dev/shm type tmpfs (rw)
/dev/hdb1 on /backup type ext3 (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
/usr/tmpDSK on /tmp type ext3 (rw,noexec,nosuid,loop=/dev/loop0)
/tmp on /var/tmp type none (rw,noexec,nosuid,bind)


Any help appreciated. Thanks

 

[Secret Agent]

is part of kernel, and you are not going to be avail to kill it. So a reboot is the way to go.

if i was you i add more cache to my my.cnf settings, so it reduce the load (query/thread).

It for sure needs some optimization. Which os you running?

Which line for query? limit? size?

 

[anup123]

mod_perl not found / enabled in httpd.conf

root@server [~]# mount
/dev/hda2 on / type ext3 (rw,usrquota)
none on /proc type proc (rw)
none on /sys type sysfs (rw)
none on /dev/pts type devpts (rw,gid=5,mode=620)
usbfs on /proc/bus/usb type usbfs (rw)
/dev/hda1 on /boot type ext3 (rw)
none on /dev/shm type tmpfs (rw)
/dev/hdb1 on /backup type ext3 (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
/usr/tmpDSK on /tmp type ext3 (rw,noexec,nosuid,loop=/dev/loop0)
/tmp on /var/tmp type none (rw,noexec,nosuid,bind)


Any help appreciated. Thanks

mode should be available in dmesg logs -- something like this:
EXT3-fs: mounted filesystem with ordered data mode in case ordered mode is used

If that be it -- reboot would be the way to go.

regarding mysql : if using persistent connections (php scripts since u do not have mod_perl), then there must be somewhere where you can disable persistent connections in php configs

Just a shot in the dark but probaly workable.

Thnx
Anup

 

[anup123]

mysql skip locking is default for Linux as Linux file locking are not yet safe. Nothing to be concerned about IMHO. The only case when you can't use --skip-locking is if you run multiple MySQL servers (not clients) on the same data, which isn't the case here.

Anup