server load...please help me trace

S

Secret Agent

Guest
I can't seem to resolve this issue with a single server I have.

Dual Xeon 2.4GHz HT
2GB ECC Memory
cpanel 10.2x
Apache 1.33x
PHP 4.3.11
MySQL 4.1

PS AUX (portion of high usage only)

Code:
mysql    19570  0.1  9.9 298700 205824 ?     S    May25   1:32 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --pid-file=/var/lib/mysql/server.mydomain.com.pid --skip-lock
mysql    19573  0.1  9.9 298700 205824 ?     S    May25   1:31 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --pid-file=/var/lib/mysql/server.mydomain.com.pid --skip-lock
mysql    19595  0.1  9.9 298700 205824 ?     S    May25   1:52 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --pid-file=/var/lib/mysql/server.mydomain.com.pid --skip-lock
mysql    19596  0.1  9.9 298700 205824 ?     S    May25   1:32 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --pid-file=/var/lib/mysql/server.mydomain.com.pid --skip-lock
mysql    12379  0.1  9.9 298700 205824 ?     S    May25   1:06 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --pid-file=/var/lib/mysql/server.mydomain.com.pid --skip-lock
mysql    12386  0.1  9.9 298700 205824 ?     S    May25   1:09 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --pid-file=/var/lib/mysql/server.mydomain.com.pid --skip-lock
mysql    12403  0.1  9.9 298700 205824 ?     S    May25   1:24 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --pid-file=/var/lib/mysql/server.mydomain.com.pid --skip-lock
mysql    12422  0.1  9.9 298700 205824 ?     S    May25   1:04 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --pid-file=/var/lib/mysql/server.mydomain.com.pid --skip-lock
mysql    12427  0.1  9.9 298700 205824 ?     S    May25   1:06 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --pid-file=/var/lib/mysql/server.mydomain.com.pid --skip-lock
mysql    12430  0.1  9.9 298700 205824 ?     S    May25   1:10 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --pid-file=/var/lib/mysql/server.mydomain.com.pid --skip-lock
mysql    12431  0.1  9.9 298700 205824 ?     S    May25   1:11 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --pid-file=/var/lib/mysql/server.mydomain.com.pid --skip-lock
root     22307  0.0  0.3 21308 8004 ?        S    May25   0:00 /usr/sbin/clamd
mailnull 22313  0.0  0.0  7752 1692 ?        S    May25   0:00 /usr/sbin/exim -bd -q60m
mailnull 22318  0.0  0.0  8460 1624 ?        S    May25   0:00 /usr/sbin/exim -tls-on-connect -bd -oX 465
root     22328  0.1  1.0 24824 22824 ?       S    May25   0:00 /usr/bin/spamd -d --allowed-ips=127.0.0.1 --pidfile=/var/run/spamd.pid --max-children=2
root     22349  0.0  0.0  3404 1580 ?        S    May25   0:00 antirelayd
infinity 27760  0.0  0.0   384   12 ?        TN   May24   0:00 /usr/local/cpanel/3rdparty/bin/english/webalizer -N 10 -D /home/infinity/tmp/webalizer/dns_cache.db -R 250 -p -n sttdesign.com -o /home
mysql    27411  0.1  9.9 298700 205824 ?     S    May24   3:25 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --pid-file=/var/lib/mysql/server.mydomain.com.pid --skip-lock
mysql    27863  0.1  9.9 298700 205824 ?     S    May24   3:43 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --pid-file=/var/lib/mysql/server.mydomain.com.pid --skip-lock
mysql    27867  0.1  9.9 298700 205824 ?     S    May24   2:35 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --pid-file=/var/lib/mysql/server.mydomain.com.pid --skip-lock
mysql    27868  0.1  9.9 298700 205824 ?     S    May24   3:33 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --pid-file=/var/lib/mysql/server.mydomain.com.pid --skip-lock
mysql    27921  0.1  9.9 298700 205824 ?     S    May24   4:00 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --pid-file=/var/lib/mysql/server.mydomain.com.pid --skip-lock
mysql    27929  0.1  9.9 298700 205824 ?     S    May24   3:36 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --pid-file=/var/lib/mysql/server.mydomain.com.pid --skip-lock
mysql    27930  0.1  9.9 298700 205824 ?     S    May24   3:44 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --pid-file=/var/lib/mysql/server.mydomain.com.pid --skip-lock
mysql    27933  0.1  9.9 298700 205824 ?     S    May24   3:37 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --pid-file=/var/lib/mysql/server.mydomain.com.pid --skip-lock
mysql    27934  0.1  9.9 298700 205824 ?     S    May24   3:56 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --pid-file=/var/lib/mysql/server.mydomain.com.pid --skip-lock
mysql    27935  0.1  9.9 298700 205824 ?     S    May24   3:46 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --pid-file=/var/lib/mysql/server.mydomain.com.pid --skip-lock
mysql    27936  0.1  9.9 298700 205824 ?     S    May24   4:02 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --pid-file=/var/lib/mysql/server.mydomain.com.pid --skip-lock
mysql    27937  0.1  9.9 298700 205824 ?     S    May24   3:19 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --pid-file=/var/lib/mysql/server.mydomain.com.pid --skip-lock
mysql    27938  0.1  9.9 298700 205824 ?     S    May24   3:32 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --pid-file=/var/lib/mysql/server.mydomain.com.pid --skip-lock
mysql    27939  0.1  9.9 298700 205824 ?     S    May24   3:56 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --pid-file=/var/lib/mysql/server.mydomain.com.pid --skip-lock
mysql    27942  0.1  9.9 298700 205824 ?     S    May24   3:40 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --pid-file=/var/lib/mysql/server.mydomain.com.pid --skip-lock
agruner   8556  0.0  0.0  2528 1128 ?        S    May24   0:01 imapd
  22380  0.1  9.9 298700 205824 ?     S    May12  26:49 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --pid-file=/var/lib/mysql/server.mydomain.com.pid --skip-lock
mysql    23995  0.1  9.9 298700 205824 ?     S    May12  26:16 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --pid-file=/var/lib/mysql/server.mydomain.com.pid --skip-lock
mysql      371  0.1  9.9 298700 205824 ?     S    May14  20:11 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --pid-file=/var/lib/mysql/server.mydomain.com.pid --skip-lock
mysql      372  0.1  9.9 298700 205824 ?     S    May14  24:34 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --pid-file=/var/lib/mysql/server.mydomain.com.pid --skip-lock
mysql      379  0.1  9.9 298700 205824 ?     S    May14  24:35 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --pid-file=/var/lib/mysql/server.mydomain.com.pid --skip-lock
root     10969  0.0  0.0  1828  680 ?        S    May15   0:00 CROND
TOP:
Code:
  805 root      25   0     0    0    0 R 99.9  0.0   5494:04 kjournald
22354 basesoft  16   0 27064  24m 2644 S  4.0  1.2   0:05.62 spamd
 5098 mailnull  16   0  8688 3656 2540 S  2.3  0.2   0:00.07 exim
 5109 mailnull  16   0     0    0    0 Z  1.7  0.0   0:00.05 exim <defunct>
 4794 root      16   0  2472 1012  724 R  0.7  0.0   0:00.57 top
32327 nobody    15   0 46008  34m 4708 S  0.3  1.7   0:00.72 httpd
How do I resolve this mysql skip locking and kjournald problem?
 
S

Secret Agent

Guest
This PID won't even die

killall kjournald
killall -9 kjournald
kill 805


Nothing will kill it
 

theuruguayan

Active Member
PartnerNOC
Jul 6, 2003
37
0
156
looks like your mysql is going crazy, probably needs optimization.

can you paste your /etc/my.cnf ?

carlos
 
S

Secret Agent

Guest
The main thing is the kjournald causing serious overload and PID won't die out. I know its kernel related but what should I do? Some on linux IRC are telling me reboot but dont' kill pid which makes no sense at all.

Code:
[mysqld]
port = 3306
skip-locking
skip-innodb
query_cache_limit=1M
query_cache_size=32M
query_cache_type=1
max_connections=500
interactive_timeout=100
wait_timeout=100
connect_timeout=10
thread_cache_size=32
key_buffer_size=128M
join_buffer_size=1M
max_allowed_packet=4M
table_cache=2048
#record_buffer=1M <- replaced with read_buffer_size
sort_buffer_size=2M
read_buffer_size=1M
max_connect_errors=10
# Try number of CPU's*2 for thread_concurrency
thread_concurrency=4
myisam_sort_buffer_size=32M
#log-bin
server-id=1

[mysqld_safe]
open_files_limit = 8192
err-log=/var/log/mysqld.log
pid-file=/var/lib/mysql/mysql.pid

[mysqldump]
quick
max_allowed_packet = 16M

[mysql]
no-auto-rehash
# Remove the next comment character if you are not familiar with SQL
#safe-updates

[isamchk]
key_buffer_size = 64M
sort_buffer_size = 64M
read_buffer_size = 4M
write_buffer_size = 4M

[myisamchk]
key_buffer_size = 64M
sort_buffer_size = 64M
read_buffer_size = 4M
write_buffer_size = 4M

[mysqlhotcopy]
interactive-timeout
 

theuruguayan

Active Member
PartnerNOC
Jul 6, 2003
37
0
156
is part of kernel, and you are not going to be avail to kill it. So a reboot is the way to go.

if i was you i add more cache to my my.cnf settings, so it reduce the load (query/thread).

It for sure needs some optimization. Which os you running?
 

anup123

Well-Known Member
Mar 29, 2004
889
1
168
This Planet
kjournald: On the ext3 filesystem there are a few different modes of journaling Any idea which mode is your system using (journal, ordered, writeback)?

httpd: Do you have mod_perl enabled in your httpd?
Persistent connections if being used results in so many mysql processes. Normal. I have 50 at any instant.

Anup
 
Last edited:
S

Secret Agent

Guest
mod_perl not found / enabled in httpd.conf

[email protected] [~]# mount
/dev/hda2 on / type ext3 (rw,usrquota)
none on /proc type proc (rw)
none on /sys type sysfs (rw)
none on /dev/pts type devpts (rw,gid=5,mode=620)
usbfs on /proc/bus/usb type usbfs (rw)
/dev/hda1 on /boot type ext3 (rw)
none on /dev/shm type tmpfs (rw)
/dev/hdb1 on /backup type ext3 (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
/usr/tmpDSK on /tmp type ext3 (rw,noexec,nosuid,loop=/dev/loop0)
/tmp on /var/tmp type none (rw,noexec,nosuid,bind)


Any help appreciated. Thanks
 
S

Secret Agent

Guest
theuruguayan said:
is part of kernel, and you are not going to be avail to kill it. So a reboot is the way to go.

if i was you i add more cache to my my.cnf settings, so it reduce the load (query/thread).

It for sure needs some optimization. Which os you running?
Which line for query? limit? size?
 

anup123

Well-Known Member
Mar 29, 2004
889
1
168
This Planet
Secret Agent said:
mod_perl not found / enabled in httpd.conf

[email protected] [~]# mount
/dev/hda2 on / type ext3 (rw,usrquota)
none on /proc type proc (rw)
none on /sys type sysfs (rw)
none on /dev/pts type devpts (rw,gid=5,mode=620)
usbfs on /proc/bus/usb type usbfs (rw)
/dev/hda1 on /boot type ext3 (rw)
none on /dev/shm type tmpfs (rw)
/dev/hdb1 on /backup type ext3 (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
/usr/tmpDSK on /tmp type ext3 (rw,noexec,nosuid,loop=/dev/loop0)
/tmp on /var/tmp type none (rw,noexec,nosuid,bind)


Any help appreciated. Thanks
mode should be available in dmesg logs -- something like this:
EXT3-fs: mounted filesystem with ordered data mode in case ordered mode is used

If that be it -- reboot would be the way to go.

regarding mysql : if using persistent connections (php scripts since u do not have mod_perl), then there must be somewhere where you can disable persistent connections in php configs

Just a shot in the dark but probaly workable.

Thnx
Anup
 

anup123

Well-Known Member
Mar 29, 2004
889
1
168
This Planet
mysql skip locking is default for Linux as Linux file locking are not yet safe. Nothing to be concerned about IMHO. The only case when you can't use --skip-locking is if you run multiple MySQL servers (not clients) on the same data, which isn't the case here.

Anup