httpd/apache going crazy, loads up to 40 and higher.

[DrGreen]

I just moved to another server but when i start apache my loads go crazy
It will start at zero and slowly grow to 40 within 10 min and will go further if i dont kill it

When i restart apache it falls back to 2. and starts getting crazy again.

any thoughts on this are appreciated

System:

Xeon EM64T 2.8 GHz
1024MB ECC Registered ram
SATA Hard disks


------------------------------------------------------------------

13:37:27 up 4 days, 17:13, 1 user, load average: 26.91, 17.23, 9.26
127 processes: 92 sleeping, 33 running, 1 zombie, 1 stopped
CPU states: cpu user nice system irq softirq iowait idle
total 200.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0%
cpu00 100.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0%
cpu01 100.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0%
Mem: 1001384k av, 975200k used, 26184k free, 0k shrd, 175964k buff
638460k actv, 165076k in_d, 13532k in_c
Swap: 2112508k av, 80504k used, 2032004k free 504872k cached

PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME CPU COMMAND
23673 root 15 0 12892 12M 2500 S 199.9 1.2 30:06 0 httpd
24547 root 18 0 1412 1412 964 R 0.3 0.1 0:00 0 top
1 root 15 0 124 88 52 S 0.0 0.0 0:33 0 init
2 root RT 0 0 0 0 SW 0.0 0.0 0:00 0 migration/0
3 root RT 0 0 0 0 SW 0.0 0.0 0:00 1 migration/1
4 root 15 0 0 0 0 SW 0.0 0.0 0:04 0 keventd
5 root 34 19 0 0 0 SWN 0.0 0.0 0:00 0 ksoftirqd/0
6 root 34 19 0 0 0 SWN 0.0 0.0 0:00 1 ksoftirqd/1
9 root 15 0 0 0 0 SW 0.0 0.0 0:00 0 bdflush
7 root 15 0 0 0 0 SW 0.0 0.0 0:41 1 kswapd
8 root 15 0 0 0 0 SW 0.0 0.0 0:06 1 kscand
10 root 15 0 0 0 0 SW 0.0 0.0 0:15 1 kupdated
11 root 25 0 0 0 0 SW 0.0 0.0 0:00 1 mdrecoveryd
18 root 25 0 0 0 0 SW 0.0 0.0 0:00 1 scsi_eh_0
19 root 25 0 0 0 0 SW 0.0 0.0 0:00 0 scsi_eh_1
22 root 15 0 0 0 0 SW 0.0 0.0 0:04 0 kjournald
77 root 25 0 0 0 0 SW 0.0 0.0 0:00 0 khubd
192 root 15 0 0 0 0 SW 0.0 0.0 0:00 1 kjournald
193 root 15 0 0 0 0 SW 0.0 0.0 0:23 1 kjournald
194 root 15 0 0 0 0 SW 0.0 0.0 0:00 0 kjournald
195 root 15 0 0 0 0 SW 0.0 0.0 0:01 0 kjournald
196 root 15 0 0 0 0 SW 0.0 0.0 0:07 0 kjournald
197 root 15 0 0 0 0 SW 0.0 0.0 0:18 1 kjournald
4224 root 15 0 172 148 100 S 0.0 0.0 0:09 0 mdadm
5064 root 15 0 580 336 212 S 0.0 0.0 0:00 0 sshd
5107 root 15 0 360 340 248 S 0.0 0.0 0:04 1 crond

--------------------------------------------------------------

ps -aux

-----------------------------------------------------------------

root 23673 0.0 1.2 38632 12892 ? S 13:21 0:00 /usr/local/apache/bin/httpd -DSSL
nobody 23679 28.9 1.3 38768 13076 ? R 13:21 4:59 /usr/local/apache/bin/httpd -DSSL
nobody 23680 29.6 1.2 38632 12968 ? R 13:21 5:07 /usr/local/apache/bin/httpd -DSSL
nobody 23681 33.6 1.2 38632 12968 ? R 13:21 5:48 /usr/local/apache/bin/httpd -DSSL
nobody 23682 6.4 1.3 38776 13068 ? R 13:21 1:06 /usr/local/apache/bin/httpd -DSSL
nobody 23683 3.6 1.3 38784 13068 ? R 13:21 0:38 /usr/local/apache/bin/httpd -DSSL
nobody 23697 3.3 1.3 38776 13084 ? R 13:22 0:33 /usr/local/apache/bin/httpd -DSSL
nobody 23699 6.7 1.3 38784 13060 ? R 13:22 1:07 /usr/local/apache/bin/httpd -DSSL
nobody 23700 0.0 1.3 38780 13148 ? S 13:22 0:00 /usr/local/apache/bin/httpd -DSSL
nobody 23755 16.2 1.2 38632 12992 ? R 13:23 2:32 /usr/local/apache/bin/httpd -DSSL
nobody 23764 3.6 1.3 38644 13020 ? R 13:23 0:33 /usr/local/apache/bin/httpd -DSSL
martinf 23782 0.0 0.9 31124 9040 ? SN 13:23 0:00 cpanellogd - http logs for martinf
martinf 23783 0.0 0.9 31124 9048 ? SN 13:23 0:00 cpanellogd - http logs for martinf
martinf 23784 0.0 0.0 4240 428 ? SN 13:23 0:00 /usr/local/cpanel/bin/logrunner 2.0 /usr/local/cpanel/3rdparty/bin/english
martinf 23785 0.0 0.1 12264 1036 ? TN 13:23 0:00 /usr/local/cpanel/3rdparty/bin/english/webalizer -N 10 -D /home/martinf/tm
nobody 23840 4.7 1.3 38784 13056 ? R 13:27 0:33 /usr/local/apache/bin/httpd -DSSL
nobody 23841 19.8 1.2 38632 12988 ? R 13:27 2:18 /usr/local/apache/bin/httpd -DSSL
nobody 23915 5.8 1.3 38784 13068 ? R 13:29 0:33 /usr/local/apache/bin/httpd -DSSL
nobody 24172 7.2 1.2 38632 12992 ? R 13:30 0:37 /usr/local/apache/bin/httpd -DSSL
nobody 24173 7.7 1.3 38784 13056 ? R 13:30 0:40 /usr/local/apache/bin/httpd -DSSL
nobody 24190 6.7 1.2 38632 12992 ? R 13:30 0:33 /usr/local/apache/bin/httpd -DSSL
nobody 24191 6.7 1.2 38632 12992 ? R 13:30 0:33 /usr/local/apache/bin/httpd -DSSL
nobody 24268 9.1 1.2 38632 12972 ? R 13:32 0:37 /usr/local/apache/bin/httpd -DSSL
nobody 24280 5.9 1.3 38784 13056 ? R 13:32 0:23 /usr/local/apache/bin/httpd -DSSL
nobody 24281 4.6 1.3 38784 13056 ? R 13:32 0:18 /usr/local/apache/bin/httpd -DSSL
nobody 24282 7.4 1.2 38632 12992 ? R 13:32 0:29 /usr/local/apache/bin/httpd -DSSL
nobody 24283 5.0 1.2 38632 12992 ? R 13:32 0:19 /usr/local/apache/bin/httpd -DSSL
nobody 24284 0.5 1.3 38784 13068 ? R 13:32 0:02 /usr/local/apache/bin/httpd -DSSL
nobody 24293 0.0 1.3 38784 13060 ? S 13:32 0:00 /usr/local/apache/bin/httpd -DSSL
nobody 24304 6.9 1.3 38784 13056 ? R 13:33 0:22 /usr/local/apache/bin/httpd -DSSL
nobody 24307 9.1 1.2 38632 12992 ? R 13:33 0:30 /usr/local/apache/bin/httpd -DSSL
nobody 24308 5.4 1.3 38784 13056 ? R 13:33 0:17 /usr/local/apache/bin/httpd -DSSL
iraklis 24309 0.0 0.2 24092 2692 ? SL 13:33 0:00 proftpd: iraklis - 84.120.1.144: IDLE
nobody 24323 0.0 1.3 38784 13056 ? S 13:33 0:00 /usr/local/apache/bin/httpd -DSSL
nobody 24352 5.1 1.2 38632 12992 ? R 13:34 0:12 /usr/local/apache/bin/httpd -DSSL
nobody 24362 5.9 1.2 38632 12992 ? R 13:35 0:13 /usr/local/apache/bin/httpd -DSSL
nobody 24384 3.8 1.3 38916 13192 ? R 13:35 0:08 /usr/local/apache/bin/httpd -DSSL
nobody 24443 3.7 1.3 38784 13056 ? R 13:36 0:06 /usr/local/apache/bin/httpd -DSSL
nobody 24451 9.4 1.2 38632 12992 ? R 13:36 0:14 /usr/local/apache/bin/httpd -DSSL
nobody 24452 8.6 1.2 38632 12972 ? R 13:36 0:13 /usr/local/apache/bin/httpd -DSSL
nobody 24453 2.6 1.2 38632 12992 ? R 13:36 0:04 /usr/local/apache/bin/httpd -DSSL
nobody 24514 0.0 0.2 23036 2308 ? SL 13:36 0:00 proftpd: connected: 127.0.0.1 (127.0.0.1:54047)
nobody 24526 3.2 1.2 38632 12992 ? R 13:36 0:03 /usr/local/apache/bin/httpd -DSSL
nobody 24531 0.0 1.2 38632 12984 ? S 13:37 0:00 /usr/local/apache/bin/httpd -DSSL
jesse 24575 0.0 0.2 24080 2672 ? SL 13:38 0:00 proftpd: jesse - 60.226.22.131: IDLE
nobody 24587 0.0 1.2 38632 12904 ? S 13:38 0:00 /usr/local/apache/bin/httpd -DSSL
nobody 24592 0.0 1.2 38632 12904 ? S 13:38 0:00 /usr/local/apache/bin/httpd -DSSL
root 24593 0.0 0.0 8472 932 pts/1 R 13:38 0:00 ps -aux

[LA_fro]

Have you checked were these connections are going to? This may be a DoS attack on a specific site. Or this could be a random thing such as a slashdoting of your server.

[DrGreen]

no, not sure how todo that.

but it happend right after the transfer so it shouldnt be a DoS attack
more likely to be a faulty script/apache ore something

[rodstewart]

is that an RH9 server?

[DrGreen]

Centos 3.3 based on Redhat enterprise editon

[rodstewart]

if your server go out of load next time login into shell and make the following

ps axm > ps.txt

and poste the ps.txt here or send me the file to abuse@shells24.com

in my way i´ve find out that tomcat makes the high server load.

[DrGreen]

I dont have tomcat installed

--------------------
PID TTY STAT TIME COMMAND
1 ? S 0:11 init
2 ? SW 0:00 [migration/0]
3 ? SW 0:00 [migration/1]
4 ? SW 0:00 [keventd]
5 ? SWN 0:00 [ksoftirqd/0]
6 ? SWN 0:00 [ksoftirqd/1]
9 ? SW 0:00 [bdflush]
7 ? SW 0:03 [kswapd]
8 ? SW 0:01 [kscand]
10 ? SW 0:15 [kupdated]
11 ? SW 0:00 [mdrecoveryd]
18 ? SW 0:00 [scsi_eh_0]
19 ? SW 0:00 [scsi_eh_1]
22 ? SW 0:02 [kjournald]
77 ? SW 0:00 [khubd]
189 ? SW 0:00 [kjournald]
190 ? SW 0:04 [kjournald]
191 ? SW 0:00 [kjournald]
192 ? SW 0:01 [kjournald]
193 ? SW 0:07 [kjournald]
194 ? SW 0:09 [kjournald]
4331 ? S 0:02 syslogd -m 0
4335 ? S 0:00 klogd -x
4346 ? S 0:00 mdadm --monitor --scan -f
4411 ? SL 0:01 proftpd: (accepting connections)
5384 ? S 0:00 /usr/sbin/sshd
5398 ? S 0:00 xinetd -stayalive -pidfile /var/run/xinetd.pid
5423 ? S 0:06 chkservd
5610 ? S 0:02 crond
5913 ? S 0:14 /usr/bin/perl /usr/local/cpanel/bin/eximstats
5914 ? S 0:21 /usr/bin/perl /usr/local/cpanel/bin/eximstats
5970 ? S 0:00 entropychat
5974 ? S 0:00 /usr/local/cpanel/bin/startmelange
6004 ? S 0:01 /usr/bin/stunnel-4.04local /usr/local/cpanel/etc/stun
6033 ? S 0:00 /usr/sbin/portsentry -tcp
6052 tty1 S 0:00 /sbin/mingetty tty1
6053 tty2 S 0:00 /sbin/mingetty tty2
6054 tty3 S 0:00 /sbin/mingetty tty3
6055 tty4 S 0:00 /sbin/mingetty tty4
6056 tty5 S 0:00 /sbin/mingetty tty5
6057 tty6 S 0:00 /sbin/mingetty tty6
6058 ? S 0:00 /usr/bin/gdm-binary -nodaemon
8368 ? S 0:00 imapd
8369 ? S 0:00 imapd
8370 ? S 0:00 imapd
8371 ? S 0:00 imapd
8372 ? S 0:00 imapd
8373 ? S 0:01 imapd
13333 ? S 0:00 /usr/bin/gdm-binary -nodaemon
20298 ? S 0:00 /bin/sh /usr/bin/mysqld_safe --datadir=/var/lib/mysql
20318 ? S 0:00 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql
20319 ? S 0:00 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql
20320 ? S 0:00 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql
20336 ? S 0:02 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql
20385 ? S 0:02 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql
20591 ? S 0:03 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql
20777 ? S 0:00 /usr/sbin/clamd
20783 ? S 0:05 /usr/sbin/exim -bd -q60m
20790 ? S 0:00 /usr/sbin/exim -tls-on-connect -bd -oX 465
20823 ? S 0:04 antirelayd
20838 ? S 0:00 /usr/bin/spamd -d --allowed-ips=127.0.0.1 --pidfile=/
20844 ? S 0:00 /usr/sbin/exim -oX 26 -bd
20845 ? S 0:05 spamd child
20846 ? S 0:06 spamd child
20847 ? S 0:06 spamd child
20848 ? S 0:05 spamd child
20849 ? S 0:06 spamd child
20881 ? SN 0:50 cpanellogd - sleeping for logs
20885 ? SN 0:42 cpanellogd - sleeping for logs
20891 ? S 0:05 cppop - accepting on port 110
20917 ? S 0:01 cpsrvd - waiting for connections
20921 ? S 0:02 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql
21009 ? S 0:02 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql
22183 ? S 0:02 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql
21668 ? S 0:00 imapd
30650 ? S 0:00 /usr/sbin/named -u named
30651 ? S 0:02 /usr/sbin/named -u named
30654 ? S 0:02 /usr/sbin/named -u named
30655 ? S 0:00 /usr/sbin/named -u named
30656 ? S 0:02 /usr/sbin/named -u named
27547 ? R 0:00 sshd: root@pts/0
27643 pts/0 S 0:00 -bash
27726 pts/0 S 0:00 su -
27734 pts/0 S 0:00 -bash
28696 ? S 0:04 /usr/local/apache/bin/httpd -DSSL
6434 ? R 2:59 /usr/local/apache/bin/httpd -DSSL
6436 ? R 6:45 /usr/local/apache/bin/httpd -DSSL
6437 ? R 4:16 /usr/local/apache/bin/httpd -DSSL
6438 ? R 3:14 /usr/local/apache/bin/httpd -DSSL
6440 ? R 2:46 /usr/local/apache/bin/httpd -DSSL
6449 ? R 1:14 /usr/local/apache/bin/httpd -DSSL
6452 ? R 6:13 /usr/local/apache/bin/httpd -DSSL
6714 ? R 2:41 /usr/local/apache/bin/httpd -DSSL
7019 ? R 1:05 /usr/local/apache/bin/httpd -DSSL
7085 ? R 2:16 /usr/local/apache/bin/httpd -DSSL
7095 ? R 2:23 /usr/local/apache/bin/httpd -DSSL
7188 ? R 2:14 /usr/local/apache/bin/httpd -DSSL
7190 ? R 2:14 /usr/local/apache/bin/httpd -DSSL
7608 ? R 0:40 /usr/local/apache/bin/httpd -DSSL
7653 ? R 1:09 /usr/local/apache/bin/httpd -DSSL
7785 ? S 0:00 /usr/local/apache/bin/httpd -DSSL
7793 ? S 0:00 /usr/local/apache/bin/httpd -DSSL
7794 ? R 0:53 /usr/local/apache/bin/httpd -DSSL
7838 ? S 0:00 /usr/local/apache/bin/httpd -DSSL
7849 ? S 0:00 /usr/local/apache/bin/httpd -DSSL
7850 ? R 0:08 /usr/local/apache/bin/httpd -DSSL
7851 ? S 0:00 /usr/local/apache/bin/httpd -DSSL
7926 ? R 0:49 /usr/local/apache/bin/httpd -DSSL
7927 ? S 0:00 /usr/local/apache/bin/httpd -DSSL
7928 ? R 0:35 /usr/local/apache/bin/httpd -DSSL
7960 ? R 0:20 /usr/local/apache/bin/httpd -DSSL
7990 ? S 0:00 /usr/local/apache/bin/httpd -DSSL
7991 ? S 0:00 /usr/local/apache/bin/httpd -DSSL
7992 ? R 0:38 /usr/local/apache/bin/httpd -DSSL
8059 ? S 0:00 /usr/local/apache/bin/httpd -DSSL
8120 ? S 0:00 /usr/local/apache/bin/httpd -DSSL
8436 ? S 0:00 /usr/local/apache/bin/httpd -DSSL
8524 ? SL 0:00 proftpd: connected: 127.0.0.1 (127.0.0.1:54401)
8608 ? S 0:00 /usr/local/apache/bin/httpd -DSSL
8610 pts/0 R 0:00 ps axm

[irchs]

Check your .htaccess files.

I have noticed that when a user tries to use hotlink protection, or do anything to the .htaccess file, the server will loop forever when accessing the directory with that .htaccess file.

Definate bug in cpanel.

Jan

[rodstewart]

have you reportet that bug to the bugzilla of cpanel http://bugzilla.cpanel.net/?
If not can you do that please and give me the ID of the report?

thanks