Server load goes every weekday on some part to more then 150.00 Why?

[boyforeigner]

3 of my servers has it Server load going every weekday on some part of the day to 150.00 or more, Why?

Goes like this:

the server is working normally, and for no reason that I can see the server load goes to 150 so I need to reboot the server and got normal again.


Does anyone is having this problem, is there any solution ?

Thanks

 

[tprice42]

what does top say the heaviest processes are at the time? Also check your dmesg for faults there.

 

[boyforeigner]

see bellow

I can't see top when this occurs but in the :

mysql/ cpu usages says:
root 3.44 31.55 1.9
Top Process %CPU 24.0 /usr/sbin/exim -Mc 1CZ4Kc-0002Oy-Go
Top Process %CPU 17.0 /usr/sbin/exim -Mc 1CZ99R-0005xV-Ge
Top Process %CPU 11.0 /usr/sbin/exim -Mc 1CZ6pU-0005dP-9W

mailnull 3.10 4.58 0.0
Top Process %CPU 41.0 [exim ]
Top Process %CPU 25.0 [exim ]
Top Process %CPU 22.0 /usr/sbin/exim -bd -q60m

mysql 1.17 87.73 0.0
Top Process %CPU 26.8 /usr/sbin/mysqld --basedir/ --datadir/var/lib/mysql --usermysql --pid-file/var/lib/mysql/server.xxxxxx.com.pid --skip-locking
Top Process %CPU 24.5 /usr/sbin/mysqld --basedir/ --datadir/var/lib/mysql --usermysql --pid-file/var/lib/mysql/server.xxxxxx.com.pid --skip-locking
Top Process %CPU 11.2 /usr/sbin/mysqld --basedir/ --datadir/var/lib/mysql --usermysql --pid-file/var/lib/mysql/server.xxxxxx.com.pid --skip-locking

tokubras xxxxxxxx.com 0.74 17.70 0.1
Top Process %CPU 95.1 /usr/bin/perl /usr/local/cpanel/3rdparty/bin/awstats.pl -config.xxxxxx.com -update





sar -q shows
11:10:00 AM 0 296 0.92 0.63
11:20:01 AM 2 304 1.60 2.13
11:30:05 AM 0 332 5.86 2.48
11:40:24 AM 0 387 16.82 15.74
11:51:09 AM 0 523 39.07 34.92
12:05:04 PM 0 494 71.48 77.55
12:13:11 PM 0 628 89.91 73.30
12:37:24 PM 0 725 129.28 139.09
12:44:44 PM 1 538 55.99 99.20
12:46:16 PM 0 330 30.52 78.83
12:50:00 PM 2 257 2.04 39.09
01:00:00 PM 0 267 0.54 6.91
01:10:01 PM 1 262 0.12 1.24
01:20:00 PM 0 285 0.47 0.60


in the dmesg I see:

00 SYN URGP=0 OPT (0204056401010402)
ip_conntrack_tcp: INVALID: invalid SYN (ignored) SRC=200.159.209.223 DST=XX.XXX.XXX.XXX LEN=48 TOS=0x00 PREC=0x00 TTL=113 ID=51
830 DF PROTO=TCP SPT=62157 DPT=110 SEQ=2511585606 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B401010402)
ip_conntrack_tcp: INVALID: invalid SYN (ignored) SRC=200.174.81.119 DST=XX.XXX.XXX.XXX LEN=48 TOS=0x00 PREC=0x00 TTL=109 ID=106
14 DF PROTO=TCP SPT=1902 DPT=25 SEQ=1861922691 ACK=0 WINDOW=65535 RES=0x00 SYN URGP=0 OPT (020405B401010402)
ip_conntrack_tcp: INVALID: invalid SYN (ignored) SRC=200.102.87.43 DST=XX.XXX.XXX.XXX LEN=48 TOS=0x00 PREC=0x00 TTL=117 ID=5285
2 DF PROTO=TCP SPT=3211 DPT=80 SEQ=1919841353 ACK=0 WINDOW=16384 RES=0x00 SYN URGP=0 OPT (020405AC01010402)
ip_conntrack_tcp: INVALID: invalid SYN (ignored) SRC=212.113.164.98 DST=XX.XXX.XXX.XXX LEN=48 TOS=0x00 PREC=0x00 TTL=52 ID=3849
0 DF PROTO=TCP SPT=58732 DPT=80 SEQ=118567052 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT (020405B401010402)
ip_conntrack_tcp: INVALID: invalid SYN (ignored) SRC=200.154.55.228 DST=XX.XXX.XXX.XXX LEN=60 TOS=0x00 PREC=0x00 TTL=49 ID=2670
8 DF PROTO=TCP SPT=44779 DPT=25 SEQ=111775196 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT (020405B40402080A1B94E6BB000000000103
0300)
ip_conntrack_tcp: INVALID: invalid SYN (ignored) SRC=200.217.5.171 DST=XX.XXX.XXX.XXX LEN=44 TOS=0x00 PREC=0x00 TTL=111 ID=3423
0 DF PROTO=TCP SPT=3035 DPT=80 SEQ=1341212634 ACK=0 WINDOW=8192 RES=0x00 SYN URGP=0 OPT (020405B4)
ip_conntrack_tcp: INVALID: invalid SYN (ignored) SRC=200.175.65.93 DST=XX.XXX.XXX.XXX LEN=60 TOS=0x00 PREC=0x00 TTL=48 ID=29870
DF PROTO=TCP SPT=50895 DPT=80 SEQ=2329425615 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT (020405B40402080A014B4E1A000000000103
0307)
ip_conntrack_tcp: INVALID: invalid SYN (ignored) SRC=200.101.121.113 DST=XX.XXX.XXX.XXX LEN=60 TOS=0x00 PREC=0x00 TTL=45 ID=479
81 DF PROTO=TCP SPT=29907 DPT=80 SEQ=361616334 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT (020405B40402080A029469CE00000000010
30300)
ip_conntrack_tcp: INVALID: invalid SYN (ignored) SRC=10.150.150.77 DST=XX.XXX.XXX.XXX LEN=60 TOS=0x00 PREC=0x00 TTL=45 ID=30134
DF PROTO=TCP SPT=58650 DPT=25 SEQ=322108490 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT (020405B40402080A7F189BFE0000000001030
300)
NET: 1 messages suppressed.
ip_conntrack_tcp: INVALID: invalid SYN (ignored) SRC=200.242.229.2 DST=XX.XXX.XXX.XXX LEN=48 TOS=0x00 PREC=0x00 TTL=109 ID=2602
1 PROTO=TCP SPT=49505 DPT=80 SEQ=17931973 ACK=0 WINDOW=8192 RES=0x00 SYN URGP=0 OPT (020405B401010402)
NET: 1 messages suppressed.
ip_conntrack_tcp: INVALID: invalid SYN (ignored) SRC=10.150.150.77 DST=XX.XXX.XXX.XXX LEN=60 TOS=0x00 PREC=0x00 TTL=43 ID=30135
DF PROTO=TCP SPT=58650 DPT=25 SEQ=322108490 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT (020405B40402080A7F18B3FE0000000001030
300)
ip_conntrack_tcp: INVALID: invalid SYN (ignored) SRC=200.203.74.97 DST=XX.XXX.XXX.XXX LEN=48 TOS=0x00 PREC=0x00 TTL=115 ID=3409
3 DF PROTO=TCP SPT=50581 DPT=110 SEQ=22583588 ACK=0 WINDOW=8192 RES=0x00 SYN URGP=0 OPT (020405B401010402)
ip_conntrack_tcp: INVALID: invalid SYN (ignored) SRC=200.157.248.57 DST=XX.XXX.XXX.XXX LEN=48 TOS=0x00 PREC=0x00 TTL=116 ID=435
49 DF PROTO=TCP SPT=1394 DPT=80 SEQ=1855846 ACK=0 WINDOW=8192 RES=0x00 SYN URGP=0 OPT (020405B401010402)
ip_conntrack_tcp: INVALID: invalid SYN (ignored) SRC=200.165.18.225 DST=XX.XXX.XXX.XXX LEN=48 TOS=0x00 PREC=0x00 TTL=111 ID=874
1 DF PROTO=TCP SPT=10117 DPT=80 SEQ=3730088550 ACK=0 WINDOW=16384 RES=0x00 SYN URGP=0 OPT (020405B401010402)
ip_conntrack_tcp: INVALID: invalid SYN (ignored) SRC=10.150.150.77 DST=XX.XXX.XXX.XXX LEN=60 TOS=0x00 PREC=0x00 TTL=43 ID=30136
DF PROTO=TCP SPT=58650 DPT=25 SEQ=322108490 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT (020405B40402080A7F18E3FE0000000001030
300)



Can you figure out anything ?

 

[boyforeigner]

See the top

15:16:49 up 2 days, 23:24, 1 user, load average: 119.32, 94.55, 70.96
784 processes: 765 sleeping, 1 running, 14 zombie, 4 stopped
CPU states: cpu user nice system irq softirq iowait idle
total 4.8% 0.0% 4.4% 0.0% 0.0% 0.0% 190.4%
cpu00 2.6% 0.0% 2.7% 0.0% 0.0% 0.0% 94.6%
cpu01 2.2% 0.0% 1.8% 0.0% 0.0% 0.0% 95.8%
Mem: 447928k av, 443912k used, 4016k free, 0k shrd, 16404k buff
185572k active, 190012k inactive
Swap: 1052216k av, 617876k used, 434340k free 31836k cached

PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME CPU COMMAND
12295 mailnull 16 0 692 548 484 S 2.9 0.1 0:18 0 exim
31417 root 19 0 1732 1716 736 R 1.2 0.3 1:42 0 top
9367 root 1 0 12388 5200 5176 S 1.0 1.1 7:35 1 httpd
5 root 14 0 0 0 0 DW 0.6 0.0 5:21 1 kswapd
10580 root 9 0 3580 3580 3508 S 0.5 0.7 0:00 1 exim
10584 root 9 0 3732 3728 3508 S 0.1 0.8 0:00 1 exim
10587 benvinda 10 0 4372 2512 1948 S 0.1 0.5 0:00 0 cpsrvd
1 root 0 0 320 288 276 S 0.0 0.0 0:10 0 init
2 root 10 0 0 0 0 SW 0.0 0.0 0:05 0 keventd
3 root 19 19 0 0 0 SWN 0.0 0.0 0:00 0 ksoftirqd_CPU
4 root 18 19 0 0 0 SWN 0.0 0.0 0:00 1 ksoftirqd_CPU
6 root 9 0 0 0 0 SW 0.0 0.0 0:00 1 bdflush
7 root 9 0 0 0 0 SW 0.0 0.0 0:15 1 kupdated


Does anyone have any idea ?

 

[linux-image]

exim is the killer.. possible spamming. check with command

exim -bpc

you will see the number of mails in the mail queue.