mysql killing server. high load. 1,000s of connections from localhost.

[nat]

This just started today.

I get hundreds of these when running netstat -na

tcp        0      0 209.123.123.123:3306     209.123.123.123:51530    TIME_WAIT   
tcp        0      0 209.123.123.123:3306     209.123.123.123:51786    TIME_WAIT   
tcp        0      0 209.123.123.123:3306     209.123.123.123:52042    TIME_WAIT   
tcp        0      0 209.123.123.123:3306     209.123.123.123:50250    TIME_WAIT   
tcp        0      0 209.123.123.123:3306     209.123.123.123:50506    TIME_WAIT   
tcp        0      0 209.123.123.123:3306     209.123.123.123:50762    TIME_WAIT   
tcp        0      0 209.123.123.123:3306     209.123.123.123:51018    TIME_WAIT   
tcp        0      0 209.123.123.123:3306     209.123.123.123:54346    TIME_WAIT   
tcp        0      0 209.123.123.123:3306     209.123.123.123:53322    TIME_WAIT   
tcp        0      0 209.123.123.123:3306     209.123.123.123:53578    TIME_WAIT   
tcp        0      0 209.123.123.123:3306     209.123.123.123:53834    TIME_WAIT   
tcp        0      0 209.123.123.123:3306     209.123.123.123:54090    TIME_WAIT 
ect.....
There are so many it takes over 5 mintues to print out.


Load has gone crazy.

209.123.123.123 is my ip.

Restarted mysql. Upgraded Cpanel. Rebooted server. Still doing it.


-----------------------


"mysqladmin processlist" normally doesn't show any mysql processes running.

+--------+------+-----------+----+---------+------+-------+------------------+
| Id     | User | Host      | db | Command | Time | State | Info             |
+--------+------+-----------+----+---------+------+-------+------------------+
| 110284 | root | localhost |    | Query   | 0    |       | show processlist |
+--------+------+-----------+----+---------+------+-------+------------------+


-----------------------


Cpanel showmysqlcpuusage only shows this

root  61.76 14.91 2.0 

Top Process %CPU 72.0 netstat -nlp 
Top Process %CPU 31.9 /usr/bin/perl /usr/local/cpanel/cpanellogd 
Top Process %CPU 13.6 /usr/bin/perl /usr/local/cpanel/cpanellogd 
mysql  12.90 6.58 0.0 
Top Process %CPU 17.5 /usr/sbin/mysqld --basedir/ --datadir/var/lib/mysql --usermysql --pid-file/var/lib/mysql/andromeda.sslnetworks.com.pid --skip-locking 
Top Process %CPU 10.0 /usr/sbin/mysqld --basedir/ --datadir/var/lib/mysql --usermysql --pid-file/var/lib/mysql/andromeda.sslnetworks.com.pid --skip-locking 

The rest are below 0.63


-----------------------

top

 5:07pm  up 23 min,  2 users,  load average: 16.80, 12.14, 9.02
187 processes: 182 sleeping, 3 running, 0 zombie, 1 stopped
CPU states: 44.0% user, 23.5% system,  0.0% nice, 32.4% idle
Mem:   509920K av,  498376K used,   11544K free,       0K shrd,  138032K buff
Swap: 1052216K av,   16628K used, 1035588K free                  127220K cached

  PID USER     PRI  NI  SIZE  RSS SHARE STAT %CPU %MEM   TIME COMMAND
 1186 mysql      9   0 40092  38M  1392 S    12.5  7.7   0:48 mysqld
  722 root       4   0 17804  16M  1016 S     8.5  3.2   1:02 httpd
 8341 user1      9   0  4044 4044  2268 S     1.7  0.7   0:00 php
 8347 user2     17   0  4088 4088  2196 S     1.7  0.8   0:00 php
    6 root       9   0     0    0     0 SW    1.3  0.0   0:17 kscand
 1141 root       9   0  1108 1108   952 S     0.5  0.2   0:09 rmcfiles
  483 root       9   0   572  536   460 S     0.3  0.1   0:01 syslogd
  566 root       9   0  1384 1232  1160 S     0.3  0.2   0:03 sshd
 8326 root      11   0  1108 1108   828 R     0.3  0.2   0:00 top
  590 root       9   0  2200 1840  1404 S     0.1  0.3   0:01 antirelayd
  663 root       9   0  2404 1212   748 S     0.1  0.2   0:00 cppop
 8171 user4      9   0  3008 2164  1264 S     0.1  0.4   0:00 cpaneld
 8259 user5      9   0  3336 2856  1440 R     0.1  0.5   0:00 cppop
    1 root       6   0   484  448   420 S     0.0  0.0   0:04 init
    2 root       9   0     0    0     0 SW    0.0  0.0   0:00 keventd
    3 root       9   0     0    0     0 SW    0.0  0.0   0:00 kapmd
   69 root       9   0     0    0     0 SW    0.0  0.0   0:00 khubd
  139 root       9   0     0    0     0 SW    0.0  0.0   0:00 kjournald
  140 root       9   0     0    0     0 SW    0.0  0.0   0:00 kjournald
  141 root       9   0     0    0     0 SW    0.0  0.0   0:01 kjournald


------------------------------


Any sugesstion why there are so many of these and how to fix?
 
tcp        0      0 209.123.123.123:3306     209.123.123.123:54346    TIME_WAIT 

cPanel.net Support Ticket Number:

[nat]

reinstalling zend helped by bringing down the load to normal but did not stop the thousands of connections shown in netstat.


using phpinfo i found my php.ini file to be:
/usr/local/Zend/etc/php.ini


commenting out everything that deals with zend with a ; fixed it:

;zend_optimizer.optimization_level=15
;zend_extension_manager.optimizer=/usr/local/Zend/lib/Optimizer-2.1.0
;zend_extension_manager.optimizer_ts=/usr/local/Zend/lib/Optimizer_TS-2.1.0
;zend_extension=/usr/local/Zend/lib/ZendExtensionManager.so
;zend_extension_ts=/usr/local/Zend/lib/ZendExtensionManager_TS.so

cPanel.net Support Ticket Number:

[erwinfa]

Have you try to tweak your mysql on /etc/my.cnf ?

cPanel.net Support Ticket Number:

[nat]

[mysqld]
# set-variable = max_connections=350

# ### custom -everything below this line ####
set-variable = max_connections=50
set-variable = interactive_timeout=100
set-variable = wait_timeout=200
set-variable = connect_timeout=30
set-variable = thread_cache_size=8
set-variable = key_buffer=192M
set-variable = max_allowed_packet=8M
set-variable = table_cache=1024
set-variable = sort_buffer=2M
set-variable = record_buffer=2M
set-variable = thread_cache=8
# Try number of CPU's*2 for thread_concurrency
# set-variable = thread_concurrency=8
set-variable = myisam_sort_buffer_size=32M
set-variable = query_cache_limit=1M
set-variable = query_cache_size=16M
set-variable = query_cache_type=1
#tmpdir=/tmp

[safe_mysqld]
err-log=/var/log/mysqld.log
open_files_limit=8192

[mysqldump]
quick
set-variable = max_allowed_packet=16M

cPanel.net Support Ticket Number: