whm crash my server

[Nikoms]

Hello everybody!

It seems that i have a little problem with WHM. I just upload a new website on my server. And it works, but, now when i'm going to the whm panel 2086), my server slow down. SSH comes very slow, and mysql crash (i receive a mail mysql failed @ Sun Dec 31 19:00:11 2006. A restart was attempted automagicly.)


here the first lines of a "top" during the connection :

top - 19:03:19 up 1:28, 1 user, load average: 17.89, 9.10, 5.05
Tasks: 100 total, 1 running, 99 sleeping, 0 stopped, 0 zombie
Cpu(s): 2.4% us, 2.7% sy, 0.0% ni, 0.0% id, 94.5% wa, 0.3% hi, 0.0% si
Mem: 239352k total, 236292k used, 3060k free, 416k buffers
Swap: 0k total, 0k used, 0k free, 7172k cached


The browser can't display whm. So i quit IE.

If I do nothing during 1 minutes, and then do "top", here is the result :

top - 19:08:37 up 1:33, 1 user, load average: 0.22, 3.60, 3.86
Tasks: 97 total, 2 running, 95 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.0% us, 0.5% sy, 0.0% ni, 99.5% id, 0.0% wa, 0.0% hi, 0.0% si
Mem: 239352k total, 234624k used, 4728k free, 1160k buffers
Swap: 0k total, 0k used, 0k free, 17692k cached




I really don't understand... I already made a upcp --force, but it didn't change anything.

Did anyone have the same problem? Do you know what can it be?

Thanks a lot!

EDIT :

Sometimes i got a "Lost connection to MySQL server during query" message and when i refresh i got "Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)"

Is that normal that i have this, in my "top"?

3007 nobody 16 0 27444 12m 348 S 2.0 5.4 0:08.54 /usr/local/apache/bin/httpd -DSSL
2634 nobody 16 0 27584 12m 348 S 1.9 5.4 0:05.61 /usr/local/apache/bin/httpd -DSSL
2580 nobody 16 0 26460 11m 344 S 1.9 5.0 0:03.06 /usr/local/apache/bin/httpd -DSSL
2582 nobody 16 0 27092 12m 440 S 1.9 5.3 0:08.51 /usr/local/apache/bin/httpd -DSSL
2583 nobody 16 0 26056 11m 348 S 1.9 4.8 0:06.07 /usr/local/apache/bin/httpd -DSSL
2585 nobody 16 0 26928 12m 348 S 1.8 5.2 0:05.98 /usr/local/apache/bin/httpd -DSSL
3030 nobody 16 0 27176 12m 348 S 1.8 5.2 0:04.25 /usr/local/apache/bin/httpd -DSSL
5894 nobody 16 0 24824 9.9m 516 R 0.8 4.2 0:00.22 /usr/local/apache/bin/httpd -DSSL
5886 nobody 16 0 25740 10m 376 R 0.7 4.7 0:00.24 /usr/local/apache/bin/httpd -DSSL
5910 nobody 17 0 24560 9976 468 R 0.6 4.2 0:00.17 /usr/local/apache/bin/httpd -DSSL

 

[mohit]

hi,
seems you have a busy site cause your usage shows Apache is serving many request.
they could be using a lot of mysql and finally crashing.

you can ask a sys admin to look into this properly cause memory usage seems high and swap usage is not visible.

you can however try these from ssh

mysqladmin processlist |wc -l

above will show you active mysql connections.

ps aux | head -1;ps aux --no-headers| sort -rn +3 | head
above would show you top memory consuming processes.

have you tried killing all mysql process and restarting the mysql service ?

see ya,
mohit

 

[Nikoms]

Thank you mohit (and happy new year )


I try you command in ssh :


ps aux | head -1;ps aux --no-headers| sort -rn +3 | head

root 2276 0.0 9.1 25652 21848 ? S 14:41 0:00 spamd child
root 2275 0.0 9.2 25916 22068 ? S 14:41 0:00 spamd child
root 2176 0.0 9.1 25652 21896 ? Ss 14:41 0:01 /usr/bin/spamd -d --allowed-ips=127.0.0.1 --pidfile=/var/run/spamd.pid --max-children=5
nobody 3261 0.3 5.2 27420 12548 ? S 14:46 0:13 /usr/local/apache/bin/httpd -DSSL
nobody 2621 0.2 5.3 27656 12784 ? S 14:41 0:12 /usr/local/apache/bin/httpd -DSSL
nobody 2620 0.3 5.3 27456 12860 ? S 14:41 0:15 /usr/local/apache/bin/httpd -DSSL
nobody 2614 0.3 5.4 27684 13076 ? S 14:41 0:14 /usr/local/apache/bin/httpd -DSSL
nobody 2568 0.3 5.2 27216 12620 ? S 14:41 0:16 /usr/local/apache/bin/httpd -DSSL
nobody 2539 0.2 5.2 27228 12560 ? S 14:41 0:13 /usr/local/apache/bin/httpd -DSSL
nobody 2538 0.3 5.2 27472 12552 ? S 14:41 0:15 /usr/local/apache/bin/httpd -DSSL

mysqladmin processlist |wc -l

Sometimes it's 5, sometimes 6 and sometimes 7

And then... Ite became very slow... And i get this

mysqladmin: process list failed; error: 'Lost connection to MySQL server during query'
0

And then of course :

mysqladmin: connect to server at 'localhost' failed
error: 'Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)'
Check that mysqld is running and that the socket: '/var/lib/mysql/mysql.sock' exists!
0

Then the server stop freezing, but mysql is down. So I (re) start mysql, and again and again

My server crash every 9 minutes and x secondes.. So bizarre


EDIT

I found this in my "/var/lib/mysql" error file:

This append verytime the server crashes

061231 06:47:57 mysqld started
InnoDB: Error: pthread_create returned 12
061231 06:48:01 mysqld ended

And ate the end of the file I see :

My server has 256Mb ram, but I only have 1 website on it, so i think it's enough no?

Number of processes running now: 0
070101 16:14:00 mysqld restarted
070101 16:14:16 InnoDB: Error: cannot allocate 8404992 bytes of
InnoDB: memory with malloc! Total allocated memory
InnoDB: by InnoDB 6359288 bytes. Operating system errno: 12
InnoDB: Check if you should increase the swap file or
InnoDB: ulimits of your operating system.
InnoDB: On FreeBSD check you have compiled the OS with
InnoDB: a big enough maximum process size.
InnoDB: Note that in most 32-bit computers the process
InnoDB: memory space is limited to 2 GB or 4 GB.
InnoDB: We keep retrying the allocation for 60 seconds...
070101 16:15:19 mysqld ended

Thanks for your help :p

And happy holiday

 

[Nikoms]

I found this in my logwatch... Maybe it's important

WARNING: Kernel Errors Present
<c04036e3> error_code+0x4f/0x54 ...: 174 Time(s)
<c04036e3> error_code+0x4f/0x54 ...: 1 Time(s)
<c05ecc16> do_page_fault+0x0/0x597 <c04036e3> error_code+0x4f/0x54 ...: 403 Time(s)
agpgart-via: probe of 0000:00:00.0 failed with error -22 ...: 3 Time(s)

 

[weedy]

I found this in my logwatch... Maybe it's important

WARNING: Kernel Errors Present
<c04036e3> error_code+0x4f/0x54 ...: 174 Time(s)
<c04036e3> error_code+0x4f/0x54 ...: 1 Time(s)
<c05ecc16> do_page_fault+0x0/0x597 <c04036e3> error_code+0x4f/0x54 ...: 403 Time(s)
agpgart-via: probe of 0000:00:00.0 failed with error -22 ...: 3 Time(s)

ahh the joys of failing hardware. check dmesg and grep your logs for other errors. also upgrade your kernel. if it doesn't go away looks like you may need to buy new part(s)