Server Use almost 100% of ram and 100% of SWAP

[inertz]

Hi,

Actually one of my server have problem with the memory issue. It seem use all of its SWAP and ram and make the server to hang. The server is a shared hosting server. In normal condition the load is not high, but when it happen, it will happen on a sudden. There is not much that i can do unless do the hard reboot.

# top
top - 15:10:20 up 2 days, 15:45, 1 user, load average: 84.80, 78.34, 63.96
Tasks: 331 total, 83 running, 247 sleeping, 0 stopped, 1 zombie
Cpu0 : 0.0%us, 99.4%sy, 0.3%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.3%si, 0.0%st
Cpu1 : 0.3%us, 99.4%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.3%si, 0.0%st
Mem: 6111608k total, 6080140k used, 31468k free, 1356k buffers
Swap: 20970736k total, 20970736k used, 0k free, 9972k cached


Does anyone experience same as me? Does it related to the memory leak? How to trace memory leak?

Adi..

 

[vpsville]

It looks like you're out of memory, 6GB of RAM and 20GB of swap!. Running top may tell you what software is eating up all the ram. You could try switching to nsd instead of bind, and dovecot which may reduce your memory usage significantly.

Anytime the system is using swap, performance will suffer. You want avoid swap usage on a production server.

 

[Kailash1]

Your server is using 26 GB memory!! There is really something wrong on your server. It could be memory leak or bad written PHP scripts by your users. If this is manage server, contact your provider to investigate this. If you are managing the server then I suggest to higher administrator.

Kailash

 

[Spiral]

If you are managing the server then I suggest to higher administrator.

I would suggest the same ...

You may have underlying issues that need to be addressed professionally.

Foregoing the above, you could also probably use a fair bit of optimizing and tuning.

This crosses into my area of expertise. If you could use the help, I would be willing to take a look and not charge you just to look around and give you you a much better idea what is actually going on and then you would be in a much better position to know where to from there.

 

[hostmedic]

chasing load

I would suggest looking @ top to determine whats happening there.

If you stop mysql does it go down?
if you stop apache - does it go down ?

Have you created a my.cnf file - or are you using the generic ?
are you firewalling off traffic - ?
have you installed mod_bandwidth ?

 

[monsignore]

Hi !

I had several time this exact problem.

Look your mail queue, that was in my case a heavy use of Exim by a spam script. I just reboot, stop exim, empty the mail queue, found the script an delete it...

This happens easilly on VPS...

 

[Promethyl]

In my case, users were using mysql_pconnect forcing new connections without setting a limit on it.

Their connections were lingering.

I set time limits on it and maximum connections. That helped.

What was the issue?

 

[napraviculom]

I have a similar problem. Load averages are high and go up to 12 and even more.

[[email protected] /]# top
top - 12:27:08 up 19 days, 16:46, 1 user, load average: 8.92, 10.71, 10.21
Tasks: 153 total, 9 running, 141 sleeping, 0 stopped, 3 zombie
Cpu(s): 85.4% us, 14.5% sy, 0.0% ni, 0.0% id, 0.0% wa, 0.2% hi, 0.0% si
Mem: 6231956k total, 5745236k used, 486720k free, 175492k buffers
Swap: 2040212k total, 5388k used, 2034824k free, 2688300k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
29025 root 15 0 1388m 226m 3616 S 120.0 3.7 129:10.71 mysqld

(mysql is causing it)

This are the red marked variables from phpMyAdmin runtime information

Slow_queries  	14 k
Innodb_buffer_pool_reads  	33
Handler_read_rnd  	24 M
Handler_read_rnd_next  	3,316.21 M
Slow_launch_threads  	2
Created_tmp_disk_tables  	3,624 
Select_full_join  	621
Select_range_check  	1 
Opened_tables  	12 k
Table_locks_waited  	17 k

Handler_read_rnd_next seems to be very high. phpmyadmin suggests that there is something wrong with indexing of tables (not using them or badly made). the table is indexed properly i believe.

If someone had a similar issue, it would help.
Any help is aprecciated!

 

[inertz]

I would suggest looking @ top to determine whats happening there.

If you stop mysql does it go down?
if you stop apache - does it go down ?

Have you created a my.cnf file - or are you using the generic ?
are you firewalling off traffic - ?
have you installed mod_bandwidth ?

If you stop mysql does it go down?
>>>yes
if you stop apache - does it go down ?
>>>yes

Have you created a my.cnf file - or are you using the generic ?
>>>Configuration that works on other server same spec.

are you firewalling off traffic - ?
>>>Firewall correctly configured

 

[Promethyl]

nap : a bug in mysql? Is there a directive to limit that memory usage? Rebuild the indexes. Find out which tables are in memory. Something is using up all the memory. Is your Sql server databases really that large?

 

[inertz]

New issue with my server. It seem hackers able to upload random php file on user account using FTP. Same as gumblar attack but his time the weird thing hackers upload random php with .htaccess.

It it do it cause server to load.

 

[cPanelDon]

New issue with my server. It seem hackers able to upload random php file on user account using FTP. Same as gumblar attack but his time the weird thing hackers upload random php with .htaccess.

It it do it cause server to load.

If seeking suggestions and help related to security I may suggest the following forums area that should offer you more pertinent feedback: cPanel and WHM Security - cPanel Forums

If seeking to hire someone to assist with an issue, I recommend the following area in the advertising forums: Server Management and Server Repair - cPanel Forums

 

[Spiral]

New issue with my server. It seem hackers able to upload random php file on user account using FTP. Same as gumblar attack but his time the weird thing hackers upload random php with .htaccess.

It it do it cause server to load.

Do you save a copy of that altered .htaccess file?

Do you think I might take a peak at the rest of your server?

What you describe sounds similar to a very common attack used to use your server as a virus distribution relay but some of what you said makes me wonder if it could possibly be something entirely new and if so then that would be something that I would definitely want to take a look at more closely as this is I research these things very heavily.

A great deal of my time (and personal life) is spent developing new anti-hacking technologies and researching new threats and probably makes up the larger bulk of the work I do daily.

With that said, it is also possible that you got hit with the attack which I'm already familiar that I just described above but would probably need to at barebones minimum see a copy of that .htaccess file to know for certain if this is something old or something new.

We can take this chat private if you would like.

 

[inertz]

Do you save a copy of that altered .htaccess file?

Do you think I might take a peak at the rest of your server?

What you describe sounds similar to a very common attack used to use your server as a virus distribution relay but some of what you said makes me wonder if it could possibly be something entirely new and if so then that would be something that I would definitely want to take a look at more closely as this is I research these things very heavily.

A great deal of my time (and personal life) is spent developing new anti-hacking technologies and researching new threats and probably makes up the larger bulk of the work I do daily.

With that said, it is also possible that you got hit with the attack which I'm already familiar that I just described above but would probably need to at barebones minimum see a copy of that .htaccess file to know for certain if this is something old or something new.

We can take this chat private if you would like.

Hi Spiral,

The file inside .htaccess contain;

Options -MultiViews
ErrorDocument 404 //chat/ikoner/188093.php

and the file inside 188093.php contain encoded code and have to decode in order to see the content.

Also some attack also upload empty file name "WP"

From the log seem like the hacker upload the file using FTP same like gumblar attack.

Any idea?

 

[Spiral]

inertz:

Per your private message, I have sent you a return reply there.

 

[GaryT]

I have rought the same issue now but not as high, Not alot though but swap is growing.

Now, I do not get this as its a strange one but memory is at 21% but swap is at 5.21%

How does this one work out ?