The Community Forums

Interact with an entire community of cPanel & WHM users!
  1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Server Use almost 100% of ram and 100% of SWAP

Discussion in 'Workarounds and Optimization' started by inertz, Apr 7, 2009.

  1. inertz

    inertz Member

    Joined:
    Nov 24, 2006
    Messages:
    19
    Likes Received:
    0
    Trophy Points:
    1
    Location:
    Oklahoma City, Oklahoma, United States
    Hi,

    Actually one of my server have problem with the memory issue. It seem use all of its SWAP and ram and make the server to hang. The server is a shared hosting server. In normal condition the load is not high, but when it happen, it will happen on a sudden. There is not much that i can do unless do the hard reboot.

    # top
    top - 15:10:20 up 2 days, 15:45, 1 user, load average: 84.80, 78.34, 63.96
    Tasks: 331 total, 83 running, 247 sleeping, 0 stopped, 1 zombie
    Cpu0 : 0.0%us, 99.4%sy, 0.3%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.3%si, 0.0%st
    Cpu1 : 0.3%us, 99.4%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.3%si, 0.0%st
    Mem: 6111608k total, 6080140k used, 31468k free, 1356k buffers
    Swap: 20970736k total, 20970736k used, 0k free, 9972k cached


    Does anyone experience same as me? Does it related to the memory leak? How to trace memory leak?

    Adi..
     
  2. vpsville

    vpsville Active Member
    PartnerNOC

    Joined:
    May 8, 2008
    Messages:
    25
    Likes Received:
    0
    Trophy Points:
    1
    It looks like you're out of memory, 6GB of RAM and 20GB of swap!. Running top may tell you what software is eating up all the ram. You could try switching to nsd instead of bind, and dovecot which may reduce your memory usage significantly.

    Anytime the system is using swap, performance will suffer. You want avoid swap usage on a production server.
     
  3. Kailash1

    Kailash1 Well-Known Member

    Joined:
    Nov 27, 2006
    Messages:
    252
    Likes Received:
    2
    Trophy Points:
    18
    cPanel Access Level:
    Root Administrator
    Twitter:
    Your server is using 26 GB memory!! There is really something wrong on your server. It could be memory leak or bad written PHP scripts by your users. If this is manage server, contact your provider to investigate this. If you are managing the server then I suggest to higher administrator.

    Kailash
     
  4. Spiral

    Spiral BANNED

    Joined:
    Jun 24, 2005
    Messages:
    2,023
    Likes Received:
    7
    Trophy Points:
    0
    I would suggest the same ...

    You may have underlying issues that need to be addressed professionally.

    Foregoing the above, you could also probably use a fair bit of optimizing and tuning.

    This crosses into my area of expertise. If you could use the help, I would be willing to take a look and not charge you just to look around and give you you a much better idea what is actually going on and then you would be in a much better position to know where to from there.
     
  5. hostmedic

    hostmedic Well-Known Member

    Joined:
    Apr 30, 2003
    Messages:
    559
    Likes Received:
    0
    Trophy Points:
    16
    Location:
    Washington Court House, Ohio, United States
    cPanel Access Level:
    DataCenter Provider
    chasing load

    I would suggest looking @ top to determine whats happening there.

    If you stop mysql does it go down?
    if you stop apache - does it go down ?

    Have you created a my.cnf file - or are you using the generic ?
    are you firewalling off traffic - ?
    have you installed mod_bandwidth ?
     
  6. monsignore

    monsignore Member

    Joined:
    Mar 15, 2008
    Messages:
    19
    Likes Received:
    0
    Trophy Points:
    1
    Location:
    Casablanca Morocco
    Hi !

    I had several time this exact problem.

    Look your mail queue, that was in my case a heavy use of Exim by a spam script. I just reboot, stop exim, empty the mail queue, found the script an delete it...

    This happens easilly on VPS...
     
  7. Promethyl

    Promethyl Well-Known Member

    Joined:
    Mar 27, 2004
    Messages:
    68
    Likes Received:
    0
    Trophy Points:
    6
    In my case, users were using mysql_pconnect forcing new connections without setting a limit on it.

    Their connections were lingering.

    I set time limits on it and maximum connections. That helped.

    What was the issue?
     
  8. napraviculom

    napraviculom Member

    Joined:
    Dec 30, 2009
    Messages:
    8
    Likes Received:
    0
    Trophy Points:
    1
    I have a similar problem. Load averages are high and go up to 12 and even more.

    Code:
    [root@server /]# top
    top - 12:27:08 up 19 days, 16:46, 1 user, load average: 8.92, 10.71, 10.21
    Tasks: 153 total, 9 running, 141 sleeping, 0 stopped, 3 zombie
    Cpu(s): 85.4% us, 14.5% sy, 0.0% ni, 0.0% id, 0.0% wa, 0.2% hi, 0.0% si
    Mem: 6231956k total, 5745236k used, 486720k free, 175492k buffers
    Swap: 2040212k total, 5388k used, 2034824k free, 2688300k cached
    
    PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
    29025 root 15 0 1388m 226m 3616 S 120.0 3.7 129:10.71 mysqld
    (mysql is causing it)

    This are the red marked variables from phpMyAdmin runtime information
    Code:
    Slow_queries  	14 k
    Innodb_buffer_pool_reads  	33
    Handler_read_rnd  	24 M
    Handler_read_rnd_next  	3,316.21 M
    Slow_launch_threads  	2
    Created_tmp_disk_tables  	3,624 
    Select_full_join  	621
    Select_range_check  	1 
    Opened_tables  	12 k
    Table_locks_waited  	17 k
    Handler_read_rnd_next seems to be very high. phpmyadmin suggests that there is something wrong with indexing of tables (not using them or badly made). the table is indexed properly i believe.

    If someone had a similar issue, it would help.
    Any help is aprecciated!
     
  9. inertz

    inertz Member

    Joined:
    Nov 24, 2006
    Messages:
    19
    Likes Received:
    0
    Trophy Points:
    1
    Location:
    Oklahoma City, Oklahoma, United States
    If you stop mysql does it go down?
    >>>yes
    if you stop apache - does it go down ?
    >>>yes

    Have you created a my.cnf file - or are you using the generic ?
    >>>Configuration that works on other server same spec.

    are you firewalling off traffic - ?
    >>>Firewall correctly configured
     
  10. Promethyl

    Promethyl Well-Known Member

    Joined:
    Mar 27, 2004
    Messages:
    68
    Likes Received:
    0
    Trophy Points:
    6
    nap : a bug in mysql? Is there a directive to limit that memory usage? Rebuild the indexes. Find out which tables are in memory. Something is using up all the memory. Is your Sql server databases really that large?
     
  11. inertz

    inertz Member

    Joined:
    Nov 24, 2006
    Messages:
    19
    Likes Received:
    0
    Trophy Points:
    1
    Location:
    Oklahoma City, Oklahoma, United States
    New issue with my server. It seem hackers able to upload random php file on user account using FTP. Same as gumblar attack but his time the weird thing hackers upload random php with .htaccess.

    It it do it cause server to load.
     
  12. cPanelDon

    cPanelDon cPanel Quality Assurance Analyst
    Staff Member

    Joined:
    Nov 5, 2008
    Messages:
    2,557
    Likes Received:
    7
    Trophy Points:
    38
    Location:
    Houston, Texas, U.S.A.
    cPanel Access Level:
    DataCenter Provider
    Twitter:
    If seeking suggestions and help related to security I may suggest the following forums area that should offer you more pertinent feedback: cPanel and WHM Security - cPanel Forums

    If seeking to hire someone to assist with an issue, I recommend the following area in the advertising forums: Server Management and Server Repair - cPanel Forums
     
  13. Spiral

    Spiral BANNED

    Joined:
    Jun 24, 2005
    Messages:
    2,023
    Likes Received:
    7
    Trophy Points:
    0
    Do you save a copy of that altered .htaccess file?

    Do you think I might take a peak at the rest of your server?

    What you describe sounds similar to a very common attack used to use your server as a virus distribution relay but some of what you said makes me wonder if it could possibly be something entirely new and if so then that would be something that I would definitely want to take a look at more closely as this is I research these things very heavily.

    A great deal of my time (and personal life) is spent developing new anti-hacking technologies and researching new threats and probably makes up the larger bulk of the work I do daily.

    With that said, it is also possible that you got hit with the attack which I'm already familiar that I just described above but would probably need to at barebones minimum see a copy of that .htaccess file to know for certain if this is something old or something new.

    We can take this chat private if you would like. :)
     
  14. inertz

    inertz Member

    Joined:
    Nov 24, 2006
    Messages:
    19
    Likes Received:
    0
    Trophy Points:
    1
    Location:
    Oklahoma City, Oklahoma, United States
    Hi Spiral,

    The file inside .htaccess contain;

    and the file inside 188093.php contain encoded code and have to decode in order to see the content.

    Also some attack also upload empty file name "WP"

    From the log seem like the hacker upload the file using FTP same like gumblar attack.

    Any idea?
     
  15. Spiral

    Spiral BANNED

    Joined:
    Jun 24, 2005
    Messages:
    2,023
    Likes Received:
    7
    Trophy Points:
    0
    inertz:

    Per your private message, I have sent you a return reply there. ;)
     
  16. GaryT

    GaryT Well-Known Member

    Joined:
    May 19, 2010
    Messages:
    321
    Likes Received:
    3
    Trophy Points:
    16
    I have rought the same issue now but not as high, Not alot though but swap is growing.

    Now, I do not get this as its a strange one but memory is at 21% but swap is at 5.21%

    How does this one work out ?
     
Loading...

Share This Page