The Community Forums

Interact with an entire community of cPanel & WHM users!
  1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

urgent: Server crash / high load

Discussion in 'General Discussion' started by Simplesi, Jan 10, 2006.

  1. Simplesi

    Simplesi Registered

    Joined:
    Mar 30, 2005
    Messages:
    3
    Likes Received:
    0
    Trophy Points:
    1
    Hi,

    I'm having huge problems with a cpanel server I'm running.

    On a near daily basis, it becomes sluggish, and needs restarting. Sometimes, the mysql process crashes and isn't restarted by cpanel - and the first I hear is users complaining that mysql is down.

    Its running FC1, as that's what our host supports (its a VPS box). However, I don't know how to work out the problem. I'm running STABLE, but am considering using RELEASE instead just incase there's a bugfix there.

    Here's an example of top when it's misbehaving:

    Code:
    22:19:45  up 17:49,  2 users,  load average: 49.16, 43.18, 23.32
    98 processes: 93 sleeping, 4 running, 1 zombie, 0 stopped
    CPU states:  cpu    user    nice  system    irq  softirq  iowait    idle
               total    1.0%    0.0%    6.2%   0.3%     0.0%   92.3%    0.0%
    Mem:   126664k av,  124680k used,    1984k free,       0k shrd,     264k buff
            63724k active,              52132k inactive
    Swap:  130040k av,  130020k used,      20k free                    3440k cached
    
      PID USER     PRI  NI  SIZE  RSS SHARE STAT %CPU %MEM   TIME CPU COMMAND
     1346 root      16   0 13552  368    56 S     2.9  0.2  10:03   0 httpd
       55 root      15   0     0    0     0 DW    0.4  0.0   1:06   0 kswapd0
    13831 mailman   18   0  5604 1956   940 D     0.4  1.5   0:03   0 python
     1598 root      18   0 10120  740   152 D     0.2  0.5   0:53   0 cppop
    13803 mailman   18   0  6020 2020   716 D     0.2  1.5   0:05   0 python
    13813 mailman   18   0  6044 1868   624 R     0.2  1.4   0:04   0 python
    13826 mailman   18   0  5620 2060   960 D     0.2  1.6   0:03   0 python
    13827 root      17   0  2244  552   288 R     0.2  0.4   0:05   0 top
    13829 mailman   18   0  5812 2124   940 D     0.2  1.6   0:03   0 python
    13837 root      18   0  7580 3768   356 D     0.2  2.9   0:02   0 mrtg
    13766 root      18   0  7996 2692   404 D     0.1  2.1   0:06   0 mrtg
    13778 root      18   0  4760 1240   480 D     0.1  0.9   0:04   0 dcpumon
    13793 root      18   0 11492  844   368 D     0.1  0.6   0:05   0 php
        1 root      17   0  1596   60    48 S     0.0  0.0   0:06   0 init
        2 root      34  19     0    0     0 SWN   0.0  0.0   0:00   0 ksoftirqd/0
        3 root      10  -5     0    0     0 SW<   0.0  0.0   0:00   0 events/0
        4 root      11  -5     0    0     0 SW<   0.0  0.0   0:00   0 khelper
        9 root      10  -5     0    0     0 SW<   0.0  0.0   0:00   0 kthread
       12 root      10  -5     0    0     0 SW<   0.0  0.0   0:00   0 kblockd/0
       56 root      20  -5     0    0     0 SW<   0.0  0.0   0:00   0 aio/0
      737 root      15   0     0    0     0 SW    0.0  0.0   0:03   0 kjournald
     1106 root      17   0  1648  104    64 S     0.0  0.0   0:09   0 syslogd
     1110 root      15   0  1600   88    48 S     0.0  0.0   0:03   0 klogd
     1205 root      17   0  3852   32     0 S     0.0  0.0   0:01   0 sshd
     1218 root      16   0  2204    4     0 S     0.0  0.0   0:00   0 xinetd
     1304 mailnull  16   0  7128    4     0 S     0.0  0.0   0:01   0 exim
     1308 mailnull  18   0  7128    4     0 S     0.0  0.0   0:00   0 exim
     1315 root      16   0  3840  268   188 S     0.0  0.2   1:05   0 antirelayd
     1354 root      17   0  1640   68    36 S     0.0  0.0   0:02   0 crond
     1428 root       0 -20     0    0     0 SW<   0.0  0.0   0:00   0 loop0
     1429 root      15   0     0    0     0 SW    0.0  0.0   0:00   0 kjournald
     1511 root      16   0 11648    0     0 SW    0.0  0.0   0:00   0 cpsrvd
     1557 root      16   0  6056    4     0 S     0.0  0.0   0:00   0 pure-ftpd
     1560 root      18   0  3756    4     0 S     0.0  0.0   0:00   0 pure-authd
     1577 cpanel    15   0  4136    4     0 S     0.0  0.0   0:00   0 stunnel-4.04loc
     1695 mailman   17   0  7588  276     0 S     0.0  0.2   0:04   0 mailmanctl
     1697 root      16   0  1596    4     0 S     0.0  0.0   0:00   0 agetty
    As you can see, it's nearly out of memory - swap and RAM - and I belive alot of the CPU is being used for paging. In fact, in that extract, the cpu doesn't look too bad - except check out the load for a 1CPU machine. But at other times, the system cpu state has up to 90% load.

    How can I work out what's consuming the resources and crashing the machine? I'm fairly used to linux, but haven't had to deal with something like this before.

    All my packages are up to date, according to cpanel.

    Thanks,
    Simon
     
  2. AndyReed

    AndyReed Well-Known Member
    PartnerNOC

    Joined:
    May 29, 2004
    Messages:
    2,222
    Likes Received:
    3
    Trophy Points:
    38
    Location:
    Minneapolis, MN
    Although this issue has been discussed hundreds of times in these forums, you need to find out who or what is causing the problem. Most likely a bad php and/or cgi/perl script. If that's true, you'll need to clean up your server, and install security patches and applications including ModeSecurity, APF, and BFD.
     
  3. Simplesi

    Simplesi Registered

    Joined:
    Mar 30, 2005
    Messages:
    3
    Likes Received:
    0
    Trophy Points:
    1
    At the moment, the server hosts just one website with an up-to-date ezpublish install - there are no other users to upload dodgy scripts with vulnerabilities.

    I'd appreciate some pointers on identifying the problem - I certainly agree that I need to deal with it!

    Simon

    PS. I've just moved from Stable to Release - but it seems there's no actual difference in package, so unfortunately that hasn't fixed things.
     
  4. codek

    codek Active Member

    Joined:
    Jul 26, 2004
    Messages:
    25
    Likes Received:
    0
    Trophy Points:
    1
    VERY interesting. I have exactly this problem on my server FC2, it developed today. Load goes through the roof, but nothing is using cpu, it's all in wait state.

    Can anyone help? I've tried everything i and my host can think of to debug this but no joy!!!

    Thanks!
    Dan
     
  5. Simplesi

    Simplesi Registered

    Joined:
    Mar 30, 2005
    Messages:
    3
    Likes Received:
    0
    Trophy Points:
    1
    It seems that for me the problem was spamd, the spam assassin daemon. When I looked closely, it was consuming 90Mb of memory - and considering I have only 128Mb ram and 128Mb swap, that soon meant that it keeled over.

    My temporary fix was to disable spamd - in the email settings, but also (and I missed this the first time round) in the service configuration, so the service definately doesn't run.

    That was yesterday, and it hasn't got anywhere near a crash since. There are quite a few posts in these forums talking about spamd consuiming loads of memory and some tips for dealing with it. In due course I'll need spamd back up and running, so I guess I'll start working through those.

    Good luck!

    Simon
     
Loading...

Share This Page