The Community Forums

Interact with an entire community of cPanel & WHM users!
  1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

new easyapache dramas - SSL + Load

Discussion in 'EasyApache' started by salubrium, Dec 12, 2007.

  1. salubrium

    salubrium Member

    Joined:
    Jun 11, 2003
    Messages:
    14
    Likes Received:
    0
    Trophy Points:
    1
    Location:
    Sydney
    cPanel Access Level:
    Root Administrator
    Hi,

    I finally decided to go ahead and upgrade from Mysql4.1 to 5.0 after reading it was fairly trivial and just required a php recompile. I figured it was going to be straight forward as I have done quite a number of easyapache runs and had it down pat, so I went ahead and did the upgrade to mysql.

    So.. then came apache - ooh.. easyapache has changed, I wish I knew that before I tried my mysql upgrade. So, I tried to keep it as simple and close to original as possible.. I wasn't able to use my old config.. and kept Apache 1.3 - I only want to do one change at a time thanks.. makes troubleshooting much easier.

    I had lots of problems with running php4 & 5 concurrently but finally got it sorted out.. DirectoryIndex load order was different,

    I had to add in main config:

    Listen 443

    and under virtual host for ssl,

    SSLEnable

    Finally.. possibly the worst.. I have been running loads of 0.6-1.2 for the last 6 months.. now today, I have had 4 times today where the load suddenly jumps from 0.7 to 30 and even as high as 120!! What is it? I can't locate it.. suddenly it starts consuming ram and is putting everything into swap (it has 2gb physical + 2gb swap) until it fills up swap and then slowly comes back down.. The load is obviously the hard drive thrashing it's head off trying to put so much data in swap.

    But, if I do a top with shift+M it shows nothing out of the ordinary.. 15 apache processes using approx 20mb each, mysql with about 392mb, some exim and mailscanner processes etc..

    Has anyone seen anything similar? the only other thing is that I previously had mod_evasive installed and after all the issues, I haven't had a chance to re-add it until just now, so I will see how that goes.
     
  2. salubrium

    salubrium Member

    Joined:
    Jun 11, 2003
    Messages:
    14
    Likes Received:
    0
    Trophy Points:
    1
    Location:
    Sydney
    cPanel Access Level:
    Root Administrator
    Apache Load fixed

    After spending some time first implementing mod_evasive, re-enabling all of my mod_sec rules, reenabling RLimits for memory and cpu, I was still getting the issue (thought to a slightly lesser degree) I managed to fix it once and for all by changing the MaxRequestsPerChild - something that has bitten me once before.

    For whatever dumb reason, the default that Cpanel sets is 0 - which means each child will never die, serving unlimited requests. Bringing this to approx 200-500 generally fixes any of the memory problems I have seen with apache. God knows why it took me so long to remember it.


    From Apache website
     
  3. cPDan

    cPDan cPanel Staff
    Staff Member

    Joined:
    Mar 9, 2004
    Messages:
    711
    Likes Received:
    3
    Trophy Points:
    18
    > I wasn't able to use my old config..

    It is used to create the ea3 one, so originally it was your old config :)

    > I had lots of problems with running php4 & 5 concurrently

    http://www.cpanel.net/support/docs/ea/ea3/index.html

    Did you read the documentation about it? 2 PHPs is very flexible and complicated and needs configured after a build.

    The httpd.conf is managed by a a new templating syste:, Cpanel::AdvConfig.

    There is documentation on how it works and how to work with it at the URL above as well.

    I'll investigate the MaxRequestsPerChild issue first thing tomorrow.

    Thanks for the details!
     
  4. salubrium

    salubrium Member

    Joined:
    Jun 11, 2003
    Messages:
    14
    Likes Received:
    0
    Trophy Points:
    1
    Location:
    Sydney
    cPanel Access Level:
    Root Administrator
    Ongoing Saga

    After a couple days of running Apache 1.3.9, PHP 4.4.7 and compiling in PHP5 afterwards as a CGI, I had still experienced a number of problems. Here's an example of what happens to the memory when Apache goes awol.

    [​IMG]

    In the above screenshot, if I try and strace -p on the process, it's too late.. it's just idle, doing nothing but consuming all RAM and Swap space. When it is being put into swap kswapd goes crazy and loads shoot up to max out around 120... in comparison nightly backups load never exceeds 7-8.

    and here's an MRTG chart of what's happened since my last recompile. Note the mysql processes.. I haven't had time to view connection status on these to see which site is creating so many connections to the database

    [​IMG]

    As of approx 11pm on Saturday night, I recompiled using ea3 upgrading this time to Apache 2.2 MPM Worker PHP 4.4.7 and left PHP5 as it was (only 2 very lightweight sites are running php5).The same thing is still occurring even though I have enable mod_cache on all sites (which works brilliantly btw - even if I might not use it full time)

    So, I am partly considering that the issue is something in PHP 4.4.7. My main mode of attack has been to try and use status-info to view what sites might be triggering the issue - or I list all log files in /home/domlogs with a timestamp of the highload occurrence. I am looking for anything but mostly for bad redirects, a search engine or a ddos attack. I have used httperf on a couple sites that I have listed to be suspect and can't reproduce this effect.

    I have used httperf on the two PHP5 sites and they don't cause any loads at all, only one site causes a high load with httperf but that just chews CPU and spawns child process because there's a particular php process that creates a javascript menu with approx 800 items.. but it still doesn't create the memory scenario seen above.

    Am thinking to try and roll back PHP versions slowly now. The hard part about this all is that I have to wait a day to see if any high load issues occure and then I have to wait until late evening to do recompiles because there are some interrupts after a recompile on the php5 sites.

    Anyway, I am willing to take on any ideas at the moment.

    here's a bit of the core of my httpd.conf - don't blame the memory cacheing. It's my latest attempt to help reduce/resolve the issue.

    Code:
    PidFile logs/httpd.pid
    LockFile logs/accept.lock
    # Defined in /var/cpanel/cpanel.config: apache_port
    Listen 0.0.0.0:80
    Timeout 300
    User nobody
    Group nobody
    ExtendedStatus On
    ServerAdmin me@mydomain.com
    ServerName my.domain.com
    LogLevel warn
    ServerSignature On
    MinSpareServers 5
    MaxSpareServers 10
    StartServers 20
    MaxClients 30
    MaxRequestsPerChild 200
    CacheDefaultExpire 3600
    CacheEnable mem /
    
    <IfModule mod_fcgid.c>
      AddHandler fcgid-script .fcgi
      SocketPath /tmp/fcgid.sock
    #  DefaultInitEnv RAILS_ENV production
      IdleTimeout 60
      ProcessLifeTime 6000
      MaxProcessCount 32
      DefaultMaxClassProcessCount 2
      IPCConnectTimeout 6
      IPCCommTimeout 6
    </IfModule>
    
    
     
    #4 salubrium, Dec 16, 2007
    Last edited: Dec 16, 2007
  5. salubrium

    salubrium Member

    Joined:
    Jun 11, 2003
    Messages:
    14
    Likes Received:
    0
    Trophy Points:
    1
    Location:
    Sydney
    cPanel Access Level:
    Root Administrator
    This morning I found the URL that was causing the issue. I managed to get a server-info response and htop to show me the same PID of apache. /server-info shows the URL being processed while 'top' or 'htop' are able to show which process is using so much RAM.

    It was a Joomla site that our company had produced for a client. The site uses SEF URL's and also a component called "Glossary" and a mosbot (code snippets) that finds words in the text of the site and highlights them with a javascript hover showing the term in the glossary component.

    It seems that there's a PHP call that sends it into infinite recursion. What's surprising to me is that this site has been running without issues for almost 2 years... why is do we suddenly get this issue? What was interesting was how quickly the Apache child would start to grab ram.. it would grab about 400mb within a second and within 5 seconds it would have consumed all RAM and have 1.7Gb of swap used.

    For the time being, I have changed the HTML causing the issue:
    HTML:
    <li><a title="Surgery - treatment" href="/treatment/surgery.html">Surgery</a></li>
    
    becomes
    HTML:
    <li><a href="/treatment/surgery.html">Surgery</a></li>
    
    which fixes the issue. I will wait to see if this fixes the issue. Hopefully it's the only site / url causing the issue for the server.
     
  6. MaraBlue

    MaraBlue Well-Known Member

    Joined:
    May 3, 2005
    Messages:
    335
    Likes Received:
    2
    Trophy Points:
    18
    Location:
    Carmichael, CA
    cPanel Access Level:
    Root Administrator
    Holy bovine, that's really something. Please post back if you find any further info on the cause. Since the site was without issue for years (and I assume running under PHP4.x), it would look like something within PHP5.

    I'm planning on running 4&5 side-by-side as well...just need enough downtime strung together to make it happen.

    Thank you for posting this...probably one of the most interesting threads I've read in here in years.
     
  7. bci

    bci Registered

    Joined:
    Apr 21, 2005
    Messages:
    1
    Likes Received:
    0
    Trophy Points:
    1
    Re: Ongoing Saga

    How did you get mod_cache to work? I recompiled with EasyApache to include mod_cache and mod_disk_cache, and added directives to httpd.conf, but caching still isn't active for me.

    What steps did you take?
     
Loading...

Share This Page