The Community Forums

Interact with an entire community of cPanel & WHM users!
  1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

chkservd restarts Apache every 10 minutes

Discussion in 'EasyApache' started by Binkmeister, Mar 2, 2003.

  1. Binkmeister

    Binkmeister Member

    Joined:
    Mar 1, 2003
    Messages:
    5
    Likes Received:
    0
    Trophy Points:
    1
    Location:
    USA
    We're having a problem with chkservd - after spending a couple of hours of digging through logs and such, it appears that chkservd is rebooting Apache like clockwork, every ten minutes, no matter what. I have no idea why it would do this.

    We have a very active forum site, running mySQL, and when we have a larger number of concurrent users (> 170 or so), the site can completely lock up for a good five minutes, spouting error messages and generally being obnoxious.

    A little more digging show chkservd trying to do a graceful restart of Apache, which means Apache and mySQL are apparently trying to communicate, but they're stumbling over each other. Watching the processlist in mySQL, we go from about current 5-10 processes to a couple of hundred. They then wait the 300 seconds or so to timeout, and mySQL starts heading back down to a normal range of under 50 processes. The web server apparently feels better at that point and the site is open again.

    This is a showstopper for us, frankly. The symptom is chkservd rebooting Apache every 10 minutes... I don't know why it is doing this. Does anybody have any ideas? If I have to I'll disable that Perl script, but I have no doubt it would have some unpleasant side effects.

    Any ideas are welcome at this point.

    Thanks,
    Kelly
    BZPower.com
     
  2. dgbaker

    dgbaker Well-Known Member
    PartnerNOC

    Joined:
    Sep 20, 2002
    Messages:
    2,578
    Likes Received:
    3
    Trophy Points:
    38
    Location:
    Toronto, Ontario Canada
    cPanel Access Level:
    DataCenter Provider
  3. Binkmeister

    Binkmeister Member

    Joined:
    Mar 1, 2003
    Messages:
    5
    Likes Received:
    0
    Trophy Points:
    1
    Location:
    USA
    Thanks very much for the link. I've gone through it a couple of times, and the following seemed to be the best clue:
    I tried implementing this but it's still going through its shutdown routine every 10 minutes. Does anyone have any other suggestions? This is really bringing the server to its knees.

    - Kelly
    BZPower.com
     
  4. jsteel

    jsteel Well-Known Member

    Joined:
    Jul 4, 2002
    Messages:
    646
    Likes Received:
    0
    Trophy Points:
    16
    Location:
    Atlanta, GA
    Have you ssh'd in and run the following from the command-line?

    /etc/rc.d/init.d/httpd stop

    /etc/rc.d/init.d/httpd start

    Check for any errors listed. It sounds like you have something in your httpd.conf file that is preventing Apache from loading altogether and chkservd is just trying to get it started every 8-10 minutes as it should.

    Jaz
     
  5. testudo

    testudo Active Member

    Joined:
    Mar 26, 2002
    Messages:
    26
    Likes Received:
    0
    Trophy Points:
    1
    Are you sure this is what's causing your problem? Check your logs to make sure it's a "graceful restart" (SIGUSR1). If so, it shouldn't be hurting anything because a graceful restart does not drop connections but simply allows Apache to close/reopen the logs and run a configtest -- it's not a kill-and-restart kind of thing. This is normal behavior in the cPanel world and I haven't had it cause the problems you describe.
     
  6. Amanita

    Amanita Member

    Joined:
    Jan 26, 2003
    Messages:
    6
    Likes Received:
    0
    Trophy Points:
    1
    Did you restart Apache after making the changes?
     
  7. Website Rob

    Website Rob Well-Known Member

    Joined:
    Mar 23, 2002
    Messages:
    1,506
    Likes Received:
    0
    Trophy Points:
    36
    Location:
    Alberta, Canada
    cPanel Access Level:
    Root Administrator
    A quick way to check your httpd.conf file is:

    # /usr/local/apache/bin/httpd -t

    Syntex OK << is all you want to see.

    Any problems will be specifically mentioned and let you know where to start looking.

    The restart every 10 Minutes does have something to do with the version of PHP being used and Cpanel. Not sure how it all works as I've just accepted it is something I have to live with, for right now anyway.
     
  8. Binkmeister

    Binkmeister Member

    Joined:
    Mar 1, 2003
    Messages:
    5
    Likes Received:
    0
    Trophy Points:
    1
    Location:
    USA
    Yep, I restarted Apache and I stopped getting the restart after 10 minutes errors. There were enormous amounts of SIGUSR1 notes in the logfile from before, and each one was followed by a noticeable slowdown of the website.

    Since this didn't actually solve the problem, I went through my site and disabled heavy SQL functions in my forums, like search, mod CP, auto-generated updates for our news, etc. That has relieved the immediate issues, and after closely watching the mySQL process list, it looks like we need to configure the mySQL settings to alleviate the locking problem we were getting. I don't want to have to run my forum in "Lobotomy" mode forever.

    So while the restarting Apache was a problem, it wasn't the problem, and I'll keep looking, this time at mySQL.

    Thanks for the help, everyone!
    - Kelly
     
  9. netwrkr

    netwrkr Well-Known Member

    Joined:
    Apr 12, 2003
    Messages:
    203
    Likes Received:
    0
    Trophy Points:
    16
    Why restart apache every 10 minutes

    What is the point of restarting apache every 10 minutes?

    Please enlighten me because if you are browsing a website when the restart occurs it -does- cause a delay; alibet small but a delay nonetheless.

    Thanks,

    Tom

    grep SIGUSR1 error_log | wc -l
    2781
     
  10. dgbaker

    dgbaker Well-Known Member
    PartnerNOC

    Joined:
    Sep 20, 2002
    Messages:
    2,578
    Likes Received:
    3
    Trophy Points:
    38
    Location:
    Toronto, Ontario Canada
    cPanel Access Level:
    DataCenter Provider
    This is what Darkorb had said in another post.

    qoute

    SIGUSR1 is NORMAL. This is required for apache to reopen log files.

    enqoute
     
  11. netwrkr

    netwrkr Well-Known Member

    Joined:
    Apr 12, 2003
    Messages:
    203
    Likes Received:
    0
    Trophy Points:
    16
    Regardless of what Dark ORB says its -not- the way Apache is meant to be run.....restarting every 10 minutes. Users visiting your website(s) -will- experience a delay if they are browsing when the server is restarting.

    Having used Linux for about 5 years now I am not aware of any other program which has to restart Apache for -any- reason.


    What log files does Apache have to reopen anyways?


    TP
     
  12. howard

    howard Well-Known Member

    Joined:
    Apr 20, 2003
    Messages:
    233
    Likes Received:
    0
    Trophy Points:
    16
    i belive the stuff about the logfiles could be a smoke screen, the only real time would be during a log rotate and you can even prevent needing to do a restart if you log files to a pipe and use the supplied rotatelogs script which comes w/apache

    I think a more reasonable explantion maybe that due to the way cpanel uses the VirtualHost entires, apache needs to be restarted each time they are modifered /added/deleted to pickup changes, plesk for example has in the past (i am not sure weather it still does as i haven't used v5/6) restarted apache after each change (e.g. delete/modify/addition/etc) and this has caused some ill feelings in their forum (this should no means imply that plesk isn't good as its very capable at what it does), cpanel maybe just trying to get around this , or maybe there another reason which is unknown to us ?
     
  13. netwrkr

    netwrkr Well-Known Member

    Joined:
    Apr 12, 2003
    Messages:
    203
    Likes Received:
    0
    Trophy Points:
    16
    >> I think a more reasonable explantion maybe that due to the way cpanel uses the VirtualHost entires, apache needs to be restarted each time they are modifered /added/deleted to pickup changes,


    Apache does need to be restarted every time you add / remove entries from httpd.conf and Cpanel works fine in this aspect.

    The question I and others are trying to figure out is why -every- 10 minutes Apache is restarted with SIGUSR1. For example, I haven't added a user in the last 24 hours, yet Apache has restarted 6 times an hour * 24 hours = 144 restarts.

    Despite filling up my logs with SIGUSR1 events (144 a day), these restarts kill performace for even a moderately loaded web server. I am using some very high end servers and the restarts are very quick; how about those using lesser equipment (ie memory, CPU etc) with a higher system load? Where my Apache restarts in 1 second does it take their boxes 2 - 3 seconds (or longer) to stop and restart Apache?

    Of all the things I love about Cpanel this is pretty alarming. I modified cpanellogd to only restart every 12 hours but the latest (and possibly) every new release overwrites my changes. Pretty annoying to say the least.

    I cannot possibly understand the rationale behind this, unless as you allude to, DarkOrb is taking a shortcut and not using pipes to rotate files. (I am not a programmer and cannot speak to the correct technical implementation of rotating logs/files -- possibly someone else on this forum is and can.)


    TP
     
  14. dgbaker

    dgbaker Well-Known Member
    PartnerNOC

    Joined:
    Sep 20, 2002
    Messages:
    2,578
    Likes Received:
    3
    Trophy Points:
    38
    Location:
    Toronto, Ontario Canada
    cPanel Access Level:
    DataCenter Provider
    SIGUSR1 should have no effect on performance.

    Apache SIGUSR1 (apachectl graceful).

    It will kill off the old generation of child processes as they finish their requests and start new ones with a fresh config. This is transparent to users.

    See http://apache.hpi.uni-potsdam.de/document/4_4Overall_Behavior.html

    and refer to 4.4.4 for differences with normal restart and gracefull restart.
     
  15. dwayrynen

    dwayrynen Member

    Joined:
    Jun 19, 2002
    Messages:
    22
    Likes Received:
    0
    Trophy Points:
    1
    What's worse about gracefully restarting Apache that often is that sending a SIGUSR1 signal to cpanel's apache causes two bugs to rear their heads:

    1) Every SIGUSR1 causes apache to eat another file descriptor - by default you only get 1024 - so after a few days, you run out of file descriptors and apache can't spawn any children (cgi scripts) without errors

    and

    2) if you are running the (unsupported) tomcat java installation, each SIGUSR1 causes apache to eat up almost a meg of ram per instance of Apache. It doesn't take long to eat through two gigabytes of ram on a lightly loaded server (for us about every 12 hours) when this is happening.


    :-(

    Darin

    cPanel.net Support Ticket Number:
     

Share This Page