chkservd restarts Apache every 10 minutes

[Binkmeister]

We're having a problem with chkservd - after spending a couple of hours of digging through logs and such, it appears that chkservd is rebooting Apache like clockwork, every ten minutes, no matter what. I have no idea why it would do this.

We have a very active forum site, running mySQL, and when we have a larger number of concurrent users (> 170 or so), the site can completely lock up for a good five minutes, spouting error messages and generally being obnoxious.

A little more digging show chkservd trying to do a graceful restart of Apache, which means Apache and mySQL are apparently trying to communicate, but they're stumbling over each other. Watching the processlist in mySQL, we go from about current 5-10 processes to a couple of hundred. They then wait the 300 seconds or so to timeout, and mySQL starts heading back down to a normal range of under 50 processes. The web server apparently feels better at that point and the site is open again.

This is a showstopper for us, frankly. The symptom is chkservd rebooting Apache every 10 minutes... I don't know why it is doing this. Does anybody have any ideas? If I have to I'll disable that Perl script, but I have no doubt it would have some unpleasant side effects.

Any ideas are welcome at this point.

Thanks,
Kelly
BZPower.com

 

[dgbaker]

See this thread

http://forums.cpanel.net/showthread.php?s=&threadid=3009&highlight=apache+restart

This is "common" behavior.

 

[Binkmeister]

Thanks very much for the link. I've gone through it a couple of times, and the following seemed to be the best clue:

It's the Perl script cpanellogd running as a daemon that's causing the 10

minute reboot. I've modified my cpanellogd to only do this once per day so

I still get log processing and bandwidth limiting.

Replace this line
sleep(300);
with
sleep(43200);

43200 seconds (12 hours) instead of 300 seconds (5 minutes). Since the

script only processes logs every other loop, this will mean log processing

only happens once every 24 hours.

If you want to run phpaccelerator, replace all instances of
system(&killall -USR1 httpd&;
with
system(&/etc/rc.d/init.d/httpd stop&;
sleep(20);
system(&/etc/rc.d/init.d/httpd start&;

The above will guarantee that your server does a hard restart. In case the

httpd start command does not work, chkserv will still restart the server.

I tried implementing this but it's still going through its shutdown routine every 10 minutes. Does anyone have any other suggestions? This is really bringing the server to its knees.

- Kelly
BZPower.com

 

[jsteel]

Have you ssh'd in and run the following from the command-line?

/etc/rc.d/init.d/httpd stop

/etc/rc.d/init.d/httpd start

Check for any errors listed. It sounds like you have something in your httpd.conf file that is preventing Apache from loading altogether and chkservd is just trying to get it started every 8-10 minutes as it should.

Jaz

 

[testudo]

Are you sure this is what's causing your problem? Check your logs to make sure it's a "graceful restart" (SIGUSR1). If so, it shouldn't be hurting anything because a graceful restart does not drop connections but simply allows Apache to close/reopen the logs and run a configtest -- it's not a kill-and-restart kind of thing. This is normal behavior in the cPanel world and I haven't had it cause the problems you describe.

 

[Amanita]

I tried implementing this but it's still going through its shutdown routine every 10 minutes. Does anyone have any other suggestions? This is really bringing the server to its knees.

Did you restart Apache after making the changes?

 

[Website Rob]

A quick way to check your httpd.conf file is:

# /usr/local/apache/bin/httpd -t

Syntex OK << is all you want to see.

Any problems will be specifically mentioned and let you know where to start looking.

The restart every 10 Minutes does have something to do with the version of PHP being used and Cpanel. Not sure how it all works as I've just accepted it is something I have to live with, for right now anyway.

 

[Binkmeister]

Yep, I restarted Apache and I stopped getting the restart after 10 minutes errors. There were enormous amounts of SIGUSR1 notes in the logfile from before, and each one was followed by a noticeable slowdown of the website.

Since this didn't actually solve the problem, I went through my site and disabled heavy SQL functions in my forums, like search, mod CP, auto-generated updates for our news, etc. That has relieved the immediate issues, and after closely watching the mySQL process list, it looks like we need to configure the mySQL settings to alleviate the locking problem we were getting. I don't want to have to run my forum in "Lobotomy" mode forever.

So while the restarting Apache was a problem, it wasn't the problem, and I'll keep looking, this time at mySQL.

Thanks for the help, everyone!
- Kelly

 

[netwrkr]

Why restart apache every 10 minutes

What is the point of restarting apache every 10 minutes?

Please enlighten me because if you are browsing a website when the restart occurs it -does- cause a delay; alibet small but a delay nonetheless.

Thanks,

Tom

grep SIGUSR1 error_log | wc -l
2781

 

[dgbaker]

This is what Darkorb had said in another post.

qoute

SIGUSR1 is NORMAL. This is required for apache to reopen log files.

enqoute

 

[netwrkr]

Regardless of what Dark ORB says its -not- the way Apache is meant to be run.....restarting every 10 minutes. Users visiting your website(s) -will- experience a delay if they are browsing when the server is restarting.

Having used Linux for about 5 years now I am not aware of any other program which has to restart Apache for -any- reason.


What log files does Apache have to reopen anyways?


TP

 

[howard]

i belive the stuff about the logfiles could be a smoke screen, the only real time would be during a log rotate and you can even prevent needing to do a restart if you log files to a pipe and use the supplied rotatelogs script which comes w/apache

I think a more reasonable explantion maybe that due to the way cpanel uses the VirtualHost entires, apache needs to be restarted each time they are modifered /added/deleted to pickup changes, plesk for example has in the past (i am not sure weather it still does as i haven't used v5/6) restarted apache after each change (e.g. delete/modify/addition/etc) and this has caused some ill feelings in their forum (this should no means imply that plesk isn't good as its very capable at what it does), cpanel maybe just trying to get around this , or maybe there another reason which is unknown to us ?

 

[netwrkr]

>> I think a more reasonable explantion maybe that due to the way cpanel uses the VirtualHost entires, apache needs to be restarted each time they are modifered /added/deleted to pickup changes,


Apache does need to be restarted every time you add / remove entries from httpd.conf and Cpanel works fine in this aspect.

The question I and others are trying to figure out is why -every- 10 minutes Apache is restarted with SIGUSR1. For example, I haven't added a user in the last 24 hours, yet Apache has restarted 6 times an hour * 24 hours = 144 restarts.

Despite filling up my logs with SIGUSR1 events (144 a day), these restarts kill performace for even a moderately loaded web server. I am using some very high end servers and the restarts are very quick; how about those using lesser equipment (ie memory, CPU etc) with a higher system load? Where my Apache restarts in 1 second does it take their boxes 2 - 3 seconds (or longer) to stop and restart Apache?

Of all the things I love about Cpanel this is pretty alarming. I modified cpanellogd to only restart every 12 hours but the latest (and possibly) every new release overwrites my changes. Pretty annoying to say the least.

I cannot possibly understand the rationale behind this, unless as you allude to, DarkOrb is taking a shortcut and not using pipes to rotate files. (I am not a programmer and cannot speak to the correct technical implementation of rotating logs/files -- possibly someone else on this forum is and can.)


TP

 

[dgbaker]

SIGUSR1 should have no effect on performance.

Apache SIGUSR1 (apachectl graceful).

It will kill off the old generation of child processes as they finish their requests and start new ones with a fresh config. This is transparent to users.

See http://apache.hpi.uni-potsdam.de/document/4_4Overall_Behavior.html

and refer to 4.4.4 for differences with normal restart and gracefull restart.

 

[dwayrynen]

What's worse about gracefully restarting Apache that often is that sending a SIGUSR1 signal to cpanel's apache causes two bugs to rear their heads:

1) Every SIGUSR1 causes apache to eat another file descriptor - by default you only get 1024 - so after a few days, you run out of file descriptors and apache can't spawn any children (cgi scripts) without errors

and

2) if you are running the (unsupported) tomcat java installation, each SIGUSR1 causes apache to eat up almost a meg of ram per instance of Apache. It doesn't take long to eat through two gigabytes of ram on a lightly loaded server (for us about every 12 hours) when this is happening.


:-(

Darin

cPanel.net Support Ticket Number: