Apache graceful restart for stats update every 2 hours - causes server hang?

[odingalt]

Greetings and thank you in advance;

This is my details: CENTOS 5.9 x86_64 WHM 11.34.1 (build 7)

I am a small time business owner who tries to run his own server. My webhost has notifications set up to alert me via cellphone txt message when my httpd stops responding. I received such an alarm at 7:05PM local server time. I did not see the message on my cell phone for nearly 2 hours.

I checked my e-mail and my e-mail server was still running, meaning the problem is probably associated with httpd app. I SSH into my server and do a "TOP" command, I do not see HTTPD on the list, nor do I see PHP or RUBY which are also usually on the list.

To determine what killed HTTPD application, I checked the apache logs, here are the last few lines leading up to the point where the httpd application gets killed. As you can see the last thing that happens is a call for graceful restart:

[Tue Feb 12 19:01:28 2013] [error] [client 69.76.45.232] File does not exist: /home/protectedinfo/protectedinfo/skin/frontend/base/protectedinfo/protectedinfo/images, referer: http://www.protectedinfo$
[Tue Feb 12 19:01:59 2013] [error] [client 99.234.160.52] File does not exist: /home/protectedinfo/protectedinfo/skin/frontend/base/protectedinfo/images/bkg_nav1.gif, referer: http://www.protectedinfok$
[Tue Feb 12 19:02:17 2013] [error] [client 69.152.227.66] File does not exist: /home/protectedinfo/protectedinfo/skin/frontend/base/protectedinfo/images/bkg_nav1.gif, referer: http://www.protectedinfok$
[Tue Feb 12 19:02:18 2013] [error] [client 69.152.227.66] File does not exist: /home/protectedinfo/protectedinfo/skin/frontend/base/protectedinfo/protectedinfo/images, referer: http://www.protectedinfoking$
[Tue Feb 12 19:02:23 2013] [error] [client 174.152.119.9] File does not exist: /home/protectedinfo/protectedinfo/skin/frontend/base/protectedinfo/images/bkg_nav1.gif, referer: http://www.protectedinfok$
[Tue Feb 12 19:02:29 2013] [error] [client 69.152.227.66] File does not exist: /home/protectedinfo/protectedinfo/skin/frontend/base/protectedinfo/images/bkg_nav1.gif, referer: http://www.protectedinfok$
[Tue Feb 12 19:02:29 2013] [error] [client 69.152.227.66] File does not exist: /home/protectedinfo/protectedinfo/skin/frontend/base/protectedinfo/images/bkg_grid.gif, referer: http://www.protectedinfok$
[Tue Feb 12 19:02:29 2013] [error] [client 69.152.227.66] File does not exist: /home/protectedinfo/protectedinfo/skin/frontend/base/protectedinfo/protectedinfo/images, referer: http://www.protectedinfoking$
[Tue Feb 12 19:02:49 2013] [error] [client 50.89.71.248] File does not exist: /home/protectedinfo/protectedinfo/skin/frontend/base/protectedinfo/images/bkg_nav1.gif, referer: http://www.protectedinfoki$
[Tue Feb 12 19:02:49 2013] [error] [client 50.89.71.248] File does not exist: /home/protectedinfo/protectedinfo/skin/frontend/base/protectedinfo/protectedinfo/images, referer: http://www.protectedinfo$
[Tue Feb 12 19:03:08 2013] [error] [client 67.1.161.144] File does not exist: /home/protectedinfo/protectedinfo/skin/frontend/base/protectedinfo/images/bkg_nav1.gif, referer: http://www.protectedinfoki$
[Tue Feb 12 19:03:09 2013] [error] [client 67.1.161.144] File does not exist: /home/protectedinfo/protectedinfo/skin/frontend/base/protectedinfo/images/bkg_grid.gif, referer: http://www.protectedinfoki$
[Tue Feb 12 19:03:09 2013] [error] [client 67.1.161.144] File does not exist: /home/protectedinfo/protectedinfo/skin/frontend/base/protectedinfo/protectedinfo/images, referer: http://www.protectedinfo$
[Tue Feb 12 19:03:20 2013] [error] [client 123.151.139.215] File does not exist: /home/protectedinfo/protectedinfo/skin/frontend/base/default/css/em_megamenu.css, referer: http://www.tin$
[Tue Feb 12 19:03:30 2013] [error] [client 67.1.161.144] File does not exist: /home/protectedinfo/protectedinfo/skin/frontend/base/protectedinfo/images/bkg_nav1.gif, referer: http://www.protectedinfoki$
[Tue Feb 12 19:03:30 2013] [error] [client 123.151.139.215] File does not exist: /home/protectedinfo/protectedinfo/skin/frontend/base/default/css/em_productlabels.css, referer: http://ww$
[Tue Feb 12 19:03:30 2013] [error] [client 67.1.161.144] File does not exist: /home/protectedinfo/protectedinfo/skin/frontend/base/protectedinfo/images/bkg_diprotectedinfoder1.gif, referer: http://www.$
[Tue Feb 12 19:03:30 2013] [error] [client 67.1.161.144] File does not exist: /home/protectedinfo/protectedinfo/skin/frontend/base/protectedinfo/protectedinfo/images, referer: http://www.protectedinfo$
[Tue Feb 12 19:03:40 2013] [error] [client 123.151.139.215] File does not exist: /home/protectedinfo/protectedinfo/skin/frontend/base/default/css/em_quickshop.css, referer: http://www.ti$
[Tue Feb 12 19:03:50 2013] [error] [client 123.151.139.215] File does not exist: /home/protectedinfo/protectedinfo/skin/frontend/base/default/css/jquery.fancybox-1.3.4.css, referer: http$
[Tue Feb 12 19:04:00 2013] [error] [client 123.151.139.215] File does not exist: /home/protectedinfo/protectedinfo/skin/frontend/base/default/slideshowwidget, referer: http://www.ti$
[Tue Feb 12 19:04:58 2013] [error] [client 67.1.161.144] File does not exist: /home/protectedinfo/protectedinfo/skin/frontend/base/protectedinfo/images/bkg_nav1.gif, referer: http://www.protectedinfoki$
[Tue Feb 12 19:04:58 2013] [error] [client 67.1.161.144] File does not exist: /home/protectedinfo/protectedinfo/skin/frontend/base/protectedinfo/images/bkg_grid.gif, referer: http://www.protectedinfoki$
[Tue Feb 12 19:04:58 2013] [error] [client 67.1.161.144] File does not exist: /home/protectedinfo/protectedinfo/skin/frontend/base/protectedinfo/protectedinfo/images, referer: http://www.protectedinfo$
[Tue Feb 12 19:05:24 2013] [error] [client 67.1.161.144] File does not exist: /home/protectedinfo/protectedinfo/skin/frontend/base/protectedinfo/images/bkg_diprotectedinfoder1.gif, referer: http://www.$
[Tue Feb 12 19:05:39 2013] [notice] Graceful restart requested, doing restart

Ok so we have confirmed that a graceful restart was called for but then httpd was killed and the server no longer responded to http requests. After further research, someone in the forums suggested running this command: grep Graceful restart requested, doing restart /usr/local/apache/logs/error_log

This grep reveals the following information:


/usr/local/apache/logs/error_log:[Tue Feb 12 01:16:05 2013] [notice] Graceful restart requested, doing restart
/usr/local/apache/logs/error_log:[Tue Feb 12 03:12:50 2013] [notice] Graceful restart requested, doing restart
/usr/local/apache/logs/error_log:[Tue Feb 12 04:11:33 2013] [notice] Graceful restart requested, doing restart
/usr/local/apache/logs/error_log:[Tue Feb 12 05:11:44 2013] [notice] Graceful restart requested, doing restart
/usr/local/apache/logs/error_log:[Tue Feb 12 07:08:29 2013] [notice] Graceful restart requested, doing restart
/usr/local/apache/logs/error_log:[Tue Feb 12 09:05:14 2013] [notice] Graceful restart requested, doing restart
/usr/local/apache/logs/error_log:[Tue Feb 12 11:01:59 2013] [notice] Graceful restart requested, doing restart
/usr/local/apache/logs/error_log:[Tue Feb 12 13:15:24 2013] [notice] Graceful restart requested, doing restart
/usr/local/apache/logs/error_log:[Tue Feb 12 15:12:09 2013] [notice] Graceful restart requested, doing restart
/usr/local/apache/logs/error_log:[Tue Feb 12 17:08:54 2013] [notice] Graceful restart requested, doing restart
/usr/local/apache/logs/error_log:[Tue Feb 12 19:05:39 2013] [notice] Graceful restart requested, doing restart

As you can see, something is calling for apache graceful restarts every 2 hours. I am told that the Cpanel will call for a graceful restart every 2 hours on purpose, and this is related to statistics updating for the webalizer or awstats programs (neither of which I use).

So I guess the problem I have is that during a routine call for a graceful restart, httpd application died. How do I prevent this as it causes web server downtime?

Does anyone have hints on how I can either:

1.) determine the cause of the httpd crash and fix it? or
2.) install script or application that will automatically continue to restart httpd any time that httpd is killed?

Kind regards and thanks in advance, and please let me know if there are more logs or grep command I can run to provide more information. Of course my goal here is to eliminate downtime, 2 hours of downtime is very damaging to our SEO and Google SERPs and thus very bad for my small business!

 

[odingalt]

Automatic Graceful restarts -- httpd hangs up?

[Tue Feb 12 19:01:28 2013] [error] [client 69.76.45.232] File does not exist: /home/protectedinfo/protectedinfo/skin/frontend/base/protectedinfo/protectedinfo/images, referer: http://www.protectedinfo$
[Tue Feb 12 19:01:59 2013] [error] [client 99.234.160.52] File does not exist: /home/protectedinfo/protectedinfo/skin/frontend/base/protectedinfo/images/bkg_nav1.gif, referer: http://www.protectedinfok$
[Tue Feb 12 19:02:17 2013] [error] [client 69.152.227.66] File does not exist: /home/protectedinfo/protectedinfo/skin/frontend/base/protectedinfo/images/bkg_nav1.gif, referer: http://www.protectedinfok$
[Tue Feb 12 19:02:18 2013] [error] [client 69.152.227.66] File does not exist: /home/protectedinfo/protectedinfo/skin/frontend/base/protectedinfo/protectedinfo/images, referer: http://www.protectedinfoking$
[Tue Feb 12 19:02:23 2013] [error] [client 174.152.119.9] File does not exist: /home/protectedinfo/protectedinfo/skin/frontend/base/protectedinfo/images/bkg_nav1.gif, referer: http://www.protectedinfok$
[Tue Feb 12 19:02:29 2013] [error] [client 69.152.227.66] File does not exist: /home/protectedinfo/protectedinfo/skin/frontend/base/protectedinfo/images/bkg_nav1.gif, referer: http://www.protectedinfok$
[Tue Feb 12 19:02:29 2013] [error] [client 69.152.227.66] File does not exist: /home/protectedinfo/protectedinfo/skin/frontend/base/protectedinfo/images/bkg_grid.gif, referer: http://www.protectedinfok$
[Tue Feb 12 19:02:29 2013] [error] [client 69.152.227.66] File does not exist: /home/protectedinfo/protectedinfo/skin/frontend/base/protectedinfo/protectedinfo/images, referer: http://www.protectedinfoking$
[Tue Feb 12 19:02:49 2013] [error] [client 50.89.71.248] File does not exist: /home/protectedinfo/protectedinfo/skin/frontend/base/protectedinfo/images/bkg_nav1.gif, referer: http://www.protectedinfoki$
[Tue Feb 12 19:02:49 2013] [error] [client 50.89.71.248] File does not exist: /home/protectedinfo/protectedinfo/skin/frontend/base/protectedinfo/protectedinfo/images, referer: http://www.protectedinfo$
[Tue Feb 12 19:03:08 2013] [error] [client 67.1.161.144] File does not exist: /home/protectedinfo/protectedinfo/skin/frontend/base/protectedinfo/images/bkg_nav1.gif, referer: http://www.protectedinfoki$
[Tue Feb 12 19:03:09 2013] [error] [client 67.1.161.144] File does not exist: /home/protectedinfo/protectedinfo/skin/frontend/base/protectedinfo/images/bkg_grid.gif, referer: http://www.protectedinfoki$
[Tue Feb 12 19:03:09 2013] [error] [client 67.1.161.144] File does not exist: /home/protectedinfo/protectedinfo/skin/frontend/base/protectedinfo/protectedinfo/images, referer: http://www.protectedinfo$
[Tue Feb 12 19:03:20 2013] [error] [client 123.151.139.215] File does not exist: /home/protectedinfo/protectedinfo/skin/frontend/base/default/css/em_megamenu.css, referer: http://www.tin$
[Tue Feb 12 19:03:30 2013] [error] [client 67.1.161.144] File does not exist: /home/protectedinfo/protectedinfo/skin/frontend/base/protectedinfo/images/bkg_nav1.gif, referer: http://www.protectedinfoki$
[Tue Feb 12 19:03:30 2013] [error] [client 123.151.139.215] File does not exist: /home/protectedinfo/protectedinfo/skin/frontend/base/default/css/em_productlabels.css, referer: http://ww$
[Tue Feb 12 19:03:30 2013] [error] [client 67.1.161.144] File does not exist: /home/protectedinfo/protectedinfo/skin/frontend/base/protectedinfo/images/bkg_diprotectedinfoder1.gif, referer: http://www.$
[Tue Feb 12 19:03:30 2013] [error] [client 67.1.161.144] File does not exist: /home/protectedinfo/protectedinfo/skin/frontend/base/protectedinfo/protectedinfo/images, referer: http://www.protectedinfo$
[Tue Feb 12 19:03:40 2013] [error] [client 123.151.139.215] File does not exist: /home/protectedinfo/protectedinfo/skin/frontend/base/default/css/em_quickshop.css, referer: http://www.ti$
[Tue Feb 12 19:03:50 2013] [error] [client 123.151.139.215] File does not exist: /home/protectedinfo/protectedinfo/skin/frontend/base/default/css/jquery.fancybox-1.3.4.css, referer: http$
[Tue Feb 12 19:04:00 2013] [error] [client 123.151.139.215] File does not exist: /home/protectedinfo/protectedinfo/skin/frontend/base/default/slideshowwidget, referer: http://www.ti$
[Tue Feb 12 19:04:58 2013] [error] [client 67.1.161.144] File does not exist: /home/protectedinfo/protectedinfo/skin/frontend/base/protectedinfo/images/bkg_nav1.gif, referer: http://www.protectedinfoki$
[Tue Feb 12 19:04:58 2013] [error] [client 67.1.161.144] File does not exist: /home/protectedinfo/protectedinfo/skin/frontend/base/protectedinfo/images/bkg_grid.gif, referer: http://www.protectedinfoki$
[Tue Feb 12 19:04:58 2013] [error] [client 67.1.161.144] File does not exist: /home/protectedinfo/protectedinfo/skin/frontend/base/protectedinfo/protectedinfo/images, referer: http://www.protectedinfo$
[Tue Feb 12 19:05:24 2013] [error] [client 67.1.161.144] File does not exist: /home/protectedinfo/protectedinfo/skin/frontend/base/protectedinfo/images/bkg_diprotectedinfoder1.gif, referer: http://www.$
[Tue Feb 12 19:05:39 2013] [notice] Graceful restart requested, doing restart

 

[quietFinn]

If you do Apache graceful restart in SSH command prompt do you get any errors? And how long does it take?
i.e.:
/etc/init.d/httpd -k graceful

 

[kdean]

While this doesn't tell you why Apache is not restarting (which you still need to determine), this can lessen the potential for downtime...

Enabling Piped Log Configuration dramatically reduces the need for Apache to restart itself.

WHM » Service Configuration » Apache Configuration » Piped Log Configuration

So try enabling that and see how it goes.

You can view an apache restart log at:

/usr/local/cpanel/logs/safeapacherestart_log

With Piped Log enabled, my apache has only restarted itself (excluding manual restarts) 8 times since the beginning of the year.

 

[odingalt]

If you do Apache graceful restart in SSH command prompt do you get any errors? And how long does it take?
i.e.:
/etc/init.d/httpd -k graceful

Greetings quietFinn. I often restart httpd when making changes to php.ini (periodically the max sizes have to be temporarily increased to apply security patches) or other config files, and haven't had any problems in the past.

I ran your command:

/etc/init.d/httpd -k graceful

By running the command, httpd restarted quietly without displaying any message or error code on my command line.

 

[odingalt]

While this doesn't tell you why Apache is not restarting (which you still need to determine), this can lessen the potential for downtime...

Enabling Piped Log Configuration dramatically reduces the need for Apache to restart itself.

WHM » Service Configuration » Apache Configuration » Piped Log Configuration

So try enabling that and see how it goes.

You can view an apache restart log at:

/usr/local/cpanel/logs/safeapacherestart_log

With Piped Log enabled, my apache has only restarted itself (excluding manual restarts) 8 times since the beginning of the year.

Hi KDean. I followed your instructions, enabled Piped Logs and we'll see how it goes. At least if I can't determine the cause of the failiure, it seems like a good idea not to be incessantly restarting httpd all of the time. Especially since I'm not always close to a computer (or my cell phone) so hopefully this will ultimately lead to less "down time". Google really penalizes small businesses in the SERPs, even just for a couple hours of downtime.

Do you happen to know if this will change where I go to view logs in the future with Piped Logs enabled?

 

[kdean]

I believe all the logs are in the same place.