Greetings and thank you in advance;
This is my details: CENTOS 5.9 x86_64 WHM 11.34.1 (build 7)
I am a small time business owner who tries to run his own server. My webhost has notifications set up to alert me via cellphone txt message when my httpd stops responding. I received such an alarm at 7:05PM local server time. I did not see the message on my cell phone for nearly 2 hours.
I checked my e-mail and my e-mail server was still running, meaning the problem is probably associated with httpd app. I SSH into my server and do a "TOP" command, I do not see HTTPD on the list, nor do I see PHP or RUBY which are also usually on the list.
To determine what killed HTTPD application, I checked the apache logs, here are the last few lines leading up to the point where the httpd application gets killed. As you can see the last thing that happens is a call for graceful restart:
[Tue Feb 12 19:01:28 2013] [error] [client 69.76.45.232] File does not exist: /home/protectedinfo/protectedinfo/skin/frontend/base/protectedinfo/protectedinfo/images, referer: http://www.protectedinfo$
[Tue Feb 12 19:01:59 2013] [error] [client 99.234.160.52] File does not exist: /home/protectedinfo/protectedinfo/skin/frontend/base/protectedinfo/images/bkg_nav1.gif, referer: http://www.protectedinfok$
[Tue Feb 12 19:02:17 2013] [error] [client 69.152.227.66] File does not exist: /home/protectedinfo/protectedinfo/skin/frontend/base/protectedinfo/images/bkg_nav1.gif, referer: http://www.protectedinfok$
[Tue Feb 12 19:02:18 2013] [error] [client 69.152.227.66] File does not exist: /home/protectedinfo/protectedinfo/skin/frontend/base/protectedinfo/protectedinfo/images, referer: http://www.protectedinfoking$
[Tue Feb 12 19:02:23 2013] [error] [client 174.152.119.9] File does not exist: /home/protectedinfo/protectedinfo/skin/frontend/base/protectedinfo/images/bkg_nav1.gif, referer: http://www.protectedinfok$
[Tue Feb 12 19:02:29 2013] [error] [client 69.152.227.66] File does not exist: /home/protectedinfo/protectedinfo/skin/frontend/base/protectedinfo/images/bkg_nav1.gif, referer: http://www.protectedinfok$
[Tue Feb 12 19:02:29 2013] [error] [client 69.152.227.66] File does not exist: /home/protectedinfo/protectedinfo/skin/frontend/base/protectedinfo/images/bkg_grid.gif, referer: http://www.protectedinfok$
[Tue Feb 12 19:02:29 2013] [error] [client 69.152.227.66] File does not exist: /home/protectedinfo/protectedinfo/skin/frontend/base/protectedinfo/protectedinfo/images, referer: http://www.protectedinfoking$
[Tue Feb 12 19:02:49 2013] [error] [client 50.89.71.248] File does not exist: /home/protectedinfo/protectedinfo/skin/frontend/base/protectedinfo/images/bkg_nav1.gif, referer: http://www.protectedinfoki$
[Tue Feb 12 19:02:49 2013] [error] [client 50.89.71.248] File does not exist: /home/protectedinfo/protectedinfo/skin/frontend/base/protectedinfo/protectedinfo/images, referer: http://www.protectedinfo$
[Tue Feb 12 19:03:08 2013] [error] [client 67.1.161.144] File does not exist: /home/protectedinfo/protectedinfo/skin/frontend/base/protectedinfo/images/bkg_nav1.gif, referer: http://www.protectedinfoki$
[Tue Feb 12 19:03:09 2013] [error] [client 67.1.161.144] File does not exist: /home/protectedinfo/protectedinfo/skin/frontend/base/protectedinfo/images/bkg_grid.gif, referer: http://www.protectedinfoki$
[Tue Feb 12 19:03:09 2013] [error] [client 67.1.161.144] File does not exist: /home/protectedinfo/protectedinfo/skin/frontend/base/protectedinfo/protectedinfo/images, referer: http://www.protectedinfo$
[Tue Feb 12 19:03:20 2013] [error] [client 123.151.139.215] File does not exist: /home/protectedinfo/protectedinfo/skin/frontend/base/default/css/em_megamenu.css, referer: http://www.tin$
[Tue Feb 12 19:03:30 2013] [error] [client 67.1.161.144] File does not exist: /home/protectedinfo/protectedinfo/skin/frontend/base/protectedinfo/images/bkg_nav1.gif, referer: http://www.protectedinfoki$
[Tue Feb 12 19:03:30 2013] [error] [client 123.151.139.215] File does not exist: /home/protectedinfo/protectedinfo/skin/frontend/base/default/css/em_productlabels.css, referer: http://ww$
[Tue Feb 12 19:03:30 2013] [error] [client 67.1.161.144] File does not exist: /home/protectedinfo/protectedinfo/skin/frontend/base/protectedinfo/images/bkg_diprotectedinfoder1.gif, referer: http://www.$
[Tue Feb 12 19:03:30 2013] [error] [client 67.1.161.144] File does not exist: /home/protectedinfo/protectedinfo/skin/frontend/base/protectedinfo/protectedinfo/images, referer: http://www.protectedinfo$
[Tue Feb 12 19:03:40 2013] [error] [client 123.151.139.215] File does not exist: /home/protectedinfo/protectedinfo/skin/frontend/base/default/css/em_quickshop.css, referer: http://www.ti$
[Tue Feb 12 19:03:50 2013] [error] [client 123.151.139.215] File does not exist: /home/protectedinfo/protectedinfo/skin/frontend/base/default/css/jquery.fancybox-1.3.4.css, referer: http$
[Tue Feb 12 19:04:00 2013] [error] [client 123.151.139.215] File does not exist: /home/protectedinfo/protectedinfo/skin/frontend/base/default/slideshowwidget, referer: http://www.ti$
[Tue Feb 12 19:04:58 2013] [error] [client 67.1.161.144] File does not exist: /home/protectedinfo/protectedinfo/skin/frontend/base/protectedinfo/images/bkg_nav1.gif, referer: http://www.protectedinfoki$
[Tue Feb 12 19:04:58 2013] [error] [client 67.1.161.144] File does not exist: /home/protectedinfo/protectedinfo/skin/frontend/base/protectedinfo/images/bkg_grid.gif, referer: http://www.protectedinfoki$
[Tue Feb 12 19:04:58 2013] [error] [client 67.1.161.144] File does not exist: /home/protectedinfo/protectedinfo/skin/frontend/base/protectedinfo/protectedinfo/images, referer: http://www.protectedinfo$
[Tue Feb 12 19:05:24 2013] [error] [client 67.1.161.144] File does not exist: /home/protectedinfo/protectedinfo/skin/frontend/base/protectedinfo/images/bkg_diprotectedinfoder1.gif, referer: http://www.$
[Tue Feb 12 19:05:39 2013] [notice] Graceful restart requested, doing restart
Ok so we have confirmed that a graceful restart was called for but then httpd was killed and the server no longer responded to http requests. After further research, someone in the forums suggested running this command: grep Graceful restart requested, doing restart /usr/local/apache/logs/error_log
This grep reveals the following information:
/usr/local/apache/logs/error_log:[Tue Feb 12 01:16:05 2013] [notice] Graceful restart requested, doing restart
/usr/local/apache/logs/error_log:[Tue Feb 12 03:12:50 2013] [notice] Graceful restart requested, doing restart
/usr/local/apache/logs/error_log:[Tue Feb 12 04:11:33 2013] [notice] Graceful restart requested, doing restart
/usr/local/apache/logs/error_log:[Tue Feb 12 05:11:44 2013] [notice] Graceful restart requested, doing restart
/usr/local/apache/logs/error_log:[Tue Feb 12 07:08:29 2013] [notice] Graceful restart requested, doing restart
/usr/local/apache/logs/error_log:[Tue Feb 12 09:05:14 2013] [notice] Graceful restart requested, doing restart
/usr/local/apache/logs/error_log:[Tue Feb 12 11:01:59 2013] [notice] Graceful restart requested, doing restart
/usr/local/apache/logs/error_log:[Tue Feb 12 13:15:24 2013] [notice] Graceful restart requested, doing restart
/usr/local/apache/logs/error_log:[Tue Feb 12 15:12:09 2013] [notice] Graceful restart requested, doing restart
/usr/local/apache/logs/error_log:[Tue Feb 12 17:08:54 2013] [notice] Graceful restart requested, doing restart
/usr/local/apache/logs/error_log:[Tue Feb 12 19:05:39 2013] [notice] Graceful restart requested, doing restart
As you can see, something is calling for apache graceful restarts every 2 hours. I am told that the Cpanel will call for a graceful restart every 2 hours on purpose, and this is related to statistics updating for the webalizer or awstats programs (neither of which I use).
So I guess the problem I have is that during a routine call for a graceful restart, httpd application died. How do I prevent this as it causes web server downtime?
Does anyone have hints on how I can either:
1.) determine the cause of the httpd crash and fix it? or
2.) install script or application that will automatically continue to restart httpd any time that httpd is killed?
Kind regards and thanks in advance, and please let me know if there are more logs or grep command I can run to provide more information. Of course my goal here is to eliminate downtime, 2 hours of downtime is very damaging to our SEO and Google SERPs and thus very bad for my small business!
This is my details: CENTOS 5.9 x86_64 WHM 11.34.1 (build 7)
I am a small time business owner who tries to run his own server. My webhost has notifications set up to alert me via cellphone txt message when my httpd stops responding. I received such an alarm at 7:05PM local server time. I did not see the message on my cell phone for nearly 2 hours.
I checked my e-mail and my e-mail server was still running, meaning the problem is probably associated with httpd app. I SSH into my server and do a "TOP" command, I do not see HTTPD on the list, nor do I see PHP or RUBY which are also usually on the list.
To determine what killed HTTPD application, I checked the apache logs, here are the last few lines leading up to the point where the httpd application gets killed. As you can see the last thing that happens is a call for graceful restart:
[Tue Feb 12 19:01:28 2013] [error] [client 69.76.45.232] File does not exist: /home/protectedinfo/protectedinfo/skin/frontend/base/protectedinfo/protectedinfo/images, referer: http://www.protectedinfo$
[Tue Feb 12 19:01:59 2013] [error] [client 99.234.160.52] File does not exist: /home/protectedinfo/protectedinfo/skin/frontend/base/protectedinfo/images/bkg_nav1.gif, referer: http://www.protectedinfok$
[Tue Feb 12 19:02:17 2013] [error] [client 69.152.227.66] File does not exist: /home/protectedinfo/protectedinfo/skin/frontend/base/protectedinfo/images/bkg_nav1.gif, referer: http://www.protectedinfok$
[Tue Feb 12 19:02:18 2013] [error] [client 69.152.227.66] File does not exist: /home/protectedinfo/protectedinfo/skin/frontend/base/protectedinfo/protectedinfo/images, referer: http://www.protectedinfoking$
[Tue Feb 12 19:02:23 2013] [error] [client 174.152.119.9] File does not exist: /home/protectedinfo/protectedinfo/skin/frontend/base/protectedinfo/images/bkg_nav1.gif, referer: http://www.protectedinfok$
[Tue Feb 12 19:02:29 2013] [error] [client 69.152.227.66] File does not exist: /home/protectedinfo/protectedinfo/skin/frontend/base/protectedinfo/images/bkg_nav1.gif, referer: http://www.protectedinfok$
[Tue Feb 12 19:02:29 2013] [error] [client 69.152.227.66] File does not exist: /home/protectedinfo/protectedinfo/skin/frontend/base/protectedinfo/images/bkg_grid.gif, referer: http://www.protectedinfok$
[Tue Feb 12 19:02:29 2013] [error] [client 69.152.227.66] File does not exist: /home/protectedinfo/protectedinfo/skin/frontend/base/protectedinfo/protectedinfo/images, referer: http://www.protectedinfoking$
[Tue Feb 12 19:02:49 2013] [error] [client 50.89.71.248] File does not exist: /home/protectedinfo/protectedinfo/skin/frontend/base/protectedinfo/images/bkg_nav1.gif, referer: http://www.protectedinfoki$
[Tue Feb 12 19:02:49 2013] [error] [client 50.89.71.248] File does not exist: /home/protectedinfo/protectedinfo/skin/frontend/base/protectedinfo/protectedinfo/images, referer: http://www.protectedinfo$
[Tue Feb 12 19:03:08 2013] [error] [client 67.1.161.144] File does not exist: /home/protectedinfo/protectedinfo/skin/frontend/base/protectedinfo/images/bkg_nav1.gif, referer: http://www.protectedinfoki$
[Tue Feb 12 19:03:09 2013] [error] [client 67.1.161.144] File does not exist: /home/protectedinfo/protectedinfo/skin/frontend/base/protectedinfo/images/bkg_grid.gif, referer: http://www.protectedinfoki$
[Tue Feb 12 19:03:09 2013] [error] [client 67.1.161.144] File does not exist: /home/protectedinfo/protectedinfo/skin/frontend/base/protectedinfo/protectedinfo/images, referer: http://www.protectedinfo$
[Tue Feb 12 19:03:20 2013] [error] [client 123.151.139.215] File does not exist: /home/protectedinfo/protectedinfo/skin/frontend/base/default/css/em_megamenu.css, referer: http://www.tin$
[Tue Feb 12 19:03:30 2013] [error] [client 67.1.161.144] File does not exist: /home/protectedinfo/protectedinfo/skin/frontend/base/protectedinfo/images/bkg_nav1.gif, referer: http://www.protectedinfoki$
[Tue Feb 12 19:03:30 2013] [error] [client 123.151.139.215] File does not exist: /home/protectedinfo/protectedinfo/skin/frontend/base/default/css/em_productlabels.css, referer: http://ww$
[Tue Feb 12 19:03:30 2013] [error] [client 67.1.161.144] File does not exist: /home/protectedinfo/protectedinfo/skin/frontend/base/protectedinfo/images/bkg_diprotectedinfoder1.gif, referer: http://www.$
[Tue Feb 12 19:03:30 2013] [error] [client 67.1.161.144] File does not exist: /home/protectedinfo/protectedinfo/skin/frontend/base/protectedinfo/protectedinfo/images, referer: http://www.protectedinfo$
[Tue Feb 12 19:03:40 2013] [error] [client 123.151.139.215] File does not exist: /home/protectedinfo/protectedinfo/skin/frontend/base/default/css/em_quickshop.css, referer: http://www.ti$
[Tue Feb 12 19:03:50 2013] [error] [client 123.151.139.215] File does not exist: /home/protectedinfo/protectedinfo/skin/frontend/base/default/css/jquery.fancybox-1.3.4.css, referer: http$
[Tue Feb 12 19:04:00 2013] [error] [client 123.151.139.215] File does not exist: /home/protectedinfo/protectedinfo/skin/frontend/base/default/slideshowwidget, referer: http://www.ti$
[Tue Feb 12 19:04:58 2013] [error] [client 67.1.161.144] File does not exist: /home/protectedinfo/protectedinfo/skin/frontend/base/protectedinfo/images/bkg_nav1.gif, referer: http://www.protectedinfoki$
[Tue Feb 12 19:04:58 2013] [error] [client 67.1.161.144] File does not exist: /home/protectedinfo/protectedinfo/skin/frontend/base/protectedinfo/images/bkg_grid.gif, referer: http://www.protectedinfoki$
[Tue Feb 12 19:04:58 2013] [error] [client 67.1.161.144] File does not exist: /home/protectedinfo/protectedinfo/skin/frontend/base/protectedinfo/protectedinfo/images, referer: http://www.protectedinfo$
[Tue Feb 12 19:05:24 2013] [error] [client 67.1.161.144] File does not exist: /home/protectedinfo/protectedinfo/skin/frontend/base/protectedinfo/images/bkg_diprotectedinfoder1.gif, referer: http://www.$
[Tue Feb 12 19:05:39 2013] [notice] Graceful restart requested, doing restart
Ok so we have confirmed that a graceful restart was called for but then httpd was killed and the server no longer responded to http requests. After further research, someone in the forums suggested running this command: grep Graceful restart requested, doing restart /usr/local/apache/logs/error_log
This grep reveals the following information:
/usr/local/apache/logs/error_log:[Tue Feb 12 01:16:05 2013] [notice] Graceful restart requested, doing restart
/usr/local/apache/logs/error_log:[Tue Feb 12 03:12:50 2013] [notice] Graceful restart requested, doing restart
/usr/local/apache/logs/error_log:[Tue Feb 12 04:11:33 2013] [notice] Graceful restart requested, doing restart
/usr/local/apache/logs/error_log:[Tue Feb 12 05:11:44 2013] [notice] Graceful restart requested, doing restart
/usr/local/apache/logs/error_log:[Tue Feb 12 07:08:29 2013] [notice] Graceful restart requested, doing restart
/usr/local/apache/logs/error_log:[Tue Feb 12 09:05:14 2013] [notice] Graceful restart requested, doing restart
/usr/local/apache/logs/error_log:[Tue Feb 12 11:01:59 2013] [notice] Graceful restart requested, doing restart
/usr/local/apache/logs/error_log:[Tue Feb 12 13:15:24 2013] [notice] Graceful restart requested, doing restart
/usr/local/apache/logs/error_log:[Tue Feb 12 15:12:09 2013] [notice] Graceful restart requested, doing restart
/usr/local/apache/logs/error_log:[Tue Feb 12 17:08:54 2013] [notice] Graceful restart requested, doing restart
/usr/local/apache/logs/error_log:[Tue Feb 12 19:05:39 2013] [notice] Graceful restart requested, doing restart
As you can see, something is calling for apache graceful restarts every 2 hours. I am told that the Cpanel will call for a graceful restart every 2 hours on purpose, and this is related to statistics updating for the webalizer or awstats programs (neither of which I use).
So I guess the problem I have is that during a routine call for a graceful restart, httpd application died. How do I prevent this as it causes web server downtime?
Does anyone have hints on how I can either:
1.) determine the cause of the httpd crash and fix it? or
2.) install script or application that will automatically continue to restart httpd any time that httpd is killed?
Kind regards and thanks in advance, and please let me know if there are more logs or grep command I can run to provide more information. Of course my goal here is to eliminate downtime, 2 hours of downtime is very damaging to our SEO and Google SERPs and thus very bad for my small business!