HTTPD Keep Restarting Every 10 Minutes

masood

Well-Known Member
Jun 14, 2002
78
0
156
[quote:0ba5286b43][i:0ba5286b43]Originally posted by bdraco[/i:0ba5286b43]
Its only going to gracefully restart apache if it processes a log file so apache will reopen the log file and begin writing to it again so you don't loose your stats. If you are getting segfaults, try commented out any extra apache modules you have added.. ie mod_perl, mod_headers .. etc as some of them are not compatible.
[/quote:0ba5286b43]

Ah! I C.. Its normal behaviour then.. :) no segfaults here
 

myros

Active Member
Dec 16, 2001
42
0
306
The server restarting apache every 10 minutes is hardly &normal& behavior ;) (And WHM sending me emails to let me know) Been stuck in that rut all day myself. Any resolution to this issue on the horizon?

Myros
 

masood

Well-Known Member
Jun 14, 2002
78
0
156
Well, it becomes a normal behaviour when the master of the cpanel say so :p
 

rpmws

Well-Known Member
Aug 14, 2001
1,798
9
318
back woods of NC, USA
[quote:0a74eadec0][i:0a74eadec0]Originally posted by myros[/i:0a74eadec0]

The server restarting apache every 10 minutes is hardly &normal& behavior ;) (And WHM sending me emails to let me know) Been stuck in that rut all day myself. Any resolution to this issue on the horizon?

Myros
[/quote:0a74eadec0]

Humm now that's weird ..I see the graceful restart every 10 minutes exactally for a long time now but not once have I seen a email from WHM ..are you talking about the normal chkservd..like.

&apache failed @ Thu May 9 23:03:58 2002. A restart was attempted automagicly.&

is that what you get every 10 minutes?
 

myros

Active Member
Dec 16, 2001
42
0
306
The error logs just before apache crashes go like this:

[Thu Jun 27 22:47:07 2002] [crit] (98)Address already in use: make_sock: could not bind to port 443
[Thu Jun 27 22:50:01 2002] [error] mod_ssl: Cannot open SSLSessionCache DBM file `/usr/local/apache/logs/ssl_scache' for status retrival (System error follows)
[Thu Jun 27 22:50:01 2002] [error] System: No such file or directory (errno: 2)
[Thu Jun 27 22:53:05 2002] [notice] SIGUSR1 received. Doing graceful restart


Its not chkservd causing the apache crash/restart as we had that disabled for quite a while today and the only difference was that it wouldnt auto restart. There are also some PHP errors as the server restarts each time. A lot still to try tomorrow to get to the bottom of this, ... but any advice would be MUCH appreciated at this point :)

How possible is it to do a rebuild of apache from the previous build version? Is there anywhere to get a hold of this? Right now Id be happy to go back just to get the servers working again.

Thanks,
Myros

edit - aye thats the email that WHM sends me every 10 minutes. The strange thing with WHM right now also - when apache is actualy running WHM lists is as &failed& but when it crashes WHM lists it as up lol. *scratches head*
 

rpmws

Well-Known Member
Aug 14, 2001
1,798
9
318
back woods of NC, USA
this one deffinately looks liek you have either a bad cert or your SSL is shot. It seems that SSL is the problem.
 

feanor

Well-Known Member
Aug 13, 2001
836
0
316
Is that directory path actually intact?

`/usr/local/apache/logs/ssl_scache' for status retrival (System error follows)


????????
That error seems pretty straightforward to me.
Either the logs directory it references got waxed somehow, or the file it is touching (or trying to) is hosed and needs to be removed. (And then have apache recreate it during restart)

If you continue to get an SSL error of that nature and all of the above things are in place- then most likely your openSSL install needs to be tweaked so that your apache build is matching up with the version it is expecting......... (more often than not this is fixed by installing the latest RPM or source of openssl for your particular distro version)


Just be careful with that! you can break a lot of things besides apache_ssl if you go down that road.

...just mentioning it as a possibility.
 

ahwheelock

Well-Known Member
Nov 5, 2001
49
0
306
I'm getting a similar, but slightly different error that I thought was a very unstable apache, but after reading this forum, am no longer sure. Apache is failing every 2-3 hours (not on a regular schedule) and the chksrvd is doing a great job of restarting it each time.

Two questions:

1) WHY! We can not find out why apache is dying and there is no log entry to help. I suspect it's related to what we've all been seeing.

2) Until #1 is answered, is there a way to have chkservd run every one min instead of every 9 minutes so that we don't have &noticable& downtime when apache (inevitably) fails?
 

ahwheelock

Well-Known Member
Nov 5, 2001
49
0
306
I'm getting a similar, but slightly different error that I thought was a very unstable apache, but after reading this forum, am no longer sure. Apache is failing every 2-3 hours (not on a regular schedule) and the chksrvd is doing a great job of restarting it each time.

Two questions:

1) WHY! We can not find out why apache is dying and there is no log entry to help. I suspect it's related to what we've all been seeing.

2) Until #1 is answered, is there a way to have chkservd run every one min instead of every 9 minutes so that we don't have &noticable& downtime when apache (inevitably) fails?
 

Juanra

Well-Known Member
Sep 22, 2001
777
0
316
Spain
[quote:d9b27857ba][i:d9b27857ba]Originally posted by myros[/i:d9b27857ba]
The error logs just before apache crashes go like this:
[Thu Jun 27 22:47:07 2002] [crit] (98)Address already in use: make_sock: could not bind to port 443
[Thu Jun 27 22:50:01 2002] [error] mod_ssl: Cannot open SSLSessionCache DBM file `/usr/local/apache/logs/ssl_scache' for status retrival (System error follows)
[Thu Jun 27 22:50:01 2002] [error] System: No such file or directory (errno: 2)
[Thu Jun 27 22:53:05 2002] [notice] SIGUSR1 received. Doing graceful restart[/quote:d9b27857ba]

We were getting errors like these too with an Apache installed on a new RedHat 7.3 system. Apache was crashing every hour. I can't find the thread now but Nick said something about a broken library in RedHat or something like that. 'Update System Software' from WHM solved the problem for us.

[b:d9b27857ba]Edit:[/b:d9b27857ba] I found the thread but can't be accessed. To be precise, what solved this problem for us was [b:d9b27857ba]not[/b:d9b27857ba] Update System Software from WHM, but running [b:d9b27857ba]/scripts/sysup[/b:d9b27857ba] from Shell.
 

myros

Active Member
Dec 16, 2001
42
0
306
[quote:cb6b8cc419][i:cb6b8cc419]Originally posted by feanor[/i:cb6b8cc419]


Is that directory path actually intact?

`/usr/local/apache/logs/ssl_scache' for status retrival (System error follows)

[/quote:cb6b8cc419]

The path is intact but there is no such file as ssl_scache, was going to try re-installing the ssl stuff today but you just scared me out of it lol, last thing I need is to hose more stuff ;)
Thanks for the input guys, will keep working at it.

Myros

EDIT - quick update to this - I watched the log area for a while and files ssl_scache.dir and ssl_scache.pag are showing up fine, my guess at this point is that that particular error is being shown because apache is restarting so much *shrugs*
 

myros

Active Member
Dec 16, 2001
42
0
306
Anton over at VO came up with this temporary fix to the &restarting apache every 10 minutes& problem -

1. Moved bin/httpd to bin/httpd.real
2. Wrote perl script with the following and named it bin/httpd:

#!/usr/bin/perl

$SIG{USR1} = IGN;

exec(&/usr/local/apache/bin/httpd.real & . join(& &, @ARGV));

3. Chmodded it 755 and restarted Apache.


Causes apache to ignore the shutdown command so would require a manual restart if anything is changed that would normaly require an apache restart.
Apache has been running 7 hours for me now without a problem, so it seems something was sending it an uneeded sigusr1. Hopefully somebody will figure out why pretty soon :)

Cheers,
Myros
 

Juanra

Well-Known Member
Sep 22, 2001
777
0
316
Spain
[quote:981bee1921][i:981bee1921]Originally posted by myros[/i:981bee1921]
Apache has been running 7 hours for me now without a problem, so it seems something was sending it an uneeded sigusr1. Hopefully somebody will figure out why pretty soon :)[/quote:981bee1921]

Myros, Nick explained it at the end of page 4 in this thread :)
 

masood

Well-Known Member
Jun 14, 2002
78
0
156
This is graceful restart. It should not create any problem. However if you are having sigfault etc., then that's a problem.
 

mesranet

Well-Known Member
May 6, 2002
133
0
316
Hmmm..

httpd restart on every 10 minutes is not a problem ? :p
But its problem to me, everytime httpd are restart, my 'Load everage' will going up to 12.00 12.00 12.00 and sometime its going more than that.
 

myros

Active Member
Dec 16, 2001
42
0
306
It sure was a problem, because it would shutdown every 10 minutes and wouldnt be restarted until the next time chkservd ran. The same thing is happening on quite a few machines over at VO. There is a real problem that wasnt covered earlier in this topic ..and it is most clearly NOT normal ;)

Httpd should run for hours if not days without any kind of restart.


Myros
 

Juanra

Well-Known Member
Sep 22, 2001
777
0
316
Spain
[quote:fc577d68b8][i:fc577d68b8]Originally posted by myros[/i:fc577d68b8]
It sure was a problem, because it would shutdown every 10 minutes and wouldnt be restarted until the next time chkservd ran. The same thing is happening on quite a few machines over at VO. There is a real problem that wasnt covered earlier in this topic ..and it is most clearly NOT normal ;)

Httpd should run for hours if not days without any kind of restart.[/quote:fc577d68b8]

Of course if your Apache can't restart after a SIGUSR1 signal then you have a problem, but by ignoring SIGUSR1 signals I think you are solving the wrong problem. If Cpanel needs to rotate the bytes logs to keep track of HTTP bandwidth, and you don't let it rotate them, then it will not keep track of HTTP bandwidth. I'd say the same applies for any other Apache log rotation.

Try running /scripts/sysup to solve your ssl_scache problem and let your Apache restart as often as it wishes :p

Note: I am more or less still a newbie, so you can safely ignore me. &One benefit of not knowing what you're talking about is that you can babble on in the full confidence that you must be saying something meaningful anyway& (Arjun Ray). ;)
 

mbarb

Member
Aug 14, 2001
23
0
301
Hmmm... Me to..

I just noticed this tonight, about every 10 minutes apache restarts. I have my error log from January when the server was put online. Back then I had the same errors but the were once per hour instead of every 10 minutes. It looks to me like June 11 was the first time it went from restarting once per hpur to restarting every 10 minutes. What changes were made around then, what task/program was changed to run from once per hour to once every 10 minutes. . I hope this helps to nail down whatever is causing this behavior.

Server Version: Apache/1.3.22 (Unix) mod_bwlimited/1.0 PHP/4.1.0 mod_log_bytes/0.3 FrontPage/5.0.2.2510 mod_ssl/2.8.5 OpenSSL/0.9.6b


I know I should upgrade to 1.3.26 but until some of the issues are resolved I'm staying here.

NOW
[Mon Jul 1 23:12:35 2002] [notice] SIGUSR1 received. Doing graceful restart
[Mon Jul 1 23:17:53 2002] [notice] SIGUSR1 received. Doing graceful restart
[Mon Jul 1 23:27:53 2002] [notice] SIGUSR1 received. Doing graceful restart
[Mon Jul 1 23:37:53 2002] [notice] SIGUSR1 received. Doing graceful restart
[Mon Jul 1 23:47:54 2002] [notice] SIGUSR1 received. Doing graceful restart
[Mon Jul 1 23:57:54 2002] [notice] SIGUSR1 received. Doing graceful restart
[Tue Jul 2 00:07:54 2002] [notice] SIGUSR1 received. Doing graceful restart
[Tue Jul 2 00:17:55 2002] [notice] SIGUSR1 received. Doing graceful restart
[Tue Jul 2 00:27:55 2002] [notice] SIGUSR1 received. Doing graceful restart
[Tue Jul 2 00:37:55 2002] [notice] SIGUSR1 received. Doing graceful restart

[Tue Jan 22 20:41:13 2002] [notice] SIGUSR1 received. Doing graceful restart
[Tue Jan 22 21:41:14 2002] [notice] SIGUSR1 received. Doing graceful restart
[Tue Jan 22 22:38:50 2002] [notice] SIGUSR1 received. Doing graceful restart
[Tue Jan 22 23:38:50 2002] [notice] SIGUSR1 received. Doing graceful restart
[Wed Jan 23 00:38:59 2002] [notice] SIGUSR1 received. Doing graceful restart
[Wed Jan 23 01:39:00 2002] [notice] SIGUSR1 received. Doing graceful restart
[Wed Jan 23 02:39:08 2002] [notice] SIGUSR1 received. Doing graceful restart
[Wed Jan 23 03:39:08 2002] [notice] SIGUSR1 received. Doing graceful restart
[Wed Jan 23 04:39:17 2002] [notice] SIGUSR1 received. Doing graceful restart
[Wed Jan 23 05:39:17 2002] [notice] SIGUSR1 received. Doing graceful restart
[Wed Jan 23 06:39:19 2002] [notice] SIGUSR1 received. Doing graceful restart
[Wed Jan 23 07:39:20 2002] [notice] SIGUSR1 received. Doing graceful restart
[Wed Jan 23 08:39:22 2002] [notice] SIGUSR1 received. Doing graceful restart
[Wed Jan 23 09:39:22 2002] [notice] SIGUSR1 received. Doing graceful restart

June 10/11
[Mon Jun 10 19:39:19 2002] [notice] SIGUSR1 received. Doing graceful restart
[Mon Jun 10 20:49:21 2002] [notice] SIGUSR1 received. Doing graceful restart
[Mon Jun 10 21:44:22 2002] [notice] SIGUSR1 received. Doing graceful restart
[Mon Jun 10 22:54:28 2002] [notice] SIGUSR1 received. Doing graceful restart
[Mon Jun 10 23:49:28 2002] [notice] SIGUSR1 received. Doing graceful restart
[Tue Jun 11 00:59:31 2002] [notice] SIGUSR1 received. Doing graceful restart
[Tue Jun 11 01:36:42 2002] [notice] SIGUSR1 received. Doing graceful restart
[Tue Jun 11 02:23:22 2002] [notice] SIGUSR1 received. Doing graceful restart
[Tue Jun 11 04:18:46 2002] [notice] SIGUSR1 received. Doing graceful restart
[Tue Jun 11 04:24:47 2002] [notice] SIGUSR1 received. Doing graceful restart
[Tue Jun 11 04:34:47 2002] [notice] SIGUSR1 received. Doing graceful restart
[Tue Jun 11 04:44:47 2002] [notice] SIGUSR1 received. Doing graceful restart
[Tue Jun 11 04:54:48 2002] [notice] SIGUSR1 received. Doing graceful restart