Why is apache restarting every two hours?

[aww]

I remember the problems people had awhile back under 11.25 with log rotation causing restarts
but this is 11.30

Almost every two hours (but not exactly) apache will restart.
It's not a segfault error, at least not any listed in the logs.

WHM 11.30.5 (build 6)
CENTOS 6.2 x86_64 xenpv
(mod_php+apc+suhosin)

I'd appreciate any advice in tracking this down - it's annoying because it's clearing the opcode cache too often

[Fri Jan 20 18:09:21 2012] [notice] Graceful restart requested, doing restart
[Fri Jan 20 20:06:09 2012] [notice] Graceful restart requested, doing restart
[Fri Jan 20 22:02:58 2012] [notice] Graceful restart requested, doing restart
[Sat Jan 21 00:18:01 2012] [notice] Graceful restart requested, doing restart
[Sat Jan 21 02:07:16 2012] [notice] Graceful restart requested, doing restart
[Sat Jan 21 04:14:56 2012] [notice] Graceful restart requested, doing restart
[Sat Jan 21 06:11:50 2012] [notice] Graceful restart requested, doing restart
[Sat Jan 21 08:08:37 2012] [notice] Graceful restart requested, doing restart
[Sat Jan 21 10:05:27 2012] [notice] Graceful restart requested, doing restart
[Sat Jan 21 12:02:16 2012] [notice] Graceful restart requested, doing restart
[Sat Jan 21 14:15:45 2012] [notice] Graceful restart requested, doing restart
[Sat Jan 21 16:12:34 2012] [notice] Graceful restart requested, doing restart
[Sat Jan 21 18:09:22 2012] [notice] Graceful restart requested, doing restart
[Sat Jan 21 20:06:10 2012] [notice] Graceful restart requested, doing restart
[Sat Jan 21 22:02:58 2012] [notice] Graceful restart requested, doing restart
[Sun Jan 22 00:23:59 2012] [notice] Graceful restart requested, doing restart
[Sun Jan 22 02:05:22 2012] [notice] Graceful restart requested, doing restart
[Sun Jan 22 04:14:51 2012] [notice] Graceful restart requested, doing restart
[Sun Jan 22 06:11:41 2012] [notice] Graceful restart requested, doing restart
[Sun Jan 22 08:08:28 2012] [notice] Graceful restart requested, doing restart
[Sun Jan 22 10:05:15 2012] [notice] Graceful restart requested, doing restart
[Sun Jan 22 12:02:03 2012] [notice] Graceful restart requested, doing restart
[Sun Jan 22 14:15:31 2012] [notice] Graceful restart requested, doing restart
[Sun Jan 22 16:12:19 2012] [notice] Graceful restart requested, doing restart
[Sun Jan 22 18:09:08 2012] [notice] Graceful restart requested, doing restart
[Sun Jan 22 20:05:55 2012] [notice] Graceful restart requested, doing restart
[Sun Jan 22 22:02:44 2012] [notice] Graceful restart requested, doing restart
[Mon Jan 23 00:16:14 2012] [notice] Graceful restart requested, doing restart
[Mon Jan 23 02:06:55 2012] [notice] Graceful restart requested, doing restart
[Mon Jan 23 04:01:13 2012] [notice] Graceful restart requested, doing restart
[Mon Jan 23 06:14:44 2012] [notice] Graceful restart requested, doing restart
[Mon Jan 23 08:11:31 2012] [notice] Graceful restart requested, doing restart
[Mon Jan 23 10:08:20 2012] [notice] Graceful restart requested, doing restart
[Mon Jan 23 12:05:10 2012] [notice] Graceful restart requested, doing restart

[aww]

Looking back further in the logs it looks like this started happening January 17th for what it's worth.

Before then restarts were "only" three time a day at 0,2 and 22 hours

So the question is did I change a setting or did cpanel change something.

[Tue Jan 10 00:10:37 2012] [notice] Graceful restart requested, doing restart
[Tue Jan 10 02:07:25 2012] [notice] Graceful restart requested, doing restart
[Tue Jan 10 22:12:35 2012] [notice] Graceful restart requested, doing restart
[Wed Jan 11 00:04:53 2012] [notice] Graceful restart requested, doing restart
[Wed Jan 11 02:01:40 2012] [notice] Graceful restart requested, doing restart
[Wed Jan 11 22:01:55 2012] [notice] Graceful restart requested, doing restart
[Thu Jan 12 00:16:05 2012] [notice] Graceful restart requested, doing restart
[Thu Jan 12 02:12:52 2012] [notice] Graceful restart requested, doing restart
[Thu Jan 12 22:13:03 2012] [notice] Graceful restart requested, doing restart
[Fri Jan 13 00:26:18 2012] [notice] Graceful restart requested, doing restart
[Fri Jan 13 02:06:25 2012] [notice] Graceful restart requested, doing restart
[Fri Jan 13 22:06:37 2012] [notice] Graceful restart requested, doing restart
[Sat Jan 14 00:11:41 2012] [notice] Graceful restart requested, doing restart
[Sat Jan 14 02:03:51 2012] [notice] Graceful restart requested, doing restart
[Sat Jan 14 22:04:02 2012] [notice] Graceful restart requested, doing restart
[Sun Jan 15 00:13:10 2012] [notice] Graceful restart requested, doing restart
[Sun Jan 15 02:02:53 2012] [notice] Graceful restart requested, doing restart
[Sun Jan 15 22:03:07 2012] [notice] Graceful restart requested, doing restart
[Mon Jan 16 00:30:20 2012] [notice] Graceful restart requested, doing restart
[Mon Jan 16 02:02:53 2012] [notice] Graceful restart requested, doing restart
[Mon Jan 16 22:03:04 2012] [notice] Graceful restart requested, doing restart

[aww]

I've been searching for possible causes

The log rotation is at the default of 300MB

While the error_log is fine, I noticed there are at least three domlogs which are way, way over 300MB

One domlog is 918801950 bytes for example

Is it possible these are causing periodic apache restart?
Why are logs failing to rotate?

researching more...


oh! Piped Apache Logs is disabled by default?
any reason why I *wouldn't* want to turn that on ?

[cPanelTristan]

I cannot think of a reason why you wouldn't want to enable piped logging. I'd actually suggest enabling that logging option in WHM > Apache Configuration > Piped Log Configuration area to see if the graceful restarts every 2 hours cease to occur.

[alphawolf50]

What executable are the logs being piped to? rotatelogs? Is the processing being spawned without a shell? The closest thing to an answer I found was:

Where does Apache pipe the logs to?

When the piped logging method is enabled Apache will pipe logs to a common target process, a program started by Apache that is listening on the pipe; the receiving process will then automatically split the logs into their expected per-domain version. Please note that this does not change the location of nor the content of any Apache log files.

"A common target process" seems like a rather generic answer

[cPanelTristan]

If you enable piped logging, here are the results on a machine:

root@host [/usr/local/cpanel/bin]# ps aux | grep split
root 9730 0.0 0.0 1364 440 ? S 01:14 0:00 /usr/local/cpanel/bin/splitlogs --main=host.daniellewallace.com --suffix=-bytes_log
root 9731 0.0 0.0 1364 452 ? S 01:14 0:00 /usr/local/cpanel/bin/splitlogs --main=host.daniellewallace.com --mainout=/usr/local/apache/logs/access_log

root@host [/usr/local/cpanel/bin]# lsof -p 9730
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
splitlogs 9730 root cwd DIR 0,78 4096 125836397 /
splitlogs 9730 root rtd DIR 0,78 4096 125836397 /
splitlogs 9730 root txt REG 0,78 1064096 131793016 /usr/local/cpanel/bin/splitlogs
splitlogs 9730 root mem REG 8,3 131793016 /usr/local/cpanel/bin/splitlogs (path dev=0,78)
splitlogs 9730 root 0r FIFO 0,6 354759040 pipe
splitlogs 9730 root 1w CHR 1,3 125836740 /dev/null
splitlogs 9730 root 2w REG 0,78 479019 138611569 /usr/local/apache/logs/error_log
splitlogs 9730 root 3w REG 0,78 108 131335242 /usr/local/cpanel/logs/splitlogs_log

Basically, enabling piped logging will add entries for splitlogs into /usr/local/apache/conf/httpd.conf file and, if you check the processes, it shows what binary is being used.

[aww]

Ah twelve hours uptime without restart - finally fixed.

I thought after 11.25 it was the default, I guess not.

Everyone with an opcode cache (which should be everyone) should have piped logging enabled.

The crazy oddity is why was it only 2-3 times a day before the 17th and every 2 hours afterwards.

Anyway, the error log is realtime and that's all that matters to me, so I don't care there's a helper splitting it later.

[cPanelBrianO]

By default, Apache will restart every 2 hours as well as every 24 hours. This is normal and intended.

Why at those intervals?

Those are the default log/statistics processing intervals for a cPanel & WHM server.

WHM -> Statistics Software Configuration
Log Processing Frequency [ 24 ] hours
Bandwidth Processing Frequency [ 2 ] hours

If your Apache is not restarting at those two configured intervals (whatever they may be) then this means either...

[1] You have Piped Logging enabled
- or -
[2] Something is broken on your server that needs investigation.


The reason the restarts occur is that Apache must be restarted to grab the new logs' file handles (since the logs being processed are rotated out during processing). If Apache were not restarted, bandwidth/stats would fail to log and that data be lost.

Piped Logging, however, aims to do two things.
[1] Reduce the number of open file handles on a server
-and-
[2] Eliminate the need for Apache to be restarted during stats/bandwidth logging processing.

In prior releases, Piped Logging was considered experimental. It no longer carries that label and is certainly beneficial. It's certainly plausible that this will become the default in the future, although I can't speak to when/if that will occur.

Note that Apache will restart in one other situation as well, when a cPanel user performs a change that involves modifying httpd.conf (adding/removing domains, for example). When that occurs, Apache must be restarted.

Any other restarts would be manually requested restarts or something like chkservd restarting the service if it believes it to be offline/down.

[aww]

Ah and that explains why it changed on the 17th, I vaguely remember tinkering with the stats processing intervals.

Pleased with piped so far, zero problems.

[alphawolf50]

Everyone with an opcode cache (which should be everyone)...

People using suPHP can't use an opcode cache