500 Error all pages after midnight

[Jake Ruston]

Hello,

Over the past month or two I've been facing an issue where shortly after midnight server time (usually at 10 - 20 seconds after 0:00) all web requests return 500 internal server errors. It does this until I restart the server, which makes it completely functional again. Without a restart the server will remain broken. This is a production server so obviously I can't keep checking the website after midnight everyday to check if it's broken.

All of the websites are built on PHP, with suPHP. Interestingly, scripts run by cron job seem to function fine. It's just web requests from clients which don't work and give a 500.

Looking at the apache log, I can logs like this:

[Wed May 18 00:00:11.812048 2016] [mpm_prefork:notice] [pid 12823] AH00169: caught SIGTERM, shutting down
[Wed May 18 00:00:13.002824 2016] [ssl:warn] [pid 31758] AH01909: <ONE OF MY DOMAIN NAMES>:443:0 server certificate does NOT include an ID which matches the server name
[Wed May 18 00:00:13.004000 2016] [suexec:notice] [pid 31758] AH01232: suEXEC mechanism enabled (wrapper: /usr/local/apache/bin/suexec)
[Wed May 18 00:00:13.004065 2016] [:notice] [pid 31758] ModSecurity for Apache/2.9.0 (http://www.modsecurity.org/) configured.
[Wed May 18 00:00:13.004071 2016] [:notice] [pid 31758] ModSecurity: APR compiled version="1.5.2"; loaded version="1.5.2"
[Wed May 18 00:00:13.004076 2016] [:notice] [pid 31758] ModSecurity: PCRE compiled version="8.38 "; loaded version="8.38 2015-11-23"
[Wed May 18 00:00:13.004086 2016] [:notice] [pid 31758] ModSecurity: LUA compiled version="Lua 5.1"
[Wed May 18 00:00:13.004096 2016] [:notice] [pid 31758] ModSecurity: LIBXML compiled version="2.9.2"
[Wed May 18 00:00:13.004099 2016] [:notice] [pid 31758] ModSecurity: Status engine is currently disabled, enable it by set SecStatusEngine to On.
[Wed May 18 00:00:14.002363 2016] [ssl:warn] [pid 31759] AH01909: <ONE OF MY DOMAIN NAMES>:443:0 server certificate does NOT include an ID which matches the server name
[Wed May 18 00:00:14.005641 2016] [mpm_prefork:notice] [pid 31759] AH00163: Apache/2.4.18 (Unix) OpenSSL/1.0.1e-fips mod_bwlimited/1.4 configured -- resuming normal operations
[Wed May 18 00:00:14.005677 2016] [core:notice] [pid 31759] AH00094: Command line: '/usr/local/apache/bin/httpd'
[Wed May 18 00:00:14.256595 2016] [core:error] [pid 31765] [client 70.181.227.211:54282] End of script output before headers: index.php

After this, there are thousands and thousands of errors saying "End of script output before headers" for every web request to every website and cPanel account on the server.

The suphp_log contains a line like this for every successful web request:

[Sat May 28 18:47:50 2016] [info] Executing "<path to script>" as UID 505, GID 516

However, when the server is returning 500 errors, there are no lines added to this file at all.

This is what I'm running:

/etc/redhat-release:CentOS release 6.8 (Final)
/usr/local/cpanel/version:11.56.0.24
/var/cpanel/envtype:kvm
CPANEL=release
Server version: Apache/2.2.31 (Unix)
Server built:   May 27 2016 05:45:13
Cpanel::Easy::Apache v3.32.13 rev9999
PHP 5.6.21 (cli) (built: May 27 2016 05:49:02) 
Copyright (c) 1997-2016 The PHP Group
Zend Engine v2.6.0, Copyright (c) 1998-2016 Zend Technologies
mysql  Ver 14.14 Distrib 5.5.50, for Linux (x86_64) using deadline 5.1

Let me know if you have any advice, this is starting to become a real problem

Thanks!

 

[twhiting9275]

Crons work because you're using the CLI. It looks like something is restarting the apache (or litespeed) server automatically at midnight, causing this.
If you have any conflicting software (litespeed, nginx, etc), you'll want to remove that, then recompile apache.

 

[cPanelMichael]

Hello,

Yes, as mentioned, the information you provided suggests another application is restarting Apache. Are there any additional entries to /usr/local/apache/logs/error_log just before the "[Wed May 18 00:00:11.812048 2016] [mpm_prefork:notice] [pid 12823] AH00169: caught SIGTERM, shutting down" entry? Are any cron jobs configured to run at this time of the night?

Thank you.