The Community Forums

Interact with an entire community of cPanel & WHM users!
  1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

500 Error all pages after midnight

Discussion in 'General Discussion' started by Jake Ruston, Jun 10, 2016.

  1. Jake Ruston

    Jake Ruston Registered

    Joined:
    Aug 26, 2015
    Messages:
    2
    Likes Received:
    0
    Trophy Points:
    1
    Location:
    Netherlands
    cPanel Access Level:
    Root Administrator
    Hello,

    Over the past month or two I've been facing an issue where shortly after midnight server time (usually at 10 - 20 seconds after 0:00) all web requests return 500 internal server errors. It does this until I restart the server, which makes it completely functional again. Without a restart the server will remain broken. This is a production server so obviously I can't keep checking the website after midnight everyday to check if it's broken.

    All of the websites are built on PHP, with suPHP. Interestingly, scripts run by cron job seem to function fine. It's just web requests from clients which don't work and give a 500.

    Looking at the apache log, I can logs like this:

    Code:
    [Wed May 18 00:00:11.812048 2016] [mpm_prefork:notice] [pid 12823] AH00169: caught SIGTERM, shutting down
    [Wed May 18 00:00:13.002824 2016] [ssl:warn] [pid 31758] AH01909: <ONE OF MY DOMAIN NAMES>:443:0 server certificate does NOT include an ID which matches the server name
    [Wed May 18 00:00:13.004000 2016] [suexec:notice] [pid 31758] AH01232: suEXEC mechanism enabled (wrapper: /usr/local/apache/bin/suexec)
    [Wed May 18 00:00:13.004065 2016] [:notice] [pid 31758] ModSecurity for Apache/2.9.0 (http://www.modsecurity.org/) configured.
    [Wed May 18 00:00:13.004071 2016] [:notice] [pid 31758] ModSecurity: APR compiled version="1.5.2"; loaded version="1.5.2"
    [Wed May 18 00:00:13.004076 2016] [:notice] [pid 31758] ModSecurity: PCRE compiled version="8.38 "; loaded version="8.38 2015-11-23"
    [Wed May 18 00:00:13.004086 2016] [:notice] [pid 31758] ModSecurity: LUA compiled version="Lua 5.1"
    [Wed May 18 00:00:13.004096 2016] [:notice] [pid 31758] ModSecurity: LIBXML compiled version="2.9.2"
    [Wed May 18 00:00:13.004099 2016] [:notice] [pid 31758] ModSecurity: Status engine is currently disabled, enable it by set SecStatusEngine to On.
    [Wed May 18 00:00:14.002363 2016] [ssl:warn] [pid 31759] AH01909: <ONE OF MY DOMAIN NAMES>:443:0 server certificate does NOT include an ID which matches the server name
    [Wed May 18 00:00:14.005641 2016] [mpm_prefork:notice] [pid 31759] AH00163: Apache/2.4.18 (Unix) OpenSSL/1.0.1e-fips mod_bwlimited/1.4 configured -- resuming normal operations
    [Wed May 18 00:00:14.005677 2016] [core:notice] [pid 31759] AH00094: Command line: '/usr/local/apache/bin/httpd'
    [Wed May 18 00:00:14.256595 2016] [core:error] [pid 31765] [client 70.181.227.211:54282] End of script output before headers: index.php
    
    After this, there are thousands and thousands of errors saying "End of script output before headers" for every web request to every website and cPanel account on the server.

    The suphp_log contains a line like this for every successful web request:

    Code:
    [Sat May 28 18:47:50 2016] [info] Executing "<path to script>" as UID 505, GID 516
    
    However, when the server is returning 500 errors, there are no lines added to this file at all.

    This is what I'm running:

    Code:
    /etc/redhat-release:CentOS release 6.8 (Final)
    /usr/local/cpanel/version:11.56.0.24
    /var/cpanel/envtype:kvm
    CPANEL=release
    Server version: Apache/2.2.31 (Unix)
    Server built:   May 27 2016 05:45:13
    Cpanel::Easy::Apache v3.32.13 rev9999
    PHP 5.6.21 (cli) (built: May 27 2016 05:49:02) 
    Copyright (c) 1997-2016 The PHP Group
    Zend Engine v2.6.0, Copyright (c) 1998-2016 Zend Technologies
    mysql  Ver 14.14 Distrib 5.5.50, for Linux (x86_64) using deadline 5.1
    
    Let me know if you have any advice, this is starting to become a real problem :)

    Thanks!
     
  2. twhiting9275

    twhiting9275 Well-Known Member

    Joined:
    Sep 26, 2002
    Messages:
    538
    Likes Received:
    15
    Trophy Points:
    18
    cPanel Access Level:
    Root Administrator
    Twitter:
    Crons work because you're using the CLI. It looks like something is restarting the apache (or litespeed) server automatically at midnight, causing this.
    If you have any conflicting software (litespeed, nginx, etc), you'll want to remove that, then recompile apache.
     
  3. cPanelMichael

    cPanelMichael Forums Analyst
    Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    30,678
    Likes Received:
    653
    Trophy Points:
    113
    cPanel Access Level:
    Root Administrator
    Hello,

    Yes, as mentioned, the information you provided suggests another application is restarting Apache. Are there any additional entries to /usr/local/apache/logs/error_log just before the "[Wed May 18 00:00:11.812048 2016] [mpm_prefork:notice] [pid 12823] AH00169: caught SIGTERM, shutting down" entry? Are any cron jobs configured to run at this time of the night?

    Thank you.
     
Loading...

Share This Page