The Community Forums

Interact with an entire community of cPanel & WHM users!
  1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Server re-boots every morning at 4am for no apparent reason!

Discussion in 'General Discussion' started by monkey64, Nov 24, 2011.

  1. monkey64

    monkey64 Well-Known Member

    Joined:
    Nov 6, 2011
    Messages:
    86
    Likes Received:
    0
    Trophy Points:
    6
    cPanel Access Level:
    Root Administrator
    According to WHM -> Server Status -> Apache Status, the Server re-boots itself at 4am each day. This is a new install and I haven't made any changes to the config. Here is the crontab:

    Code:
    13,28,43,58 * * * * /usr/local/cpanel/whostmgr/bin/dnsqueue > /dev/null 2>&1
    34 4 * * * /usr/local/cpanel/whostmgr/docroot/cgi/cpaddons_report.pl --notify
    22 13 * * * cd /usr/local/cpanel/whostmgr/docroot/cgi/fantastico/scripts/ ; /usr/local/cpanel/3rdparty/bin/php cron.php > /dev/null 2>&1
    */5 * * * * /usr/local/cpanel/bin/dcpumon >/dev/null 2>&1
    30 */4 * * * /usr/bin/test -x /usr/local/cpanel/scripts/update_db_cache && /usr/local/cpanel/scripts/update_db_cache
    45 */8 * * * /usr/bin/test -x /usr/local/cpanel/bin/optimizefs && /usr/local/cpanel/bin/optimizefs
    7 0 * * * /usr/local/cpanel/scripts/upcp --cron
    0 1 * * * /usr/local/cpanel/scripts/cpbackup
    35 * * * * /usr/bin/test -x /usr/local/cpanel/bin/tail-check && /usr/local/cpanel/bin/tail-check
    30 */2 * * * /usr/local/cpanel/bin/mysqluserstore >/dev/null 2>&1
    15 */2 * * * /usr/local/cpanel/bin/dbindex >/dev/null 2>&1
    0 6 * * * /usr/local/cpanel/scripts/exim_tidydb > /dev/null 2>&1
    0 6 * * * /usr/local/cpanel/scripts/exim_tidydb > /dev/null 2>&1
    45 */4 * * * /usr/bin/test -x /usr/local/cpanel/scripts/update_mailman_cache && /usr/local/cpanel/scripts/update_mailman_cache
    15 */6 * * * /usr/local/cpanel/scripts/recoverymgmt >/dev/null 2>&1
    2,58 * * * * /usr/local/bandmin/bandmin
    0 0 * * * /usr/local/bandmin/ipaddrmap
    
    Here is the page from the the WHM Apache Status page:

    Code:
    Current Time: Thursday, 24-Nov-2011 08:01:21 GMT
    Restart Time: Thursday, 24-Nov-2011 [B][COLOR=#ff0000]04:02:11[/COLOR][/B] GMT
    Parent Server Generation: 7
    Server uptime:  3 hours 59 minutes 10 seconds
    Total accesses: 288 - Total Traffic: 1.3 MB
    CPU Usage: u.03 s.01 cu.16 cs0 - .00139% CPU load.0201 requests/sec - 96 B/second - 4817 B/request
    1 requests currently being processed, 9 idle workers
    
    This is from the /var/log/messages file:

    Code:
    Nov 24 04:02:10 myserver syslogd 1.4.1: restart.
    Nov 24 04:02:47 myserver pure-ftpd: (?@127.0.0.1) [INFO] New connection from 127.0.0.1
    Nov 24 04:02:48 myserver pure-ftpd: (?@127.0.0.1) [INFO] __cpanel__service__auth__ftpd__QohI2zUdulD07RlmNytXymYpMFNMAwjVf16maMVngAEZWOqeluCTxhxMLDmoinau is now logged in
    Nov 24 04:02:48 myserver pure-ftpd: (__cpanel__service__auth__ftpd__QohI2zUdulD07RlmNytXymYpMFNMAwjVf16maMVngAEZWOqeluCTxhxMLDmoinau@127.0.0.1) [INFO] Logout.
    Nov 24 04:07:49 myserver pure-ftpd: (?@127.0.0.1) [INFO] New connection from 127.0.0.1
    Nov 24 04:07:49 myserver pure-ftpd: (?@127.0.0.1) [INFO] __cpanel__service__auth__ftpd__TlM4BHGCZidAgq3Xrujnvo5Gu8yfmUMj9VT9oI7om21IJoQH8LBwVBzQFd6IXPyP is now logged in
    Nov 24 04:07:49 myserver pure-ftpd: (__cpanel__service__auth__ftpd__TlM4BHGCZidAgq3Xrujnvo5Gu8yfmUMj9VT9oI7om21IJoQH8LBwVBzQFd6IXPyP@127.0.0.1) [INFO] Logout.
    Nov 24 04:12:49 myserver pure-ftpd: (?@127.0.0.1) [INFO] New connection from 127.0.0.1
    Nov 24 04:12:50 myserver pure-ftpd: (?@127.0.0.1) [INFO] __cpanel__service__auth__ftpd__L0GfKtEYUNVLsxpLvtX5Gwn5ES0MtZidSH2xWwNrGbVx7bG4GBL40hnKmV4KJ4LT is now logged in
    Nov 24 04:12:50 myserver pure-ftpd: (__cpanel__service__auth__ftpd__L0GfKtEYUNVLsxpLvtX5Gwn5ES0MtZidSH2xWwNrGbVx7bG4GBL40hnKmV4KJ4LT@127.0.0.1) [INFO] Logout.
    Nov 24 04:17:50 myserver pure-ftpd: (?@127.0.0.1) [INFO] New connection from 127.0.0.1
    Nov 24 04:17:51 myserver pure-ftpd: (?@127.0.0.1) [INFO] __cpanel__service__auth__ftpd__b8ruaxRBjExoLRzjmDQnvtgmCT1uIXO3XqYg14H691rkSNFJKauSUJYamlJ3gYmY is now logged in
    Nov 24 04:17:51 myserver pure-ftpd: (__cpanel__service__auth__ftpd__b8ruaxRBjExoLRzjmDQnvtgmCT1uIXO3XqYg14H691rkSNFJKauSUJYamlJ3gYmY@127.0.0.1) [INFO] Logout.
    Nov 24 04:22:51 myserver pure-ftpd: (?@127.0.0.1) [INFO] New connection from 127.0.0.1
    Nov 24 04:22:52 myserver pure-ftpd: (?@127.0.0.1) [INFO] __cpanel__service__auth__ftpd__1mCaeh1SXp6Nl1osVR9Z8PBZcDeTW1yjPOFfAJ0AlHdu5T_V7b7CvOxicyKARdpo is now logged in
    Nov 24 04:22:52 myserver pure-ftpd: (__cpanel__service__auth__ftpd__1mCaeh1SXp6Nl1osVR9Z8PBZcDeTW1yjPOFfAJ0AlHdu5T_V7b7CvOxicyKARdpo@127.0.0.1) [INFO] Logout.
    Nov 24 04:27:53 myserver pure-ftpd: (?@127.0.0.1) [INFO] New connection from 127.0.0.1
    Nov 24 04:27:53 myserver pure-ftpd: (?@127.0.0.1) [INFO] __cpanel__service__auth__ftpd__zWmjlW74b87cciS5P1ED2HEnEntypWtR09hRSRKp8GKU_ftnkPgZTV6x0_bzhmJJ is now logged in
    Nov 24 04:27:54 myserver pure-ftpd: (__cpanel__service__auth__ftpd__zWmjlW74b87cciS5P1ED2HEnEntypWtR09hRSRKp8GKU_ftnkPgZTV6x0_bzhmJJ@127.0.0.1) [INFO] Logout.
    Nov 24 04:32:54 myserver pure-ftpd: (?@127.0.0.1) [INFO] New connection from 127.0.0.1
    Nov 24 04:32:54 myserver pure-ftpd: (?@127.0.0.1) [INFO] __cpanel__service__auth__ftpd__zmKwSxY1tEz62O7C_l8eDkZZaJmnkDT2PtXXIseGAaoiea7F7ftQe4_0LvOfbC8O is now logged in
    Nov 24 04:32:54 myserver pure-ftpd: (__cpanel__service__auth__ftpd__zmKwSxY1tEz62O7C_l8eDkZZaJmnkDT2PtXXIseGAaoiea7F7ftQe4_0LvOfbC8O@127.0.0.1) [INFO] Logout.
    Nov 24 04:37:54 myserver pure-ftpd: (?@127.0.0.1) [INFO] New connection from 127.0.0.1
    Nov 24 04:37:55 myserver pure-ftpd: (?@127.0.0.1) [INFO] __cpanel__service__auth__ftpd__vXt2EQbhNpO9mXSw5Hg7Irf0OunDLuFCpLgc3OgG2whgQlJ7lq2I0LRY2nezjNW9 is now logged in
    Nov 24 04:37:55 myserver pure-ftpd: (__cpanel__service__auth__ftpd__vXt2EQbhNpO9mXSw5Hg7Irf0OunDLuFCpLgc3OgG2whgQlJ7lq2I0LRY2nezjNW9@127.0.0.1) [INFO] Logout.
    Nov 24 04:42:55 myserver pure-ftpd: (?@127.0.0.1) [INFO] New connection from 127.0.0.1
    Nov 24 04:42:55 myserver pure-ftpd: (?@127.0.0.1) [INFO] __cpanel__service__auth__ftpd__3nZUxE6DtJ0EeJAOn4Yby3oIWOdB8W97Pao0oLNDeFRTa4BEIxBFndf1kfjbAEaJ is now logged in
    Nov 24 04:42:56 myserver pure-ftpd: (__cpanel__service__auth__ftpd__3nZUxE6DtJ0EeJAOn4Yby3oIWOdB8W97Pao0oLNDeFRTa4BEIxBFndf1kfjbAEaJ@127.0.0.1) [INFO] Logout.
    
    The Restart time is always 04:02:xx. Does anyone have any ideas?
     
    #1 monkey64, Nov 24, 2011
    Last edited: Nov 24, 2011
  2. cPanelTristan

    cPanelTristan Quality Assurance Analyst
    Staff Member

    Joined:
    Oct 2, 2010
    Messages:
    7,623
    Likes Received:
    21
    Trophy Points:
    38
    Location:
    somewhere over the rainbow
    cPanel Access Level:
    Root Administrator
    Is this a VPS machine or a dedicated machine?

    What does last show for the server, since it should show whether it is a reboot or what it might be.

    Next, if you believe it to be a server-base cron due to when it is happening, what shows up in /var/log/cron right before the reboot occurs?
     
  3. monkey64

    monkey64 Well-Known Member

    Joined:
    Nov 6, 2011
    Messages:
    86
    Likes Received:
    0
    Trophy Points:
    6
    cPanel Access Level:
    Root Administrator
    This is a Virtual Private Server.

    The command "last" gives me no clue as to whether it is a reboot. Nothing is listed at that time.
    I spoke to the hosting company who confirm that the hardware has not been reset at this time.

    I found two cron files. The one I added in my first post, and another one: mailman. In each of these, I have "#" 'd out each line to work out if it is a command which is causing the re-boot. Nothing helped.

    I also looked in the /var/log/cron file. The file only shows one day's worth of logs, and they start at 04:05, after the server has been re-booted.

    Any ideas?
     
  4. cPanelTristan

    cPanelTristan Quality Assurance Analyst
    Staff Member

    Joined:
    Oct 2, 2010
    Messages:
    7,623
    Likes Received:
    21
    Trophy Points:
    38
    Location:
    somewhere over the rainbow
    cPanel Access Level:
    Root Administrator
    /var/log/cron should be archiving, so look at the archived cron file /var/log/cron.1 or /var/log/cron.2 to see the information prior to the reboot.
     
  5. monkey64

    monkey64 Well-Known Member

    Joined:
    Nov 6, 2011
    Messages:
    86
    Likes Received:
    0
    Trophy Points:
    6
    cPanel Access Level:
    Root Administrator
    Right. Thanks for that Tristan. I found the log files and they are all quite similar. The last entry before the restart is this line:

    "Nov 27 04:02:01 myserver anacron[9833]: Updated timestamp for job `cron.daily' to 2011-11-27"

    So this command was executed at 04:02:01 [27th November], and the restart time as shown in "Server Status -> Apache Status" is 04:02:07 [27th November]. That's just 6 seconds later.

    Code:
    Nov 27 04:00:01 myserver crond[9626]: (root) CMD (/usr/local/cpanel/bin/dcpumon >/dev/null 2>&1)
    Nov 27 04:00:01 myserver crond[9627]: (root) CMD (/usr/lib/sa/sa1 1 1)
    Nov 27 04:01:01 myserver crond[9699]: (root) CMD (run-parts /etc/cron.hourly)
    Nov 27 04:02:01 myserver crond[9825]: (root) CMD (/usr/local/bandmin/bandmin)
    Nov 27 04:02:01 myserver crond[9826]: (root) CMD (run-parts /etc/cron.daily)
    Nov 27 04:02:01 myserver anacron[9833]: Updated timestamp for job `cron.daily' to 2011-11-27
    
    Where do I go from here? Thanks Tristan
     
  6. cPanelTristan

    cPanelTristan Quality Assurance Analyst
    Staff Member

    Joined:
    Oct 2, 2010
    Messages:
    7,623
    Likes Received:
    21
    Trophy Points:
    38
    Location:
    somewhere over the rainbow
    cPanel Access Level:
    Root Administrator
    logrotate is likely running daily at that time, rotating the log files. Maybe, it is somehow appearing to reboot when the machine isn't actually rebooting. Does uptime actually show the server having rebooted that day?
     
  7. monkey64

    monkey64 Well-Known Member

    Joined:
    Nov 6, 2011
    Messages:
    86
    Likes Received:
    0
    Trophy Points:
    6
    cPanel Access Level:
    Root Administrator
    Uptime actually shows the server rebooted that day.
    Changing the file etc/logrotate.conf to: "rotate log files weekly" instead of daily has temporarily stopped the rebooting issue but it will still happen each week.

    Where is the crontab to schedule logrotate? If I could re-schedule it for a time I was awake (not 4 am!), I could see whether the Apache was actually re-booting or whether it was a false positive. Can I run logrotate manually?

    I looked up logrotate and the definition was "logrotate is designed to ease administration of systems that generate large numbers of log files." I only have 8 sites in my VPS, and I can't think I'm going to generate large log files.
     
  8. cPanelTristan

    cPanelTristan Quality Assurance Analyst
    Staff Member

    Joined:
    Oct 2, 2010
    Messages:
    7,623
    Likes Received:
    21
    Trophy Points:
    38
    Location:
    somewhere over the rainbow
    cPanel Access Level:
    Root Administrator
    The cron for /etc/cron.weekly is at /etc/crontab location for the time interval. You could change it there to any time you would like.

    Next, you could run it manually with this command:

    Code:
    logrotate -vf /etc/logrotate.conf
     
  9. cPanelTristan

    cPanelTristan Quality Assurance Analyst
    Staff Member

    Joined:
    Oct 2, 2010
    Messages:
    7,623
    Likes Received:
    21
    Trophy Points:
    38
    Location:
    somewhere over the rainbow
    cPanel Access Level:
    Root Administrator
  10. monkey64

    monkey64 Well-Known Member

    Joined:
    Nov 6, 2011
    Messages:
    86
    Likes Received:
    0
    Trophy Points:
    6
    cPanel Access Level:
    Root Administrator
    Tristan

    I ran the logrotate command "logrotate -vf /etc/logrotate.conf" manually, and it worked fine.

    In answer to your question, I don't believe I have snmpd installed. I read the thread and the file /etc/logrotate.d/snmpd is not present in my installation.

    In var/log/messages.x.gz, I found this entry at the time Apache rebooted:

    Code:
    Nov 24 04:02:10 myserver syslogd 1.4.1: restart.
    
    I googled and found this page which suggests that it is acceptable for a restart after the logs have rotated. As a noob, I am not sure though.

    Some of my sites are International, and 4am is peak time in their country. I can't allow the server to restart itself whenever it wants to logrotate. I just don't know where else to look!!!!
     
  11. cPanelTristan

    cPanelTristan Quality Assurance Analyst
    Staff Member

    Joined:
    Oct 2, 2010
    Messages:
    7,623
    Likes Received:
    21
    Trophy Points:
    38
    Location:
    somewhere over the rainbow
    cPanel Access Level:
    Root Administrator
    You would likely need to be on the server at the time the supposed reboot occurs and also check uptime to see the results. If it is logrotate, which changing the cron to weekly appears to have shown might be the cause, then I would say you'll need to change the logrotate cron time to one that isn't peak server time, then change back to daily to troubleshoot. Be on the server when it occurs and then see what happens.
     
  12. monkey64

    monkey64 Well-Known Member

    Joined:
    Nov 6, 2011
    Messages:
    86
    Likes Received:
    0
    Trophy Points:
    6
    cPanel Access Level:
    Root Administrator
    Tristan

    I rescheduled the logrotate cron time to 8am, and went to a couple of the sites I host. The sites were all up and running for the time logrotate ran. However, the WHM panel still reports a restart during this period which is strange considering that no restart occurred???

    Oh well at least I know that I do not have a more serious issue.

    Just one thing. When I edited the etc/crontab file to reschedlue logrotate cron time, I did it using the text editor on WinSCP and rebooted Apache for the changes to take effect. Was that neccessary? I'm used to editing user crontabs in the command line using "crontab -e", but if I wanted to edit the etc/crontab, what syntax would I use and would I still need a reboot?

    Tristan. Thanks very much for your help.
     
  13. cPanelTristan

    cPanelTristan Quality Assurance Analyst
    Staff Member

    Joined:
    Oct 2, 2010
    Messages:
    7,623
    Likes Received:
    21
    Trophy Points:
    38
    Location:
    somewhere over the rainbow
    cPanel Access Level:
    Root Administrator
    You shouldn't have to reboot Apache for the changes to take effect. As for the syntax, it is the same as editing /var/spool/cron/root (which is the same file that "crontab -e" command issues). All crontabs take the same type of syntax for interval and command entries, so /etc/crontab would be the same as any other cron. You shouldn't need to reboot anything not even crond itself.
     
  14. djalpha

    djalpha Active Member

    Joined:
    Jan 20, 2003
    Messages:
    25
    Likes Received:
    0
    Trophy Points:
    1
    0 6 * * * /usr/local/cpanel/scripts/exim_tidydb > /dev/null 2>&1
    0 6 * * * /usr/local/cpanel/scripts/exim_tidydb > /dev/null 2>&1


    double
     
  15. cPanelTristan

    cPanelTristan Quality Assurance Analyst
    Staff Member

    Joined:
    Oct 2, 2010
    Messages:
    7,623
    Likes Received:
    21
    Trophy Points:
    38
    Location:
    somewhere over the rainbow
    cPanel Access Level:
    Root Administrator
    This post would be a separate issue from the thread discussion. Simply remove the double entry in your crontab.
     
Loading...

Share This Page