Server re-boots every morning at 4am for no apparent reason!

[monkey64]

According to WHM -> Server Status -> Apache Status, the Server re-boots itself at 4am each day. This is a new install and I haven't made any changes to the config. Here is the crontab:

13,28,43,58 * * * * /usr/local/cpanel/whostmgr/bin/dnsqueue > /dev/null 2>&1
34 4 * * * /usr/local/cpanel/whostmgr/docroot/cgi/cpaddons_report.pl --notify
22 13 * * * cd /usr/local/cpanel/whostmgr/docroot/cgi/fantastico/scripts/ ; /usr/local/cpanel/3rdparty/bin/php cron.php > /dev/null 2>&1
*/5 * * * * /usr/local/cpanel/bin/dcpumon >/dev/null 2>&1
30 */4 * * * /usr/bin/test -x /usr/local/cpanel/scripts/update_db_cache && /usr/local/cpanel/scripts/update_db_cache
45 */8 * * * /usr/bin/test -x /usr/local/cpanel/bin/optimizefs && /usr/local/cpanel/bin/optimizefs
7 0 * * * /usr/local/cpanel/scripts/upcp --cron
0 1 * * * /usr/local/cpanel/scripts/cpbackup
35 * * * * /usr/bin/test -x /usr/local/cpanel/bin/tail-check && /usr/local/cpanel/bin/tail-check
30 */2 * * * /usr/local/cpanel/bin/mysqluserstore >/dev/null 2>&1
15 */2 * * * /usr/local/cpanel/bin/dbindex >/dev/null 2>&1
0 6 * * * /usr/local/cpanel/scripts/exim_tidydb > /dev/null 2>&1
0 6 * * * /usr/local/cpanel/scripts/exim_tidydb > /dev/null 2>&1
45 */4 * * * /usr/bin/test -x /usr/local/cpanel/scripts/update_mailman_cache && /usr/local/cpanel/scripts/update_mailman_cache
15 */6 * * * /usr/local/cpanel/scripts/recoverymgmt >/dev/null 2>&1
2,58 * * * * /usr/local/bandmin/bandmin
0 0 * * * /usr/local/bandmin/ipaddrmap

Here is the page from the the WHM Apache Status page:

Current Time: Thursday, 24-Nov-2011 08:01:21 GMT
Restart Time: Thursday, 24-Nov-2011 [B][COLOR=#ff0000]04:02:11[/COLOR][/B] GMT
Parent Server Generation: 7
Server uptime:  3 hours 59 minutes 10 seconds
Total accesses: 288 - Total Traffic: 1.3 MB
CPU Usage: u.03 s.01 cu.16 cs0 - .00139% CPU load.0201 requests/sec - 96 B/second - 4817 B/request
1 requests currently being processed, 9 idle workers

This is from the /var/log/messages file:

Nov 24 04:02:10 myserver syslogd 1.4.1: restart.
Nov 24 04:02:47 myserver pure-ftpd: ([email protected]) [INFO] New connection from 127.0.0.1
Nov 24 04:02:48 myserver pure-ftpd: ([email protected]) [INFO] __cpanel__service__auth__ftpd__QohI2zUdulD07RlmNytXymYpMFNMAwjVf16maMVngAEZWOqeluCTxhxMLDmoinau is now logged in
Nov 24 04:02:48 myserver pure-ftpd: (__cpanel__service__auth__ftpd__Q[email protected]127.0.0.1) [INFO] Logout.
Nov 24 04:07:49 myserver pure-ftpd: ([email protected]) [INFO] New connection from 127.0.0.1
Nov 24 04:07:49 myserver pure-ftpd: ([email protected]) [INFO] __cpanel__service__auth__ftpd__TlM4BHGCZidAgq3Xrujnvo5Gu8yfmUMj9VT9oI7om21IJoQH8LBwVBzQFd6IXPyP is now logged in
Nov 24 04:07:49 myserver pure-ftpd: (__cpanel__service__auth__ftpd__T[email protected]127.0.0.1) [INFO] Logout.
Nov 24 04:12:49 myserver pure-ftpd: ([email protected]) [INFO] New connection from 127.0.0.1
Nov 24 04:12:50 myserver pure-ftpd: ([email protected]) [INFO] __cpanel__service__auth__ftpd__L0GfKtEYUNVLsxpLvtX5Gwn5ES0MtZidSH2xWwNrGbVx7bG4GBL40hnKmV4KJ4LT is now logged in
Nov 24 04:12:50 myserver pure-ftpd: (__cpanel__service__auth__ftpd__L[email protected]127.0.0.1) [INFO] Logout.
Nov 24 04:17:50 myserver pure-ftpd: ([email protected]) [INFO] New connection from 127.0.0.1
Nov 24 04:17:51 myserver pure-ftpd: ([email protected]) [INFO] __cpanel__service__auth__ftpd__b8ruaxRBjExoLRzjmDQnvtgmCT1uIXO3XqYg14H691rkSNFJKauSUJYamlJ3gYmY is now logged in
Nov 24 04:17:51 myserver pure-ftpd: (__cpanel__service__auth__ftpd__b[email protected]127.0.0.1) [INFO] Logout.
Nov 24 04:22:51 myserver pure-ftpd: ([email protected]) [INFO] New connection from 127.0.0.1
Nov 24 04:22:52 myserver pure-ftpd: ([email protected]) [INFO] __cpanel__service__auth__ftpd__1mCaeh1SXp6Nl1osVR9Z8PBZcDeTW1yjPOFfAJ0AlHdu5T_V7b7CvOxicyKARdpo is now logged in
Nov 24 04:22:52 myserver pure-ftpd: (__cpanel__service__auth__ftpd__1[email protected]127.0.0.1) [INFO] Logout.
Nov 24 04:27:53 myserver pure-ftpd: ([email protected]) [INFO] New connection from 127.0.0.1
Nov 24 04:27:53 myserver pure-ftpd: ([email protected]) [INFO] __cpanel__service__auth__ftpd__zWmjlW74b87cciS5P1ED2HEnEntypWtR09hRSRKp8GKU_ftnkPgZTV6x0_bzhmJJ is now logged in
Nov 24 04:27:54 myserver pure-ftpd: (__cpanel__service__auth__ftpd__z[email protected]127.0.0.1) [INFO] Logout.
Nov 24 04:32:54 myserver pure-ftpd: ([email protected]) [INFO] New connection from 127.0.0.1
Nov 24 04:32:54 myserver pure-ftpd: ([email protected]) [INFO] __cpanel__service__auth__ftpd__zmKwSxY1tEz62O7C_l8eDkZZaJmnkDT2PtXXIseGAaoiea7F7ftQe4_0LvOfbC8O is now logged in
Nov 24 04:32:54 myserver pure-ftpd: (__cpanel__service__auth__ftpd__z[email protected]127.0.0.1) [INFO] Logout.
Nov 24 04:37:54 myserver pure-ftpd: ([email protected]) [INFO] New connection from 127.0.0.1
Nov 24 04:37:55 myserver pure-ftpd: ([email protected]) [INFO] __cpanel__service__auth__ftpd__vXt2EQbhNpO9mXSw5Hg7Irf0OunDLuFCpLgc3OgG2whgQlJ7lq2I0LRY2nezjNW9 is now logged in
Nov 24 04:37:55 myserver pure-ftpd: (__cpanel__service__auth__ftpd__v[email protected]127.0.0.1) [INFO] Logout.
Nov 24 04:42:55 myserver pure-ftpd: ([email protected]) [INFO] New connection from 127.0.0.1
Nov 24 04:42:55 myserver pure-ftpd: ([email protected]) [INFO] __cpanel__service__auth__ftpd__3nZUxE6DtJ0EeJAOn4Yby3oIWOdB8W97Pao0oLNDeFRTa4BEIxBFndf1kfjbAEaJ is now logged in
Nov 24 04:42:56 myserver pure-ftpd: (__cpanel__service__auth__ftpd__3[email protected]127.0.0.1) [INFO] Logout.

The Restart time is always 04:02:xx. Does anyone have any ideas?

 

[cPanelTristan]

Is this a VPS machine or a dedicated machine?

What does last show for the server, since it should show whether it is a reboot or what it might be.

Next, if you believe it to be a server-base cron due to when it is happening, what shows up in /var/log/cron right before the reboot occurs?

 

[monkey64]

This is a Virtual Private Server.

The command "last" gives me no clue as to whether it is a reboot. Nothing is listed at that time.
I spoke to the hosting company who confirm that the hardware has not been reset at this time.

I found two cron files. The one I added in my first post, and another one: mailman. In each of these, I have "#" 'd out each line to work out if it is a command which is causing the re-boot. Nothing helped.

I also looked in the /var/log/cron file. The file only shows one day's worth of logs, and they start at 04:05, after the server has been re-booted.

Any ideas?

 

[cPanelTristan]

/var/log/cron should be archiving, so look at the archived cron file /var/log/cron.1 or /var/log/cron.2 to see the information prior to the reboot.

 

[monkey64]

Right. Thanks for that Tristan. I found the log files and they are all quite similar. The last entry before the restart is this line:

"Nov 27 04:02:01 myserver anacron[9833]: Updated timestamp for job `cron.daily' to 2011-11-27"

So this command was executed at 04:02:01 [27th November], and the restart time as shown in "Server Status -> Apache Status" is 04:02:07 [27th November]. That's just 6 seconds later.

Nov 27 04:00:01 myserver crond[9626]: (root) CMD (/usr/local/cpanel/bin/dcpumon >/dev/null 2>&1)
Nov 27 04:00:01 myserver crond[9627]: (root) CMD (/usr/lib/sa/sa1 1 1)
Nov 27 04:01:01 myserver crond[9699]: (root) CMD (run-parts /etc/cron.hourly)
Nov 27 04:02:01 myserver crond[9825]: (root) CMD (/usr/local/bandmin/bandmin)
Nov 27 04:02:01 myserver crond[9826]: (root) CMD (run-parts /etc/cron.daily)
Nov 27 04:02:01 myserver anacron[9833]: Updated timestamp for job `cron.daily' to 2011-11-27

Where do I go from here? Thanks Tristan

 

[cPanelTristan]

logrotate is likely running daily at that time, rotating the log files. Maybe, it is somehow appearing to reboot when the machine isn't actually rebooting. Does uptime actually show the server having rebooted that day?

 

[monkey64]

Uptime actually shows the server rebooted that day.
Changing the file etc/logrotate.conf to: "rotate log files weekly" instead of daily has temporarily stopped the rebooting issue but it will still happen each week.

Where is the crontab to schedule logrotate? If I could re-schedule it for a time I was awake (not 4 am!), I could see whether the Apache was actually re-booting or whether it was a false positive. Can I run logrotate manually?

I looked up logrotate and the definition was "logrotate is designed to ease administration of systems that generate large numbers of log files." I only have 8 sites in my VPS, and I can't think I'm going to generate large log files.

 

[cPanelTristan]

The cron for /etc/cron.weekly is at /etc/crontab location for the time interval. You could change it there to any time you would like.

Next, you could run it manually with this command:

logrotate -vf /etc/logrotate.conf

 

[cPanelTristan]

Also, could you check if snmpd is installed on the server? Here's a forum thread that might relate to logrotate plus snmpd causing this to happen:

linux snmpd restart during logrotate triggers false reboot alert weekly - thwack

 

[monkey64]

Tristan

I ran the logrotate command "logrotate -vf /etc/logrotate.conf" manually, and it worked fine.

In answer to your question, I don't believe I have snmpd installed. I read the thread and the file /etc/logrotate.d/snmpd is not present in my installation.

In var/log/messages.x.gz, I found this entry at the time Apache rebooted:

Nov 24 04:02:10 myserver syslogd 1.4.1: restart.

I googled and found this page which suggests that it is acceptable for a restart after the logs have rotated. As a noob, I am not sure though.

Some of my sites are International, and 4am is peak time in their country. I can't allow the server to restart itself whenever it wants to logrotate. I just don't know where else to look!!!!

 

[cPanelTristan]

You would likely need to be on the server at the time the supposed reboot occurs and also check uptime to see the results. If it is logrotate, which changing the cron to weekly appears to have shown might be the cause, then I would say you'll need to change the logrotate cron time to one that isn't peak server time, then change back to daily to troubleshoot. Be on the server when it occurs and then see what happens.

 

[monkey64]

Tristan

I rescheduled the logrotate cron time to 8am, and went to a couple of the sites I host. The sites were all up and running for the time logrotate ran. However, the WHM panel still reports a restart during this period which is strange considering that no restart occurred???

Oh well at least I know that I do not have a more serious issue.

Just one thing. When I edited the etc/crontab file to reschedlue logrotate cron time, I did it using the text editor on WinSCP and rebooted Apache for the changes to take effect. Was that neccessary? I'm used to editing user crontabs in the command line using "crontab -e", but if I wanted to edit the etc/crontab, what syntax would I use and would I still need a reboot?

Tristan. Thanks very much for your help.

 

[cPanelTristan]

You shouldn't have to reboot Apache for the changes to take effect. As for the syntax, it is the same as editing /var/spool/cron/root (which is the same file that "crontab -e" command issues). All crontabs take the same type of syntax for interval and command entries, so /etc/crontab would be the same as any other cron. You shouldn't need to reboot anything not even crond itself.

 

[djalpha]

0 6 * * * /usr/local/cpanel/scripts/exim_tidydb > /dev/null 2>&1
0 6 * * * /usr/local/cpanel/scripts/exim_tidydb > /dev/null 2>&1


double

 

[cPanelTristan]

This post would be a separate issue from the thread discussion. Simply remove the double entry in your crontab.