Server crashes every night at approximately 4:02 am

[oblivio]

Hello all, i hope someone can help me with this. I am the first to admit that my skill with Centos is still not up to my Windows Server skills, but i am learning.

I am running CENTOS 5.7 64bit in hyper-v. Cpanel Version: WHM 11.30.6 (build 4). This system worked great for almost a year now but recently every night (like clock work) the server crashes at 4:02 AM. I can still log into the SSH and even ping the web sites but the file system becomes Read Only. Can anyone lead me into the right direction to solving the issue?

Ps. I found out that the crash time happens at about 4:02 by looking at messages under var/log and /var/log/cron
The only item i see right before the crash is anacron[6248]: Updated timestamp for job `cron.daily' to 2012-04-04

 

[Infopro]

I can still log into the SSH and even ping the web sites but the file system becomes Read Only.

Crashing and becoming read only might be hardware issues if I was to guess. No way of knowing by your post whats going on I don't think but, if you have backups running and many large accounts on the server, late night at the same exact time sounds like backups are running.

You might want to have your Hosting Provider look at the server for hardware issues.

 

[flexfive]

We are having the exact same problem on our server. Right at 4am Apache becomes unresponsive, yet I can still ping server. I must reboot the server to get it to come back up, which usually takes quite a while. We are running WHM 11.32.2 (build 18), CentOS 5.8 x86_64

 

[AvatarPortal]

I'm in the same boat, with my crashes occurring a bit before 11pm EST. My host is Neosurge, but no word back from them on this yet. This has been occurring for about a week. Sorry if I am of little help, I run a VPS but have minimal experience in it. I only hope to see if this is a more prevalent problem with a similar solution.

 

[Infopro]

From within your WHM, top right corner of any page is a menu titled Support. On that menu is a link to "Contact cPanel" use that. Get in touch with cPanel Support, provide full login details to the server so they can take a closer look at this.

They'll get you sorted. cPanel Support are the best!

 

[flexfive]

We've solved this on our server. There was something causing issue with APF firewall when nightly cron jobs ran at 4am. Restarting only the APF firewall resolved the issue each time. Didn't seem to be anything wrong with APF configuration, so rather than spend too much time troubleshooting APF firewall, tech suggested changing to CSF firewall instead since it is more up to date and better functionality anyway. We did this and our 4am problem went away.

 

[NetMantis]

CSF is definitely a much better firewall than APF though you need to make sure you configure the CSF Firewall configuration correctly ...

With regard to your crashing problem, I believe APF was only one single contributor in a lot of different variables. You need to take a look at the /etc/cron.daily folder and see what all you have running at that location as those are the scripts that would be executed at your 4:02 time clock as you said previously.

Now with regard of you mentioning that your file system goes read only, it sounds to me like you are over running your /tmp drive which I would wager is a custom volume using a file for the backend. You can check /etc/fstab to see how it is setup. May need to setup a larger /tmp volume than what you currently have right now.

It also would not hurt to do a boot time single user mode file system scan (fsck) to make sure you don't have any corrupted data sitting on your hard drive possibly messing you up further.
Drives can be flipped over to "read only" automatically in Linux when data errors are detected.

 

[cPanelTristan]

Hello,

Additionally, whenever you suspect a possible hardware issue (such as the file system becoming read-only), reviewing /var/log/messages at the time of the crash can be fruitful. If the machine might be having temperature fluctuations or segmentation faults, those can log to /var/log/messages and would be helpful in tracking it down as a hardware issue that your datacenter, NOC or provider would need to assist in resolving.

Thanks!