The Community Forums

Interact with an entire community of cPanel & WHM users!
  1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Server crashes every night at approximately 4:02 am

Discussion in 'General Discussion' started by oblivio, Apr 5, 2012.

  1. oblivio

    oblivio Registered

    Joined:
    Apr 5, 2012
    Messages:
    3
    Likes Received:
    0
    Trophy Points:
    1
    cPanel Access Level:
    Root Administrator
    Hello all, i hope someone can help me with this. I am the first to admit that my skill with Centos is still not up to my Windows Server skills, but i am learning.

    I am running CENTOS 5.7 64bit in hyper-v. Cpanel Version: WHM 11.30.6 (build 4). This system worked great for almost a year now but recently every night (like clock work) the server crashes at 4:02 AM. I can still log into the SSH and even ping the web sites but the file system becomes Read Only. Can anyone lead me into the right direction to solving the issue?

    Ps. I found out that the crash time happens at about 4:02 by looking at messages under var/log and /var/log/cron
    The only item i see right before the crash is anacron[6248]: Updated timestamp for job `cron.daily' to 2012-04-04
     
  2. Infopro

    Infopro cPanel Sr. Product Evangelist
    Staff Member

    Joined:
    May 20, 2003
    Messages:
    14,446
    Likes Received:
    195
    Trophy Points:
    63
    Location:
    Pennsylvania
    cPanel Access Level:
    Root Administrator
    Twitter:
    Crashing and becoming read only might be hardware issues if I was to guess. No way of knowing by your post whats going on I don't think but, if you have backups running and many large accounts on the server, late night at the same exact time sounds like backups are running.

    You might want to have your Hosting Provider look at the server for hardware issues.
     
  3. flexfive

    flexfive Registered

    Joined:
    Apr 18, 2012
    Messages:
    2
    Likes Received:
    0
    Trophy Points:
    1
    cPanel Access Level:
    Root Administrator
    We are having the exact same problem on our server. Right at 4am Apache becomes unresponsive, yet I can still ping server. I must reboot the server to get it to come back up, which usually takes quite a while. We are running WHM 11.32.2 (build 18), CentOS 5.8 x86_64
     
  4. AvatarPortal

    AvatarPortal Member

    Joined:
    Apr 18, 2012
    Messages:
    10
    Likes Received:
    0
    Trophy Points:
    1
    cPanel Access Level:
    Root Administrator
    I'm in the same boat, with my crashes occurring a bit before 11pm EST. My host is Neosurge, but no word back from them on this yet. This has been occurring for about a week. Sorry if I am of little help, I run a VPS but have minimal experience in it. I only hope to see if this is a more prevalent problem with a similar solution.
     
  5. Infopro

    Infopro cPanel Sr. Product Evangelist
    Staff Member

    Joined:
    May 20, 2003
    Messages:
    14,446
    Likes Received:
    195
    Trophy Points:
    63
    Location:
    Pennsylvania
    cPanel Access Level:
    Root Administrator
    Twitter:
    From within your WHM, top right corner of any page is a menu titled Support. On that menu is a link to "Contact cPanel" use that. Get in touch with cPanel Support, provide full login details to the server so they can take a closer look at this.

    They'll get you sorted. cPanel Support are the best!
     
  6. flexfive

    flexfive Registered

    Joined:
    Apr 18, 2012
    Messages:
    2
    Likes Received:
    0
    Trophy Points:
    1
    cPanel Access Level:
    Root Administrator
    We've solved this on our server. There was something causing issue with APF firewall when nightly cron jobs ran at 4am. Restarting only the APF firewall resolved the issue each time. Didn't seem to be anything wrong with APF configuration, so rather than spend too much time troubleshooting APF firewall, tech suggested changing to CSF firewall instead since it is more up to date and better functionality anyway. We did this and our 4am problem went away.
     
    #6 flexfive, Apr 24, 2012
    Last edited: Apr 24, 2012
  7. NetMantis

    NetMantis BANNED

    Joined:
    Apr 22, 2012
    Messages:
    117
    Likes Received:
    0
    Trophy Points:
    0
    Location:
    Utah
    cPanel Access Level:
    DataCenter Provider
    CSF is definitely a much better firewall than APF though you need to make sure you configure the CSF Firewall configuration correctly ...

    With regard to your crashing problem, I believe APF was only one single contributor in a lot of different variables. You need to take a look at the /etc/cron.daily folder and see what all you have running at that location as those are the scripts that would be executed at your 4:02 time clock as you said previously.

    Now with regard of you mentioning that your file system goes read only, it sounds to me like you are over running your /tmp drive which I would wager is a custom volume using a file for the backend. You can check /etc/fstab to see how it is setup. May need to setup a larger /tmp volume than what you currently have right now.

    It also would not hurt to do a boot time single user mode file system scan (fsck) to make sure you don't have any corrupted data sitting on your hard drive possibly messing you up further.
    Drives can be flipped over to "read only" automatically in Linux when data errors are detected.
     
    #7 NetMantis, Apr 24, 2012
    Last edited: Apr 24, 2012
  8. cPanelTristan

    cPanelTristan Quality Assurance Analyst
    Staff Member

    Joined:
    Oct 2, 2010
    Messages:
    7,623
    Likes Received:
    21
    Trophy Points:
    38
    Location:
    somewhere over the rainbow
    cPanel Access Level:
    Root Administrator
    Hello,

    Additionally, whenever you suspect a possible hardware issue (such as the file system becoming read-only), reviewing /var/log/messages at the time of the crash can be fruitful. If the machine might be having temperature fluctuations or segmentation faults, those can log to /var/log/messages and would be helpful in tracking it down as a hardware issue that your datacenter, NOC or provider would need to assist in resolving.

    Thanks!
     
Loading...

Share This Page