The Community Forums

Interact with an entire community of cPanel & WHM users!
  1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Unlinked inode... Remounting filesystem read-only - AGAIN?

Discussion in 'General Discussion' started by deieno, Jan 25, 2010.

  1. deieno

    deieno Well-Known Member

    Joined:
    Nov 16, 2003
    Messages:
    69
    Likes Received:
    0
    Trophy Points:
    6
    Location:
    Floripa - Brazil
    Hi...
    I have this server that is becaming on filesystem read-only mode EVERY 2:00 (around).

    logs show:


    Jan 25 02:50:31 server pure-ftpd: (?@127.0.0.1) [INFO] New connection from 127.0.0.1
    Jan 25 02:50:42 server pure-ftpd: (?@127.0.0.1) [INFO] __cpanel__service__auth__ftpd__GjI_22MRgfyqZTv544N TA0j7feqymxLP0P1XXzWvvbWzZrNHfkR89BAtuEKbnNQP is now logged in
    Jan 25 02:50:43 server pure-ftpd: (__cpanel__service__auth__ftpd__GjI_22MRgfyqZTv544 NTA0j7feqymxLP0P1XXzWvvbWzZrNHfkR89BAtuEKbnNQP@127 .0.0.1) [INFO] Logout.
    Jan 25 02:51:44 server kernel: EXT3-fs error (device sda8): ext3_lookup: unlinked inode 159222206 in dir #159220259
    Jan 25 02:51:44 server kernel: Aborting journal on device sda8.
    Jan 25 02:51:44 server kernel: EXT3-fs error (device sda8): ext3_lookup: unlinked inode 159222215 in dir #159220259
    Jan 25 02:51:44 server kernel: EXT3-fs error (device sda8): ext3_lookup: unlinked inode 159222143 in dir #159220259
    Jan 25 02:51:44 server kernel: EXT3-fs error (device sda8): ext3_lookup: unlinked inode 159222125 in dir #159220259
    Jan 25 02:51:44 server kernel: EXT3-fs error (device sda8): ext3_lookup: unlinked inode 159222250 in dir #159220259
    Jan 25 02:51:44 server kernel: EXT3-fs error (device sda8): ext3_lookup: unlinked inode 159222235 in dir #159220259
    Jan 25 02:51:44 server kernel: EXT3-fs error (device sda8): ext3_lookup: unlinked inode 159222127 in dir #159220259
    Jan 25 02:51:44 server kernel: EXT3-fs error (device sda8): ext3_lookup: unlinked inode 159222210 in dir #159220259
    Jan 25 02:51:44 server kernel: EXT3-fs error (device sda8): ext3_lookup: unlinked inode 159222150 in dir #159220259
    Jan 25 02:51:45 server kernel: EXT3-fs error (device sda8): ext3_lookup: unlinked inode 159222236 in dir #159220259
    Jan 25 02:51:45 server kernel: ext3_abort called.
    Jan 25 02:51:45 server kernel: EXT3-fs error (device sda8): ext3_journal_start_sb: Detected aborted journal
    Jan 25 02:51:45 server kernel: Remounting filesystem read-only


    root@server [/var/log]# grep only messages
    Jan 25 02:51:45 server kernel: Remounting filesystem read-only
    Jan 25 07:56:19 server kernel: Write protecting the kernel read-only data: 392k
    Jan 25 07:56:21 server kernel: EXT3-fs: INFO: recovery required on readonly filesystem.



    Ok.. I scheduled a FSCK... but the day after that ... same issue.

    What worries me is not the problem itself... but, last week this happened in TWO servers... One at Softlayer.. one at theplanet... Yes.. I thought it was to much coincidence... but, "things" happens..

    The solution for the last week case was a OS RELOAD and THAN... restoring everything ... well.. A nightmare.. all night awake restoring the data.

    I thought my nighs were safe.. at least for more time...
    But now, another client server with the SAME issue.

    Strangely.. it turn just the /home partition on read-only mode.. the way it happened with the other servers that had the os reload.

    Before start to trying things... I would like to have some opinions here. What this could be?

    A cpanel issue? a centos issue? someone trying to hurt the server?
    or it's really just happens and the fact it happen twice on other servers last week it's just coincidence?

    I appreciate any help...

    Thank you
     
  2. madaboutlinux

    madaboutlinux Well-Known Member

    Joined:
    Jan 24, 2005
    Messages:
    1,052
    Likes Received:
    2
    Trophy Points:
    38
    Location:
    Earth
    This looks like a kernel problem. What kernel are you using on this server and was it the same kernel on the other 2 servers when the problem occurred last week?

    You can find the kernel by executing the command:

     
  3. deieno

    deieno Well-Known Member

    Joined:
    Nov 16, 2003
    Messages:
    69
    Likes Received:
    0
    Trophy Points:
    6
    Location:
    Floripa - Brazil
    Hi...
    kernel is
    Linux hostname 2.6.18-92.el5PAE #1 SMP Tue Jun 10 19:22:41 EDT 2008 i686 i686 i386 GNU/Linux


    thank you
     
  4. zigzam

    zigzam Well-Known Member

    Joined:
    May 9, 2005
    Messages:
    206
    Likes Received:
    0
    Trophy Points:
    16
    I have run into this exact same issue. Happens one time daily at roughly 2 am. I am using kernel 2.6.18-164.11.1.el5 #1 SMP Wed Jan 20 07:32:21 EST 2010 x86_64 x86_64 x86_64 GNU/Linux


    Any ideas?
     
  5. rackaid

    rackaid Active Member

    Joined:
    Jan 18, 2003
    Messages:
    42
    Likes Received:
    1
    Trophy Points:
    8
    Location:
    Jacksonville, FL
    cPanel Access Level:
    DataCenter Provider
    Would these systems happen to have Adaptec RAId controllers?

    I've seen this issue if the RAID never fully initializes.


    I've also seen issues with ext3 that can be fixed by downgrading to ext2. Run a fsck. Then go back to ext3. Re-run fsck.

    Lastly, perhaps you do have bad blocks in an area and something at that time happens to hit that area.
     
  6. zigzam

    zigzam Well-Known Member

    Joined:
    May 9, 2005
    Messages:
    206
    Likes Received:
    0
    Trophy Points:
    16
    Mine is a Adaptec 2405

    Everything is reported as optimal.
     
Loading...

Share This Page