The Community Forums

Interact with an entire community of cPanel & WHM users!
  1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

read-only file system

Discussion in 'General Discussion' started by phantom, May 2, 2004.

  1. phantom

    phantom Well-Known Member

    Joined:
    Sep 17, 2002
    Messages:
    123
    Likes Received:
    0
    Trophy Points:
    16
    Hi,
    This has happened to us before and a simple reboot fixed it but this time it wont fix it. WHM does not come up and when I SSH in and try to execute anything or edit anything I get a "Read-Only file system".

    Also, when I first log into ssh as root, the following line is above the command prompt....

    /tmp/RsaWcHVp: Read-only file system

    I checked the /tmp/ directory and the above is not there. Only an empty folder named .apf-3366

    Any advise? We already did the /scripts/securetmp

    Thanks!
     
  2. chirpy

    chirpy Well-Known Member

    Joined:
    Jun 15, 2002
    Messages:
    13,475
    Likes Received:
    20
    Trophy Points:
    38
    Location:
    Go on, have a guess
    It could be that the actual tmp file that is created by /scripts/securetmp has become corrupted somehow.

    To create a new one:

    /etc/init.d/mysql stop
    umount /tmp
    umount /var/tmp
    cd /usr
    mv tmpDSK tmpDSK.old # you can delete tmpDSK.old later
    /scripts/securetmp
    cd /tmp
    ln -s /var/lib/mysql/mysql.sock
    /etc/init.d/mysql start

    Then try restarting cPanel.
     
  3. phantom

    phantom Well-Known Member

    Joined:
    Sep 17, 2002
    Messages:
    123
    Likes Received:
    0
    Trophy Points:
    16
    Thanks for taking the time to post help but it did not work. I think the problem may be a little deeper.

    I think the box has been hacked.

    Look at how the file name in the /tmp/ directory changes if I just type in 'su root'

    root@matrix [/usr/local/apache/logs]# su root
    /tmp/RsGzaSCm: Read-only file system
    root@matrix [/usr/local/apache/logs]# su root
    /tmp/RsasNrMo: Read-only file system
    root@matrix [/usr/local/apache/logs]# su root
    /tmp/Rsi7aGGC: Read-only file system
    root@matrix [/usr/local/apache/logs]# su root
    /tmp/RsEYAJId: Read-only file system
    root@matrix [/usr/local/apache/logs]# su root
    /tmp/Rswmz4lo: Read-only file system
    root@matrix [/usr/local/apache/logs]# su root
    /tmp/RsHYib5j: Read-only file system


    They all start with Rs. Don't know what that means. Also, none of these files or folders are in the /tmp/
    directory. There is only one empty folder in the tmp directory. It's named .apf-3366

    Plus, in the access log, about 5 entries before the end
    (because it cannot be written to anymore), there is a very large and strange line of code

    69.46.*.* - - [02/May/2004:06:25:16 -0400] "SEARCH /\x90\x02\xb1\x02\xb1\x02\xb1\x02\xb1\x02\xb1\x02\xb1\x02\xb1\x02\xb1\x02\xb1\
    \x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90"
    414 353

    Except there is about 250 lines of the \x90\ stuff.

    I starred out the IP in case it is innocent. If someone can tell me if they've seen anything like this and it is some sort of hack, let me know so I can go after this person.

    Thanks!
     
  4. nitromax

    nitromax Well-Known Member

    Joined:
    Feb 12, 2002
    Messages:
    189
    Likes Received:
    0
    Trophy Points:
    16
    Hey phantom,

    Did you ever find a fix for this problem?

    We are having a problem with Apache shutting down every couple of minitues and it's because someone is maxing out the apache connection max of 256. But I am seeing huge lines of code in the access_log like the ones you describe... Like this:

    66.98.60.19 - - [02/Nov/2004:21:06:43 -0600] "SEARCH /\x90\x02\xb1\x02\xb1\x02\xb1\x02\xb1\x02\xb1\x02\xb1\x02\xb1\x02\xb1\x02\xb1\x02\xb1\x02\xb1\x02\xb1\x02\xb1\x02\xb1\x02\xb1\x02\xb1\x02\xb1\x02\xb1\x02\xb1\x02\xb1\x02\xb1\x02\xb1\x02\xb1\x02\xb1\x02\xb1\x02\xb1\x02\xb1\x02\xb1\x02\xb1\x02\xb1\x02\xb1\x02\xb1\x02\xb1\x02\xb1\x02\xb1\x02\xb1\x02\xb1\x02\xb1\x02\xb1\x02\xb1\x02\xb1\x02\xb1\x02\xb1\x02\xb1\x02\xb1\x02\xb1\x02\xb1\x02\xb1\x02\xb1\x02\xb1\x02\xb1\x02\xb1\x02\xb1\x02\xb1\x02\xb1\x02\xb1\x02\xb1\x02\xb1\x02\x

    I can't figure out what this is. Any ideas?
     
  5. PWSowner

    PWSowner Well-Known Member

    Joined:
    Nov 10, 2001
    Messages:
    2,948
    Likes Received:
    3
    Trophy Points:
    38
    Location:
    ON, Canada
    I see lines like that now and then. I asked about it a while back and found out it's a Windows attack which is harmless to a linux system other than server load if they go crazy with it.

    When I get it, it's usually only a few times over a minute or so, then dot again for hours or days.
     
  6. nitromax

    nitromax Well-Known Member

    Joined:
    Feb 12, 2002
    Messages:
    189
    Likes Received:
    0
    Trophy Points:
    16
    I was wondering if it might of been something like that. I did the following to find the offending IP address: netstat -an | grep :80


    The I used the following to have iptables keep that IP from accessing the server:

    iptables -I INPUT -s 69.3.64.218 -j DROP


    That seems to have done the trick for now, but I'll have to wait and see. Thanks for your reply.
     
  7. PWSowner

    PWSowner Well-Known Member

    Joined:
    Nov 10, 2001
    Messages:
    2,948
    Likes Received:
    3
    Trophy Points:
    38
    Location:
    ON, Canada
    No problem. The IP will change once in a while though.
     
  8. chirpy

    chirpy Well-Known Member

    Joined:
    Jun 15, 2002
    Messages:
    13,475
    Likes Received:
    20
    Trophy Points:
    38
    Location:
    Go on, have a guess
    Yup, windows attacks. There are posts kicking around with help on alleviating the load of them from your log files and mod_security and/or mod_dosevasive would probably help out if it's a persistent problem.
     
  9. sh4ka

    sh4ka Well-Known Member

    Joined:
    May 12, 2005
    Messages:
    442
    Likes Received:
    0
    Trophy Points:
    16
    Location:
    US
    cPanel Access Level:
    DataCenter Provider
    same problem :(

    I have the same problem in one of my boxes...
    When I login as root i got this and then the ssh session get's locked :S ..

    sh4ka@machine:~$ ssh root@xxx.xxx.xx.x
    root@xxx.xxx.xx.xx's password:
    stdin: is not a tty
    /tmp/RsKgCzBO: Read-only file system
    stdin: is not a tty

    Any ideas about this ? , i think i got owned :(
     
  10. sh4ka

    sh4ka Well-Known Member

    Joined:
    May 12, 2005
    Messages:
    442
    Likes Received:
    0
    Trophy Points:
    16
    Location:
    US
    cPanel Access Level:
    DataCenter Provider
    Fixed!!

    Now i know why this was happening...
    FSTAB (/etc/fstab) got corrupted.. high level techs from DC fixed:
    "I've recreated the /etc/fstab file and your server is now online."

    If it happens again to anyone, now you'll know why.
    I'm so happy, my server wasn't owned! :D
     
  11. Snowman30

    Snowman30 Well-Known Member
    PartnerNOC

    Joined:
    Apr 7, 2002
    Messages:
    681
    Likes Received:
    0
    Trophy Points:
    16
    cPanel Access Level:
    DataCenter Provider
    One of our US servers had this issue last week after a reboot. We ran a manual forced fsck a couple of times and it resolved itself.

    Im still thinking it may have been a buggy or failing drive so we swapped it out...
     
Loading...

Share This Page