/etc/passwd getting corrupted

[EWD]

Hi Guys,

I have a weird situation here that started last night.

On 2 servers now I've had the /etc/passwd file corrupted.
Each time the following is what is found in the file:

pst0^E^G^C^@^@^@^U^D^B^@^@^@^@^@^@^@^KDEADDOMAINS
        unlimited^@^@^@^FMAXSUB
        127.0.0.1^@^@^@^BIP
^@^@^@^@^FDOMAIN
        unlimited^@^@^@^FMAXSQL
        unlimited^@^@^@^FMAXFTP^H^À^@^@^@^FHASCGI^H^À^@^@^@^DDEMO
^\../../../../../../etc/passwd^@^@^@^DUSER
^Gdefault^@^@^@^KFEATURELIST^D^B^@^@^@^@^@^@^@^GDOMAINS
        unlimited^@^@^@^GBWLIMIT

0000000000^@^@^@        STARTDATE
^Droot^@^@^@^EOWNER^H^À^@^@^@^GMAXPARK
^Bx3^@^@^@^BRS
        undefined^@^@^@^DPLAN^H^À^@^@^@^HMAXADDON
^Genglish^@^@^@^DLANG
        unlimited^@^@^@^FMAXLST
        unlimited^@^@^@^FMAXPOP

All seems to be data from hosting packages.
Nothing in the logs other then errors about users not existing(because /etc/passwd is wiped)

Anyone seen anything like this happening?

Thanks

 

[nazmy]

yeah we faced this too, hope cpanel got this fixed ASAP this is a serious problem

 

[EWD]

Ok do you have a ticket in on this?

I am going to put one in now and reference this thread.

Glad it is not just me :D

 

[gasxtreme]

Yes, I can confirm I have had this occur on multiple servers of mine........

I have found a (atleast temporary) fix:

cd /etc/;mv passwd passwd.old;mv passwd.OLD passwd;reboot;

-steve

 

[LS_Drew]

I've had this happen on two machines today, one of them twice.

The second time, /etc/passwd, /etc/group, and /etc/shadow were all empty.

 

[Todd Mitchell]

Can you post the cPanel versions that are being affected by this?

 

[LS_Drew]

WHM 11.2.0 cPanel 11.8.0-C15689

Centos 4.5

That's one of them.

the other one that did it twice today, I downgraded to stable after it did it again. I should have noted the version, but I didn't.

When I did the downgrade, I got an email to update perl, which I did immediately.

 

[Todd Mitchell]

I recommend updating to the latest version of the build. The only thing I can see so far with this issue is that all reported cases of this are running an out of date build.

The latest builds are:

11.11.0-STABLE_16999
11.15.0-RELEASE_17665
11.15.0-CURRENT_17700
11.15.0-EDGE_17700

We are still looking into this issue to pinpoint the exact cause.

 

[EWD]

Just had it happen to another one:

WHM 11.2.0 cPanel 11.11.0-R16709
CENTOS Enterprise 4.5 i686 - WHM X v3.1.0

 

[cPanelNick]

Just had it happen to another one:

WHM 11.2.0 cPanel 11.11.0-R16709
CENTOS Enterprise 4.5 i686 - WHM X v3.1.0

Have you seen it happen on any builds newer then 16800 ?

 

[EWD]

Have you seen it happen on any builds newer then 16800 ?

Hi Nick,

So far all older.

 

[cPanelNick]

Hi Nick,

So far all older.

I would advise updating to 16800+ on all machines if you haven't already done so (as this is the safest course of action at this point). These builds have much safer locking code. However I'm not sure if its 16800+ or a later version that solves the problem at this time. Likewise the latest builds appear immune if this is related.

Update: It has now has something to do with changing passwords, however this is not yet confirmed.

 

[cPanelNick]

Can you check the rlog on /etc/passwd? That might be helpful in determining the cause.

You should be able to restore the latest editted version of /etc/passwd by running

co /etc/passwd
chmod 644 /etc/passwd

 

[EWD]

Can you check the rlog on /etc/passwd?

Nothing really showing at the time it happened:

[email protected] [~]# rlog /etc/passwd

RCS file: /etc/passwd,v
Working file: /etc/passwd
head: 1.56
branch:
locks: strict
        root: 1.56
access list:
symbolic names:
keyword substitution: kv
total revisions: 56;    selected revisions: 56
description:
Init by /scripts/realadduser
----------------------------
revision 1.56   locked by: root;
date: 2007/10/10 12:56:25;  author: root;  state: Exp;  lines: +0 -1
Modified by /usr/local/cpanel/whostmgr/bin/whostmgr .
----------------------------
revision 1.55
date: 2007/08/31 23:30:49;  author: root;  state: Exp;  lines: +1 -0
Modified by /scripts/realadduser .
----------------------------
revision 1.54
date: 2007/07/01 14:23:56;  author: root;  state: Exp;  lines: +1 -1
Modified by /usr/local/cpanel/whostmgr/bin/whostmgr .
----------------------------

 

[mweb]

One server:

WHM 11.2.0 cPanel 11.8.0-C15845
CENTOS Enterprise 3.9 i686 - WHM X v3.1.0

Coincidentally, it just recently started receiving regular nessus scanning from scanalert.com, and just resumed web log processing after a short hiatus. Not sure if that matters, but it does put ../../../../../etc/passwd in the error logs.

 

[eperdeme]

Hmm, same problem also.

WHM 11.2.0 cPanel 11.6.0-R15076
REDHAT Enterprise 5 i686 - WHM X v3.1.0

Also started on 20th after months of no problems. Appears to be a possible exploit in the wild I would presume causing this.

 

[lbccserv]

I have the same issue:

WHM 11.2.0 cPanel 11.7.0-E14936
FEDORA 2 i686 - WHM X v3.1.0

moving passwd.OLD ---> passwd fixed the issue for me as well

 

[handsonhosting]

Hey Emerson (fancy meeting you here! )

Just thought I'd document another incident for this issue.

cPanel Version 11.8.0-C16276

Running an upgrade on that server currently. What bothers me is that there was NO changes made to that server since that version installation MONTHS ago.

There shouldn't have been ANY password changes etc, as there is only ONE accounton the machine and it's used strictly as a storage space for myself.

 

[VeZoZ]

Had this happen at the beginning of September

At the time I was told this could have been caused by anything from a Race condition, corrupt library or to a Load issue at the time the /etc/passwd file was modified when this occurred. It was also unlikely to occur again. I was also the only one to experience this problem and was told it was not a cPanel bug.

Well thanks to this post I at least know it more than likely wasn't something I could have stopped.

 

[EWD]

Hey Emerson (fancy meeting you here! )
-- Sup dude

Just thought I'd document another incident for this issue.

cPanel Version 11.8.0-C16276

Running an upgrade on that server currently. What bothers me is that there was NO changes made to that server since that version installation MONTHS ago.
-- Same here. The machines that have been affected did not have any changes done.

There shouldn't have been ANY password changes etc, as there is only ONE accounton the machine and it's used strictly as a storage space for myself.
-- What do you store in there, you fishing gear? :D