Where's the data?

[wcs4web]

The other day, one of our servers was hacked. We immediately had a fresh OS installed, slaved the old OS and rebuilt the machine. We also had a secondary drive which contained all of our CPanel backups, but since there was only two slots for drives, the backup drive had to be removed to install the old OS drive.

Once the new OS was installed, the data from the old OS was copied and we had a workable machine, we had the old OS drive removed and the backup drive put back in.

Here is were the problem starts. When we went to do a restore from the backup drive, it was empty, nothing at all on the drive. The backups have been running nightly up until the hack occurred.

We thought that one of the following may have caused it:

• 
  When the machine was brought backup after putting in the backup drive, the backup drive was mounted as an ext3 drive, but was formatted and used with the old OS as an ext2 filesystem.

• 
  It is possible that the hacker deleted all of the backups, but he didn't delete anything from the old OS drive, so it is questionable whether this was the cause

• 
  The data center whipped the drive, but would not admit doing so or put in a new drive that was empty.

But here's where it gets confusing...The drive shows no data when we do a directory listing (ls -la), but when we do a DF -h command it shows that 85mb have been used on the drive and that was the same usage prior to the hacks according to our logwatch reports.

So based on the fact that the system is showing data usage on the drive, it appears that the correct drive was put in. There is only one partition on the drive. So what happened to the data?

I am looking to see if there are any answers to this issue and to these questions:

• Could the partition have been corrupt by mounting as an ext3 instead of ext2 when it was previously formatted as an ext2?

• Could the hacker have made the cpbackup folder hidden from view of the root account, and if so how can it be unhidden?

• Why would the system show disk usage when nothing is appearing on the drive

Any thoughts or insights that would help clarify this situation would be appreciated.

Regards

George

 

[chirpy]

It's not easy mounting a drive in ext3 when it's been formatted as ext2 so I doubt that would be the issue. It's always possible that they disk was knocked/damaged when they had the server open which caused the problems - that's not difficult to do.

A very good tool to use on disks that may have partition table problems is:
http://www.cgsecurity.org/wiki/TestDisk

 

[wcs4web]

Thanks Jonathan,

I did download that tool but it did not find any additional partitions, I was able to recover several of the backup files off the hard drive using this tool, although some of the files appeared to be corrupt and could not get all the data back.

Any idea why it would still show 85 meg used even though there appeared to be nothing visible on that partition and only the one partition on the drive.

Regards,

George Wilson

Also, Thanks Jonathan for recovering our machine, you did a great job.