Help with Finding Disk Exceeded Error and Potential Hacker

[GoreDefex]

Hello,

I am currently having a large problem with a VPS hosting that I am administrating. It is a reseller account and we as a company have many clients within our CPanel with varied bandwidth and HDD limitations on each. I recently went to log in to my CPanel WHM account and I could not even view the login panel due to a 500 error and a Disk quota exceeded error at /usr/local/cpanel/Cpanel/Session.

It took many calls to CPanel and my hosting provider to discover that no one is interested in helping us since we do not have a managed account... CPanel support has tried but so far they could not do much besides help me resolve the error in the present moment. This was helpful since I discovered that this error had been due to having my entire HDD space filled (even though this is not physically possible since we have a massive package and not even a third of the space normally used). We quite literally had 2/3 of our huge account space free last week and now its full....

At first I thought it was a bad allocation of INodes on the VPS causing a phantom "filled" effect however since I could still SSH into the VPS a few df and du commands revealed that sure enough the acutal HDD somehow was full. Now the tricky part of this problem.

In order to stop this problem temporarily I removed (by the advice of a CPanel agent) a few large file sized backups from some time ago that could be removed via SSH and it let me back into WHM. This was a happy moment however three times since then I have went to log in and found that I was locked out again due to the exact same error. I am running out of useless files on our server to delete in order to get back into WHM and the error keeps happening. There is nothing relevant to our problem within an of the CPanel logs and no customer support can help it seems. I have tried reviewing the /apache/logs, the logs within CPanel UI and also my own server maintenance logs from our weekly backups.

Now, since all this stuff has been happening, the only server log that is related to this error I can find is in different software. Unbeknownst to me we had received two emails from one of our security systems flagging two different blocked email chains from one of our clients accounts for email spamming. I was wondering if perhaps this is causing the problem and someone is spam emails us, or spam comment attacking us or anything similar. That and also where would I locate these spam emails being saved somewhere?

I am a new administrator to the field and fresh out of school. I am unsure how to approach this anyone have any thoughts?

 

[Mckenzielaa]

surely you just need to find the cPanel user who had uploaded a lot of information and delete there account?

 

[Infopro]

It took many calls to CPanel and my hosting provider to discover that no one is interested in helping us since we do not have a managed account

Suggestion: As a new user, starting out with an unmanaged server is going to be full of surprises for you. Some, not much fun at all trying to ride out on your own. You might consider moving to a managed solution for now until you've got a more clear idea of what's needed and how to go about it.

In order to stop this problem temporarily I removed (by the advice of a CPanel agent) a few large file sized backups from some time ago that could be removed via SSH and it let me back into WHM. This was a happy moment however three times since then I have went to log in and found that I was locked out again due to the exact same error.

Where were these backups located? On the account itself, or in the automated server account backups directory?

 

[GoreDefex]

Yes the backups are within the server and were done automatically and shut off a long time ago. Since then they have sat on the server because we have (had) so much space that it was not an issue. Now however, it is keeping me a float as a happy accident....

Since I have posted this the server has gone down three more times. Each time its the same thing. I remove something worth 200 mgs or so and then after 6 hours or so I get a disk exceeded error all over again. I am completely out of stuff I can't afford to delete lol. It's getting intense.

I have no idea how to figure out which account is doing this but I have some clue from my spam detector security software. However when I look at the monthly/weekly transfer rates from them I see nothing large.

 

[Infopro]

Since I have posted this the server has gone down three more times. Each time its the same thing. I remove something worth 200 mgs or so and then after 6 hours

Could you be a bit more specific? What files are you removing here, backups again or no?

 

[GoreDefex]

Ya backups are not needed for us. I came into the company and all this was already set up for us and the previous guy had chosen to do backups. I stopped this once I joined because we do weekly and nightly backups ourselves to both cloud repo's as well as a time machine so there is no need for the third option.

I keep deleting these backups because as long as they are there I have something able to be deleted when this spam nonsense occurs and shuts us down momentarily.

- - - Updated - - -

I have tried blocking the account with the problem and its just not working. I put a bandwidth limit on it of 4 g and we have over 200g of FREE space on our HDD lol. Its all just confusing. Anyway the problem account is at 95% of this 4g limit and I hope it stops the problem. I went into this accounts emails like the spam detector had mentioned and sure enough there is alot of "new" emails that say they weren't delievered and the email content is click here to win $800,000. Also, all these spams are aprently coming from steve.[email protected] .... curious lol.

 

[Infopro]

Still unclear on the situation here.

If backups are enabled on the server, in WHM, and the configuration of said backups is set to backup all accounts to the same HDD, you should either change that, or disable backups proper in WHM.

As to what sounds like a second issue, a user on your server sending spam, suspend the account, or, remove it outright.