Cannot stop user accounts from spamming and cannot limit scripts from emailing.

[roosterv]

Hi everyone - I am really struggling trying to fix a spammer problem. Hopefully someone can assist or show me the error of my ways.

I have a VPS with a number of websites running, mainly Joomla. Recently, a few sites have been hacked. The spammers managed to place some scripts on the root directory.

Sites getting hacked is not the problem...

The real issue is that when/if a site gets hacked, my entire server's email gets OWNED. They jack my allocation and screw all my other accounts in 12 seconds.

***** NOTE *****
I have included one of the thousands of emails sent out by the rouge script below. I replaced the domain with a placeholder.
***************

So the idea here is that I'm trying to minimize the damage an account can do to my other accounts if it gets hacked.

I have done the tweak settings approach without luck.

And I do have CSF running on the site, btw.

Anyway, silly me :p:p:p You see I thought when tweak settings stated: "Max hourly emails per domain" that meant "Max hourly emails per domain". I set it to 5 and it stopped at 1000 because that's all that I'm allowed per day for ALL domains. NOT what I would have expected.

And I also thought that when tweak settings said: "Maximum percentage of failed or deferred messages a domain may send per hour", and I set that to 3% and had emails by domain set to 5, then I should NOT expect to see 32,000 emails in my mail Queue.

So obviously the account owner's have privileges that are too high.

I tried some other things as well:

1) Restrict outgoing SMTP to root, exim, and mailman (FKA SMTP Tweak)"

Result: There was an error updating Restrict outgoing SMTP to root, exim, and mailman (FKA SMTP Tweak).

(BTW, what differene does it make? You can see below that the spammer script is setting itself to localhost and cPanel is ignoring the idea that the domain is not allowed to do this.

2) I changed CFS to NOT limit SMTP and tried the above. Same result so I changed it back.

So, to summarize:

- How do I limit users so under their domains they cannot send out more emails than they are allowed?

I'm sure that there is some setting or configuration that will actually limit user emails, but for the life of me I cannot get this working. I would be very grateful for any assistance.

Thanks in advance.

=================================
Event: success
User: THISUSERNAME
Domain: THISDOMAINNAME.org
Sender: [email protected]
Sent Time: May 18, 2013 3:25:03 AM
Sender Host: localhost
Sender IP: 127.0.0.1
Authentication: localuser
Spam Score:
Recipient: [email protected]
Delivered To: [email protected]
Delivery User: -remote-
Delivery Domain:
Router: send_to_smart_host
Transport: remote_smtp
Out Time: May 18, 2013 3:25:03 AM
ID: 1UdeKl-0006EE-QG
Delivery Host: dedrelay.where.secureserver.net
Delivery IP: 208.109.80.54
Size: 884 bytes
Result: Message accepted
=======================================

 

[Veeble-Adam]

Worth a read : How to: Prevent Email Abuse

 

[cPanelMichael]

Hello

The following document is in-fact very helpful for preventing SPAM:

Prevent Email Abuse

In particular, steps four and five should be helpful in your case, and it includes information on how to configure these options:

Step 4: Configure the max hourly emails settings
Step 5: Configure high failure rate protection

Thank you.

 

[roosterv]

Hello

The following document is in-fact very helpful for preventing SPAM:

Prevent Email Abuse

In particular, steps four and five should be helpful in your case, and it includes information on how to configure these options:

Step 4: Configure the max hourly emails settings
Step 5: Configure high failure rate protection

Thank you.

Hi - and thanks for the feedback.

I did indeed configure all of those settings. Doesn't work. They blow past them without blinking and then all of my accounts are screwed.

Do you have any other suggestions?

Thanks in advance.

Mark

 

[Serra]

Big clue here:

Sender Host: localhost
Sender IP: 127.0.0.1

That means that the mail is coming from INSIDE the server, which is important. You can also have mail coming from outside, which is just as bad, but you'd need to look else where for a solution.

Things to look at:

1. Localhost mail can come from Roundcube too. A hacked account might be actually sending mail.
2. Normally local host exploits are sendmail in origin. i.e. a php script.

Try chmod'ing the /home/user directory to 000 and see if that stops the spam. If scripts can't run and the spamming stops, then you 100% know it is a script.

3. mod_security does a good job of stopping the hack in the first place. Install mod_security with a good set of rules. (like got_root)

4. Change all of your passwords, for root and for each of the domains. Change all of your email passwords too.

5. Make sure csf is blocking people who are attacking passwords over and over again.

6. Use the find command:

find /home/domain -mtime -1 -print

to look for changed files. Maybe you can find where they start and that will give you an idea of where they are going.

6. The configserver utilities have some settings that let you cap user domains from sending an amount of emails. Make sure you set it so that if a user exceeds their cap, the mail is saved on the queue. Then look on the queue at the email headers and see if they are sending it from sendmail or from some where else.

7. Look at the log and see if there is any ftp activity involved with the hack, if so, block that IP in csf.


Go to configserver.com and get all of their free utilities. They will be greatly helpful in seeing what is going on.

 

[roosterv]

Big clue here:

Sender Host: localhost
Sender IP: 127.0.0.1

That means that the mail is coming from INSIDE the server, which is important. You can also have mail coming from outside, which is just as bad, but you'd need to look else where for a solution.

Things to look at:

1. Localhost mail can come from Roundcube too. A hacked account might be actually sending mail.
2. Normally local host exploits are sendmail in origin. i.e. a php script.

Try chmod'ing the /home/user directory to 000 and see if that stops the spam. If scripts can't run and the spamming stops, then you 100% know it is a script.

3. mod_security does a good job of stopping the hack in the first place. Install mod_security with a good set of rules. (like got_root)

4. Change all of your passwords, for root and for each of the domains. Change all of your email passwords too.

5. Make sure csf is blocking people who are attacking passwords over and over again.

6. Use the find command:

find /home/domain -mtime -1 -print

to look for changed files. Maybe you can find where they start and that will give you an idea of where they are going.

6. The configserver utilities have some settings that let you cap user domains from sending an amount of emails. Make sure you set it so that if a user exceeds their cap, the mail is saved on the queue. Then look on the queue at the email headers and see if they are sending it from sendmail or from some where else.

7. Look at the log and see if there is any ftp activity involved with the hack, if so, block that IP in csf.


Go to configserver.com and get all of their free utilities. They will be greatly helpful in seeing what is going on.

I'm obviously not making myself clear. Sorry about that.

It is true, scripts were placed on an account's site. I found those scripts and corrected the problem. The problem is not about who/where the email are originating. The problem is that a single account was able to use up my email quote for the entire server.

Let me see if I can illustrate:

Server: Example Server has an email allocation of 1000 emails per day.

Setup in tweak settings:

Account A set to 10 emails per hour x 24 = 240/day.
Account B set to 10 emails per hour x 24 = 240/day.
Account C set to 10 emails per hour x 24 = 240/day.
Account D set to 10 emails per hour x 24 = 240/day.
Account E set to 10 emails per hour x 24 = 240/day.

Total per hour 50 and 1000 per day for the server due to quota.

Here is the problem.

Let's say that Account A gets hacked. The hacker places a script on the site that sends out spam. Because it happens so fast, I cannot stop the entire allocation of 1000 emails from being used up.

To ME, the hacker should only be able to send 240 emails max in a day on that account, no matter what. The other accounts should not be in jeopardy. But they are!!!!!

So what's happening? When account A was hacked, the hacker sent the entire 1000 emails in 2 minutes AND placed 40,000 email in my mail queue. I didn't find out until on of the other accounts (B,C,D or E) sent me personal email complaining that the website scripts for their website wasn't working at all. (Actually, all my other accounts got screwed for 3 days).

Why does the server let that hacked account override the tweak settings? The bottom line is: I myself can go into any account as that account user and send more than the hourly alloted rate.

What am I doing wrong? What good are tweak settings? Under my current configuration, tweak setting hourly rate reminds me of a busy box on a baby's crib. Nice steering wheel. But the baby is not actually driving the crib :p. Obviously my configuration is wrong, else cPanel would STOP Account A from screwing acounts B,C,D and E when it hit the 10 emails per hour (240 per day).

It is my opinion that the server's PHP code cannot differentiate between the account user and the root. Email is sent as root under localhost, and thus the quotas are bypassed. That is a guess on my part, but if true, how do I keep an account's php code from sending as root? I have also tried tiering the users, creating a user for account 'A' that is a reseller: ExampleDomain.Local, then assigning the actual domain under the .Local account. No difference, I can still send out more emails than the hour allocation allows.

This MUST be a problem for others users as well. I think this is a good discussion not just for me, but for others who believe that cPanel will stop an account from going rouge, only to find out that it will not actually stay inside the hourly quotas.

Again, thanks so much for your thoughts on this. I very much appreciate your input.

 

[cPanelMichael]

Could you open a support ticket so we can take a closer look at what settings you have enabled and provide you with a more accurate solution? You can open a ticket via:

Submit A Ticket

Please post the ticket number here so we can update this thread with the outcome.

Thank you.

 

[Serra]

I'm obviously not making myself clear. Sorry about that.

Sorry, I went off on a tangent there. The system should be able to tell if root or a user is sending mail. You should not be able to send mail as root if your settings are set that way.

It seems to me that the real problem here is the server getting hacked, not the email issue. The email issue is a symptom of the problem of hacking.

I would STOP THE HACKING! Forget the email issue!

The other issue is the 1000 email limit for the server. I realize it is a VPN and you have to live with the limits placed on the VPN by your host, but it just seems a bit difficult. One hack and your server is toast.

Obviously another solution would be to not do on server mail. Or at least not handle outgoing mail via the VPN at all. Two solutions I can think of are:

1. Switch the domains over to gmail, which you can get for fee for less than 6 users.
2. Have your clients send mail through a secondary email sender rather than through the VPN. Then when/if you get hacked, the server mail limit is blown, but who cares, the server doesn't send mail anyway. One such solution I can think of is Sendgrid. They can up your limit from 1000 to 10,000 for about $10.00 a month. It also makes it possible for your users to send mass emails directly from their outbox, which is cool.

 

[stdout]

Let's stay on track. You need to monitor your mail queue more closely (I don't mean that in a derogatory way).
I hope this can help you keep an eye on your server's mail/hack activity. The 1 liner below will:

1. Log the mail queue to /root/mailq.log
2. A per-account "count" is run on the mail queue to determine how many emails each account has sent.
3. If the account has more than 1,000 emails sitting in the mail queue -- those emails are automatically removed.
4. An email is automatically sent to [email protected] containing the full mail queue (prior to the removal).

exim -bp > /root/mailq.log; cat /root/mailq.log|awk {'print $4'} | sort | uniq -c | awk {'if ($1 > 1000) if ($2 != "<>") if ($2 != "") system("exiqgrep -i -f $2 | xargs exim -Mrm;mail -s \"Hacked Account\" [email protected] </root/mailq.log ")'}

Add the above code to a cronjob -- a 6-12hr cronjob is a fair window?

 

[iceman101184]

Let's stay on track. You need to monitor your mail queue more closely (I don't mean that in a derogatory way).
I hope this can help you keep an eye on your server's mail/hack activity. The 1 liner below will:

1. Log the mail queue to /root/mailq.log
2. A per-account "count" is run on the mail queue to determine how many emails each account has sent.
3. If the account has more than 1,000 emails sitting in the mail queue -- those emails are automatically removed.
4. An email is automatically sent to [email protected] containing the full mail queue (prior to the removal).

exim -bp > /root/mailq.log; cat /root/mailq.log|awk {'print $4'} | sort | uniq -c | awk {'if ($1 > 1000) if ($2 != "<>") if ($2 != "") system("exiqgrep -i -f $2 | xargs exim -Mrm;mail -s \"Hacked Account\" [email protected] </root/mailq.log ")'}

Add the above code to a cronjob -- a 6-12hr cronjob is a fair window?

This looks like a great script. I tried the code provided as root and got this reply:

[email protected] [/home/sdempsey]# exim -bp > /root/mailq.log; cat /root/mailq.log|awk {'print $4'} | sort | uniq -c | awk {'if ($1 > 1000) if ($2 != "<>") if ($2 != "") system("exiqgrep -i -f $2 | xargs exim -Mrm;mail -s \"Hacked Account\" - e-mail address removed - </root/mailq.log ")'}
bash: exim: command not found

Any ideas?

BTW - I think this is a GREAT thread. It's funny how many time the admin had to explain himself. I think the question is a good one and a real limitation of cpanel/whm that scripts run with PHP are executed as nobody/root and thus the limiting of email by account has 0% success. This is how 99.99% of hackers send spam these days - by accessing an account, injecting a malicious spam script, and using it to send the mail. So all the replies indicating things like "you need to get a better handle on what's emailing on your box" etc are ridiculous. We have 100 accounts on a box. The only way we know half the time when we're hacked if when a blacklist tells us and then we have to dig to find it. This is backwards. The WHM/cPanel utility should be able to limit the amount of php mail a user's account can send per hour. Right now it's a gap in the system. And with so much open-source php software out there (e.g. wordpress modules, cms tools, phpbb3, etc etc) it's hard to keep a system secure and prevent spammers from getting in now and then...!

 

[JaredR.]

Friendly Moderator Note

I removed the actual e-mail address from your output, and wrapped it in code tags to preserve the formatting.