The Community Forums

Interact with an entire community of cPanel & WHM users!
  1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Outgoing Email Abuse from localhost

Discussion in 'E-mail Discussions' started by ruzbehraja, Aug 10, 2016.

Tags:
  1. ruzbehraja

    ruzbehraja Well-Known Member

    Joined:
    May 19, 2011
    Messages:
    383
    Likes Received:
    7
    Trophy Points:
    18
    cPanel Access Level:
    Root Administrator
    I recently noticed that some accounts on one of our servers, were sending out spam mail.
    Code:
    1 (compromiseddomain.com)[::1] info@compromiseddomain.com alt2.gmail-smtp-in.l.google.com[173.194.213.26] jii93@gmail.com
    1 (compromiseddomain.com)[::1] info@compromiseddomain.com alt4.gmail-smtp-in.l.google.com[209.85.203.27] jignre@gmail.com
    1 (compromiseddomain.com)[::1] info@compromiseddomain.com gmail-smtp-in.l.google.com[74.125.129.27] jien@gmail.com
    1 (compromiseddomain.com)[::1] info@compromiseddomain.com gmail-smtp-in.l.google.com[74.125.129.27] jibsney@gmail.com
    1 (compromiseddomain.com)[::1] info@compromiseddomain.com gmail-smtp-in.l.google.com[74.125.129.27] jig78@gmail.com
    
    However, the strange part is that the logs were showing that the mail was coming from our IPv6 loopback address i.e. ::1

    Assuming that webmail maybe compromised, I investigated further.

    But I noticed that most of these accounts did not have any email address or forwarders on our server - they were using Google Apps. Hence there was no way of authenticating to send out mail.

    All the emails are going out from info@ accounts only.

    The maillog shows:
    Code:
    2016-08-10 07:31:18 1bXIpa-003UXt-QG <= info@compromiseddomain H=(compromiseddomain) [::1]:48468 P=esmtp S=1617 id=2025813250.1694955.1470794479702@compromiseddomain T="howre you?" for pashl23@orange.fr
    2016-08-10 07:31:19 1bXIpa-003UXt-QG From: header (rewritten was: [info@compromiseddomain], actual sender is not the same system user) original=[info@compromiseddomain] actual_sender=[nobody@myserver.com]
    2016-08-10 07:31:19 SMTP connection from (compromiseddomain) [::1]:48468 lost
    
    Other info:
    • SMTP Authentication is enabled on the server.
    • Nobody user is prevented from sending out mail.
    • CSF is enabled. CSF did not send any mail alerts.
    • IPv6 is enabled in Exim. This is happening only from the ::1 IPv6 Address.
    • This is happening to about 7 - 10 domains on the server.
    • This is happening only to the INFO@ email address for all those domains.
    • Some of the accounts are using Google Apps for mail and don't even have an email address configured on my server.
    • One of the accounts was suspended on 1-Aug and still remains suspended, still mails went out from it.
    • The Firewall logs show multiple unsuccessful brute force attempts for all those accounts a day prior to the mail spamming. Besides that nothing suspicious.
    • Running CLOUDLINUX 6.8 x86_64 with WHM 58.0 (build 13)

    Any ideas on how this could be happening?
     
    #1 ruzbehraja, Aug 10, 2016
    Last edited: Aug 10, 2016
  2. cPanelMichael

    cPanelMichael Forums Analyst
    Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    30,854
    Likes Received:
    675
    Trophy Points:
    113
    cPanel Access Level:
    Root Administrator
    Hello,

    I recommend running the following command to rule out the possibility that SPAM is coming from a script uploaded to an account:

    Code:
    awk '/cwd=\/home\// {print $3}' /var/log/exim_mainlog|sort|uniq -c|sort -n
    Look for directories with a large send count and review the files in those directories to see if any scripts are able to send email.

    Thank you.
     
  3. reddot

    reddot Member

    Joined:
    Jan 14, 2008
    Messages:
    5
    Likes Received:
    0
    Trophy Points:
    1
    My problem is similar.

    Below is exim_mainlog extract.

    Code:
    2016-08-17 01:39:05 1bZiKO-0007WK-GW H=(userdomain.com) [::1]:56891 Warning: Message has been scanned: no virus or other harmful content was found
    2016-08-17 01:39:05 1bZiKO-0007WK-GW <= info@userdomain.com H=(userdomain.com) [::1]:56891 P=esmtp S=2256 id=2128087589.1190747.1471369143281@userdomain.com T="Fw:  Hi there." for recipient@aol.com
    2016-08-17 01:39:05 cwd=/var/spool/exim 3 args: /usr/sbin/exim -Mc 1bZiKO-0007WK-GW
    2016-08-17 01:39:05 1bZiKO-0007WK-GW SMTP connection identification H= A=::1 P=56891 M=1bZiKO-0007WK-GW U=nobody ID=99 S=nobody B=authenticated_local_user
    2016-08-17 01:39:05 1bZiKO-0007WK-GW From: header (rewritten was: [info@userdomain.com], actual sender is not the same system user) original=[info@userdomain.com] actual_sender=[nobody@myserver.com]
    2016-08-17 01:39:10 1bZiKO-0007WK-GW => recipient@aol.com R=lookuphost T=remote_smtp H=mailin-02.mx.aol.com [152.163.0.99] X=TLSv1:DHE-RSA-AES256-SHA:256 CV=yes C="250 2.0.0 Ok: queued as 05C7B700000BF"
    2016-08-17 01:39:10 1bZiKO-0007WK-GW Completed
    
    Prevent “nobody” from sending mail is "ON". How is the server still allowing emails to be sent by nobody to remote address?

    "cwd" from the log above shows it is not from a script.

    Please help. Thank you.
     
  4. ruzbehraja

    ruzbehraja Well-Known Member

    Joined:
    May 19, 2011
    Messages:
    383
    Likes Received:
    7
    Trophy Points:
    18
    cPanel Access Level:
    Root Administrator
    Great. You have the exact same issue that I did....

    It was solved by enabling mod_ruid2.

    This helped us determine the user that was sending out the mails and also the script.

    The cause was a compromised WordPress plugin i.e. Revolution Slider. It was using the cPanel password to authenticate the mail going out.

    This was one of the latest versions of Revolution Slider which has no known vulnerability. Removing the entire folder was the only solution, as every sub-folder in the templates folder was infected.

    Resetting the cPanel password stopped the mails from authenticating.
     
    #4 ruzbehraja, Aug 19, 2016
    Last edited: Aug 19, 2016
  5. ruzbehraja

    ruzbehraja Well-Known Member

    Joined:
    May 19, 2011
    Messages:
    383
    Likes Received:
    7
    Trophy Points:
    18
    cPanel Access Level:
    Root Administrator
    Now coming to the point about the Tweak Setting to prevent the nobody user from sending out mail.

    When I reported this to cPanel, this was their response:

    "...
    The tweak setting 'Prevent "nobody" from sending mail' is a restriction that only applies to emails sent with /usr/sbin/sendmail and does not restrict emails sent as SMTP through a local TCP port.

    If you wanted to disallow sending SMTP emails through localhost, you would need to enable the 'SMTP Restrictions' feature in the Security Center in WHM. This would add firewall rules to block connecting to the SMTP ports locally, forcing PHP scripts to use sendmail (and honor the tweak setting mentioned earlier). It is worth noting that if you use CSF to configure your firewall, the 'SMTP Restrictions' setting in WHM will not work. You would need to enable the following features in the CSF firewall settings to achieve the same effect: SMTP_BLOCK = ON, SMTP_ALLOWLOCAL = OFF.
    ..."​


    Also, when they investigated the issue on the server, the Support Rep's response was:

    "...
    Thanks again for the thorough follow up. To clarify though, in my previous testing, while accessing the "nobody" user, it didn't seem to matter whether I connected to the server on the IPv4 or IPv6 addresses, the messages were blocked if I didn't first authenticate with another user, which seems to be a deliberate choice to prevent servers using DSO or CGI from blocking every source of mail.

    Reviewing this a bit further, I found a much older case where we found users having similar problems while on DSO were encountering conflicts with the following options in Exim:

    ----------------------------------------------------
    "Query Apache server status to determine the sender of email sent from processes running as nobody"
    "Trust X-PHP-Script headers to determine the sender of email sent from processes running as nobody"
    ----------------------------------------------------

    Both are enabled by default, but in the past, have allowed some user's scripts to send mail after having been identified by Exim as not actually being sent as nobody. Though this server shouldn't encounter this problem again with the most recent changes to its settings, if you would like to go back to the DSO handler, I'd recommend disabling both of these options as well and expect things will work more as you'd expect.
    ..."​
     
  6. reddot

    reddot Member

    Joined:
    Jan 14, 2008
    Messages:
    5
    Likes Received:
    0
    Trophy Points:
    1
    I have no idea how the emails can be sent if what cPanel support said is working.

    My server is not using CSF.
    SMTP Restrictions is enabled.
    Prevent "nobody" from sending mail is ON.

    I might just switch to mod_ruid2 like what you did or force reset the cPanel password.

    Thank you very much for the update, ruzbehraja.
     
  7. ruzbehraja

    ruzbehraja Well-Known Member

    Joined:
    May 19, 2011
    Messages:
    383
    Likes Received:
    7
    Trophy Points:
    18
    cPanel Access Level:
    Root Administrator
    The emails are being authenticated with a username and password. In my case it was a cPanel username and password. Mails were going out from a script which was in a WordPress plugin folder.

    To find out which user was being used to authenticate the mails, after you install mod_ruid2 grep the logs again.

    I think what cPanel really needs to highlight in the Tweak Settings option explanation is that "The tweak setting 'Prevent "nobody" from sending mail' is a restriction that only applies to emails sent with /usr/sbin/sendmail and does not restrict emails sent as SMTP through a local TCP port."

    If you still can't find out, open a support ticket with cPanel and do post back here if you bump into something interesting.
     
  8. reddot

    reddot Member

    Joined:
    Jan 14, 2008
    Messages:
    5
    Likes Received:
    0
    Trophy Points:
    1
    Thank you for the explanation, ruzbehraja.

    I was looking at the response given by cPanel saying that enabling SMTP Restrictions will disallow sending SMTP emails through localhost when coupled with 'Prevent "nobody" from sending mail'.

    I looked at the description in WHM -> Server Configuration -> Tweak Settings ...

    and WHM -> Security Center -> SMTP Restrictions

    Do the descriptions mean that the SMTP Restrictions feature is only to disallow connections to remote SMTP servers? If yes, does it mean that the cPanel response is wrong in saying that it forces PHP scripts to use sendmail?

    My log shows it is using /usr/sbin/exim.

    Code:
    2016-08-17 01:39:05 cwd=/var/spool/exim 3 args: /usr/sbin/exim -Mc 1bZiKO-0007WK-GW
    I am kind of confused.
     
  9. ruzbehraja

    ruzbehraja Well-Known Member

    Joined:
    May 19, 2011
    Messages:
    383
    Likes Received:
    7
    Trophy Points:
    18
    cPanel Access Level:
    Root Administrator
    Block outgoing SMTP (except for root, exim and mailman) forces scripts/users to use the exim/sendmail binary instead of sockets access i.e. directly to a remote servers port 25

    When they are forced to use sendmail, they have to send out as the nobody user. Which is when the nobody "Tweak Setting" will take effect or be triggered.

    If they are not forced to use sendmail, they can connect to your localhost random port and send through exim as if they are an authenticated local user, as what happened. i.e. they connected to the ::1 loopback and as your log says:

    Code:
    U=nobody ID=99 S=nobody B=authenticated_local_user
    They authenticated as a local user but sent the mail as 'nobody'.

    I am also not 100% clear on how this can be prevented and whether a similar tweak should be made applicable to prevent this in future.

    I am still a bit skeptical on whether or not it is related to the IPv6 address, because till date I have never faced such an issue with the IPv4 loopback, having the same settings.

    Also, when I went to the hostslists loopback Advanced Settings in Exim, and removed ::1, the spam stopped going out, however, webmail's connection also broke, so I couldn't test it for long.
     
  10. reddot

    reddot Member

    Joined:
    Jan 14, 2008
    Messages:
    5
    Likes Received:
    0
    Trophy Points:
    1
    Thank you for clearing up on SMTP Restrictions.

    All the spam is going through ::1 using an authenticated user and nobody@myserver.com as the actual sender.

    Code:
    2016-08-17 01:39:05 1bZiKO-0007WK-GW SMTP connection identification H= A=::1 P=56891 M=1bZiKO-0007WK-GW U=nobody ID=99 S=nobody B=authenticated_local_user
    Would it be possible to write a custom filter to drop such emails? Anyone?

    cPanel has so many users, but it seems that this thread is only you and me. LOL
     
  11. ruzbehraja

    ruzbehraja Well-Known Member

    Joined:
    May 19, 2011
    Messages:
    383
    Likes Received:
    7
    Trophy Points:
    18
    cPanel Access Level:
    Root Administrator
    The modus operandi of this attack is to send out such few mails frequently that they don't trigger an alarm.

    Just about 10-20 mails are sent out every hour, from different domains from the info@ email address (which was obviously spoofed).
    Surprisingly all the email addresses must be valid, because I didn't notice anything (bounces) in the mail queue - all got delivered perfectly.

    This may have been happening for a couple of days, but only when I manually saw the mail statistics did I notice the bunch of ::1 loopback addresses all together. Hence I am pretty sure it is happening to others, but they have not realized it yet.

    Also, people maybe under a false impression that the nobody tweak setting should be blocking this. Which even I was under and I guess so were you. :)

    I don't think you need a filter for this because the mails were authenticated with a stolen password. Only thing is that it was spoofing the nobody user as the sender.
     
  12. cPanelMichael

    cPanelMichael Forums Analyst
    Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    30,854
    Likes Received:
    675
    Trophy Points:
    113
    cPanel Access Level:
    Root Administrator
  13. reddot

    reddot Member

    Joined:
    Jan 14, 2008
    Messages:
    5
    Likes Received:
    0
    Trophy Points:
    1
    Code:
    2016-08-17 01:39:05 cwd=/var/spool/exim 3 args: /usr/sbin/exim -Mc 1bZiKO-0007WK-GW
    2016-08-17 01:39:05 1bZiKO-0007WK-GW SMTP connection identification H= A=::1 P=56891 M=1bZiKO-0007WK-GW U=nobody ID=99 S=nobody B=authenticated_local_user
    2016-08-17 01:39:05 1bZiKO-0007WK-GW From: header (rewritten was: [info@userdomain.com], actual sender is not the same system user) original=[info@userdomain.com] actual_sender=[nobody@myserver.com]
    Is this considered enabled?

    The only thing I have not done is to use Mod_Ruid2 or suPHP or DSO with MPM ITK to force the scripts to run as the user in order to identify.

    Is it the only way?

    Is there any log for such kind of authentication?
     
  14. cPanelMichael

    cPanelMichael Forums Analyst
    Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    30,854
    Likes Received:
    675
    Trophy Points:
    113
    cPanel Access Level:
    Root Administrator
    Yes, it does look enabled based on that output.

    It's not required, but it will make figuring out the source of the SPAM leaving your system easier.

    Thank you.
     
  15. ruzbehraja

    ruzbehraja Well-Known Member

    Joined:
    May 19, 2011
    Messages:
    383
    Likes Received:
    7
    Trophy Points:
    18
    cPanel Access Level:
    Root Administrator
    In this specific case, this option did not help, because it rewrote the header to show the nobody@serverhostname.com as the sender.

    Even I had pasted the logs showing this above.

    It seems like this is the only way. You have to enable atleast mod_ruid2 to identify the user with which it is authenticating.
     
Loading...

Share This Page