The Community Forums

Interact with an entire community of cPanel & WHM users!
  1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Server overloaded - SPAM? Exim processes

Discussion in 'General Discussion' started by postcd, Aug 9, 2011.

  1. postcd

    postcd Well-Known Member

    Joined:
    Oct 22, 2010
    Messages:
    621
    Likes Received:
    6
    Trophy Points:
    18
    Hello i need help,

    i can see exim processes like this on my server:
    27869 mailnull 25 0 7700 852 532 R 4 0.0 0:17.53 /usr/sbin/exim -bd -q60m

    There are load spikes and swap is at 50-100%

    Mail Queue is at maximum 4000+ messages and tail -f /var/log/exim_mainlog shows some like spammy esmtp threads.


    2011-08-10 00:04:12 1QquPD-00061H-TD <= ***@****.org H=ns35.****.com [254.254.254.254] P=esmtps X=TLSv1:AES256-SHA:256 S=3117 id=E1QquUw-0002Z5-DQ@ns35.****.com

    Im not able to discover what is the cause and what i need to ban, how to protect server, so im asking you there. There are more details from cPanel mail stats:

    Time spent on the queue: all messagesTime Messages Percentage Cumulative Percentage
    Under 1m 58312 44.1% 44.1%
    5m 47 0.0% 44.1%
    3h 4 0.0% 44.1%
    6h 1 0.0% 44.1%
    12h 6 0.0% 44.1%
    1d 8 0.0% 44.2%
    Over 1d 73844 55.8% 100.0%

    Top 50 mail rejection reasons by message countMessages Mail rejection reason
    8264 Unknown
    2327 Rejected RCPT: Sender verify failed
    492 Rejected MAIL: Access denied - Invalid HELO name (See RFC2821 4.1.1.1)

    Top 50 mail temporary rejection reasons by message countMessages Mail temporary rejection reason
    6499 Temporarily rejected RCPT: Could not complete sender verify

    Top 50 rejected ips by message countMessages Rejected ip
    7493 local
    1630 [*.*.*.*]
    68 [*.*.*.*]
    61 [*.*.*.*]

    PLEASE, can anyone help me what exactly to do to discover source of this issue and eliminate it? It must be also helpfull for more members.

    Thank you,
    P.
     
  2. postcd

    postcd Well-Known Member

    Joined:
    Oct 22, 2010
    Messages:
    621
    Likes Received:
    6
    Trophy Points:
    18
    Please can You help me and all who have same issue to recognize what is the cause?
     
  3. syslint

    syslint Well-Known Member

    Joined:
    Oct 9, 2006
    Messages:
    249
    Likes Received:
    6
    Trophy Points:
    18
    Location:
    India
    cPanel Access Level:
    Root Administrator
    Twitter:
    You may better enable spam cops RBL checking from whm -> Exim configuration editor
     
  4. postcd

    postcd Well-Known Member

    Joined:
    Oct 22, 2010
    Messages:
    621
    Likes Received:
    6
    Trophy Points:
    18
    -
    Thank you, i enabled this today, and added my server IP to the whitelist on same config page. But even before i enabled this, LOAD was quite low. I restarted httpd, exim etc, at first look nothing changed.. So LOAD is no longer problem maybe, but there are still those SMTP senders mentioned in first post..! Please how can i eliminate them and stop using blacklist external service?

    Swap Used 96.29% (1,012,080 of 1,051,064)

    Thank you
     
    #4 postcd, Aug 19, 2011
    Last edited: Aug 19, 2011
  5. postcd

    postcd Well-Known Member

    Joined:
    Oct 22, 2010
    Messages:
    621
    Likes Received:
    6
    Trophy Points:
    18
    lfd: The exim delivery queue size is 19940

    I need to note that i have high number of incoming emails, like 100 per minute and most coming into non existing email and are relayed to existing email address.
     
    #5 postcd, Aug 19, 2011
    Last edited: Aug 19, 2011
  6. cPanelTristan

    cPanelTristan Quality Assurance Analyst
    Staff Member

    Joined:
    Oct 2, 2010
    Messages:
    7,623
    Likes Received:
    21
    Trophy Points:
    38
    Location:
    somewhere over the rainbow
    cPanel Access Level:
    Root Administrator
    What does "come into non-existing email and are relayed to existing email mean" exactly? Are they forwarded by accounts on the machine that exist? What is the full route for one of the messages?

    Code:
    exigrep messageID /var/log/exim_mainlog
    Where messageID is the exim ID given to the message. For example, 1QuRrF-0007sJ-WF would be a message ID for a message on my machine.
     
  7. tdens

    tdens Member

    Joined:
    Aug 18, 2011
    Messages:
    19
    Likes Received:
    0
    Trophy Points:
    1
    Swapping like that is bad, obviously.

    Sounds like you may have catchall addresses enabled. They're on by default, so if you didn't disable them, anything sent to any username at a real domain will be sent to the default user for that domain. Look under server configuration - tweak settings - default catch-all. If that's not the case, please describe the problem better so others can help (I'm a complete WHM newb, sorry).
     
  8. postcd

    postcd Well-Known Member

    Joined:
    Oct 22, 2010
    Messages:
    621
    Likes Received:
    6
    Trophy Points:
    18
    --
    Hello, this is full route:

    What i meant is that i have Catch all mail. But in fact i realised that i deleted all mailboxes on that domain and set :blackhole:

    Even when i suspend all suspicious cPanel accounts, those email entries coming into exim_mainlog.

    Server load is quite ok, except 95-100% full swap and overloaded mailqueue 19k mails
     
    #8 postcd, Aug 19, 2011
    Last edited: Aug 19, 2011
  9. cPanelTristan

    cPanelTristan Quality Assurance Analyst
    Staff Member

    Joined:
    Oct 2, 2010
    Messages:
    7,623
    Likes Received:
    21
    Trophy Points:
    38
    Location:
    somewhere over the rainbow
    cPanel Access Level:
    Root Administrator
    This entry doesn't show it was relayed. It shows that it went to :blackhole: to be deleted.
     
  10. postcd

    postcd Well-Known Member

    Joined:
    Oct 22, 2010
    Messages:
    621
    Likes Received:
    6
    Trophy Points:
    18
    So it was not relayed, what i can do about it? My simple question is how to discover what script causing the issue? Or what is the cause, how i can discover it exactly. If anyone can help me, i would be gratefull.
     
  11. cPanelTristan

    cPanelTristan Quality Assurance Analyst
    Staff Member

    Joined:
    Oct 2, 2010
    Messages:
    7,623
    Likes Received:
    21
    Trophy Points:
    38
    Location:
    somewhere over the rainbow
    cPanel Access Level:
    Root Administrator
    If these are incoming emails, there isn't a script causing the issue that I can see. Rather than worrying about the emails that are going to :blackhole: and not in the mail queue, it would be more helpful to see an email (header and exigrep details) that is one of those 195k in your mail queue. WHM > Mail Queue Manager area has the emails where you can view one of them.
     
  12. postcd

    postcd Well-Known Member

    Joined:
    Oct 22, 2010
    Messages:
    621
    Likes Received:
    6
    Trophy Points:
    18
    Thanks,
    It showed me around 4K emails on that queue, total its 19k.

    Its probably sorted by date message is in queue. all these are in queue for 45-46days probably? There is 45d for example
    All these originating from one account on my server. And are certinly spam which i did not sent.

    Is there any command which will delete all emails from queue that match one particular cpanel account or are older than X number days?
    What i can do?

    When i did exigrep, it returned nothing on this email.
     
    #12 postcd, Aug 19, 2011
    Last edited: Aug 19, 2011
  13. cPanelTristan

    cPanelTristan Quality Assurance Analyst
    Staff Member

    Joined:
    Oct 2, 2010
    Messages:
    7,623
    Likes Received:
    21
    Trophy Points:
    38
    Location:
    somewhere over the rainbow
    cPanel Access Level:
    Root Administrator
    To remove all messages older than 5 days from the mail queue:

    Code:
    exiqgrep -o 432000 -i | xargs exim -Mrm
    Here 86400 * 5 = 432000 seconds, so this is the number of seconds in 5 days. If you want to delete everything older than a day, use 86400 or 86400 x # for whatever number of days old.

    For all emails sent to a certain domain, you'd run:

    Code:
    exiqgrep -ir domain.com | xargs exim -Mrm
    For all emails sent from a certain domain, you'd run:

    Code:
    exiqgrep -if domain.com | xargs exim -Mrm
     
  14. postcd

    postcd Well-Known Member

    Joined:
    Oct 22, 2010
    Messages:
    621
    Likes Received:
    6
    Trophy Points:
    18
    Thanks,
    i used the command to delete all queued more than 5 days.
    Now queue has around 43 messages and SWAP used is 100%..

    There are entries added into exim_mainlog where there is text like "Warning: Sender rate 12.3 / 1h" etc
     
  15. cPanelTristan

    cPanelTristan Quality Assurance Analyst
    Staff Member

    Joined:
    Oct 2, 2010
    Messages:
    7,623
    Likes Received:
    21
    Trophy Points:
    38
    Location:
    somewhere over the rainbow
    cPanel Access Level:
    Root Administrator
    Yes, the sender rates are set due to WHM > Exim Configuration Editor > Ratelimit suspicious SMTP servers being set to "On"
     
  16. tdens

    tdens Member

    Joined:
    Aug 18, 2011
    Messages:
    19
    Likes Received:
    0
    Trophy Points:
    1
    That appears to be locally generated mail outbound, probably generated by a cgi or php script. Running "top" from a command line and then hitting either > or M will sort procs by memory usage. Also run 'iostat' a few times to get an idea of what your i/o load looks like.
     
  17. postcd

    postcd Well-Known Member

    Joined:
    Oct 22, 2010
    Messages:
    621
    Likes Received:
    6
    Trophy Points:
    18
    Thank You, These are quite top processes i got. Spamd and Clamd being on the top....

    TOP:
    Iostat
    Can you see anythink alarming in that? What can i do to discover what is using 100% SWAP?
     
  18. tdens

    tdens Member

    Joined:
    Aug 18, 2011
    Messages:
    19
    Likes Received:
    0
    Trophy Points:
    1
    No, that looks normal - at least that snapshot in time does. Mysql is using a little memory, but nothing else around it is using much to speak of. Of your used mem, most of it is cached. Your io wait is 11.8%. The swap usage probably came from trying to run a queue with 19 thousand files in it (if I understand what you were saying). You can reset those swap values two ways.

    1 - reboot your server during a maintenance window
    2 - from the cli, as root, run 'swapoff -a ; swapon -a'

    #2 turns swap off momentarily and then turns it back on. Boilerplate: with 8 gigs of ram, during low usage periods this shouldn't be an issue, but do this at your own risk, yadda, yadda. I've personally done this on several machines over the years when troubleshooting, but I can't swear it won't result in a crash of some sort.

    You're going to want to find out what is/was generating those queued email messages, since it's unlikely to stop, and it's possible that your box is being used to spam people. This isn't nice, and it may lead to your IP(s) being blacklisted. Two places to look at mail server reputation - plug your IP(s) in to these websites:

    Cisco IronPort SenderBase Security Network
    Multi-RBL Check | The Anti-Abuse Project

    Also, just a heads up, but you might consider using fail rather than blackhole unless you have a really good reason to blackhole mail. Blackhole still accepts the mail and sends it to /dev/null. This means that the message still uses your bandwidth, still uses your ram, still needs to be processed, and still uses cpu cycles. You may want to research the differences, and if you have a large number of already active domains, you may want to look at how to change the settings for the existing domain files in /etc/valiases as well.
     
    #18 tdens, Aug 20, 2011
    Last edited: Aug 20, 2011
  19. tdens

    tdens Member

    Joined:
    Aug 18, 2011
    Messages:
    19
    Likes Received:
    0
    Trophy Points:
    1
    I posted a reply, but it looks like since it contains URLs, it needs moderator approval. Long story short, looks ok.
     
  20. postcd

    postcd Well-Known Member

    Joined:
    Oct 22, 2010
    Messages:
    621
    Likes Received:
    6
    Trophy Points:
    18
    Thank you for the message, it was usefull. When i restarted mysql 12 hours earlier, SWAP was freed to 6% usage.
    Mail queue is perfect, no mail so far.

    So i learnt that i need to look into mail queue into emails header and discover what is the originating account of the mail. Thats important.

    I checked those blacklist servers and im blocked on b.barracudacentral.org
    And at dyna.spamrats.com im also blocked, it says: "Does IP Address comply reverse hostname naming convention... Failed!"
     
Loading...

Share This Page