Server overloaded - SPAM? Exim processes

[postcd]

Hello i need help,

i can see exim processes like this on my server:
27869 mailnull 25 0 7700 852 532 R 4 0.0 0:17.53 /usr/sbin/exim -bd -q60m

There are load spikes and swap is at 50-100%

Mail Queue is at maximum 4000+ messages and tail -f /var/log/exim_mainlog shows some like spammy esmtp threads.


2011-08-10 00:04:12 1QquPD-00061H-TD <= ***@****.org H=ns35.****.com [254.254.254.254] P=esmtps X=TLSv1:AES256-SHA:256 S=3117 id=E1QquUw-0002Z5-DQ@ns35.****.com

Im not able to discover what is the cause and what i need to ban, how to protect server, so im asking you there. There are more details from cPanel mail stats:

Time spent on the queue: all messagesTime Messages Percentage Cumulative Percentage
Under 1m 58312 44.1% 44.1%
5m 47 0.0% 44.1%
3h 4 0.0% 44.1%
6h 1 0.0% 44.1%
12h 6 0.0% 44.1%
1d 8 0.0% 44.2%
Over 1d 73844 55.8% 100.0%

Top 50 mail rejection reasons by message countMessages Mail rejection reason
8264 Unknown
2327 Rejected RCPT: Sender verify failed
492 Rejected MAIL: Access denied - Invalid HELO name (See RFC2821 4.1.1.1)

Top 50 mail temporary rejection reasons by message countMessages Mail temporary rejection reason
6499 Temporarily rejected RCPT: Could not complete sender verify

Top 50 rejected ips by message countMessages Rejected ip
7493 local
1630 [*.*.*.*]
68 [*.*.*.*]
61 [*.*.*.*]

PLEASE, can anyone help me what exactly to do to discover source of this issue and eliminate it? It must be also helpfull for more members.

Thank you,
P.

 

[postcd]

Please can You help me and all who have same issue to recognize what is the cause?

 

[syslint]

You may better enable spam cops RBL checking from whm -> Exim configuration editor

 

[postcd]

You may better enable spam cops RBL checking from whm -> Exim configuration editor

-
Thank you, i enabled this today, and added my server IP to the whitelist on same config page. But even before i enabled this, LOAD was quite low. I restarted httpd, exim etc, at first look nothing changed.. So LOAD is no longer problem maybe, but there are still those SMTP senders mentioned in first post..! Please how can i eliminate them and stop using blacklist external service?

Swap Used 96.29% (1,012,080 of 1,051,064)

Thank you

 

[postcd]

lfd: The exim delivery queue size is 19940

I need to note that i have high number of incoming emails, like 100 per minute and most coming into non existing email and are relayed to existing email address.

 

[cPanelTristan]

What does "come into non-existing email and are relayed to existing email mean" exactly? Are they forwarded by accounts on the machine that exist? What is the full route for one of the messages?

exigrep messageID /var/log/exim_mainlog

Where messageID is the exim ID given to the message. For example, 1QuRrF-0007sJ-WF would be a message ID for a message on my machine.

 

[tdens]

Swapping like that is bad, obviously.

Sounds like you may have catchall addresses enabled. They're on by default, so if you didn't disable them, anything sent to any username at a real domain will be sent to the default user for that domain. Look under server configuration - tweak settings - default catch-all. If that's not the case, please describe the problem better so others can help (I'm a complete WHM newb, sorry).

 

[postcd]

What does "come into non-existing email and are relayed to existing email mean" exactly? Are they forwarded by accounts on the machine that exist? What is the full route for one of the messages?

--
Hello, this is full route:

2011-08-19 20:**:40 1QuTp5-0000LF-Ub <= [email protected] H=home.soka.ac.jp [150.37.251.**] P=esmtps X=TLSv1:AES256-SHA:256 S=1342 id=20110***1829.p7JIT*****[email protected] T="[CETL STAFF BBS-No.54109]Satellite Direct" from <[email protected]> for *****@mydomain.info
2011-08-19 20:**:40 1QuTp5-0000LF-Ub => :blackhole: <*****@mydomain.info> R=virtual_aliases
2011-08-19 20:**:40 1QuTp5-0000LF-Ub Completed

What i meant is that i have Catch all mail. But in fact i realised that i deleted all mailboxes on that domain and set :blackhole:

Even when i suspend all suspicious cPanel accounts, those email entries coming into exim_mainlog.

Server load is quite ok, except 95-100% full swap and overloaded mailqueue 19k mails

 

[cPanelTristan]

This entry doesn't show it was relayed. It shows that it went to :blackhole: to be deleted.

 

[postcd]

So it was not relayed, what i can do about it? My simple question is how to discover what script causing the issue? Or what is the cause, how i can discover it exactly. If anyone can help me, i would be gratefull.

 

[cPanelTristan]

If these are incoming emails, there isn't a script causing the issue that I can see. Rather than worrying about the emails that are going to :blackhole: and not in the mail queue, it would be more helpful to see an email (header and exigrep details) that is one of those 195k in your mail queue. WHM > Mail Queue Manager area has the emails where you can view one of them.

 

[postcd]

Thanks,
It showed me around 4K emails on that queue, total its 19k.

Its probably sorted by date message is in queue. all these are in queue for 45-46days probably? There is 45d for example
All these originating from one account on my server. And are certinly spam which i did not sent.

Return-path: <[email protected]>
Received: from myaccname by server.mysite.info with local (Exim 4.69)
(envelope-from <[email protected]>)
id 1QdwZ1-0002Hz-Qo
for ***[email protected]; Tue, 05 Jul 2011 05:44:43 +0200
To: ***[email protected]
Subject: INFORMATION AND NEED URGENT REPLY
From: Taif Bin *** <taif_***@rediffmail.com>
Reply-To: taifbin***@yahoo.com

Is there any command which will delete all emails from queue that match one particular cpanel account or are older than X number days?
What i can do?

When i did exigrep, it returned nothing on this email.

 

[cPanelTristan]

To remove all messages older than 5 days from the mail queue:

exiqgrep -o 432000 -i | xargs exim -Mrm

Here 86400 * 5 = 432000 seconds, so this is the number of seconds in 5 days. If you want to delete everything older than a day, use 86400 or 86400 x # for whatever number of days old.

For all emails sent to a certain domain, you'd run:

exiqgrep -ir domain.com | xargs exim -Mrm

For all emails sent from a certain domain, you'd run:

exiqgrep -if domain.com | xargs exim -Mrm

 

[postcd]

Thanks,
i used the command to delete all queued more than 5 days.
Now queue has around 43 messages and SWAP used is 100%..

There are entries added into exim_mainlog where there is text like "Warning: Sender rate 12.3 / 1h" etc

 

[cPanelTristan]

There are entries added into exim_mainlog where there is text like "Warning: Sender rate 12.3 / 1h" etc

Yes, the sender rates are set due to WHM > Exim Configuration Editor > Ratelimit suspicious SMTP servers being set to "On"

 

[tdens]

That appears to be locally generated mail outbound, probably generated by a cgi or php script. Running "top" from a command line and then hitting either > or M will sort procs by memory usage. Also run 'iostat' a few times to get an idea of what your i/o load looks like.

 

[postcd]

Thank You, These are quite top processes i got. Spamd and Clamd being on the top....

TOP:

Mem: 8245744k total, 7792904k used, 452840k free, 159528k buffers
Swap: 1051064k total, 1038716k used, 12348k free, 5364204k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
19285 mysql 20 0 2894m 1.7g 4104 S 2.7 21.4 1903:10 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --log-error=/var/lib/mysql/server.camer
28508 root 20 0 156m 97m 1188 S 0.0 1.2 1:08.15 /usr/sbin/clamd
28605 root 20 0 40968 36m 1800 S 0.0 0.5 0:06.98 spamd child
28597 root 20 0 33388 29m 1884 S 0.0 0.4 0:05.20 /usr/bin/spamd -d --allowed-ips=127.0.0.1 --pidfile=/var/run/spamd.pid --max-children=5
28606 root 20 0 33388 28m 1052 S 0.0 0.4 0:00.02 spamd child

Iostat

avg-cpu: %user %nice %system %iowait %steal %idle
9.82 0.53 1.53 11.18 0.00 76.93

Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 128.29 696.50 98.58 2875667954 407029216
sda1 35.45 112.00 21.62 462413736 89272672
sda2 92.49 576.30 71.96 2379398282 297094040
sda3 0.35 8.20 5.00 33855232 20662504
sdb 121.97 405.73 96.90 1675144858 400085032
sdb1 34.96 105.26 21.62 434573730 89272672
sdb2 86.86 296.32 71.96 1223414376 297094040
sdb3 0.14 4.16 3.32 17156048 13718320
md2 275.43 175.27 60.33 723651674 249098216
md1 145.88 217.24 11.36 896932226 46913304

Can you see anythink alarming in that? What can i do to discover what is using 100% SWAP?

 

[tdens]

No, that looks normal - at least that snapshot in time does. Mysql is using a little memory, but nothing else around it is using much to speak of. Of your used mem, most of it is cached. Your io wait is 11.8%. The swap usage probably came from trying to run a queue with 19 thousand files in it (if I understand what you were saying). You can reset those swap values two ways.

1 - reboot your server during a maintenance window
2 - from the cli, as root, run 'swapoff -a ; swapon -a'

#2 turns swap off momentarily and then turns it back on. Boilerplate: with 8 gigs of ram, during low usage periods this shouldn't be an issue, but do this at your own risk, yadda, yadda. I've personally done this on several machines over the years when troubleshooting, but I can't swear it won't result in a crash of some sort.

You're going to want to find out what is/was generating those queued email messages, since it's unlikely to stop, and it's possible that your box is being used to spam people. This isn't nice, and it may lead to your IP(s) being blacklisted. Two places to look at mail server reputation - plug your IP(s) in to these websites:

Cisco IronPort SenderBase Security Network
Multi-RBL Check | The Anti-Abuse Project

Also, just a heads up, but you might consider using fail rather than blackhole unless you have a really good reason to blackhole mail. Blackhole still accepts the mail and sends it to /dev/null. This means that the message still uses your bandwidth, still uses your ram, still needs to be processed, and still uses cpu cycles. You may want to research the differences, and if you have a large number of already active domains, you may want to look at how to change the settings for the existing domain files in /etc/valiases as well.

 

[tdens]

I posted a reply, but it looks like since it contains URLs, it needs moderator approval. Long story short, looks ok.

 

[postcd]

I posted a reply, but it looks like since it contains URLs, it needs moderator approval. Long story short, looks ok.

Thank you for the message, it was usefull. When i restarted mysql 12 hours earlier, SWAP was freed to 6% usage.
Mail queue is perfect, no mail so far.

So i learnt that i need to look into mail queue into emails header and discover what is the originating account of the mail. Thats important.

I checked those blacklist servers and im blocked on b.barracudacentral.org
And at dyna.spamrats.com im also blocked, it says: "Does IP Address comply reverse hostname naming convention... Failed!"