Server overloaded - SPAM? Exim processes

[tdens]

That appears to be locally generated mail outbound, probably generated by a cgi or php script. Running "top" from a command line and then hitting either > or M will sort procs by memory usage. Also run 'iostat' a few times to get an idea of what your i/o load looks like.

[postcd]

Thank You, These are quite top processes i got. Spamd and Clamd being on the top....

TOP:

Mem: 8245744k total, 7792904k used, 452840k free, 159528k buffers
Swap: 1051064k total, 1038716k used, 12348k free, 5364204k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
19285 mysql 20 0 2894m 1.7g 4104 S 2.7 21.4 1903:10 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --log-error=/var/lib/mysql/server.camer
28508 root 20 0 156m 97m 1188 S 0.0 1.2 1:08.15 /usr/sbin/clamd
28605 root 20 0 40968 36m 1800 S 0.0 0.5 0:06.98 spamd child
28597 root 20 0 33388 29m 1884 S 0.0 0.4 0:05.20 /usr/bin/spamd -d --allowed-ips=127.0.0.1 --pidfile=/var/run/spamd.pid --max-children=5
28606 root 20 0 33388 28m 1052 S 0.0 0.4 0:00.02 spamd child

Iostat

avg-cpu: %user %nice %system %iowait %steal %idle
9.82 0.53 1.53 11.18 0.00 76.93

Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 128.29 696.50 98.58 2875667954 407029216
sda1 35.45 112.00 21.62 462413736 89272672
sda2 92.49 576.30 71.96 2379398282 297094040
sda3 0.35 8.20 5.00 33855232 20662504
sdb 121.97 405.73 96.90 1675144858 400085032
sdb1 34.96 105.26 21.62 434573730 89272672
sdb2 86.86 296.32 71.96 1223414376 297094040
sdb3 0.14 4.16 3.32 17156048 13718320
md2 275.43 175.27 60.33 723651674 249098216
md1 145.88 217.24 11.36 896932226 46913304

Can you see anythink alarming in that? What can i do to discover what is using 100% SWAP?

[tdens]

No, that looks normal - at least that snapshot in time does. Mysql is using a little memory, but nothing else around it is using much to speak of. Of your used mem, most of it is cached. Your io wait is 11.8%. The swap usage probably came from trying to run a queue with 19 thousand files in it (if I understand what you were saying). You can reset those swap values two ways.

1 - reboot your server during a maintenance window
2 - from the cli, as root, run 'swapoff -a ; swapon -a'

#2 turns swap off momentarily and then turns it back on. Boilerplate: with 8 gigs of ram, during low usage periods this shouldn't be an issue, but do this at your own risk, yadda, yadda. I've personally done this on several machines over the years when troubleshooting, but I can't swear it won't result in a crash of some sort.

You're going to want to find out what is/was generating those queued email messages, since it's unlikely to stop, and it's possible that your box is being used to spam people. This isn't nice, and it may lead to your IP(s) being blacklisted. Two places to look at mail server reputation - plug your IP(s) in to these websites:

Cisco IronPort SenderBase Security Network
Multi-RBL Check | The Anti-Abuse Project

Also, just a heads up, but you might consider using fail rather than blackhole unless you have a really good reason to blackhole mail. Blackhole still accepts the mail and sends it to /dev/null. This means that the message still uses your bandwidth, still uses your ram, still needs to be processed, and still uses cpu cycles. You may want to research the differences, and if you have a large number of already active domains, you may want to look at how to change the settings for the existing domain files in /etc/valiases as well.

[tdens]

I posted a reply, but it looks like since it contains URLs, it needs moderator approval. Long story short, looks ok.

[postcd]

I posted a reply, but it looks like since it contains URLs, it needs moderator approval. Long story short, looks ok.

Thank you for the message, it was usefull. When i restarted mysql 12 hours earlier, SWAP was freed to 6% usage.
Mail queue is perfect, no mail so far.

So i learnt that i need to look into mail queue into emails header and discover what is the originating account of the mail. Thats important.

I checked those blacklist servers and im blocked on b.barracudacentral.org
And at dyna.spamrats.com im also blocked, it says: "Does IP Address comply reverse hostname naming convention... Failed!"

[k-planethost]

that means that reverse dns entries are missing

[postcd]

Thanks, only knowing how to exactly fix it in cpanel.

[tdens]

If you're authoritative for your IP space:

Configuring Reverse DNS in WHM

if someone else is, you'll need to ask them to setup an appropriate reverse entry for you.