Services Failing, SPAM Increase, SMTP Failures... And More :(

[coalescefl]

We're in a bit over our heads, the past couple days our E-mail has been going crazy. The only changes were the install of MailScanner (Which I don't think did this) and some security tweaking in WHM following the guide on the Cpanel Forum.

Here is our diagnosis:
- Getting Failed Services, Spamd and Exim, Imap
- SPAM has increased
- Certain Accounts aren't receiving messages
- Sometimes Outbox Failures, SMTP Failures.

I don't really know where to start. We went throught the exim logs, but aren't sure exactly what we're looking for.

Any suggestions on where to start?

Thanks!!!

 

[coalescefl]

Maybe even a server management company to call ???

 

[ckh]

http://configserver.com/

Chirpy comes highly recommended.

 

[coalescefl]

Thank you, however, I already contacted them and I was told they can only handle package issues at the moment, not a custom problem like we have...

Any other recommendations or ideas???

Thank you

 

[coalescefl]

I have a further diagnosis. It seems exim has stopped failing and has stabilized itself. However, clients are having trouble with SMTP, it seems like you have to try to send a message about 5 times, and sometimes it just never goes out.

Once I restarted I got this message:

exim status
mailnull  4212  0.0  0.1  6640 1312 ?        S    Oct17   0:42 eximstats
mailnull 11284  0.2  0.1  6640 1824 ?        S    11:28   0:00 /usr/sbin/exim -bd -q60m
mailnull 11290  0.0  0.1  6620 1800 ?        S    11:28   0:00 /usr/sbin/exim -tls-on-connect -bd -oX 465
mailnull 11307  1.2  0.3  7468 3556 ?        S    11:28   0:00 /usr/sbin/exim -bd -q60m
mailnull 11324  0.7  0.2  6772 2540 ?        S    11:28   0:00 /usr/sbin/exim -bd -q60m
mailnull 11330  1.2  0.3  7468 3556 ?        S    11:28   0:00 /usr/sbin/exim -bd -q60m
mailnull 11339  2.7  0.4  7568 4224 ?        S    11:28   0:00 /usr/sbin/exim -bd -q60m
mailnull 11343  0.7  0.3  7460 3576 ?        S    11:28   0:00 /usr/sbin/exim -bd -q60m
mailnull 11346  1.6  0.4  7548 4136 ?        S    11:28   0:00 /usr/sbin/exim -bd -q60m
mailnull 11352  1.6  0.3  7468 3556 ?        S    11:28   0:00 /usr/sbin/exim -bd -q60m
mailnull 11359  1.6  0.3  7468 3560 ?        S    11:28   0:00 /usr/sbin/exim -bd -q60m
mailnull 11375  0.0  0.1  6656 1916 ?        S    11:28   0:00 /usr/sbin/exim -bd -q60m
mailnull 11382  0.3  0.1  6656 1932 ?        S    11:28   0:00 /usr/sbin/exim -bd -q60m
mailnull 11388  1.3  0.4  7552 4140 ?        S    11:28   0:00 /usr/sbin/exim -bd -q60m
mailnull 11417  0.3  0.2  6752 2396 ?        S    11:28   0:00 /usr/sbin/exim -bd -q60m
mailnull 11424  0.0  0.1  6656 1916 ?        R    11:28   0:00 /usr/sbin/exim -bd -q60m
mailnull 11425  0.0  0.1  6656 1932 ?        S    11:28   0:00 /usr/sbin/exim -bd -q60m
mailnull 11432  1.3  0.3  7448 3472 ?        R    11:28   0:00 /usr/sbin/exim -bd -q60m
mailnull 11436  1.5  0.3  7468 3556 ?        S    11:28   0:00 /usr/sbin/exim -bd -q60m
mailnull 11437  0.0  0.1  6648 1844 ?        S    11:28   0:00 /usr/sbin/exim -bd -q60m
mailnull 11438  0.0  0.1  6648 1844 ?        S    11:28   0:00 /usr/sbin/exim -bd -q60m
mailnull 11439  0.0  0.1  6648 1844 ?        S    11:28   0:00 /usr/sbin/exim -bd -q60m
mailnull 11443  0.0  0.1  6648 1844 ?        S    11:28   0:00 /usr/sbin/exim -bd -q60m
mailnull 11444  0.0  0.1  6648 1844 ?        S    11:28   0:00 /usr/sbin/exim -bd -q60m
mailnull 11445  0.0  0.1  6648 1844 ?        S    11:28   0:00 /usr/sbin/exim -bd -q60m
mailnull 11446  0.0  0.1  6648 1844 ?        S    11:28   0:00 /usr/sbin/exim -bd -q60m
mailnull 11447  0.0  0.1  6648 1844 ?        S    11:28   0:00 /usr/sbin/exim -bd -q60m
mailnull 11448  0.0  0.1  6648 1844 ?        S    11:28   0:00 /usr/sbin/exim -bd -q60m
mailnull 11449  0.0  0.1  6648 1844 ?        S    11:28   0:00 /usr/sbin/exim -bd -q60m
mailnull 11451  0.0  0.1  6648 1844 ?        S    11:28   0:00 /usr/sbin/exim -bd -q60m
mailnull 11453  0.0  0.1  6656 1924 ?        R    11:28   0:00 /usr/sbin/exim -bd -q60m
mailnull 11454  0.0  0.1  6648 1844 ?        S    11:28   0:00 /usr/sbin/exim -bd -q60m
mailnull 11456  4.0  0.3  7452 3480 ?        R    11:28   0:00 /usr/sbin/exim -bd -q60m
mailnull 11457  0.0  0.1  6656 1916 ?        S    11:28   0:00 /usr/sbin/exim -bd -q60m
mailnull 11458  0.0  0.1  6648 1844 ?        S    11:28   0:00 /usr/sbin/exim -bd -q60m
mailnull 11459  0.0  0.1  6648 1844 ?        S    11:28   0:00 /usr/sbin/exim -bd -q60m
mailnull 11460  0.0  0.1  6648 1844 ?        S    11:28   0:00 /usr/sbin/exim -bd -q60m
mailnull 11461  0.0  0.1  6648 1844 ?        S    11:28   0:00 /usr/sbin/exim -bd -q60m
mailnull 11463  0.0  0.1  6648 1844 ?        S    11:28   0:00 /usr/sbin/exim -bd -q60m
root     11398  1.3  0.1  2772 1436 ?        S    11:28   0:00 antirelayd

Any idea what all this means? It seems like a lot of instances of exim might have been running? Once I restarted it works fine, until eventually becoming clogged?

Im stumped...

 

[coalescefl]

Another interesting thing when tracing the exim process:

write(6, "2006-10-19 11:36:01 Connection f"..., 83) = 83
write(4, "421 Too many concurrent SMTP con"..., 67) = 67
close(4)

 

[coalescefl]

OK, another log. This is after grepping a user who caused a SPAM issue in the past:

2006-10-15 09:31:38 1GZ71b-0001XV-8w <= [email protected] U=myclient P=local-bsmtp S=4833 [email protected]g
2006-10-15 09:31:38 1GZ71b-0001XV-8w => /dev/null <[email protected]> R=central_user_filter T*bypassed**
2006-10-15 09:31:38 1GZ71a-0001XN-Ja => myclient <[email protected]> R=sa_localuser T=local_sa_delivery
2006-10-15 09:31:38 1GZ71c-0001Xe-5s <= [email protected] U=myclient P=local-bsmtp S=2315 [email protected]
2006-10-15 09:31:38 1GZ71c-0001Xe-5s => myclient <[email protected]> R=localuser T=local_delivery
2006-10-15 09:31:38 1GZ71a-0001XC-Ge => myclient <[email protected]> R=sa_localuser T=local_sa_delivery
2006-10-15 09:31:48 H=(friend) [69.143.76.19] F=<[email protected]> rejected RCPT <[email protected]>: Sender verify failed
2006-10-15 09:32:15 1GZ72D-0001Y7-H3 <= <> U=myclient P=local-bsmtp S=81507

Not sure if that leads anywhere either... The first one on there shows a fake email address coming from my clients main account...

 

[rpmws]

across all 11 of my boxes I have noticed a HUGE increase in overall emails (spams) in the last 2-3 days especially and spamd in my case is going nuts keeping up. My MRTG on all servers show huge jump ..almost 2 times what it was last week. If I look at my weekly mrtg graphs on emails in general i see a bump on the last few days. I have a couple of shared boxes with loads about 5 ..jumping to 8-10. If I kill exim or spamd the loads drop to .9 .7 . I think the spammers are killing us from the outside.

 

[mickalo]

Thank you, however, I already contacted them and I was told they can only handle package issues at the moment, not a custom problem like we have...

Any other recommendations or ideas???

Thank you

Tom at Linux Tech Networks is very good, we've used them serveral times in the past for various issues on our server that we couldn't handle. Very quick to response and take care of most any type of server problems.

I would recommend them as another choice to look into.

Mickalo

 

[coalescefl]

Thanks for the recommendation.

rpmws, what you're talking about sounds quite scary...

 

[coalescefl]

We're still trying to resolve the issue.

Current diagnosis is Spamd and Exim fail about every 5-10 minutes... ::sigh::

 

[coalescefl]

What a week...

Tom from Linux Tech seems to have us back on track. He raised the amount of outgoing connections.

Apparently no SPAM has been coming from our server. As for how to fix these symptoms, you'll have to call Tom