how to check server is spamming ?

Dear Beanth.
There are millions of ways to do that. If your IP is blacklisted - most likely your server is sending a lot of spam. And if mail queue is very large -also..

Anyway, here are points to check:
1. check your mailqueue, especially bounces to nobody and emails from nobody.
2. make sure, you have installed "MailHeaders" extension for PHP and that nobody servermail is being redirected to some existing account and that you actually check these bounces to identify source of spam. This PHP extension will add path to the file, which has generated that mail.
3. Make sure you have some limit of how many emails your users are allowed to send per hour.
4. Install and configure ConfigServer Firewall. This group of scripts will periodically notify, if some script is sending too much emails.

If you are unsure about all this, contact me and I can look and fix these things for you.

Anton.