All emails still stuck in queue after a spam detected and cleaned

[psytanium]

Hello,

3 days ago, 1 of the emails was sending thousands of spams, i suspended the email address, fixed the issue from the client device (He wasn't using the SSL) then changed his password and all is fine.

Lots of emails were stuck on the queue, most of it spams, i cleaned the spams and delivered the others.

But the problem, is now all outgoing emails still stuck on queue, I have to deliver it manually.

Thanks

 

[ZenHostingTravis]

The below notes from cPanel's documentation may assist you further.

To release or delete outgoing mail held in the queue, perform the following actions in cPanel's Email Accounts interface (cPanel >> Home >> Email >> Email Accounts):

1. Click Manage Suspension.
2. Select Allow for the Send option.
3. If applicable, click Delete messages from the mail queue to remove any queued messages.
4. Click Save.

 

[psytanium]

The problem is not specific to 1 user, all 70 accounts are affected, I have emails in queue from all domain names and users.

 

[andrew.n]

Have you checked if your server IP became blacklisted? A good place to check this is at Email Blacklist Check - IP Blacklist Check - See if your server is blacklisted

 

[cPanelLauren]

You can also check the reason the email is queued but obtaining the message ID of a queued message and checking the exim logs. The following is what I use frequently:

exigrep $msgid /var/log/exim_mainlog

 

[psytanium]

The problem resolved on its own, after 3 days of manual deliver. I don't know how.

 

[andrew.n]

At least we glad to hear its resolved now

 

[psytanium]

Hello,

Again, outgoing emails from various users end up in the Mail Queue, and I have to manually select and deliver, since yesterday.

I tried to check the exim log for one specific email, i got this :

[root@server ~]# exigrep OF88395AED.D3082EEF-ONC12587D4.0057D133-C12587D4.00582AD8@samospa.net /var/log/exim_mainlog
+++ 1nC1ok-00025r-Jj has not completed +++
2022-01-24 18:03:42 1nC1ok-00025r-Jj <= [email protected] H=mx2.samo.it (mx2.samoindustries.com) [77.241.5.94]:55276 P=esmtps X=TLS1.2:ECDHE-RSA-AES256-GCM-SHA384:256 CV=no S=566517 id=OF88395AED.D3082EEF-ONC12587D4.0057D133-C12587D4.00582AD8@samospa.net T="Rif: RE: RE: Rif: Sara Riyadh - AV258C mirror inquiry" for [email protected]
2022-01-24 18:03:42 1nC1ok-00025r-Jj no immediate delivery: load average 74.12

Also I noticed my server is slow when loading pages. I don't know what's happening.

 

[quietFinn]

no immediate delivery: load average 74.12

Exim is not delivering messages because server load is too high.
Command:
grep "deliver_queue_load_max" /etc/exim.conf
shows the maximum.

 

[psytanium]

Exim is not delivering messages because server load is too high.
Command:
grep "deliver_queue_load_max" /etc/exim.conf
shows the maximum.

Returned : deliver_queue_load_max = 12
in red

 

[quietFinn]

That means that Exim will continue delivering when load goes under 12.
You can change that setting in WHM -> Service Configuration -> Exim Configuration Manager -> Advanced editor

 

[cPRex]

Your server's load is too high for mail to be delivered at that time. You'll want to investigate the load on the machine to determine the source, and once the load is less the mail will function normally.

 

[psytanium]

Today the server laod is back to normal on its own.. any recommended documentation how to find and fix the reason of high load in a VPS server ? Thanks

 

[cPRex]

It's nearly impossible to find the cause of high load after it has happened - it's really best to try and troubleshoot that in real-time. I like to check the following things:

top -c
mysqladmin proc status
apachectl status
OR
apachectl fullstatus

Those commands will give you an idea of where the server usage is, if MySQL is using too many resources or taking too long to handle queries, and if Apache is overloaded. One of those things usually gets me down the right path.

 

[psytanium]

It's nearly impossible to find the cause of high load after it has happened - it's really best to try and troubleshoot that in real-time. I like to check the following things:

top -c
mysqladmin proc status
apachectl status
OR
apachectl fullstatus

Those commands will give you an idea of where the server usage is, if MySQL is using too many resources or taking too long to handle queries, and if Apache is overloaded. One of those things usually gets me down the right path.

mysqladmin proc status returned this output

+---------+------------------+-----------+------------------+---------+------+--------------+------------------------------------------------------------------------------------------------------+
| Id      | User             | Host      | db               | Command | Time | State        | Info             |
+---------+------------------+-----------+------------------+---------+------+--------------+------------------------------------------------------------------------------------------------------+
| 2231939 | houseoli_user | localhost | houseoli_user | Sleep   | 0    |              |             |
| 2231940 | macrotro_suser    | localhost | macrotro_user    | Query   | 0    | Sending data | select count(p.products_id) as count  from (products p)
                            join (products_t |
| 2231941 | elements_user | localhost | elements_user | Sleep   | 0    |              |             |
| 2231943 | root             | localhost |                  | Query   | 0    | starting     | show processlist             |
+---------+------------------+-----------+------------------+---------+------+--------------+------------------------------------------------------------------------------------------------------+
Uptime: 137032  Threads: 4  Questions: 106930006  Slow queries: 0  Opens: 5423017  Flush tables: 1  Open tables: 2000  Queries per second avg: 780.328

apachectl status returned this output

Apache Server Status for localhost (via ::1)

   Server Version: Apache/2.4.52 (cPanel) OpenSSL/1.1.1m mod_bwlimited/1.4

   Server MPM: prefork

   Server Built: Jan 12 2022 01:58:29

   --------------------------------------------------------------------------

   Current Time: Wednesday, 26-Jan-2022 08:24:30 EET

   Restart Time: Saturday, 22-Jan-2022 20:34:55 EET

   Parent Server Config. Generation: 3

   Parent Server MPM Generation: 2

   Server uptime: 3 days 11 hours 49 minutes 34 seconds

   Server load: 1.90 1.69 1.78

   Total accesses: 5579109 - Total Traffic: 92.7 GB - Total Duration:
   19335594938

   CPU Usage: u9.1 s219.25 cu612714 cs157174 - 255% CPU load

   18.5 requests/sec - 322.2 kB/second - 17.4 kB/request - 3465.71 ms/request

   29 requests currently being processed, 4 idle workers

R_RCWKWW_C..KR..CC.........K.._...R.R..RK......RRRR..R.R.R_.....
.K...K..R.CR....................................................
......................

   Scoreboard Key:
   "_" Waiting for Connection, "S" Starting up, "R" Reading Request,
   "W" Sending Reply, "K" Keepalive (read), "D" DNS Lookup,
   "C" Closing connection, "L" Logging, "G" Gracefully finishing,
   "I" Idle cleanup of worker, "." Open slot with no current process

Anything suspicious in those reports ?
I will try your commands if the problem occur again. Thank you fir your support.

 

[cPRex]

Both of those results look normal to me. If you can get more details during the time an issue happens that would likely be more helpful as that would show the state of the server while the commands gather their data.

 

[psytanium]

Same problem happened again, today all day, my server is very slow, and emails stuck in queue.

I received a notification email from the server.

Subjext : The process “pkgacct - macrotro - av: 4 - write compressed stream” stalled while it ran for user “root”.

The process has run for “12 hours, 1 minute, and 35.96 seconds”.
At the time that the system generated this notification, the process had a PID of “2176”, and it consumed 0.46% of the system’s memory and 0.01% of the system’s CPU time.

Running this command " mysqladmin proc status " returned this output:

+---------+----------------------+-----------+------------------+---------+------+-----------------------+------------------------------------------------------------------------------------------------------+
| Id      | User                 | Host      | db               | Command | Time | State                 | Info                          |
+---------+----------------------+-----------+------------------+---------+------+-----------------------+------------------------------------------------------------------------------------------------------+
| 3774003 | user_account         | localhost | user_account     | Sleep   | 0    |                       |                          |
| 3774026 | user_account         | localhost | user_account     | Sleep   | 0    |                       |                          |
| 3774037 | user_account         | localhost | user_account     | Sleep   | 0    |                       |                          |
| 3774060 | user_account         | localhost | user_account     | Sleep   | 1    |                       |                          |
| 3774065 | user_account         | localhost | user_account     | Sleep   | 0    |                       |                          |
| 3774067 | user_account         | localhost | user_account     | Sleep   | 1    |                       |                          |
| 3774068 | user_account         | localhost | user_account     | Sleep   | 0    |                       |                          |
| 3774080 | user_account         | localhost | user_account     | Sleep   | 1    |                       |                          |
| 3774083 | user_account         | localhost | user_account     | Sleep   | 0    |                       |                          |
| 3774084 | user_account         | localhost | user_account     | Sleep   | 1    |                       |                          |
| 3774089 | macrotro_shop        | localhost | user_account     | Query   | 0    | Sending data          | select count(p.products_id) as count  from (products p)
                            join (products_t |
| 3774092 | user_account         | localhost | user_account     | Sleep   | 1    |                       |                          |
| 3774093 | user_account         | localhost | user_account     | Sleep   | 1    |                       |                          |
| 3774094 | user_account         | localhost | user_account     | Sleep   | 0    |                       |                          |
| 3774097 | user_account         | localhost | user_account     | Sleep   | 6    |                       |                          |
| 3774099 | user_account         | localhost | user_account     | Sleep   | 5    |                       |                          |
| 3774104 | user_account         | localhost | user_account     | Sleep   | 1    |                       |                          |
| 3774105 | user_account         | localhost | user_account     | Sleep   | 0    |                       |                          |
| 3774114 | user_account         | localhost | user_account     | Sleep   | 0    |                       |                          |
| 3774116 | user_account         | localhost | user_account     | Sleep   | 0    |                       |                          |
| 3774117 | user_account         | localhost | user_account     | Sleep   | 0    |                       |                          |
| 3774118 | user_account         | localhost | user_account     | Sleep   | 0    |                       |                          |
| 3774124 | user_account         | localhost | user_account     | Query   | 0    | Sending data          | SELECT COUNT(DISTINCT p.product_id) AS total FROM oc_category_path cp LEFTJOIN oc_product_to_catego |
| 3774125 | user_account         | localhost | user_account     | Sleep   | 0    |                       |                          |
| 3774127 | user_account         | localhost | user_account     | Query   | 0    | Sending data          | SELECT pd.tag FROM ins_product_description pd
                                                INNER JOIN ins_product p ON pd.product_id = p.pr |
| 3774131 | user_account         | localhost | user_account     | Sleep   | 0    |                       |                          |
| 3774133 | user_account         | localhost | user_account     | Sleep   | 3    |                       |                          |
| 3774134 | user_account         | localhost | user_account     | Sleep   | 0    |                       |                          |
| 3774135 | user_account         | localhost | user_account     | Sleep   | 2    |                       |                          |
| 3774138 | user_account         | localhost | user_account     | Sleep   | 0    |                       |                          |
| 3774139 | user_account         | localhost | user_account     | Sleep   | 1    |                       |                          |
| 3774141 | user_account         | localhost | user_account     | Sleep   | 2    |                       |                          |
| 3774145 | user_account         | localhost | user_account     | Sleep   | 1    |                       |                          |
| 3774154 | user_account         | localhost | user_account     | Sleep   | 1    |                       |                          |
| 3774156 | user_account         | localhost | user_account     | Sleep   | 0    |                       |                          |
| 3774157 | user_account         | localhost | user_account     | Sleep   | 0    |                       |                          |
| 3774158 | user_account         | localhost | user_account     | Sleep   | 0    |                       |                          |
| 3774160 | user_account         | localhost | user_account     | Sleep   | 0    |                       |                          |
| 3774161 | user_account         | localhost | user_account     | Sleep   | 0    |                       |                          |
| 3774162 | user_account         | localhost | user_account     | Sleep   | 0    |                       |                          |
| 3774163 | root                 | localhost |                  | Query   | 0    | starting              | show processlist                          |
| 3774164 | unauthenticated user | localhost |                  | Connect | 0    | Receiving from client |                          |
+---------+----------------------+-----------+------------------+---------+------+-----------------------+------------------------------------------------------------------------------------------------------+
Uptime: 338304  Threads: 42  Questions: 451501310  Slow queries: 0  Opens: 18817021  Flush tables: 1  Open tables: 2000  Queries per second avg: 1334.602

Running this command "apachectl status" returned this output:

Apache Server Status for localhost (via ::1)

   Server Version: Apache/2.4.52 (cPanel) OpenSSL/1.1.1m mod_bwlimited/1.4

   Server MPM: prefork

   Server Built: Jan 12 2022 01:58:29

   --------------------------------------------------------------------------

   Current Time: Friday, 28-Jan-2022 16:26:58 EET

   Restart Time: Thursday, 27-Jan-2022 02:22:37 EET

   Parent Server Config. Generation: 1

   Parent Server MPM Generation: 0

   Server uptime: 1 day 14 hours 4 minutes 20 seconds

   Server load: 33.66 36.18 36.18

   Total accesses: 1516768 - Total Traffic: 40.8 GB - Total Duration:
   3798260523

   CPU Usage: u8.38 s98.73 cu245198 cs57556.5 - 221% CPU load

   11.1 requests/sec - 312.5 kB/second - 28.2 kB/request - 2504.18 ms/request

   80 requests currently being processed, 10 idle workers

 WRWRRC_CRWCCKRW.K.WWC_WWKWKKWKW_RK__W_RCRKKRR.WRWWW.WKR_KW.WRWW_
 RRKRRRKRR_WWWKWK.KR...WR.K.RRKR_.K...KWK.R......................
 ......................

   Scoreboard Key:
   "_" Waiting for Connection, "S" Starting up, "R" Reading Request,
   "W" Sending Reply, "K" Keepalive (read), "D" DNS Lookup,
   "C" Closing connection, "L" Logging, "G" Gracefully finishing,
   "I" Idle cleanup of worker, "." Open slot with no current process

Please I need some help to identify what is consuming my server resources.

 

[cPRex]

It would have also been good to get the output from "top -c" during that top to see what the top processes were. Although troubleshooting load is typically left to the server admin, it would be worth submitting a ticket to our team so we could check the server, or have your hosting provider examine the system for issues on their side.

 

[psytanium]

It would have also been good to get the output from "top -c" during that top to see what the top processes were. Although troubleshooting load is typically left to the server admin, it would be worth submitting a ticket to our team so we could check the server, or have your hosting provider examine the system for issues on their side.

If i run "top -c", its always mysql on top of the list, during the high load or not

31740 mysql     20   0 2527856 808876   7704 S 115.5 10.1 862:15.51 /usr/sbin/mysqld --daemonize --pid-file=/var/run/mysqld/mysqld.pid

 

[psytanium]

What does it mean "The process “pkgacct - macrotro - av: 4 - write compressed stream” stalled while it ran for user “root”.
I receive this notification email every few hours.