All emails still stuck in queue after a spam detected and cleaned

cPanel & WHM Version
CENTOS 7.8 kvm

psytanium

Well-Known Member
Jun 6, 2014
343
22
68
Lebanon
cPanel Access Level
Root Administrator
Hello,

3 days ago, 1 of the emails was sending thousands of spams, i suspended the email address, fixed the issue from the client device (He wasn't using the SSL) then changed his password and all is fine.

Lots of emails were stuck on the queue, most of it spams, i cleaned the spams and delivered the others.

But the problem, is now all outgoing emails still stuck on queue, I have to deliver it manually.

Thanks :)
 

ZenHostingTravis

Well-Known Member
PartnerNOC
May 22, 2020
273
92
28
Australia
cPanel Access Level
Root Administrator
The below notes from cPanel's documentation may assist you further.

To release or delete outgoing mail held in the queue, perform the following actions in cPanel's Email Accounts interface (cPanel >> Home >> Email >> Email Accounts):

  1. Click Manage Suspension.
  2. Select Allow for the Send option.
  3. If applicable, click Delete messages from the mail queue to remove any queued messages.
  4. Click Save.
 

andrew.n

Well-Known Member
Jun 9, 2020
983
363
63
EU
cPanel Access Level
Root Administrator
  • Like
Reactions: cPRex

cPanelLauren

Product Owner II
Staff member
Nov 14, 2017
13,266
1,304
363
Houston
You can also check the reason the email is queued but obtaining the message ID of a queued message and checking the exim logs. The following is what I use frequently:

Code:
exigrep $msgid /var/log/exim_mainlog
 

psytanium

Well-Known Member
Jun 6, 2014
343
22
68
Lebanon
cPanel Access Level
Root Administrator
Hello,

Again, outgoing emails from various users end up in the Mail Queue, and I have to manually select and deliver, since yesterday.

I tried to check the exim log for one specific email, i got this :

Code:
[root@server ~]# exigrep OF88395AED.D3082EEF-ONC12587D4.0057D133-C12587D4.00582AD8@samospa.net /var/log/exim_mainlog
+++ 1nC1ok-00025r-Jj has not completed +++
2022-01-24 18:03:42 1nC1ok-00025r-Jj <= [email protected] H=mx2.samo.it (mx2.samoindustries.com) [77.241.5.94]:55276 P=esmtps X=TLS1.2:ECDHE-RSA-AES256-GCM-SHA384:256 CV=no S=566517 id=OF88395AED.D3082EEF-ONC12587D4.0057D133-C12587D4.00582AD8@samospa.net T="Rif: RE: RE: Rif: Sara Riyadh - AV258C mirror inquiry" for [email protected]
2022-01-24 18:03:42 1nC1ok-00025r-Jj no immediate delivery: load average 74.12
Also I noticed my server is slow when loading pages. I don't know what's happening.
 
Last edited:

quietFinn

Well-Known Member
Feb 4, 2006
2,042
553
493
Finland
cPanel Access Level
Root Administrator
no immediate delivery: load average 74.12
Exim is not delivering messages because server load is too high.
Command:
grep "deliver_queue_load_max" /etc/exim.conf
shows the maximum.
 

quietFinn

Well-Known Member
Feb 4, 2006
2,042
553
493
Finland
cPanel Access Level
Root Administrator
That means that Exim will continue delivering when load goes under 12.
You can change that setting in WHM -> Service Configuration -> Exim Configuration Manager -> Advanced editor
 

cPRex

Jurassic Moderator
Staff member
Oct 19, 2014
16,634
2,629
363
cPanel Access Level
Root Administrator
It's nearly impossible to find the cause of high load after it has happened - it's really best to try and troubleshoot that in real-time. I like to check the following things:

top -c
mysqladmin proc status
apachectl status
OR
apachectl fullstatus

Those commands will give you an idea of where the server usage is, if MySQL is using too many resources or taking too long to handle queries, and if Apache is overloaded. One of those things usually gets me down the right path.
 

psytanium

Well-Known Member
Jun 6, 2014
343
22
68
Lebanon
cPanel Access Level
Root Administrator
It's nearly impossible to find the cause of high load after it has happened - it's really best to try and troubleshoot that in real-time. I like to check the following things:

top -c
mysqladmin proc status
apachectl status
OR
apachectl fullstatus

Those commands will give you an idea of where the server usage is, if MySQL is using too many resources or taking too long to handle queries, and if Apache is overloaded. One of those things usually gets me down the right path.
mysqladmin proc status returned this output

Code:
+---------+------------------+-----------+------------------+---------+------+--------------+------------------------------------------------------------------------------------------------------+
| Id      | User             | Host      | db               | Command | Time | State        | Info             |
+---------+------------------+-----------+------------------+---------+------+--------------+------------------------------------------------------------------------------------------------------+
| 2231939 | houseoli_user | localhost | houseoli_user | Sleep   | 0    |              |             |
| 2231940 | macrotro_suser    | localhost | macrotro_user    | Query   | 0    | Sending data | select count(p.products_id) as count  from (products p)
                            join (products_t |
| 2231941 | elements_user | localhost | elements_user | Sleep   | 0    |              |             |
| 2231943 | root             | localhost |                  | Query   | 0    | starting     | show processlist             |
+---------+------------------+-----------+------------------+---------+------+--------------+------------------------------------------------------------------------------------------------------+
Uptime: 137032  Threads: 4  Questions: 106930006  Slow queries: 0  Opens: 5423017  Flush tables: 1  Open tables: 2000  Queries per second avg: 780.328
apachectl status returned this output

Code:
Apache Server Status for localhost (via ::1)

   Server Version: Apache/2.4.52 (cPanel) OpenSSL/1.1.1m mod_bwlimited/1.4

   Server MPM: prefork

   Server Built: Jan 12 2022 01:58:29

   --------------------------------------------------------------------------

   Current Time: Wednesday, 26-Jan-2022 08:24:30 EET

   Restart Time: Saturday, 22-Jan-2022 20:34:55 EET

   Parent Server Config. Generation: 3

   Parent Server MPM Generation: 2

   Server uptime: 3 days 11 hours 49 minutes 34 seconds

   Server load: 1.90 1.69 1.78

   Total accesses: 5579109 - Total Traffic: 92.7 GB - Total Duration:
   19335594938

   CPU Usage: u9.1 s219.25 cu612714 cs157174 - 255% CPU load

   18.5 requests/sec - 322.2 kB/second - 17.4 kB/request - 3465.71 ms/request

   29 requests currently being processed, 4 idle workers

R_RCWKWW_C..KR..CC.........K.._...R.R..RK......RRRR..R.R.R_.....
.K...K..R.CR....................................................
......................

   Scoreboard Key:
   "_" Waiting for Connection, "S" Starting up, "R" Reading Request,
   "W" Sending Reply, "K" Keepalive (read), "D" DNS Lookup,
   "C" Closing connection, "L" Logging, "G" Gracefully finishing,
   "I" Idle cleanup of worker, "." Open slot with no current process
Anything suspicious in those reports ?
I will try your commands if the problem occur again. Thank you fir your support.
 

psytanium

Well-Known Member
Jun 6, 2014
343
22
68
Lebanon
cPanel Access Level
Root Administrator
Same problem happened again, today all day, my server is very slow, and emails stuck in queue.

I received a notification email from the server.

Subjext : The process “pkgacct - macrotro - av: 4 - write compressed stream” stalled while it ran for user “root”.

The process has run for “12 hours, 1 minute, and 35.96 seconds”.
At the time that the system generated this notification, the process had a PID of “2176”, and it consumed 0.46% of the system’s memory and 0.01% of the system’s CPU time.

Running this command " mysqladmin proc status " returned this output:

Code:
+---------+----------------------+-----------+------------------+---------+------+-----------------------+------------------------------------------------------------------------------------------------------+
| Id      | User                 | Host      | db               | Command | Time | State                 | Info                          |
+---------+----------------------+-----------+------------------+---------+------+-----------------------+------------------------------------------------------------------------------------------------------+
| 3774003 | user_account         | localhost | user_account     | Sleep   | 0    |                       |                          |
| 3774026 | user_account         | localhost | user_account     | Sleep   | 0    |                       |                          |
| 3774037 | user_account         | localhost | user_account     | Sleep   | 0    |                       |                          |
| 3774060 | user_account         | localhost | user_account     | Sleep   | 1    |                       |                          |
| 3774065 | user_account         | localhost | user_account     | Sleep   | 0    |                       |                          |
| 3774067 | user_account         | localhost | user_account     | Sleep   | 1    |                       |                          |
| 3774068 | user_account         | localhost | user_account     | Sleep   | 0    |                       |                          |
| 3774080 | user_account         | localhost | user_account     | Sleep   | 1    |                       |                          |
| 3774083 | user_account         | localhost | user_account     | Sleep   | 0    |                       |                          |
| 3774084 | user_account         | localhost | user_account     | Sleep   | 1    |                       |                          |
| 3774089 | macrotro_shop        | localhost | user_account     | Query   | 0    | Sending data          | select count(p.products_id) as count  from (products p)
                            join (products_t |
| 3774092 | user_account         | localhost | user_account     | Sleep   | 1    |                       |                          |
| 3774093 | user_account         | localhost | user_account     | Sleep   | 1    |                       |                          |
| 3774094 | user_account         | localhost | user_account     | Sleep   | 0    |                       |                          |
| 3774097 | user_account         | localhost | user_account     | Sleep   | 6    |                       |                          |
| 3774099 | user_account         | localhost | user_account     | Sleep   | 5    |                       |                          |
| 3774104 | user_account         | localhost | user_account     | Sleep   | 1    |                       |                          |
| 3774105 | user_account         | localhost | user_account     | Sleep   | 0    |                       |                          |
| 3774114 | user_account         | localhost | user_account     | Sleep   | 0    |                       |                          |
| 3774116 | user_account         | localhost | user_account     | Sleep   | 0    |                       |                          |
| 3774117 | user_account         | localhost | user_account     | Sleep   | 0    |                       |                          |
| 3774118 | user_account         | localhost | user_account     | Sleep   | 0    |                       |                          |
| 3774124 | user_account         | localhost | user_account     | Query   | 0    | Sending data          | SELECT COUNT(DISTINCT p.product_id) AS total FROM oc_category_path cp LEFTJOIN oc_product_to_catego |
| 3774125 | user_account         | localhost | user_account     | Sleep   | 0    |                       |                          |
| 3774127 | user_account         | localhost | user_account     | Query   | 0    | Sending data          | SELECT pd.tag FROM ins_product_description pd
                                                INNER JOIN ins_product p ON pd.product_id = p.pr |
| 3774131 | user_account         | localhost | user_account     | Sleep   | 0    |                       |                          |
| 3774133 | user_account         | localhost | user_account     | Sleep   | 3    |                       |                          |
| 3774134 | user_account         | localhost | user_account     | Sleep   | 0    |                       |                          |
| 3774135 | user_account         | localhost | user_account     | Sleep   | 2    |                       |                          |
| 3774138 | user_account         | localhost | user_account     | Sleep   | 0    |                       |                          |
| 3774139 | user_account         | localhost | user_account     | Sleep   | 1    |                       |                          |
| 3774141 | user_account         | localhost | user_account     | Sleep   | 2    |                       |                          |
| 3774145 | user_account         | localhost | user_account     | Sleep   | 1    |                       |                          |
| 3774154 | user_account         | localhost | user_account     | Sleep   | 1    |                       |                          |
| 3774156 | user_account         | localhost | user_account     | Sleep   | 0    |                       |                          |
| 3774157 | user_account         | localhost | user_account     | Sleep   | 0    |                       |                          |
| 3774158 | user_account         | localhost | user_account     | Sleep   | 0    |                       |                          |
| 3774160 | user_account         | localhost | user_account     | Sleep   | 0    |                       |                          |
| 3774161 | user_account         | localhost | user_account     | Sleep   | 0    |                       |                          |
| 3774162 | user_account         | localhost | user_account     | Sleep   | 0    |                       |                          |
| 3774163 | root                 | localhost |                  | Query   | 0    | starting              | show processlist                          |
| 3774164 | unauthenticated user | localhost |                  | Connect | 0    | Receiving from client |                          |
+---------+----------------------+-----------+------------------+---------+------+-----------------------+------------------------------------------------------------------------------------------------------+
Uptime: 338304  Threads: 42  Questions: 451501310  Slow queries: 0  Opens: 18817021  Flush tables: 1  Open tables: 2000  Queries per second avg: 1334.602

Running this command "apachectl status" returned this output:

Code:
Apache Server Status for localhost (via ::1)

   Server Version: Apache/2.4.52 (cPanel) OpenSSL/1.1.1m mod_bwlimited/1.4

   Server MPM: prefork

   Server Built: Jan 12 2022 01:58:29

   --------------------------------------------------------------------------

   Current Time: Friday, 28-Jan-2022 16:26:58 EET

   Restart Time: Thursday, 27-Jan-2022 02:22:37 EET

   Parent Server Config. Generation: 1

   Parent Server MPM Generation: 0

   Server uptime: 1 day 14 hours 4 minutes 20 seconds

   Server load: 33.66 36.18 36.18

   Total accesses: 1516768 - Total Traffic: 40.8 GB - Total Duration:
   3798260523

   CPU Usage: u8.38 s98.73 cu245198 cs57556.5 - 221% CPU load

   11.1 requests/sec - 312.5 kB/second - 28.2 kB/request - 2504.18 ms/request

   80 requests currently being processed, 10 idle workers

 WRWRRC_CRWCCKRW.K.WWC_WWKWKKWKW_RK__W_RCRKKRR.WRWWW.WKR_KW.WRWW_
 RRKRRRKRR_WWWKWK.KR...WR.K.RRKR_.K...KWK.R......................
 ......................

   Scoreboard Key:
   "_" Waiting for Connection, "S" Starting up, "R" Reading Request,
   "W" Sending Reply, "K" Keepalive (read), "D" DNS Lookup,
   "C" Closing connection, "L" Logging, "G" Gracefully finishing,
   "I" Idle cleanup of worker, "." Open slot with no current process
Please I need some help to identify what is consuming my server resources.
 

cPRex

Jurassic Moderator
Staff member
Oct 19, 2014
16,634
2,629
363
cPanel Access Level
Root Administrator
It would have also been good to get the output from "top -c" during that top to see what the top processes were. Although troubleshooting load is typically left to the server admin, it would be worth submitting a ticket to our team so we could check the server, or have your hosting provider examine the system for issues on their side.
 

psytanium

Well-Known Member
Jun 6, 2014
343
22
68
Lebanon
cPanel Access Level
Root Administrator
It would have also been good to get the output from "top -c" during that top to see what the top processes were. Although troubleshooting load is typically left to the server admin, it would be worth submitting a ticket to our team so we could check the server, or have your hosting provider examine the system for issues on their side.
If i run "top -c", its always mysql on top of the list, during the high load or not

Code:
31740 mysql     20   0 2527856 808876   7704 S 115.5 10.1 862:15.51 /usr/sbin/mysqld --daemonize --pid-file=/var/run/mysqld/mysqld.pid
 

psytanium

Well-Known Member
Jun 6, 2014
343
22
68
Lebanon
cPanel Access Level
Root Administrator
What does it mean "The process “pkgacct - macrotro - av: 4 - write compressed stream” stalled while it ran for user “root”.
I receive this notification email every few hours.