SOLVED [CPANEL-26002] LMTP errors after upgrading to cPanel & WHM Version 78

adamreece.webbox

Well-Known Member
Nov 3, 2016
51
18
8
Penarth, United Kingdom
cPanel Access Level
Root Administrator
Moderator Notice: Please see this post for a description of why this error message occurs.

Since one of our servers updated to v78.0.11 this morning emails are completely failing to deliver.

Example snippet from `exim -qff -v`:

Code:
LOG: MAIN
  == [email protected] R=dkim_lookuphost T=dkim_remote_smtp defer (111): Connection refused
delivering 1gx6xU-0045fg-So (queue run pid 26838)
  LMTP<< 220 server.example.co.uk Dovecot ready.
  LMTP>> LHLO server.example.co.uk
  LMTP<< 250-server.example.co.uk
  LMTP<< 250-8BITMIME
  LMTP<< 250-CHUNKING
  LMTP<< 250-ENHANCEDSTATUSCODES
  LMTP<< 250-PIPELINING
  LMTP<< 250-STARTTLS
  LMTP<< 250 VRFY
  LMTP>> MAIL FROM:<>
  LMTP<< 250 2.1.0 OK
  LMTP>> RCPT TO:<[email protected]>
  LMTP<< 451 4.3.0 <[email protected]> Temporary internal error
  LMTP>> QUIT
  LMTP<< 221 2.0.0 Bye
I've tried restarting exim+spamd, restarting the server completely, and repairing mailbox permissions. Looks specific to LMTP. (Emails going out to foreign exchanges seem to go through.)

This issue has also impacted another person on Discord (Ross). I raised a ticket on this (11496321) but with 160 clients suffering, and a mail queue at 1330 messages and growing, I need to get this resolved ASAP.

Thanks for reading.

Edit: Also tried `/scripts/buildeximconf` then restarting Exim, no improvement.
 
Last edited by a moderator:

adamreece.webbox

Well-Known Member
Nov 3, 2016
51
18
8
Penarth, United Kingdom
cPanel Access Level
Root Administrator
Looks like it's actually a quota issue due to the server being a virtual machine. Andrea at cPanel found this log entry in "/var/log/maillog":

Feb 22 13:49:57 demeter dovecot: lmtp(128010): Error: mailbox_get_status(INBOX, STATUS_CHECK_OVER_QUOTA) failed: Mailbox INBOX: quota: Failed to get quota resource STORAGE_BYTES for INBOX: quota-fs: quotactl(Q_GETQUOTA, /dev/simfs) failed: No such device
What I'm not sure of is why it only impacted this VM. We've got 15 of them and it only happened to this one today with the v78 update.

Edit: Support have got back to me to say the quota setup is fine.
 
Last edited:
  • Like
Reactions: azadeh69

kernow

Well-Known Member
Jul 23, 2004
1,018
55
178
cPanel Access Level
Root Administrator
We also had this problem on all our servers after upgrading to 78.0.11. We have configservers MailScanner installed and if you do too, the fix is to close MailScanner, check and kill any MailScanner processes still running, then restart MailScanner.
 

cPanelMichael

Administrator
Staff member
Apr 11, 2011
47,909
2,222
463
We also had this problem on all our servers after upgrading to 78.0.11. We have configservers MailScanner installed and if you do too, the fix is to close MailScanner, check and kill any MailScanner processes still running, then restart MailScanner.
Hello @kernow,

We confirmed that MailScanner was not installed on the affected system as part of investigation in the support ticket that was opened by the original poster.

Can you provide more information about the specific error message you encountered prior to restarting MailScanner, along with the steps you used to reproduce that error? Also, please post the output from the following commands:

Code:
cat /var/cpanel/envtype
mount|grep simfs
Thank you.
 

kernow

Well-Known Member
Jul 23, 2004
1,018
55
178
cPanel Access Level
Root Administrator
Can't remember the exact error prior to restarting MailScanner, we had this same problem about a year ago after a cpanel update so we knew what to check. Checking the status of MailScanner said something like, "MailScanner: Could not use Custom Function code /usr/mailscanner/usr/share/MailScanner/perl/custom/MailControl.pm" It also said "SpamAssassin is not installed. This problem affected CloudLinux 6 & 7 servers.
Code:
cat /var/cpanel/envtype
[email protected]
Code:
mount|grep simfs
Returns nothing
 

cPanelMichael

Administrator
Staff member
Apr 11, 2011
47,909
2,222
463
Hello @kernow,

This doesn't appear to relate to the issue described by the original poster. I recommend reporting this behavior to the developers of the MailScanner plugin you are using to verify if it's an issue they can address.

Thank you.
 

adamreece.webbox

Well-Known Member
Nov 3, 2016
51
18
8
Penarth, United Kingdom
cPanel Access Level
Root Administrator
For us the issue was that there was leftover Virtuozzo packages installed after the VM had been migrated to KVM. I don't know why it only started causing a problem now, but when cPanel's support removed these, and instructed me to do a reboot, the Exim queue delivered to Dovecot via LMTP successfully.

Hi there,

With the assistance of my colleague, we were able to determine the issue.

Was this KVM instance migrated from a VZ instance?

I ask because there are remnants of VZ configuration files on this server:

-----
[14:31:29 demeter [email protected] ~]cPs# stat /etc/rc.d/init.d/vzquota
File: `/etc/rc.d/init.d/vzquota'
Size: 919 Blocks: 8 IO Block: 4096 regular file
Device: fd01h/64769d Inode: 12046 Links: 1
Access: (0755/-rwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2019-02-21 19:14:03.460000071 +0000
Modify: 2017-09-30 00:38:28.000000000 +0100
Change: 2017-12-11 12:29:20.327019231 +0000
-----

-----
[14:32:03 demeter [email protected] ~]cPs# rpm -qf /etc/rc.d/init.d/vzquota
file /etc/rc.d/init.d/vzquota is not owned by any package
-----

-----
[14:32:06 demeter [email protected] ~]cPs# cat /etc/rc.d/init.d/vzquota
#!/bin/sh
# chkconfig: 2345 10 90
# description: prepare container to use OpenVZ 2nd-level disk quotas

### BEGIN INIT INFO
# Provides: vzquota
# Required-Start: $local_fs $time $syslog
# Required-Stop: $local_fs
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
# Short-Description: Start vzquota at the end of boot
# Description: Configure OpenVZ disk quota for a container.
### END INIT INFO

start() {
if [ ! -L /etc/mtab ]; then
rm -f /etc/mtab >/dev/null 2>&1
ln -sf /proc/mounts /etc/mtab
fi
dev=$(awk '($2 == "/") && ($4 ~ /usrquota/) && ($4 ~ /grpquota/) {print $1}' /etc/mtab)
if test -z "$dev"; then
dev="/dev/simfs"
rm -f /etc/mtab >/dev/null 2>&1
echo "/dev/simfs / reiserfs rw,usrquota,grpquota 0 0" > /etc/mtab
grep -v " / " /proc/mounts >> /etc/mtab 2>/dev/null
chmod 644 /etc/mtab
fi
[ -e "$dev" ] || mknod $dev b 0 22
quotaon -aug
}

case "$1" in
start)
start
;;
*)
exit
esac
-----

I have moved this file out of the way. I would recommend that you reboot the server so that KVM configurations are used.

You may want to speak with your data center regarding the remnant files.

If you have anymore questions, please feel free to ask.

Thank you,
After rebooting I ran `exim -qff -v` and everything delivered fine.

Hope this helps someone else. :)
 
  • Like
Reactions: cPanelMichael

cPAusaf

Linux Technical Analyst III
Staff member
Aug 24, 2016
41
11
83
Houston, TX
cPanel Access Level
Root Administrator
This seems to be a permissions/ownership issue. If another user experiences this, then I would recommend running:

/scripts/mailperm --verbose $username

If you happen to see 'Unable to fix ownership', then please open a support ticket so that we may take a closer look.
 
  • Like
Reactions: cPanelMichael

Niclas Hedfors

Registered
Feb 26, 2019
4
0
1
Kalmar, Sweden
cPanel Access Level
Root Administrator
Hello,

Our LIVE production server was updated to 78.0.12 last night.
After that most of our clients wont receive any email.
In "track Delivery" we se the following error.
Code:
LMTP error after RCPT TO:<[email protected]>: 451 4.3.0 <[email protected]> Temporary internal error
I have submitted a ticket to support but as this is affecting a lots of clients i want it resolved as soon as possible and trying to get help wherever i can.
 

Niclas Hedfors

Registered
Feb 26, 2019
4
0
1
Kalmar, Sweden
cPanel Access Level
Root Administrator
Tried to edit my last post but apparently it was spam-related?

It's quota related.
As I change the quota for the cPanel-account to "unlimited" it starts to work properly.
For now, i've changed the quota on all accounts to "unlimited", so its not that urgent any more, but i still want a solution to this.
 

keat63

Well-Known Member
Nov 20, 2014
1,913
259
113
cPanel Access Level
Root Administrator
If you could update this thread with your ticket number, one of the mods will update us with the outcome.
It may help someone in the future or coming days.
Seems odd to get two very similar errors in such close proximity.
 

Niclas Hedfors

Registered
Feb 26, 2019
4
0
1
Kalmar, Sweden
cPanel Access Level
Root Administrator
My support ticket number: 11530727

According to the maillog, there is some problem with an NFS mount on our server that not report quota at all. What we can see in the maillog this didn't work before the update either, but it seems like it is handled a bit different after the update.
Got this from cPanel Support:
It does appear that an upstream change to dovecot now treats these LMTP errors as fatal. Reviewing the changelogs, I don't see anything specific in dovecot 2.3 about LMTP errors, but I do see there was lots of changes made to LMTP.
Locating it to a quota problem made me solve it temporary by set all customers to "unlimited" as i wrote earlier.
All customers are up and running.
For now, since quota never worked properly, having customers on unlimited quota practically doesn't make much of a differens.

I will try to resolve the NFS part, with my colleagues who know more about the NFS connection, to make the quota work properly.
 

cPanelMichael

Administrator
Staff member
Apr 11, 2011
47,909
2,222
463
Hello Everyone,

We published the following Dovecot security update in cPanel & WHM version 78.0.10:

[security] Fixed case CPANEL-25461: Update cpanel-dovecot to 2.3.4.1-1 for CVE-2019-3814.

Included with this update is a change to the way the Dovecot Local Mail Transfer Protocol (LMTP) handles local delivery attempts when it's unable to determine if quotas are enabled. Prior to this change, Dovecot quota errors like the ones below were logged to /var/log/maillog:

Feb 25 03:10:11 server1 dovecot: lmtp(866): Error: Mailbox INBOX: quota: Failed to get quota resource STORAGE_BYTES for INBOX: quota-fs: quotactl(Q_GETQUOTA, /dev/simfs) failed: No such device
Feb 25 03:10:11 server1 dovecot: lmtp(866): Error: mailbox_get_status(INBOX, STATUS_CHECK_OVER_QUOTA) failed: Mailbox INBOX: quota: Failed to get quota resource STORAGE_BYTES for INBOX: quota-fs: quotactl(Q_GETQUOTA, /dev/simfs) failed: No such device
Feb 25 03:10:11 server1 dovecot: lmtp(866): Disconnect from local: Client has quit the connection (state=RCPT TO)
Feb 25 03:12:46 server1 dovecot: lmtp(1164): Connect from local
Feb 25 03:12:46 server1 dovecot: lmtp(1164): Disconnect from local: Client has quit the connection (state=GREETING
Prior to the update, these errors were non-fatal and the local delivery attempt succeeded.

The updated Dovecot version now considers these types of quota check errors to be fatal and thus aborts deliveries with an error message like this in /var/log/exim_mainlog:

T=dovecot_virtual_delivery defer (-44): LMTP error after RCPT TO:<[email protected]>: 451 4.3.0 <[email protected]> Temporary internal error
We're currently evaluating reports of this issue as part of internal case CPANEL-26002. I'll monitor this case and update this thread with more information as it becomes available.

In the meantime, the workaround is to address the underlying cause of the quota errors seen in /var/log/maillog.

The specific cause of the quota errors can vary from system to system, however most of the support requests we've received thus far stem from the vzquota init script incorrectly running at boot time on virtual servers that were migrated from the vzfs backend to the ploop storage backend.

You can verify if that's the cause of the issue on your particular server by accessing the server via SSH as root and running the following commands:

Code:
chkconfig --list | grep vzquota
diff -u /etc/mtab /proc/mounts
The output from the above commands will resemble the following on affected systems:

# chkconfig --list | grep vzquota
vzquota 0: off 1: off 2: on 3: on 4: on 5: on 6: off
# diff -u /etc/mtab /proc/mounts
--- /etc/mtab 2019-02-25 12:38:42.637938881 +0100
+++ /proc/mounts 2019-02-26 00:55:11.314517060 +0100
@@ -1,4 +1,4 @@
-/dev/simfs / reiserfs rw,usrquota,grpquota 0 0
+/dev/ploop11777p1 / ext4
rw,relatime,barrier=1,data=ordered,balloon_ino=12,jqfmt=vfsv0,usrjquota=aquota.user,grpjquota=aquota.group 0 0
proc /proc proc rw,relatime 0 0
sysfs /sys sysfs rw,relatime 0 0
nfsd /proc/fs/nfsd nfsd rw,relatime 0 0
....
Notice that vzquota is set to "on" at run levels 2, 3, 4, and 5 (this confirms it runs at boot time). Then, notice how /etc/mtab uses /dev/simfs, whereas /proc/mounts uses /dev/ploop$$. The difference exists because the vzquota init script incorrectly changes "ploop" to "simfs" in the /etc/mtab file at boot time time (/etc/mtab should instead exist as a symbolic link to /proc/mounts on these systems).

If the output does not look like this on your system, then it's likely there's a different underlying cause of the quota errors on your system. In such cases, feel free to respond to this thread with the error message you see in /var/log/maillog and we'll be happy to help you diagnose the problem.

Upon confirming the system is affected by this particular issue, you can run the following commands to fix this:

Code:
chkconfig --level 234 vzquota off
mv -v /etc/mtab{,.old}
ln -s /proc/mounts /etc/mtab
reboot
These commands disable vzquota at boot time, setup /etc/mtab is a symbolic link to /proc/mounts, and reboot the server.

Let us know if you have any questions.

Thank you.

Edit Log:
03/06/2019: Updated case number to CPANEL-26002.
03/01/2019: Updated the workaround instructions to note the commands should be ran on the individual server.
 
Last edited:

deddy

Well-Known Member
Oct 13, 2003
146
1
168
To gain some time I did the following:

vi 20-lmtp.conf

# Verify quota before replying to RCPT TO. This adds a small overhead.
# lmtp_rcpt_check_quota = no

changed to

# Verify quota before replying to RCPT TO. This adds a small overhead.
lmtp_rcpt_check_quota = no

service dovecot restart
 
  • Like
Reactions: Mohamed Mohsen

cPanelMichael

Administrator
Staff member
Apr 11, 2011
47,909
2,222
463
# Verify quota before replying to RCPT TO. This adds a small overhead.
# lmtp_rcpt_check_quota = no

changed to

# Verify quota before replying to RCPT TO. This adds a small overhead.
lmtp_rcpt_check_quota = no
Hello @deddy,

While this may act as a temporary workaround, keep in mind this is an untested change and can potentially lead to additional delivery issues if a cPanel account or individual email account is at it's quota limit.

Thank you.
 

shaun

Well-Known Member
PartnerNOC
Verifed Vendor
Nov 9, 2001
708
1
318
San Clemente, Ca
cPanel Access Level
DataCenter Provider
Twitter
We've seen this issue too recently on a number of customer servers. The cause so far looks to be permissions related, mainly ownership. Here's a the fix we've been using...

Run the following command
Code:
/scripts/mailperm --verbose username |grep "ownership" | grep "should be"
If you see output that looks simliar to the following

Code:
Unable to fix ownership on /home/username/mail/domain.tld/.Sent/cur/1484153449.M120071P468622.server1.domain.tld,S=64721,W=65681:2,S: currently (556:556), should be (618:618)
Then you need to correct the permissions on that account. The important part of the output is the "should be" part with the numbers in the parenthesis. In my example thats 556:556 and 618:618. You can then do the following

Code:
cd /home/username/mail
find ./ -user 556 -exec chown 616 {} \;
find ./ -group 556 -exec chgrp 618  {} \;
Now you should run /scripts/mailperm --verbose username two more times. The first time you may see output showing the script fixing permissions. The second run you should not see any output. If you do, there are more issues that need to be fixed.

Hope that helps
 

cPanelMichael

Administrator
Staff member
Apr 11, 2011
47,909
2,222
463
Hi @shaun,

Can you access one of the affected servers and look at the /etc/fstab file to see if the root filesystem (e.g. "/") is associated with a device that doesn't exist? For instance, many of the affected systems we've seen have an entry like this in their /etc/fstab file despite /dev/simfs not existing as a device:

Code:
/dev/simfs / reiserfs rw,usrquota,grpquota 0 0
Here's a command you can use to help identify if an invalid device name is utilized:

Code:
diff -u /etc/mtab /proc/mounts
Thank you.