Please whitelist cPanel in your adblocker so that you’re able to see our version release promotions, thanks!

The Community Forums

Interact with an entire community of cPanel & WHM users!

Pending Publication [CPANEL-26002] LMTP errors after upgrading to cPanel & WHM Version 78

Discussion in 'E-mail Discussion' started by adamreece.webbox, Feb 22, 2019.

  1. adamreece.webbox

    adamreece.webbox Member

    Joined:
    Nov 3, 2016
    Messages:
    23
    Likes Received:
    9
    Trophy Points:
    3
    Location:
    Penarth, United Kingdom
    cPanel Access Level:
    Root Administrator
    Moderator Notice: Please see this post for a description of why this error message occurs.

    Since one of our servers updated to v78.0.11 this morning emails are completely failing to deliver.

    Example snippet from `exim -qff -v`:

    Code:
    LOG: MAIN
      == sender@example.com R=dkim_lookuphost T=dkim_remote_smtp defer (111): Connection refused
    delivering 1gx6xU-0045fg-So (queue run pid 26838)
      LMTP<< 220 server.example.co.uk Dovecot ready.
      LMTP>> LHLO server.example.co.uk
      LMTP<< 250-server.example.co.uk
      LMTP<< 250-8BITMIME
      LMTP<< 250-CHUNKING
      LMTP<< 250-ENHANCEDSTATUSCODES
      LMTP<< 250-PIPELINING
      LMTP<< 250-STARTTLS
      LMTP<< 250 VRFY
      LMTP>> MAIL FROM:<>
      LMTP<< 250 2.1.0 OK
      LMTP>> RCPT TO:<recipient-account@server.example.co.uk>
      LMTP<< 451 4.3.0 <recipient-account@server.example.co.uk> Temporary internal error
      LMTP>> QUIT
      LMTP<< 221 2.0.0 Bye
    I've tried restarting exim+spamd, restarting the server completely, and repairing mailbox permissions. Looks specific to LMTP. (Emails going out to foreign exchanges seem to go through.)

    This issue has also impacted another person on Discord (Ross). I raised a ticket on this (11496321) but with 160 clients suffering, and a mail queue at 1330 messages and growing, I need to get this resolved ASAP.

    Thanks for reading.

    Edit: Also tried `/scripts/buildeximconf` then restarting Exim, no improvement.
     
    #1 adamreece.webbox, Feb 22, 2019
    Last edited by a moderator: Feb 26, 2019
  2. keat63

    keat63 Well-Known Member

    Joined:
    Nov 20, 2014
    Messages:
    1,209
    Likes Received:
    77
    Trophy Points:
    28
    cPanel Access Level:
    Root Administrator
    azadeh69 likes this.
  3. adamreece.webbox

    adamreece.webbox Member

    Joined:
    Nov 3, 2016
    Messages:
    23
    Likes Received:
    9
    Trophy Points:
    3
    Location:
    Penarth, United Kingdom
    cPanel Access Level:
    Root Administrator
    Looks like it's actually a quota issue due to the server being a virtual machine. Andrea at cPanel found this log entry in "/var/log/maillog":

    What I'm not sure of is why it only impacted this VM. We've got 15 of them and it only happened to this one today with the v78 update.

    Edit: Support have got back to me to say the quota setup is fine.
     
    #3 adamreece.webbox, Feb 22, 2019
    Last edited: Feb 22, 2019
    azadeh69 likes this.
  4. kernow

    kernow Well-Known Member

    Joined:
    Jul 23, 2004
    Messages:
    938
    Likes Received:
    18
    Trophy Points:
    168
    cPanel Access Level:
    Root Administrator
    We also had this problem on all our servers after upgrading to 78.0.11. We have configservers MailScanner installed and if you do too, the fix is to close MailScanner, check and kill any MailScanner processes still running, then restart MailScanner.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  5. cPanelMichael

    cPanelMichael Technical Support Community Manager Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    47,009
    Likes Received:
    2,123
    Trophy Points:
    363
    cPanel Access Level:
    Root Administrator
    Twitter:
    Hello @kernow,

    We confirmed that MailScanner was not installed on the affected system as part of investigation in the support ticket that was opened by the original poster.

    Can you provide more information about the specific error message you encountered prior to restarting MailScanner, along with the steps you used to reproduce that error? Also, please post the output from the following commands:

    Code:
    cat /var/cpanel/envtype
    mount|grep simfs
    Thank you.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  6. kernow

    kernow Well-Known Member

    Joined:
    Jul 23, 2004
    Messages:
    938
    Likes Received:
    18
    Trophy Points:
    168
    cPanel Access Level:
    Root Administrator
    Can't remember the exact error prior to restarting MailScanner, we had this same problem about a year ago after a cpanel update so we knew what to check. Checking the status of MailScanner said something like, "MailScanner: Could not use Custom Function code /usr/mailscanner/usr/share/MailScanner/perl/custom/MailControl.pm" It also said "SpamAssassin is not installed. This problem affected CloudLinux 6 & 7 servers.
    Code:
    cat /var/cpanel/envtype
    standardroot@servername
    Code:
    mount|grep simfs
    Returns nothing
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  7. cPanelMichael

    cPanelMichael Technical Support Community Manager Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    47,009
    Likes Received:
    2,123
    Trophy Points:
    363
    cPanel Access Level:
    Root Administrator
    Twitter:
    Hello @kernow,

    This doesn't appear to relate to the issue described by the original poster. I recommend reporting this behavior to the developers of the MailScanner plugin you are using to verify if it's an issue they can address.

    Thank you.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  8. adamreece.webbox

    adamreece.webbox Member

    Joined:
    Nov 3, 2016
    Messages:
    23
    Likes Received:
    9
    Trophy Points:
    3
    Location:
    Penarth, United Kingdom
    cPanel Access Level:
    Root Administrator
    For us the issue was that there was leftover Virtuozzo packages installed after the VM had been migrated to KVM. I don't know why it only started causing a problem now, but when cPanel's support removed these, and instructed me to do a reboot, the Exim queue delivered to Dovecot via LMTP successfully.

    After rebooting I ran `exim -qff -v` and everything delivered fine.

    Hope this helps someone else. :)
     
    cPanelMichael likes this.
  9. cPAusaf

    cPAusaf Linux Technical Analyst III Staff Member

    Joined:
    Aug 24, 2016
    Messages:
    33
    Likes Received:
    7
    Trophy Points:
    83
    Location:
    Houston, TX
    cPanel Access Level:
    Root Administrator
    This seems to be a permissions/ownership issue. If another user experiences this, then I would recommend running:

    /scripts/mailperm --verbose $username

    If you happen to see 'Unable to fix ownership', then please open a support ticket so that we may take a closer look.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
    cPanelMichael likes this.
  10. Niclas Hedfors

    Niclas Hedfors Registered

    Joined:
    Feb 26, 2019
    Messages:
    4
    Likes Received:
    0
    Trophy Points:
    1
    Location:
    Kalmar, Sweden
    cPanel Access Level:
    Root Administrator
    Hello,

    Our LIVE production server was updated to 78.0.12 last night.
    After that most of our clients wont receive any email.
    In "track Delivery" we se the following error.
    Code:
    LMTP error after RCPT TO:<xxx@xxx.xx>: 451 4.3.0 <xxx@xxx.xx> Temporary internal error
    I have submitted a ticket to support but as this is affecting a lots of clients i want it resolved as soon as possible and trying to get help wherever i can.
     
  11. keat63

    keat63 Well-Known Member

    Joined:
    Nov 20, 2014
    Messages:
    1,209
    Likes Received:
    77
    Trophy Points:
    28
    cPanel Access Level:
    Root Administrator
  12. Niclas Hedfors

    Niclas Hedfors Registered

    Joined:
    Feb 26, 2019
    Messages:
    4
    Likes Received:
    0
    Trophy Points:
    1
    Location:
    Kalmar, Sweden
    cPanel Access Level:
    Root Administrator
    Tried to edit my last post but apparently it was spam-related?

    It's quota related.
    As I change the quota for the cPanel-account to "unlimited" it starts to work properly.
    For now, i've changed the quota on all accounts to "unlimited", so its not that urgent any more, but i still want a solution to this.
     
  13. Niclas Hedfors

    Niclas Hedfors Registered

    Joined:
    Feb 26, 2019
    Messages:
    4
    Likes Received:
    0
    Trophy Points:
    1
    Location:
    Kalmar, Sweden
    cPanel Access Level:
    Root Administrator
  14. keat63

    keat63 Well-Known Member

    Joined:
    Nov 20, 2014
    Messages:
    1,209
    Likes Received:
    77
    Trophy Points:
    28
    cPanel Access Level:
    Root Administrator
    If you could update this thread with your ticket number, one of the mods will update us with the outcome.
    It may help someone in the future or coming days.
    Seems odd to get two very similar errors in such close proximity.
     
  15. Niclas Hedfors

    Niclas Hedfors Registered

    Joined:
    Feb 26, 2019
    Messages:
    4
    Likes Received:
    0
    Trophy Points:
    1
    Location:
    Kalmar, Sweden
    cPanel Access Level:
    Root Administrator
    My support ticket number: 11530727

    According to the maillog, there is some problem with an NFS mount on our server that not report quota at all. What we can see in the maillog this didn't work before the update either, but it seems like it is handled a bit different after the update.
    Got this from cPanel Support:
    Locating it to a quota problem made me solve it temporary by set all customers to "unlimited" as i wrote earlier.
    All customers are up and running.
    For now, since quota never worked properly, having customers on unlimited quota practically doesn't make much of a differens.

    I will try to resolve the NFS part, with my colleagues who know more about the NFS connection, to make the quota work properly.
     
  16. cPanelMichael

    cPanelMichael Technical Support Community Manager Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    47,009
    Likes Received:
    2,123
    Trophy Points:
    363
    cPanel Access Level:
    Root Administrator
    Twitter:
    Hello Everyone,

    We published the following Dovecot security update in cPanel & WHM version 78.0.10:

    [security] Fixed case CPANEL-25461: Update cpanel-dovecot to 2.3.4.1-1 for CVE-2019-3814.

    Included with this update is a change to the way the Dovecot Local Mail Transfer Protocol (LMTP) handles local delivery attempts when it's unable to determine if quotas are enabled. Prior to this change, Dovecot quota errors like the ones below were logged to /var/log/maillog:

    Prior to the update, these errors were non-fatal and the local delivery attempt succeeded.

    The updated Dovecot version now considers these types of quota check errors to be fatal and thus aborts deliveries with an error message like this in /var/log/exim_mainlog:

    We're currently evaluating reports of this issue as part of internal case CPANEL-26002. I'll monitor this case and update this thread with more information as it becomes available.

    In the meantime, the workaround is to address the underlying cause of the quota errors seen in /var/log/maillog.

    The specific cause of the quota errors can vary from system to system, however most of the support requests we've received thus far stem from the vzquota init script incorrectly running at boot time on virtual servers that were migrated from the vzfs backend to the ploop storage backend.

    You can verify if that's the cause of the issue on your particular server by accessing the server via SSH as root and running the following commands:

    Code:
    chkconfig --list | grep vzquota
    diff -u /etc/mtab /proc/mounts
    The output from the above commands will resemble the following on affected systems:

    Notice that vzquota is set to "on" at run levels 2, 3, 4, and 5 (this confirms it runs at boot time). Then, notice how /etc/mtab uses /dev/simfs, whereas /proc/mounts uses /dev/ploop$$. The difference exists because the vzquota init script incorrectly changes "ploop" to "simfs" in the /etc/mtab file at boot time time (/etc/mtab should instead exist as a symbolic link to /proc/mounts on these systems).

    If the output does not look like this on your system, then it's likely there's a different underlying cause of the quota errors on your system. In such cases, feel free to respond to this thread with the error message you see in /var/log/maillog and we'll be happy to help you diagnose the problem.

    Upon confirming the system is affected by this particular issue, you can run the following commands to fix this:

    Code:
    chkconfig --level 234 vzquota off
    mv -v /etc/mtab{,.old}
    ln -s /proc/mounts /etc/mtab
    reboot
    These commands disable vzquota at boot time, setup /etc/mtab is a symbolic link to /proc/mounts, and reboot the server.

    Let us know if you have any questions.

    Thank you.

    Edit Log:
    03/06/2019: Updated case number to CPANEL-26002.
    03/01/2019: Updated the workaround instructions to note the commands should be ran on the individual server.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
    #16 cPanelMichael, Feb 26, 2019
    Last edited: Mar 6, 2019
  17. deddy

    deddy Well-Known Member

    Joined:
    Oct 13, 2003
    Messages:
    143
    Likes Received:
    1
    Trophy Points:
    168
    To gain some time I did the following:

    vi 20-lmtp.conf

    # Verify quota before replying to RCPT TO. This adds a small overhead.
    # lmtp_rcpt_check_quota = no

    changed to

    # Verify quota before replying to RCPT TO. This adds a small overhead.
    lmtp_rcpt_check_quota = no

    service dovecot restart
     
    Mohamed Mohsen likes this.
  18. cPanelMichael

    cPanelMichael Technical Support Community Manager Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    47,009
    Likes Received:
    2,123
    Trophy Points:
    363
    cPanel Access Level:
    Root Administrator
    Twitter:
    Hello @deddy,

    While this may act as a temporary workaround, keep in mind this is an untested change and can potentially lead to additional delivery issues if a cPanel account or individual email account is at it's quota limit.

    Thank you.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  19. shaun

    shaun Well-Known Member Verifed Vendor PartnerNOC

    Joined:
    Nov 9, 2001
    Messages:
    708
    Likes Received:
    1
    Trophy Points:
    318
    Location:
    San Clemente, Ca
    cPanel Access Level:
    DataCenter Provider
    Twitter:
    We've seen this issue too recently on a number of customer servers. The cause so far looks to be permissions related, mainly ownership. Here's a the fix we've been using...

    Run the following command
    Code:
    /scripts/mailperm --verbose username |grep "ownership" | grep "should be"
    
    If you see output that looks simliar to the following

    Code:
    Unable to fix ownership on /home/username/mail/domain.tld/.Sent/cur/1484153449.M120071P468622.server1.domain.tld,S=64721,W=65681:2,S: currently (556:556), should be (618:618)
    
    Then you need to correct the permissions on that account. The important part of the output is the "should be" part with the numbers in the parenthesis. In my example thats 556:556 and 618:618. You can then do the following

    Code:
    cd /home/username/mail
    find ./ -user 556 -exec chown 616 {} \;
    find ./ -group 556 -exec chgrp 618  {} \;
    
    Now you should run /scripts/mailperm --verbose username two more times. The first time you may see output showing the script fixing permissions. The second run you should not see any output. If you do, there are more issues that need to be fixed.

    Hope that helps
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  20. cPanelMichael

    cPanelMichael Technical Support Community Manager Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    47,009
    Likes Received:
    2,123
    Trophy Points:
    363
    cPanel Access Level:
    Root Administrator
    Twitter:
    Hi @shaun,

    Can you access one of the affected servers and look at the /etc/fstab file to see if the root filesystem (e.g. "/") is associated with a device that doesn't exist? For instance, many of the affected systems we've seen have an entry like this in their /etc/fstab file despite /dev/simfs not existing as a device:

    Code:
    /dev/simfs / reiserfs rw,usrquota,grpquota 0 0
    Here's a command you can use to help identify if an invalid device name is utilized:

    Code:
    diff -u /etc/mtab /proc/mounts
    Thank you.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
Loading...

Share This Page

  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice