[Case 82261] 11.40 incomplete/hung update [symptom: exim & imap services down]

[x-origin]

I have a problem with the new version 11.40.0, the services exim & imap are down. Is there any solution for this issue?
I quote the log from the upgrade report.

LOG 

[20131106.120948]   Detected version '11.40.0.16' from version file.
[20131106.120948]   Using mirror '74.200.65.162' for host 'httpupdate.cpanel.net'.
[20131106.120949]   Target version set to '11.40.0.16'
[20131106.120949]   Checking license
[20131106.120950]   License file check complete
[20131106.120950]   Checking that the RPM DB is OK...
[20131106.120951]   glibc-2.5-118.el5_10.2
[20131106.120951]   glibc-2.5-118.el5_10.2
[20131106.120951]   OK: RPM DB is responding to queries
[20131106.120951]   Testing if rpm_is_working RPM is installed
[20131106.120951]   package rpm_is_working is not installed
[20131106.120951]   Testing if it's possible to install a simple RPM
[20131106.120951]   Preparing...                ##################################################
[20131106.120951]   rpm_is_working              ##################################################
[20131106.120952]   Switching to version '11.40.0.16' of updatenow to perform sync.
[20131106.120955]   All Needed RPMS are installed.
[20131106.120955]   Staging 11.40.0.16 cpanelsync files prior to updating /usr/local/cpanel
[20131106.120955]   Staging x3 theme changes
[20131106.120955]   Retrieving and staging /cpanelsync/11.40.0.16/x3/.cpanelsync.bz2
[20131106.120956]   Using mirror '159.253.142.50' for host 'httpupdate.cpanel.net'.
[20131106.120956]   Staging files for x3
[20131106.120958]   Retrieving and staging /cpanelsync/11.40.0.16/x3/js/filemanager_editors_optimized-es_es.js.bz2
[20131106.120958]   Retrieving and staging /cpanelsync/11.40.0.16/x3/js2/ssl/install-es_es.js.bz2
[20131106.120959]   Retrieving and staging /cpanelsync/11.40.0.16/x3/js2/mail/delegatelist-es_es.js.bz2
[20131106.121000]   Retrieving and staging /cpanelsync/11.40.0.16/x3/js2/webdav/accounts_webdav-es_es.js.bz2
[20131106.121001]   Retrieving and staging /cpanelsync/11.40.0.16/x3/js2/ssl/crts-es_es.js.bz2
[20131106.121003]   Retrieving and staging /cpanelsync/11.40.0.16/x3/js/filemanager_editors-es_es.js.bz2
[20131106.121004]   Retrieving and staging /cpanelsync/11.40.0.16/x3/js2/ssl/csrs-es_es.js.bz2
[20131106.121004]   Retrieving and staging /cpanelsync/11.40.0.16/x3/js2/mail/archive-es_es.js.bz2
[20131106.121005]   Retrieving and staging /cpanelsync/11.40.0.16/x3/js2/mail/filters/editfilter-es_es.js.bz2
[20131106.121005]   Retrieving and staging /cpanelsync/11.40.0.16/x3mail/.cpanelsync.bz2
[20131106.121005]   Using mirror '159.253.142.50' for host 'httpupdate.cpanel.net'.
[20131106.121005]   Staging files for x3mail
[20131106.121005]   Staging cpanel changes
[20131106.121005]   Retrieving and staging /cpanelsync/11.40.0.16/binaries/linux-x86_64/.cpanelsync.bz2
[20131106.121005]   Using mirror '159.253.142.50' for host 'httpupdate.cpanel.net'.
[20131106.121005]   Retrieving and staging /cpanelsync/11.40.0.16/cpanel/.cpanelsync.bz2
[20131106.121005]   Staging files for binaries/linux-x86_64
[20131106.121006]   Staging files for cpanel
[20131106.121008]   Retrieving and staging /cpanelsync/11.40.0.16/cpanel/base/sharedjs/emailstats_search-es_es.js.bz2
[20131106.121009]   Retrieving and staging /cpanelsync/11.40.0.16/cpanel/whostmgr/docroot/js2/managehooks-es_es.js.bz2
[20131106.121012]   Retrieving and staging /cpanelsync/11.40.0.16/cpanel/base/cjt/cpanel-all-min-es_es.js.bz2
[20131106.121013]   Retrieving and staging /cpanelsync/11.40.0.16/cpanel/base/cjt/ajaxapp-es_es.js.bz2
[20131106.121016]   Retrieving and staging /cpanelsync/11.40.0.16/cpanel/whostmgr/docroot/js2/manageservicecrts-es_es.js.bz2
[20131106.121016]   Retrieving and staging /cpanelsync/11.40.0.16/cpanel/base/cjt/datatable-es_es.js.bz2
[20131106.121019]   Retrieving and staging /cpanelsync/11.40.0.16/cpanel/whostmgr/docroot/js2/minpwstrength-es_es.js.bz2
[20131106.121021]   Retrieving and staging /cpanelsync/11.40.0.16/cpanel/base/cjt/password-es_es.js.bz2
[20131106.121022]   Retrieving and staging /cpanelsync/11.40.0.16/cpanel/base/sharedjs/sslinstall-es_es.js.bz2
[20131106.121023]   Retrieving and staging /cpanelsync/11.40.0.16/cpanel/whostmgr/docroot/templates/backup/restore.min-es_es.js.bz2
[20131106.121024]   Retrieving and staging /cpanelsync/11.40.0.16/cpanel/base/sharedjs/email_ui_control-es_es.js.bz2
[20131106.121024]   Retrieving and staging /cpanelsync/11.40.0.16/cpanel/whostmgr/docroot/js2/listsslhosts-es_es.js.bz2
[20131106.121026]   Retrieving and staging /cpanelsync/11.40.0.16/cpanel/scripts/check_cpanel_rpms.bz2
[20131106.121026]   Set permissions on /usr/local/cpanel/scripts/check_cpanel_rpms-cpanelsync to 0755
[20131106.121027]   Retrieving and staging /cpanelsync/11.40.0.16/cpanel/whostmgr/docroot/templates/backup/restore-es_es.js.bz2
[20131106.121028]   Retrieving and staging /cpanelsync/11.40.0.16/cpanel/base/sharedjs/email_ui_control_optimized-es_es.js.bz2
[20131106.121034]   Retrieving and staging /cpanelsync/11.40.0.16/cpanel/whostmgr/docroot/templates/accounts/createacct-es_es.js.bz2
[20131106.121036]   Retrieving and staging /cpanelsync/11.40.0.16/cpanel/whostmgr/docroot/templates/backup/users.min-es_es.js.bz2
[20131106.121037]   Retrieving and staging /cpanelsync/11.40.0.16/cpanel/base/sharedjs/fetch_mail_queue-es_es.js.bz2
[20131106.121038]   Retrieving and staging /cpanelsync/11.40.0.16/cpanel/base/sharedjs/form_submit_handler_optimized-es_es.js.bz2
[20131106.121041]   Retrieving and staging /cpanelsync/11.40.0.16/cpanel/base/sharedjs/emailstats_summary-es_es.js.bz2
[20131106.121041]   Retrieving and staging /cpanelsync/11.40.0.16/cpanel/whostmgr/docroot/js2/edituser-es_es.js.bz2
[20131106.121044]   Retrieving and staging /cpanelsync/11.40.0.16/cpanel/whostmgr/docroot/templates/accounts/createacct.min-es_es.js.bz2
[20131106.121044]   Retrieving and staging /cpanelsync/11.40.0.16/cpanel/base/sharedjs/ssltable-es_es.js.bz2
[20131106.121044]   Retrieving and staging /cpanelsync/11.40.0.16/cpanel/base/sharedjs/fetch_mail_queue_optimized-es_es.js.bz2
[20131106.121044]   Retrieving and staging /cpanelsync/11.40.0.16/cpanel/base/cjt/widgets-es_es.js.bz2
[20131106.121044]   Retrieving and staging /cpanelsync/11.40.0.16/cpanel/base/sharedjs/ssltable_optimized-es_es.js.bz2
[20131106.121046]   Retrieving and staging /cpanelsync/11.40.0.16/cpanel/base/sharedjs/emailstats_summary_optimized-es_es.js.bz2
[20131106.121046]   Retrieving and staging /cpanelsync/11.40.0.16/cpanel/base/sharedjs/sslinstall_optimized-es_es.js.bz2
[20131106.121047]   Retrieving and staging /cpanelsync/11.40.0.16/cpanel/whostmgr/docroot/templates/backup/config-es_es.js.bz2
[20131106.121047]   Retrieving and staging /cpanelsync/11.40.0.16/cpanel/whostmgr/docroot/templates/listips/listips-es_es.js.bz2
[20131106.121048]   Retrieving and staging /cpanelsync/11.40.0.16/cpanel/base/sharedjs/sslwidgets-es_es.js.bz2
[20131106.121048]   Retrieving and staging /cpanelsync/11.40.0.16/cpanel/base/cjt/api-es_es.js.bz2
[20131106.121051]   Retrieving and staging /cpanelsync/11.40.0.16/cpanel/base/cjt/ajaxapp-min-es_es.js.bz2
[20131106.121052]   Retrieving and staging /cpanelsync/11.40.0.16/cpanel/base/sharedjs/emailstats_search_optimized-es_es.js.bz2
[20131106.121052]   Retrieving and staging /cpanelsync/11.40.0.16/cpanel/base/sharedjs/sslwidgets_optimized-es_es.js.bz2
[20131106.121052]   Retrieving and staging /cpanelsync/11.40.0.16/cpanel/base/sharedjs/form_submit_handler-es_es.js.bz2
[20131106.121052]   Retrieving and staging /cpanelsync/11.40.0.16/cpanel/whostmgr/docroot/templates/backup/users-es_es.js.bz2
[20131106.121053]   Retrieving and staging /cpanelsync/11.40.0.16/cpanel/whostmgr/docroot/js2/force_password_change-es_es.js.bz2
[20131106.121053]   Retrieving and staging /cpanelsync/11.40.0.16/cpanel/whostmgr/docroot/templates/listips/listips.min-es_es.js.bz2
[20131106.121053]   Retrieving and staging /cpanelsync/11.40.0.16/cpanel/whostmgr/docroot/js2/addipform-es_es.js.bz2
[20131106.121054]   Retrieving and staging /cpanelsync/11.40.0.16/cpanel/whostmgr/docroot/templates/backup/config.min-es_es.js.bz2
[20131106.121054]   Staging new RPMS
[20131106.121055]   Downloading [url]http://httpupdate.cpanel.net/RPM/11.36/centos/5/x86_64/rpm.md5[/url]
[20131106.121055]   cpanel-perl-514-Devel-NYTProf-5.05-1.cp1136.x86_64.rpm already downloaded
[20131106.121055]   All files Staged
[20131106.121055]   Testing if the newly downloaded RPMS can be installed without conflict
[20131106.121055]   Testing RPM transaction
[20131106.121057]   	package cpanel-perl-514-Devel-NYTProf-5.05-1.cp1136.x86_64 is already installed
[20131106.121057] W Exit Code: 2
[20131106.121057] ***** FATAL: Test install failed: 	package cpanel-perl-514-Devel-NYTProf-5.05-1.cp1136.x86_64 is already installed
[20131106.121057]   The Administrator will be notified to review this output when this script completes
[20131106.121057] ***** FATAL: Error testing if the RPMs will install: Died at /usr/local/cpanel/Cpanel/RPM/Versions/File.pm line 850.
[20131106.121057]   see [url=http://go.cpanel.net/rpmcheckfailed]How to Fix RPM Installation Failures in WHM[/url] for more information
[20131106.121057]   The Administrator will be notified to review this output when this script completes
[20131106.121057] E Detected events which require user notification during updatenow. Will send iContact the log

 

[cPanelMichael]

re: [Case 82261] whm/cpanel 11.40.0 update problem with exim & imap services

Hello

Please run the following command and let us know the output:

/scripts/check_cpanel_rpms

Do you notice any error messages when restarting Exim or IMAP? Are there any errors during the restart in the /var/log/exim_paniclog file?

Also, since you have services down, please open a support ticket so we can take a closer look:

Submit A Ticket

Post the ticket number here so we can update this thread with the outcome.

Thank you.

 

[x-origin]

re: [Case 82261] whm/cpanel 11.40.0 update problem with exim & imap services

cPanelMichael thank you for your reply,

The ticket number is 4388071, and i quote all the informations in the ticket.
Special thanks.

 

[cPanelMichael]

re: [Case 82261] whm/cpanel 11.40.0 incomplete/hung update [symptom: exim & imap services]

To update, there is an internal case open to address the issue where Dovecot does not function properly after updating to cPanel version 11.40. The internal case number is 82261. While the specific cause of this issue is not yet known (possibly attributed to issues with RPMs), the current workaround to resolve the issue with Dovecot is to correct the ownership and permission values via:

chown root.dovecot /usr/local/cpanel/bin/dovecot-wrap ; chmod 4750 /usr/local/cpanel/bin/dovecot-wrap

Note: The issue reported with the RPMs on this thread was resolved by removing some duplicate RPMs and running:

/usr/local/cpanel/scripts/check_cpanel_rpms

Thank you.

 

[High Voltage]

re: [Case 82261] whm/cpanel 11.40.0 incomplete/hung update [symptom: exim & imap services]

Thanks cPanelMichael.
Your workaround got my mail server up & running again.

 

[sahostking]

re: [Case 82261] whm/cpanel 11.40.0 incomplete/hung update [symptom: exim & imap services]

Thanks this may have helped

 

[cPanelNick]

re: [Case 82261] whm/cpanel 11.40.0 incomplete/hung update [symptom: exim & imap services]

In the instances we have seen this problem, the update was terminated in the middle of the process (out of memory, server reboot, or killed off in most instances). The permissions are normally set on the dovecot wrapper closer to the end of the update. If the update does not fully complete you can manually fix the problem with the following:

chown root:dovecot /usr/local/cpanel/bin/dovecot-wrap
chmod 4750 /usr/local/cpanel/bin/dovecot-wrap
yum clean all
yum update
/usr/local/cpanel/scripts/check_cpanel_rpms --fix

Manually fix any problems with the rpm database as needed.

/scripts/upcp --force

If you are unable to determine and correct the reason for the update being terminated prematurely, please do open a ticket (link in my signature) so we can help you look into the matter. Its is important that the problem causing the update to be incomplete is resolved to prevent seemingly random problems from occurring. Its very likely that the next major update will not be able to complete as well if the underlying cause is not resolved now. The primary cause we see is the server running out of memory.

 

[tss]

re: [Case 82261] whm/cpanel 11.40.0 incomplete/hung update [symptom: exim & imap services]

The exact same problem happened to my server with this update. Never had an update problem before. Here's what happens: update fails, no exim, no imap. I found out about this when the mail server stopped delivering and sending emails. Hundreds of notification emails followed, but because I didn't subscribe on my cell phone to the particular email address they were sent, I didn't see them until much later. Fortunately, it didn't cause any problems with Apache, PHP, MySQL, pure-ftp, or sshd. They kept running, so at least my clients' sites didn't go down.

The first email had the subject of "cPanel update hanging" telling me "/usr/local/cpanel/scripts/upcp" was running for longer than 6 hours. The next email, 10 minutes later, had the subject of "Cpanel version change is blocked", it was fatal, and that my "RPM database appears unstable".

About 12 hours after these messages were sent I noticed that I wasn't receiving expected emails at certain addresses. When I fired up my laptop and tried to log in to webmail on port 2096, I received a white screen instead of my expected SquirrelMail client. I tried a different email address on the same server with the same result. For some reason, only after these failed attempts to log in did I start getting 2 messages, every 5 minutes, from the cPanel Service monitor. "FAILED: imap on <server name and IP address here>" and "FAILED: exim on <server name and IP address here>". The messages was fairly detailed, but didn't really tell me what the underlying problem was. From the imap message:

TCP Transaction Log:
<< * OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE STARTTLS AUTH=PLAIN AUTH=LOGIN] Dovecot ready.
>> A001 LOGIN __cpanel__service__auth__imap__<hash inserted here>
<< A001 NO [UNAVAILABLE] Temporary authentication failure. [<server name>:2013-11-08 19:30:01]
imap: ** [A001 NO [UNAVAILABLE] Temporary authentication failure. [<server name>:2013-11-08 19:30:01] != A001 OK]
: Died at /usr/local/cpanel/Cpanel/TailWatch/ChkServd.pm line 894, <$socket_scc> line 2.

The confusing part was that exim and dovecot services (at least some of them) appeared on the running processes page of WHM.

The exim message was similar. A few details were:

<< 535 Incorrect authentication data
exim: ** [535 Incorrect authentication data != 2]
: Died at /usr/local/cpanel/Cpanel/TailWatch/ChkServd.pm line 894, <$socket_scc> line 11.

and

Starting clamd: [ OK ]
Starting exim: [ OK ]
0 processes (antirelayd) sent signal 9
/usr/local/cpanel/scripts/update_sa_rules: running in background

So exim and dovecot were running, but the authentication service wasn't authenticating.

I finally saw that the update had failed, and hoped that was the cause of the problem, so I attempted a manual forced update. That's when I saw a virtually identical log to the one shown in x-origin's original post above. However, the solution for me came from a Russian forum (cPanel 11.40.0.19 (RELEASE) -) which I found by searching for "Test install failed: package cpanel-perl-514-Devel-NYTProf-5.05-1.cp1136.x86_64 is already installed". Thanks to google translate and someone's comment, I realized that I needed remove the RPMs that were causing the problem with the command "rpm -e".

I tried 2 different manual updates; both failed on different RPMs. The first time the log had the identical message as shown above:

Downloading http://httpupdate.cpanel.net/RPM/11....x86_64/rpm.md5
cpanel-perl-514-Devel-NYTProf-5.05-1.cp1136.x86_64.rpm already downloaded
All files Staged
Testing if the newly downloaded RPMS can be installed without conflict
Testing RPM transaction
package cpanel-perl-514-Devel-NYTProf-5.05-1.cp1136.x86_64 is already installed
W Exit Code: 2
***** FATAL: Test install failed: package cpanel-perl-514-Devel-NYTProf-5.05-1.cp1136.x86_64 is already installed
The Administrator will be notified to review this output when this script completes
***** FATAL: Error testing if the RPMs will install: Died at /usr/local/cpanel/Cpanel/RPM/Versions/File.pm line 850.

So I ran the command:

rpm -e cpanel-perl-514-Devel-NYTProf-5.05-1.cp1136.x86_64.rpm

and tried the update again. It failed again, this time citing 2 problem RPMs:

cpanel-perl-514-Archive-Tar-Builder-1.4-1.cp1136.x86_64
cpanel-git-1.8.3.4-2.cp1136.x86_64

So I ran the commands:

rpm -e cpanel-perl-514-Archive-Tar-Builder-1.4-1.cp1136.x86_64

followed by

rpm -e cpanel-git-1.8.3.4-2.cp1136.x86_64

I tried the manual update again, expecting that I'd see more failures, but I didn't. The update continued until successful completion, and suddenly my mail server was functioning again.

I hope this will help anyone who experiences the frustration of finding their mail services not working after a cPanel update. Perhaps it will help in troubleshooting what the actual issue is, so that the problem can be fixed, or at least the documentation can be updated.

 

[cPanelNick]

re: [Case 82261] whm/cpanel 11.40.0 incomplete/hung update [symptom: exim & imap services]

The first email had the subject of "cPanel update hanging" telling me "/usr/local/cpanel/scripts/upcp" was running for longer than 6 hours. The next email, 10 minutes later, had the subject of "Cpanel version change is blocked", it was fatal, and that my "RPM database appears unstable".

Please do open a ticket so we can determine why the update was hung up on your server to ensure it doesn't happen again next update.