11.30.6.6 — massive cpsrvd-ssl problem

[accretor]

I encountered a serious problem with cpsrvd-ssl on both the cPanel/WHM servers I administer. I have been able to fix the problem, but since I do not understand the cause or solution, I can't be completely sure it'll never recur. I also could not find any discussion of the problem, even though it happened on two similar but unrelated servers and only with 11.30.6.6. I thought discussing it here might help cPanel support and users.

I will monitor this thread, so if there is any further info I can provide, please let me know.

Server 1
xenpv VPS (Slicehost)
CentOS 5.8
1 GB RAM
cPanel/WHM VPS Optimized 11.30.6.6 (upgraded from 11.30.6.3)

root@lush1 [/usr/local/cpanel/logs]# grep -H '' /etc/*release /usr/local/cpanel/version /var/cpanel/envtype
/etc/redhat-release:CentOS release 5.8 (Final)
/usr/local/cpanel/version:11.30.6.6
/var/cpanel/envtype:xen pv
root@lush1 [/usr/local/cpanel/logs]# grep "numacctlist\|conserve\|gzip" /var/cpanel/cpanel.config
numacctlist=30
conserve_memory=1
cpsrvd-gzip=1
root@lush1 [/usr/local/cpanel/logs]# perl -v | grep "v[0-9]"
This is perl, v5.8.8 built for x86_64-linux
root@lush1 [/usr/local/cpanel/logs]# uname -a
Linux lush1.sustainablylush.ca 2.6.35.4-rscloud #8 SMP Mon Sep 20 15:54:33 UTC 2010 x86_64 x86_64 x86_64 GNU/Linux

Server 2
xenpv VPS (Rackspace Cloud Server)
CentOS 6.2
1 GB RAM
cPanel/WHM VPS Optimized 11.30.6.6 (fresh install)

root@harbour1 [/home/pdswoodw/.cpaddons]# grep -H '' /etc/*release /usr/local/cpanel/version /var/cpanel/envtype; grep "numacctlist\|conserve\|gzip" /var/cpanel/cpanel.config; perl -v | grep "v[0-9]"; uname -a
/etc/centos-release:CentOS release 6.2 (Final)
/etc/redhat-release:CentOS release 6.2 (Final)
/etc/system-release:CentOS release 6.2 (Final)
/usr/local/cpanel/version:11.30.6.6
/var/cpanel/envtype:xen pv
numacctlist=30
conserve_memory=1
cpsrvd-gzip=1
This is perl, v5.8.8 built for x86_64-linux
Linux harbour1.wpwebmasters.com 2.6.32-220.4.1.el6.x86_64 #1 SMP Tue Jan 24 02:13:44 GMT 2012 x86_64 x86_64 x86_64 GNU/Linux

Symptoms

• All cPanel/WHM unresponsive when accessed over SSL. They load slowly and often will not load completely.
• cPanel/WHM unaffected over HTTP (non-SSL)
• regular HTTP and HTTPS hosting for sites unaffected
• cpsrvd-ssl refuses to shutdown and has to be terminated: Waiting for cpsrvd-ssl to shutdown ... ........... terminating cpsrvd-ssl .....Done
• rebooting server or restarting cPanel does not fix the problem (may initially be a bit more responsive, but problem reappears within minutes).
• there is no correlation with CPU or memory use (i.e. it is not caused by CPU or memory over-utilization)

Solution

Unsuccessful: I attempted a server reboot. This seemed to improve performance but only briefly.

Unsuccessful: I attempted:

service cpanel restart

What seems to solve the problem is to repeatedly toggle (with saves) Force SSL and restart cPanel. Eventually this caused the problem behaviour to go away and not reoccur (over 24 hours on one server, the other server had the problem before and it's been quite a bit more than 24 hours with no recurrence).

 

[Infopro]

...
I will monitor this thread, so if there is any further info I can provide, please let me know.
...

Have you reported this to cPanel? If you feel there is a problem here you should report it so that cPanel can take a closer look.

http://go.cpanel.net/bugs

If you do, and they fix or find the issue, please do post back here with the result.

Thanks.

 

[accretor]

Thanks for the suggestion, Infopro. My intent is to get the attention of cPanel so that they can investigate further if this could be an issue for a certain class of users (there are some points of commonality between the two servers I had that experienced the problem, but they are not otherwise related — they are not cloned nor do they interact in any way with each other). My secondary intent is to create a record where someone having the problem may have a chance to resolve the issue.

That bugs link is new to me, so I am setting up an account. I wasn't aware that I could report potential bugs in that way, so I was just hoping that someone at cPanel might see this thread and take an interest. I will definitely try to get a response that way. I promise to follow-up in this thread one way or another. I hate it when you go looking for help and find these threads left unresolved from years back.

 

[Infopro]

cPanel tries to not leave anything unresolved if possible. But that does not mean every thread is updated when something gets resolved.

For future reference, the link I provided here in this thread is also on the top of every page of these forums.


Thanks!

 

[accretor]

Ha, so it is. I'm a doofus.

And I wasn't trying to blame cPanel support. I see a lot of cases where they respond to a forum post but the user never follows up or the follow-up is unhelpful, such as a declaration that the user resolved the problem but no information about how the issues was resolved.

 

[Infopro]

And I wasn't trying to blame cPanel support.

Yes of course, I know. I just didn't want to leave this thread unresolved. :p

 

[accretor]

I've not registered (or checked) for bug reports yet. I've been really swamped. But I want to cry because now I'm seeing the same problem on 11.32.1.15 DNSONLY. And DNSONLY doesn't have the Force SSL setting for me to toggle which has fixed other servers so far. So I have no way to fix the problem. It's almost impossible to get anything done in WHM for this server because it's so slow.

Once again:

Stopping cPanel services:  Waiting for cpsrvd to shutdown ... ...Done
Waiting for cpsrvd-ssl to shutdown ... ........... terminating cpsrvd-ssl .....Done

Just sharing and showing that I haven't forgotten about the issue or the thread. I do still instead to ensure that a proper bug report has been submitted.

 

[accretor]

I'm sorry for the delay. The ticket has now been submitted for "Complimentary Support". I chose to use this option instead of attempting to submit it through our cPanel license vendor for a few reasons. I'm not sure how much longer it will take to get a resolution/response as a result of this, but I will relay the response to this thread as soon as I can to update the community.

My hope is that we can get to a root cause, an officially sanctioned work-around (preferably better than mine), and a resolution of the problem in the cPanel package.

I have now confirmed that upgrading to 11.32.2.15 does not resolve the problem, but I have no idea if the problem would affect a fresh install of 11.32.2.15. I've also now confirmed that DNSONLY is also affected.