Server crashed -- many "whmhttps connected from" in kernel log

[Stefaans]

Our one server crashed today. The server has been experiencing stability problems lately, and I have been unable to pinpoint to cause.

On scanning /var/log/messages, I found a several hundreds of lines like the following, spread over a few seconds, immediately before the server went down:

Jul 27 05:16:33 www-14 stunnel[19649]: whmhttps connected from 11.22.33.44:2156
Jul 27 05:16:33 www-14 stunnel[19649]: Connection closed: 86 bytes sent to SSL, 524 bytes sent to socket

The IP address 11.22.33.44 represents my IP address. My browser was open on WHM as root at the time.

Any ideas what is going one here?

 

[chirpy]

Do you have a cPanel DNS cluster? If so, then one of the members may be having problems with dnsadmin looping which would fit this evidence.

 

[Stefaans]

Thanks for the DNS pointer (pun intended). We do have DNS clustering in place: this box (that crashed) and one other. However, the IP address shown in the error messages is that of our office, not the other DNS server.

Could you possibly give me some further advice on where to check for signs of the potential "dnsadmin looping" problem? Should I just check both servers' /var/log/messages files, or are there some other clues that I could follow up on as well.

Thanks

 

[Stefaans]

I just saw this post http://forums.cpanel.net/showthread.php?t=41761

I will follow the steps in there, regardless of what else I can dig up.

 

[chirpy]

It may well have nothing to do with dnsadmin, then. Though it could certainly be a looping script since you were in WHM at the time.

One idea might be to use PRM if you don't already have it installed (though it has its limitations). Another would be to enable the Fork/Bomb protection if that is not enabled (though since WHM runs under root it probably won't help as root is excluded, IIRC).

Lastly, have you checked that the laus rpm is not installed?

 

[Stefaans]

Thanks for the advice Jonathan.

PRM is not installed. I will get working on that shortly

Fork/Bomb protection is already enabled.

Checking for laus, if find

rpm -qa | grep laus
laus-libs-0.1-70RHEL3

Is its presense good or bad? I believe laus stands for "Linux Audit-Subsystem user space tools and daemon".

 

[stasd]

root     11311  0.5  0.3  8064 6464 ?        S    08:34   0:00 dnsadmin - REMOVEZONE - KqxIwE6Hrxe46e8zISP8VNEjorbQhpQd           
root     23448  0.6  0.3  8064 6464 ?        S    08:34   0:00 dnsadmin - REMOVEZONE - uhc7uR1uh5soAIm6hu3BtPZ5feYxL5_D           
root     20236  0.5  0.3  8064 6464 ?        S    08:34   0:00 dnsadmin - REMOVEZONE - aioYlYtJ4NpE7mnb5w1T0xB2LmQz5UVZ           
root      7090  0.6  0.3  8064 6472 ?        S    08:34   0:00 dnsadmin - REMOVEZONE - i3m8MHLSfxdTdPNEuYGGF3pjykvLPzVJ           
root     12561  0.0  0.3 10744 8104 ?        S    08:34   0:00 dnsadmin - REMOVEZONE - KqxIwE6Hrxe46e8zISP8VNEjorbQhpQd           
root     22059  0.0  0.3  8064 6464 ?        S    08:34   0:00 dnsadmin - REMOVEZONE - KqxIwE6Hrxe46e8zISP8VNEjorbQhpQd           
root     20470  0.0  0.3  8064 6464 ?        S    08:34   0:00 dnsadmin - REMOVEZONE - KqxIwE6Hrxe46e8zISP8VNEjorbQhpQd - locking /e
root     11129  0.0  0.3 10744 8100 ?        S    08:34   0:00 dnsadmin - REMOVEZONE - uhc7uR1uh5soAIm6hu3BtPZ5feYxL5_D           
root     30315  0.0  0.3  8064 6464 ?        S    08:34   0:00 dnsadmin - REMOVEZONE - uhc7uR1uh5soAIm6hu3BtPZ5feYxL5_D           
root     18472  0.0  0.3  8064 6464 ?        S    08:34   0:00 dnsadmin - REMOVEZONE - uhc7uR1uh5soAIm6hu3BtPZ5feYxL5_D - locking /e
root      1957  0.0  0.3 10876 8116 ?        S    08:34   0:00 dnsadmin - REMOVEZONE - i3m8MHLSfxdTdPNEuYGGF3pjykvLPzVJ           
root     23481  0.0  0.3  8064 6472 ?        S    08:34   0:00 dnsadmin - REMOVEZONE - i3m8MHLSfxdTdPNEuYGGF3pjykvLPzVJ           
root     29758  0.0  0.3  8064 6472 ?        S    08:34   0:00 dnsadmin - REMOVEZONE - i3m8MHLSfxdTdPNEuYGGF3pjykvLPzVJ - locking /e
root     11408  0.0  0.3 10876 8108 ?        S    08:34   0:00 dnsadmin - REMOVEZONE - aioYlYtJ4NpE7mnb5w1T0xB2LmQz5UVZ           
root     18831  0.0  0.3  8064 6464 ?        S    08:34   0:00 dnsadmin - REMOVEZONE - aioYlYtJ4NpE7mnb5w1T0xB2LmQz5UVZ           
root     11201  0.0  0.3  8064 6464 ?        S    08:34   0:00 dnsadmin - REMOVEZONE - aioYlYtJ4NpE7mnb5w1T0xB2LmQz5UVZ - locking /e
root     11159  0.0  0.2  9728 5784 ?        S    08:34   0:00 whostmgrd - serving 67.15.14.90
root     14325  0.1  0.8 19568 17964 ?       S    08:34   0:00 /usr/local/cpanel/whostmgr/bin/whostmgr2 ./removezone_local
root      2212  0.4  0.3  8064 6472 ?        S    08:34   0:00 dnsadmin - REMOVEZONE - UqDIhHARe7i8Ghef1YMpQ9VjeehO8RVJ           
root      8610  0.0  0.3 10876 8112 ?        S    08:34   0:00 dnsadmin - REMOVEZONE - UqDIhHARe7i8Ghef1YMpQ9VjeehO8RVJ           
root     21560  0.0  0.3  8064 6472 ?        S    08:34   0:00 dnsadmin - REMOVEZONE - UqDIhHARe7i8Ghef1YMpQ9VjeehO8RVJ           
root     20904  0.0  0.3  8064 6472 ?        S    08:34   0:00 dnsadmin - REMOVEZONE - UqDIhHARe7i8Ghef1YMpQ9VjeehO8RVJ - locking /e
root      3480  0.0  0.2  9728 5780 ?        S    08:34   0:00 whostmgrd - serving 67.15.14.90
root     22402  0.1  0.8 19568 17956 ?       S    08:34   0:00 /usr/local/cpanel/whostmgr/bin/whostmgr2 ./removezone_local
root      7154  0.5  0.3  8064 6476 ?        S    08:34   0:00 dnsadmin - REMOVEZONE - KMQiWfMWl6WJxaLlyMqCUu8UpKa1DRjx           
root      2213  0.0  0.3 10876 8124 ?        S    08:34   0:00 dnsadmin - REMOVEZONE - KMQiWfMWl6WJxaLlyMqCUu8UpKa1DRjx           
root     21081  0.0  0.3  8064 6476 ?        S    08:34   0:00 dnsadmin - REMOVEZONE - KMQiWfMWl6WJxaLlyMqCUu8UpKa1DRjx           
root      1560  0.0  0.3  8064 6476 ?        S    08:34   0:00 dnsadmin - REMOVEZONE - KMQiWfMWl6WJxaLlyMqCUu8UpKa1DRjx - locking /e
root     21955  0.0  0.2  9728 5780 ?        S    08:34   0:00 whostmgrd - serving 67.15.14.90
root     11243  0.2  0.8 19568 17960 ?       S    08:34   0:00 /usr/local/cpanel/whostmgr/bin/whostmgr2 ./removezone_local

All servers in the sluster crashing

 

[chirpy]

Is its presense good or bad? I believe laus stands for "Linux Audit-Subsystem user space tools and daemon".

That's it, yes. It's presence often makes cPanel server unstable and you should remove it. Have a search for laus on the forum if you have it on how best to remove it properly.

 

[Stefaans]

Thanks, I am learning something new every day

Seems like laus is not installed or running as a service:

service audit status
audit: unrecognized service

rpm -e laus
error: package laus is not installed

Some laus libraries are installed, and required by other packages

rpm -q --whatrequires laus-libs-0.1-70RHEL3
no package requires laus-libs-0.1-70RHEL3

rpm -e laus-libs-0.1-70RHEL3
error: Failed dependencies:
liblaus.so.1 is needed by (installed) passwd-0.68-3.1
liblaus.so.1 is needed by (installed) pam-0.75-64
liblaus.so.1 is needed by (installed) shadow-utils-4.0.3-23.08
liblaus.so.1 is needed by (installed) vixie-cron-3.0.1-76_EL3

So, innocent or not, the laus libraries probably need to stay