Failed: sshd after /upcp

[kernow]

Were getting emails about sshd failing on several of our servers, this started shortly after upcp ran. Anyone else having same?

Service check failed to completesshd: [ != SSH]
refused connect from 127.0.0.1 (127.0.0.1)
Server listening on 0.0.0.0 port xxx

 

[ES - George]

Not seen this on any of our boxes before. Have you tried restarting SSH?

 

[Veeble-Adam]

Sounds strange. Anything in the logs?

You may try a forceful upcp. /scripts/upcp --force

 

[kernow]

We did try a restart of sshd but it didn't fix the problem, its occurring on a few boxes. However ssh is not failing as we can login at any time and on two boxes we have left top -s running during the ssh failure emails. Interestingly the cpanel service email contains a line giving the number of restarts attempted and this matches the number of emails received, so the first email said ssh failed with one restart attempted and the 27th email says restarts attempted 27 etc.
The logs contain the same info the email does, ie ;

Service check failed to completesshd: [ != SSH]
refused connect from 127.0.0.1 (127.0.0.1)
Server listening on 0.0.0.0 port xxx

 

[MTBiker]

I've also started receiving these email since last night but SSH is working fine

I've restarted the ssh service using "service sshd restart" and it restarts fine, in cpanel I get this error:

Restarting SSH Daemon

Waiting for sshd to restart....finished.

sshd (/usr/sbin/sshd) running as root with PID 7170 (pidfile check method)

sshd: [ != SSH]
sshd has failed, please contact the sysadmin.

I restarted the server and also tried "/scripts/upcp --force" but made no difference.

Here is the email I receive:

Server:
x.x.x

Primary IP:
x.x.x.x

Service:
sshd

Notification Type:
failed

Notification:
sshd failed @ Wed May 8 12:56:20 2013. A restart was attempted automagically.

Service Check Method:
[socket connect]

Reason:
Service check failed to completesshd: [ != SSH]



Number of Restart Attempts:
54

Startup Log:
Starting sshd: [ OK ]



Syslog Messages:
May 8 12:56:20 dedicated sshd[6458]: refused connect from 127.0.0.1 (127.0.0.1)
May 8 12:52:13 dedicated sshd[5979]: refused connect from 127.0.0.1 (127.0.0.1)
May 8 12:52:13 dedicated sshd[5977]: Server listening on 0.0.0.0 port 22.
May 8 12:52:13 dedicated sshd[5977]: Server listening on :: port 22.
May 8 12:52:13 dedicated sshd[5246]: Received signal 15; terminating.


Memory Information:
Used: 905MB
Available: 1065MB
Installed: 1970MB


Load Information:
0.18 1.01 1.59

Uptime:
0 days, 4 hours, 29 seconds

IOStat Information:
avg-cpu: %user %nice %system %iowait %steal %idle
5.29 0.22 2.13 5.35 0.00 87.00
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 1.15 22.09 55.17 357473 892792
sda1 0.01 0.26 0.00 4160 8
sda2 0.00 0.08 0.02 1296 320
sda3 1.02 6.50 55.15 105185 892464
sdb 84.01 2050.23 1482.66 33175848 23991744
sdb1 83.92 2039.94 1482.66 33009338 23991744
sdb2 0.00 0.08 0.00 1360 0
sdb3 0.00 0.05 0.00 814 0


ChkServd Version:
15.2

I also love the my server is "automagically" restarted and not "automatically", sounds like there is magic involved

 

[tommy_pap]

I am having the same problem since last nights update.

I am also getting "The security token is missing from your request." error when refreshing a WHM Login or WebMial Login.

 

[ochlocracy]

Another one with the same issue. Started getting sshd FAILED texts in the middle of the night, likely after upcp ran.

sshd is running fine. Disabled monitoring in the meantime.

additional info:
It's reporting the failure with the primary IP, but we run sshd on a single IP that is not the primary server IP. We also moved it from port 22. If this is related to anyone else's issues. csf+lfd is also on this server, therefore controlling access to sshd via the ip and port.

 

[schoeps]

Another one with the same issue. Started getting sshd FAILED texts in the middle of the night, likely after upcp ran.

sshd is running fine. Disabled monitoring in the meantime.

additional info:
It's reporting the failure with the primary IP, but we run sshd on a single IP that is not the primary server IP. We also moved it from port 22. If this is related to anyone else's issues. csf+lfd is also on this server, therefore controlling access to sshd via the ip and port.

Getting the same messages on 2/5 of our servers, should we wait for a patch?

 

[CitizenK]

Can you please try adding this line above any deny rules in WHM Home »Security Center »Host Access Control to see if whitelisting Local connections resolves this issue.

sshd LOCAL allow
sshd 127.0.0.1 allow

If LOCAL is not whitelisted the ssh connection in the chksrvd test will fail because localhost will not be able to connect to the ssh server if you have an 'sshd ALL deny' rule in Host Access Control.

Please then wait 5 minutes to allow for chksrvd checks to re-run, and then check the 'Server Status' page to verify the sshd check ran successfully.

Please let us know if this resolves your issue.

Thanks!

 

[MTBiker]

Thanks cpCitizenK, that tip works for me

 

[kernow]

If LOCAL is not whitelisted the ssh connection in the chksrvd test will fail because localhost will not be able to connect to the ssh server if you have an 'sshd ALL deny' rule in Host Access Control.
Thanks!

White listing LOCAL was never needed before, what changed in the last /upcp ?

 

[schoeps]

worded on one, not the other

 

[kernow]

Nope, the suggested fix from cpCitizenK has not worked for us

 

[CitizenK]

Previously chksrvd ssh check only checked for sshd in the process list. In 11.38 cPanel is now attempting to make a connection to the sshd service to ensure that the service is up. Because of this if you have an deny all statement in your host access control without whitelisting local connections, host access control is blocking the chksrvd sshd check in some cases.

If the above fix did not work for you can you please open a ticket using the ticket link in my signature and post your ticket number here so we can investigate this issue further. In your support ticket make sure to reference this forum thread.

Thanks,

 

[kernow]

Ticket # 4089155

 

[kernow]

OK, fixed thanks to cpanel tech Juan-Carlos Newton. Make sure to add to /etc/hosts.allow:
sshd : 127.0.0.1 : allow
sshd : local : allow

 

[schoeps]

OK, fixed thanks to cpanel tech Juan-Carlos Newton. Make sure to add to /etc/hosts.allow:
sshd : 127.0.0.1 : allow
sshd : local : allow

This worked