SOLVED Chkservd continues restarting Redis

Operating System & Version
CENTOS 7.9 kvm
cPanel & WHM Version
92.0.10

orvisoft

Registered
Feb 10, 2021
4
0
1
United Kigdom
cPanel Access Level
Root Administrator
I had followed this forum post

And created monitoring for my redis service as
Bash:
echo "service[redis]=6379,QUIT,220,systemctl restart redis.service,redis-server,redis" > /etc/chkserv.d/redis
The problem that is really annoying is that system continues throwing errors in
Code:
/var/log/chkservd.log
With messages like this
Bash:
[check command:+][socket connect:-][socket failure threshold:163/3][fail count:161]Restarting redis....
I am now at blind spot to fix this continued issue in service manager. If I remove the monitoring, the Redis keeps working fine, but the purpose of adding in monitoring is to ensure the 100% availability of services in absence of administrator which is why the chkserv is there.

Any help is really appreciated.
 
Last edited by a moderator:

cPRex

Jurassic Moderator
Staff member
Oct 19, 2014
4,777
588
273
cPanel Access Level
Root Administrator
Hey there! From the server itself, are you able to reach port 6379? If not, that would cause the service check to fail.

We also have more official documentation on those settings here:

 

orvisoft

Registered
Feb 10, 2021
4
0
1
United Kigdom
cPanel Access Level
Root Administrator
Hi,

Yes my application is able to reach on port 6379 and application is working like a charm.

From

Bash:
[[email protected] serviceauth]# netstat -tulnp | grep "6379"
tcp        0      0 127.0.0.1:6379          0.0.0.0:*               LISTEN      14374/redis-server
[[email protected] serviceauth]#
And also the lsof lists this successfully

Bash:
[[email protected] serviceauth]# lsof -i | grep ":6379"
redis-ser 14374         redis    4u  IPv4 12664598      0t0  TCP localhost:6379 (LISTEN)
redis-ser 14374         redis    7u  IPv4 12682030      0t0  TCP localhost:6379->localhost:46860 (ESTABLISHED)
redis-ser 14374         redis    8u  IPv4 12684077      0t0  TCP localhost:6379->localhost:46862 (ESTABLISHED)
redis-ser 14374         redis   11u  IPv4 12676978      0t0  TCP localhost:6379->localhost:45950 (ESTABLISHED)
redis-ser 14374         redis   15u  IPv4 12679730      0t0  TCP localhost:6379->localhost:45926 (ESTABLISHED)
redis-ser 14374         redis   16u  IPv4 12679733      0t0  TCP localhost:6379->localhost:45928 (ESTABLISHED)
redis-ser 14374         redis   19u  IPv4 12676980      0t0  TCP localhost:6379->localhost:45952 (ESTABLISHED)
ea-php72  15217  example    3u  IPv4 12683668      0t0  TCP localhost:45926->localhost:6379 (ESTABLISHED)
ea-php72  15217  example    4u  IPv4 12683669      0t0  TCP localhost:45928->localhost:6379 (ESTABLISHED)
ea-php72  15217  example    8u  IPv4 12682526      0t0  TCP localhost:45950->localhost:6379 (ESTABLISHED)
ea-php72  15217  example    9u  IPv4 12682527      0t0  TCP localhost:45952->localhost:6379 (ESTABLISHED)
httpd     15241  example   19u  IPv4 12680104      0t0  TCP localhost:46860->localhost:6379 (ESTABLISHED)
httpd     15241  example   20u  IPv4 12680105      0t0  TCP localhost:46862->localhost:6379 (ESTABLISHED)
[[email protected] serviceauth]#
Also the telnet have same results as expected
Bash:
[[email protected] serviceauth]# telnet localhost 6379
Trying ::1...
telnet: connect to address ::1: Connection refused
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
This is the only reason why I am at blind spot.

Hey there! From the server itself, are you able to reach port 6379? If not, that would cause the service check to fail.

We also have more official documentation on those settings here:

 

cPRex

Jurassic Moderator
Staff member
Oct 19, 2014
4,777
588
273
cPanel Access Level
Root Administrator
Thanks for the additional details. I do agree that the command you've provided should work as expected. You may want to consider submitting a ticket to our team so we can check that directly on the server, as this could indicate an issue with chkservd. If you do submit a ticket please post the number here so I can follow along and keep this thread updated.
 

ffeingol

Well-Known Member
PartnerNOC
Nov 9, 2001
599
192
343
cPanel Access Level
DataCenter Provider
It's just a guess, but is it possible that the issue because Redis is only listing on localhost? It's hard to tell if chkservd is connecting to localhost or the server IP for the checks.
 

orvisoft

Registered
Feb 10, 2021
4
0
1
United Kigdom
cPanel Access Level
Root Administrator
I had opened a ticket with ticket id #94225166

Thanks for the additional details. I do agree that the command you've provided should work as expected. You may want to consider submitting a ticket to our team so we can check that directly on the server, as this could indicate an issue with chkservd. If you do submit a ticket please post the number here so I can follow along and keep this thread updated.

Yeah I believe so, but I am unsure how to fix this as my application works as expected without any issue.
It's just a guess, but is it possible that the issue because Redis is only listing on localhost? It's hard to tell if chkservd is connecting to localhost or the server IP for the checks.
 

orvisoft

Registered
Feb 10, 2021
4
0
1
United Kigdom
cPanel Access Level
Root Administrator
So from ticket, the solution is to validate the response.

cPanel team input is as below. (Might be useful for peoples searching for a solution to similar issues)

I can see that the config file appears to be using an improper command to check the service, and it is expecting an incorrect response. With redis, you will want to ensure that the proper call and check commands are used in the command.​

Bash:
[07:24:26 m2production [email protected] ~]cPs# redis-cli ping[/INDENT]
[INDENT]PONG

These are the proper call and response for redis. You will want to ensure that you alter your config to include these so that when the call is made, it is returning the proper response.​
For example on what you should be looking to change the config to, this is another config that is working, but they have a custom file for the restartsrv script in their command.​
Bash:
[16:49:22 [email protected][REDACTED] ~]cPs# cat /etc/chkserv.d/redis[/INDENT]
[INDENT]service[redis]=6379,ping,\+PONG,/scripts/restartsrv_redis,redis,redis

Based on the feedback provided by cPanel team, I made following configuration changes and its working like a charm. Can see its running from past 7 hours approx.

Bash:
echo "service[redis]=6379,ping,\+PONG,systemctl restart redis.service,redis,redis" > /etc/chkserv.d/redis