SOLVED Chkservd continues restarting Redis

[orvisoft]

I had followed this forum post

chkserv with one application and multiple processes

Let's say I've got 3 redis servers running on 1 machine. Each runs on a different port (1111, 2222, 3333). I create each chkserv.d file as follows: echo "service[redis_1111]=x,x,x,systemctl restart redis_1111.service,redis-server,redis_user" > /etc/chkserv.d/redis_1111 echo...

forums.cpanel.net

And created monitoring for my redis service as

echo "service[redis]=6379,QUIT,220,systemctl restart redis.service,redis-server,redis" > /etc/chkserv.d/redis

The problem that is really annoying is that system continues throwing errors in

/var/log/chkservd.log

With messages like this

[check command:+][socket connect:-][socket failure threshold:163/3][fail count:161]Restarting redis....

I am now at blind spot to fix this continued issue in service manager. If I remove the monitoring, the Redis keeps working fine, but the purpose of adding in monitoring is to ensure the 100% availability of services in absence of administrator which is why the chkserv is there.

Any help is really appreciated.

 

[cPRex]

Hey there! From the server itself, are you able to reach port 6379? If not, that would cause the service check to fail.

We also have more official documentation on those settings here:

TailWatch | cPanel & WHM Documentation

The TailWatch driver monitors a log file for certain activities and then performs actions based on the activity.

docs.cpanel.net

 

[orvisoft]

Hi,

Yes my application is able to reach on port 6379 and application is working like a charm.

From

[[email protected] serviceauth]# netstat -tulnp | grep "6379"
tcp        0      0 127.0.0.1:6379          0.0.0.0:*               LISTEN      14374/redis-server
[[email protected] serviceauth]#

And also the lsof lists this successfully

[[email protected] serviceauth]# lsof -i | grep ":6379"
redis-ser 14374         redis    4u  IPv4 12664598      0t0  TCP localhost:6379 (LISTEN)
redis-ser 14374         redis    7u  IPv4 12682030      0t0  TCP localhost:6379->localhost:46860 (ESTABLISHED)
redis-ser 14374         redis    8u  IPv4 12684077      0t0  TCP localhost:6379->localhost:46862 (ESTABLISHED)
redis-ser 14374         redis   11u  IPv4 12676978      0t0  TCP localhost:6379->localhost:45950 (ESTABLISHED)
redis-ser 14374         redis   15u  IPv4 12679730      0t0  TCP localhost:6379->localhost:45926 (ESTABLISHED)
redis-ser 14374         redis   16u  IPv4 12679733      0t0  TCP localhost:6379->localhost:45928 (ESTABLISHED)
redis-ser 14374         redis   19u  IPv4 12676980      0t0  TCP localhost:6379->localhost:45952 (ESTABLISHED)
ea-php72  15217  example    3u  IPv4 12683668      0t0  TCP localhost:45926->localhost:6379 (ESTABLISHED)
ea-php72  15217  example    4u  IPv4 12683669      0t0  TCP localhost:45928->localhost:6379 (ESTABLISHED)
ea-php72  15217  example    8u  IPv4 12682526      0t0  TCP localhost:45950->localhost:6379 (ESTABLISHED)
ea-php72  15217  example    9u  IPv4 12682527      0t0  TCP localhost:45952->localhost:6379 (ESTABLISHED)
httpd     15241  example   19u  IPv4 12680104      0t0  TCP localhost:46860->localhost:6379 (ESTABLISHED)
httpd     15241  example   20u  IPv4 12680105      0t0  TCP localhost:46862->localhost:6379 (ESTABLISHED)
[[email protected] serviceauth]#

Also the telnet have same results as expected

[[email protected] serviceauth]# telnet localhost 6379
Trying ::1...
telnet: connect to address ::1: Connection refused
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.

This is the only reason why I am at blind spot.

Hey there! From the server itself, are you able to reach port 6379? If not, that would cause the service check to fail.

We also have more official documentation on those settings here:

TailWatch | cPanel & WHM Documentation

The TailWatch driver monitors a log file for certain activities and then performs actions based on the activity.

docs.cpanel.net

 

[cPRex]

Thanks for the additional details. I do agree that the command you've provided should work as expected. You may want to consider submitting a ticket to our team so we can check that directly on the server, as this could indicate an issue with chkservd. If you do submit a ticket please post the number here so I can follow along and keep this thread updated.

 

[ffeingol]

It's just a guess, but is it possible that the issue because Redis is only listing on localhost? It's hard to tell if chkservd is connecting to localhost or the server IP for the checks.

 

[orvisoft]

I had opened a ticket with ticket id #94225166

Thanks for the additional details. I do agree that the command you've provided should work as expected. You may want to consider submitting a ticket to our team so we can check that directly on the server, as this could indicate an issue with chkservd. If you do submit a ticket please post the number here so I can follow along and keep this thread updated.

Yeah I believe so, but I am unsure how to fix this as my application works as expected without any issue.

It's just a guess, but is it possible that the issue because Redis is only listing on localhost? It's hard to tell if chkservd is connecting to localhost or the server IP for the checks.

 

[orvisoft]

So from ticket, the solution is to validate the response.

cPanel team input is as below. (Might be useful for peoples searching for a solution to similar issues)

I can see that the config file appears to be using an improper command to check the service, and it is expecting an incorrect response. With redis, you will want to ensure that the proper call and check commands are used in the command.​

[07:24:26 m2production [email protected] ~]cPs# redis-cli ping[/INDENT]
[INDENT]PONG

These are the proper call and response for redis. You will want to ensure that you alter your config to include these so that when the call is made, it is returning the proper response.​

​

https://docs.cpanel.net/knowledge-base/cpanel-product/tailwatch/#add-a-new-monitor​

​

For example on what you should be looking to change the config to, this is another config that is working, but they have a custom file for the restartsrv script in their command.​

[16:49:22 [email protected][REDACTED] ~]cPs# cat /etc/chkserv.d/redis[/INDENT]
[INDENT]service[redis]=6379,ping,\+PONG,/scripts/restartsrv_redis,redis,redis

Based on the feedback provided by cPanel team, I made following configuration changes and its working like a charm. Can see its running from past 7 hours approx.

echo "service[redis]=6379,ping,\+PONG,systemctl restart redis.service,redis,redis" > /etc/chkserv.d/redis

 

[cPRex]

I'm glad we were able to help get that resolved!