SOLVED cpHulk failed attempts list is now permanently empty?

[Benjamin D.]

Hi, not sure if this is directly related to the server switch I made 2 months ago from CentOS 6 to CentOS 7.5 but cpHulk failed attempts list is now permanently empty, altough cpHulk is activated. I'm used to seeing at least 10 failures an hour from China and such, so there's something wrong. What are the steps to find out why this list now remains empty at all time? I haven't changed any software on the server and for years I've been using ConfigServer Security & Firewall which is still there on my new server. What could cause this?

1) cPHulk is Enabled as seen in the cpHulk page in WHM.

2) cpHulkd is running as seen in top: cPhulkd - processor - dormant mode - accepting connections

 

[Benjamin D.]

Well, I just toggled it off and then on and a minute later I checked the cpHulk history tab and now it has 7 entries, so it works again... no idea what happened, but it works now. You can close this thread, thanks... weird?

 

[cPanelMichael]

Hello @Benjamin D.,

I'm glad to see it's working again. Let us know if you encounter any further problems.

Thank you.

 

[WynandGrobler]

I am also experiencing this exact issue. I have been experiencing it for about 4 months now, and every couple of days, the failed login list is empty. The blocked IP addresses and one day blocks are also empty. The I then have to disable, and enable cpHulk for it to start logging again, but it will only work for a day or 2 before I have to disable and enable it again.

CENTOS 6.10 kvm
v74.0.9

 

[cPanelMichael]

Hello @WynandGrobler,

Have you received any notifications about the cPHulk Daemon failing? Do you notice any output in /usr/local/cpanel/logs/error_log related to cphulk? It's possible the issue you notice relates to a case that's fixed in cPanel & WHM version 76:

Fixed case CPANEL-21626: cPHulk's dbprocessor is more resilient to crashes.

Thank you.

 

[WynandGrobler]

I used "grep hulk" on the log file and went through it. Lines that look like they may be relevant are below:
I'm unsure if this helps. If there is something else I can do to make it easier to see the cause, please let me know.

[2018-08-31 13:05:25 +0200] warn [pureauth] Failed to call hulk pre_login for [email protected] (system) at bin/pureauth.pl line 136.
[2018-08-31 13:07:39 +0200] die [cPhulkd] 1
[2018-08-31 13:07:46 +0200] die [cPhulkd] 1
[2018-08-31 13:07:46 +0200] die [cPhulkd] 1
[2018-08-31 13:07:48 +0200] die [cPhulkd] 1
[2018-08-31 13:07:51 +0200] warn [queueprocd] Bad starting address
[2018-09-03 13:03:11 +0200] die [cPhulkd] 1
[2018-09-03 13:03:32 +0200] die [cPhulkd] 1
[2018-09-03 13:03:33 +0200] die [cPhulkd] 1
[2018-09-03 13:03:48 +0200] warn [queueprocd] Bad starting address
[2018-09-04 09:51:57 +0200] warn [cPhulkd] safekill_single_pid failed to send TERM to pid: 2860: No such process at /usr/local/cpanel/Cpanel/Kill/Single.pm line 71.
        Cpanel::Hulkd::run_daemon(Cpanel::Hulkd=HASH(0x1e287a8)) called at libexec/cphulkd.pl line 32
[2018-09-04 09:52:47 +0200] info [xml-api] Whostmgr::Services::_restart_services: cphulkd
null user passed to hulk.pm can_login: info: Cpanel::Hulk
2018-09-18 09:33:25 +0200] warn [cpaneld] Brute force checking was skipped because cphulkd failed to process “invaliduser” from “145.249.104.232” for the “system” service. at /usr/local/cpanel/Cpanel/Server.pm line 1952.
        Cpanel::Server::connect_cphulkd(Cpanel::Server=HASH(0x1c95130)) called at /usr/local/cpanel/Cpanel/Server.pm line 497
[2018-09-19 10:50:52 +0200] die [cPhulkd] 1
[2018-09-19 10:50:55 +0200] die [cPhulkd] 1
[2018-09-19 10:50:56 +0200] die [cPhulkd] 1
[2018-09-19 10:50:59 +0200] die [cPhulkd] 1
[2018-09-19 10:51:01 +0200] die [cPhulkd] 1
[2018-09-19 10:51:04 +0200] die [cPhulkd] 1
[2018-09-19 10:51:04 +0200] die [cPhulkd] 1
[2018-09-19 10:51:06 +0200] die [cPhulkd] 1
[2018-09-19 10:51:06 +0200] die [cPhulkd] 1
[2018-09-19 10:51:08 +0200] die [cPhulkd] 1
[2018-09-19 10:51:08 +0200] die [cPhulkd] 1
[2018-09-19 10:51:11 +0200] die [cPhulkd] 1
[2018-09-19 10:51:12 +0200] die [cPhulkd] 1
[2018-09-19 10:51:14 +0200] die [cPhulkd] 1
[2018-09-19 10:51:16 +0200] die [cPhulkd] 1
[2018-09-19 10:51:18 +0200] die [cPhulkd] 1
[2018-09-19 10:51:22 +0200] die [cPhulkd] 1
[2018-09-19 10:51:23 +0200] die [cPhulkd] 1
[2018-09-19 10:51:26 +0200] die [cPhulkd] 1
[2018-09-19 10:51:30 +0200] die [cPhulkd] 1
[2018-09-19 10:51:30 +0200] die [cPhulkd] 1
[2018-09-19 10:51:31 +0200] die [cPhulkd] 1
[2018-09-19 10:51:32 +0200] warn [queueprocd] Bad starting address
        Cpanel::Server::connect_cphulkd(Cpanel::Server=HASH(0x1d1f6d0)) called at /usr/local/cpanel/Cpanel/Server.pm line 1574
        Cpanel::Server::check_hulk_reject_login_and_exit_if_brute(Cpanel::Server=HASH(0x1d1f6d0), "authok", 1, "user", "root") called at /usr/local/cpanel/Cpanel/Server.pm line 1987
        Cpanel::Server::do_hulk_checks_after_successful_auth_if_needed(Cpanel::Server=HASH(0x1d1f6d0), "authok", 1, "user", "root") called at cpsrvd.pl line 5419
[2018-09-19 13:05:06 +0200] warn [cPhulkd] safekill_single_pid failed to send TERM to pid: 29910: No such process at /usr/local/cpanel/Cpanel/Kill/Single.pm line 71.
        Cpanel::Hulkd::run_daemon(Cpanel::Hulkd=HASH(0xbc2e60)) called at libexec/cphulkd.pl line 32
null user passed to hulk.pm can_login: info: Cpanel::Hulk
[2018-10-01 08:19:23 +0200] info [xml-api] Whostmgr::Services::_restart_services: cphulkd
[2018-10-01 08:43:43 +0200] warn [cpaneld] Brute force checking was skipped because cphulkd failed to process “validuser” from “x.x.x.x” for the “system” service. at /usr/local/cpanel/Cpanel/Server.pm line 1952.
        Cpanel::Server::connect_cphulkd(Cpanel::Server=HASH(0x1b6d378)) called at /usr/local/cpanel/Cpanel/Server.pm line 1574
        Cpanel::Server::check_hulk_reject_login_and_exit_if_brute(Cpanel::Server=HASH(0x1b6d378), "user", "validuser", "authok", 1) called at /usr/local/cpanel/Cpanel/Server.pm line 1987
        Cpanel::Server::do_hulk_checks_after_successful_auth_if_needed(Cpanel::Server=HASH(0x1b6d378), "authok", 1, "user", "validuser") called at cpsrvd.pl line 3470

 

[cPanelMichael]

Hello @WynandGrobler,

Can you open a support ticket so we can take a closer look at your system to see what's happening? You can post the ticket number here and we will update this thread with the outcome.

Thank you.

 

[WynandGrobler]

Thank you. I need the login details to our main cPanel account. Will get it from the MD on Monday and post the ticket number here.

 

[WynandGrobler]

I'm actually struggling to get a ticket logged. I don't actually know if it worked. Server shows open Ticket ID: 10649877 but don't see an open ticket on the portal.

 

[cPanelMichael]

I'm actually struggling to get a ticket logged. I don't actually know if it worked. Server shows open Ticket ID: 10649877 but don't see an open ticket on the portal.

Hello @WynandGrobler,

I do see the login details were verified. Can you login to Manage2 Login and verify if you see the ticket there?

Thank you.

 

[WynandGrobler]

I don't have access to that portal. Our login details don't work there.
I resubmitted the ticket from the tickets.cpanel.net page and had to generate a new SSH key.
Support Request ID is: 10657107
It has logged successfully now.

 

[WynandGrobler]

Amanda has been looking at the ticket and confirmed this error indicates that the issue is related to an issue that was fixed in cPanel 76.

connect(/var/run/cphulkd_db.sock): Connection refused
The socket is not setup in the Cpanel::Hulk object. Do you need to call connect() first? at /usr/local/cpanel/Cpanel/Hulk.pm line 118, <GEN24332> line 36.

An internal case has been filed regarding this behavior under case ID: CPANEL-21626 and has a fix that has been applied in cPanel version 76 which should be hitting the release update tier shortly.

The temporary work around this issue is to restart queueprocd and then the cphulkd services. I have gone ahead and done this on your server at this time.

So for now, I will just need to regularly restart the services until the update reaches release update tier.

Thank you for your assistance.

 

[cPanelMichael]

Hello @WynandGrobler,

Thank you for sharing the outcome. To summarize, internal case CPANEL-21626 is included with cPanel & WHM version 76 and should address the reported issue:

Fixed case CPANEL-21626: cPHulk's dbprocessor is more resilient to crashes.

Version 76 is tentatively planned for publication to the RELEASE build tier this week.

Thank you.