CPHulk locked all access

kcorbin · Feb 4, 2020

We enabled country blocking on our server and CPhulk blocked everything, web and ssh, even blocked access to hosted sites. I tried the following;

Code:

/usr/local/cpanel/bin/cphulk_pam_ctl --disable
/usr/local/cpanel/etc/init/stopcphulkd

but cphulk would restart. I then tried;

Code:

whmapi1 configureservice service=cphulkd enabled=0 monitord=0
iptables -F cphulk && /usr/local/cpanel/3rdparty/bin/sqlite3 /var/cpanel/hulkd/cphulk.sqlite "DELETE FROM login_track;"

and was able to get access to WHM and SSH again, so I have a few questions.

What did I do wrong? I went to Home > Security Center > Cphulk Brute force protection and selected every nation except the US and chose blacklist. I didn't chose anything else, but I'm sure I probably missed something.
Is there a way to reset cphulk through cli back to its defaults? I would like to restart it and use it again.
What can I do to prevent this from happening again?

Thank you all.

cPanelLauren · Feb 5, 2020

Hello,

As far as managing cPHulk we have the following documentation that might be helpful: cPHulk Management on the Command Line | cPanel & WHM Documentation

In order to know how to prevent the issue from occurring again, we'd have to understand what happened specifically, the log files might be helpful for this which can be found here:

/usr/local/cpanel/logs/cphulkd.log
/usr/local/cpanel/logs/cphulkd_errors.log

To manage cPHulk's config ove the CLI you'd need to use the API which can be found here: WHM API 1 Functions - set_cphulk_config_key - Developer Documentation - cPanel Documentation

You can still make changes to the configuration in WHM with cPHulk's IP management disabled though and it might be easiest to make those modifications there.

kcorbin · Feb 6, 2020

Lauren thank you for the reply. Here is the last few entries in the cphulkd_errors.log

Code:

[2020-02-01 21:30:38 -0500] die [cPhulkd] Timeout while waiting for response at /usr/local/cpanel/Cpanel/Hulkd.pm line 487.
        Cpanel::Hulkd::die(Cpanel::Hulkd=HASH(0xb02130), "Timeout while waiting for response") called at /usr/local/cpanel/Cpanel/Hulkd.pm line 417
        Cpanel::Hulkd::__ANON__(__CPANEL_HIDDEN__) called at /usr/local/cpanel/Cpanel/Hulkd/Processor.pm line 289
        eval {...} called at /usr/local/cpanel/Cpanel/Hulkd/Processor.pm line 289
        Cpanel::Hulkd::Processor::run(Cpanel::Hulkd::Processor=HASH(0xe00cf8), undef) called at /usr/local/cpanel/Cpanel/Hulkd.pm line 423
        Cpanel::Hulkd::__ANON__() called at /usr/local/cpanel/3rdparty/perl/528/lib/perl5/cpanel_lib/Try/Tiny.pm line 97
        eval {...} called at /usr/local/cpanel/3rdparty/perl/528/lib/perl5/cpanel_lib/Try/Tiny.pm line 88
        Try::Tiny::try(CODE(0xd50888), Try::Tiny::Catch=REF(0xc655a8)) called at /usr/local/cpanel/Cpanel/Hulkd.pm line 427
        Cpanel::Hulkd::handle_one_connection(Cpanel::Hulkd=HASH(0xb02130), Cpanel::Socket::INET=GLOB(0xc65b18), undef) called at /usr/local/cpanel/Cpanel/Hulkd.pm line 594
        Cpanel::Hulkd::_handle_accepted_socket_and_reset_idleloops(Cpanel::Hulkd=HASH(0xb02130), Cpanel::Socket::INET=GLOB(0xc65b18)) called at /usr/local/cpanel/Cpanel/Hulkd.pm line 341
        Cpanel::Hulkd::main_loop(Cpanel::Hulkd=HASH(0xb02130), Cpanel::Socket::UNIX=GLOB(0xb3a718), Cpanel::Socket::INET=GLOB(0xb3aa30)) called at /usr/local/cpanel/Cpanel/Hulkd.pm line 244
        Cpanel::Hulkd::processor_run(Cpanel::Hulkd=HASH(0xb02130)) called at /usr/local/cpanel/Cpanel/Hulkd.pm line 159
        Cpanel::Hulkd::__ANON__(__CPANEL_HIDDEN__) called at /usr/local/cpanel/Cpanel/Hulkd.pm line 170
        Cpanel::Hulkd::launcher(Cpanel::Hulkd=HASH(0xb02130), 0) called at /usr/local/cpanel/Cpanel/Hulkd.pm line 139
        Cpanel::Hulkd::start_daemon(Cpanel::Hulkd=HASH(0xb02130), 0) called at /usr/local/cpanel/Cpanel/Hulkd.pm line 93
        Cpanel::Hulkd::run_daemon(Cpanel::Hulkd=HASH(0xb02130)) called at libexec/cphulkd.pl line 32
[2020-02-01 21:30:38 -0500] info [cPhulkd] The system encountered an error while processing a request: exit level [die] [pid=20503] (Timeout while waiting for response)

[2020-02-03 14:38:31 -0500] info [cPhulkd] The service:[login] unexpectedly sent the invalid remote IP address:[0.0.0.0]. (Consider disabling DNS resolution for this service)
[2020-02-03 14:38:31 -0500] info [cPhulkd] The service:[login] unexpectedly sent the invalid remote IP address:[0.0.0.0]. (Consider disabling DNS resolution for this service)

Here is some of cphulkd.log around the time the first user blocks happened.

Code:

[2020-02-03 14:02:18 -0500] info [cPhulkd] DB processor shutdown via SIGTERM with pid 11073
[2020-02-03 14:07:10 -0500] info [cPhulkd] processor startup with pid 9091
[2020-02-03 14:07:10 -0500] info [cPhulkd] DB processor startup with pid 13789
[2020-02-03 14:08:04 -0500] info [cPhulkd] DB processor shutdown via SIGTERM with pid 13789
[2020-02-03 14:08:04 -0500] info [cPhulkd] processor startup with pid 9091
[2020-02-03 14:08:04 -0500] info [cPhulkd] DB processor startup with pid 13889
[2020-02-03 14:12:33 -0500] info [cPhulkd] Login Blocked: IP reached maximum auth failures for a one day block [Service]=[cpaneld] [Local IP Address]=[74.50.46.249] [Local Port]=[2083] [Remote IP Address]=[195.24.207.249] [Remote Port]=[33544] [Authentication Database]=[system] [Username]=[testsite] (1/30 failures) (blocked until [Tue Feb  4 19:12:33 2020 UTC/Tue Feb  4 14:12:33 2020 LOCAL])
[2020-02-03 14:12:33 -0500] info [cPhulkd] Login Blocked: The country is blacklisted. [Service]=[cpaneld] [Local IP Address]=[74.50.46.249] [Local Port]=[2083] [Remote IP Address]=[195.24.207.249] [Remote Port]=[33544] [Authentication Database]=[system] [Username]=[testsite]
[2020-02-03 14:23:04 -0500] info [cPhulkd] DB processor shutdown via SIGTERM with pid 13889
[2020-02-03 14:36:33 -0500] info [cPhulkd] processor startup with pid 9091
[2020-02-03 14:36:33 -0500] info [cPhulkd] DB processor startup with pid 17495
[2020-02-03 14:36:33 -0500] info [cPhulkd] Login Blocked: IP reached maximum auth failures for a one day block [Service]=[sshd] [Remote IP Address]=[10.140.140.28] [Authentication Database]=[system] [Username]=[cruckrie] (1/30 failures) (blocked until [Tue Feb  4 19:36:33 2020 UTC/Tue Feb  4 14:36:33 2020 LOCAL])
[2020-02-03 14:36:33 -0500] info [cPhulkd] Login Blocked: The country is blacklisted. [Service]=[sshd] [Remote IP Address]=[10.140.140.28] [Authentication Database]=[system] [Username]=[XXXXXXX]
[2020-02-03 14:37:15 -0500] info [cPhulkd] Login Blocked: IP reached maximum auth failures for a one day block [Service]=[sshd] [Remote IP Address]=[10.140.140.5] [Authentication Database]=[system] [Username]=[root] (1/30 failures) (blocked until [Tue Feb  4 19:37:15 2020 UTC/Tue Feb  4 14:37:15 2020 LOCAL])
[2020-02-03 14:37:15 -0500] info [cPhulkd] Login Blocked: The country is blacklisted. [Service]=[sshd] [Remote IP Address]=[10.140.140.5] [Authentication Database]=[system] [Username]=[root]
[2020-02-03 14:37:26 -0500] info [cPhulkd] Login Blocked: IP reached maximum auth failures for a one day block [Service]=[sshd] [Remote IP Address]=[10.140.140.5] [Authentication Database]=[system] [Username]=[root] (1/30 failures) (blocked until [Tue Feb  4 19:37:26 2020 UTC/Tue Feb  4 14:37:26 2020 LOCAL])
[2020-02-03 14:37:26 -0500] info [cPhulkd] Login Blocked: The country is blacklisted. [Service]=[sshd] [Remote IP Address]=[10.140.140.5] [Authentication Database]=[system] [Username]=[root]
[2020-02-03 14:37:27 -0500] info [cPhulkd] Login Blocked: IP reached maximum auth failures for a one day block [Service]=[sshd] [Remote IP Address]=[10.140.140.31] [Authentication Database]=[system] [Username]=[cluescher] (1/30 failures) (blocked until [Tue Feb  4 19:37:27 2020 UTC/Tue Feb  4 14:37:27 2020 LOCAL])
[2020-02-03 14:37:27 -0500] info [cPhulkd] Login Blocked: The country is blacklisted. [Service]=[sshd] [Remote IP Address]=[10.140.140.31] [Authentication Database]=[system] [Username]=[XXXXXXX]
[2020-02-03 14:38:09 -0500] info [cPhulkd] Login Blocked: IP reached maximum auth failures for a one day block [Service]=[sshd] [Remote IP Address]=[10.140.140.5] [Authentication Database]=[system] [Username]=[root] (1/30 failures) (blocked until [Tue Feb  4 19:38:09 2020 UTC/Tue Feb  4 14:38:09 2020 LOCAL])
[2020-02-03 14:38:09 -0500] info [cPhulkd] Login Blocked: The country is blacklisted. [Service]=[sshd] [Remote IP Address]=[10.140.140.5] [Authentication Database]=[system] [Username]=[root]
[2020-02-03 14:43:16 -0500] info [cPhulkd] Login Blocked: IP reached maximum auth failures for a one day block [Service]=[pure-ftpd] [Local IP Address]=[74.50.46.253] [Local Port]=[21] [Remote IP Address]=[78.131.193.167] [Authentication Database]=[system] [Username]=[bluegrasswireless.com] (1/30 failures) (blocked until [Tue Feb  4 19:43:16 2020 UTC/Tue Feb  4 14:43:16 2020 LOCAL])
[2020-02-03 14:43:16 -0500] info [cPhulkd] Login Blocked: The country is blacklisted. [Service]=[pure-ftpd] [Local IP Address]=[74.50.46.253] [Local Port]=[21] [Remote IP Address]=[78.131.193.167] [Authentication Database]=[system] [Username]=[bluegrasswireless.com]

This is about the time I turned on country blocking, I have blocked out some usernames.

Thank you.

cPanelLauren · Feb 13, 2020

Hello,

This is interesting because it's seeing your private IP address - 10.140.140.31 is not a public IP address. This means it's an IP that should/would only be accessible within your network. I assume your server is NAT routed, based on that I wonder if it's being properly recognized.

Without listing your public IP address can you tell me if this file exists and properly maps your Private IPs to your Public IPs?

Code:

cat /var/cpanel/cpnat

kcorbin · Feb 14, 2020

There are no private IP's in the cpnat file only the public IP's assigned to the server. The main reason I came to the forums with the issue is that it blocked private IPs too. No matter were anyone tried to login in from it would block the IP and maybe the account too, I couldn't tell at the time. If it wasn't for the vSphere console we would have lost all access to the server. Let me add these were not repeated failed logins, accounts were blocked even after a successful login. Their IP's were also blocked from hosted websites too.

Thank you for your help.

cPanelLauren · Feb 14, 2020

Hello,

I believe there's a bit of a misunderstanding and to clarify: on 1:1 NAT Routed systems (the only NAT configuration we support) the private IP address like 10.140.140.31 is mapped to a public IP address. The file at /var/cpanel/cpnat shows this mapping and is a reference for cPanel to recognize whether or not the system is NAT routed.

cPHulk should not be seeing the private IP address to block and this is why I asked the question, in that output you provided the IP is a private IP and should never be listed. What I believe is happening is one of two things:

1. The NAT routing on the server is misconfigured

2. cPanel isn't seeing that the system is NAT routed - you could confirm if this is the case fairly easily by running the following:

Code:

/scripts/build_cpnat

If you're not NAT routed at all, this becomes a bit confusing as it would even further the point that something is misconfigured as private IP's aren't able to connect outside of the local network.

kcorbin · Feb 14, 2020

The server is not NAT routed and only has public IP's assigned to it, I believe we sys admins are to public IP's. I did run the script you provided and got the following.

Code:

info [build_cpnat] All publicly routeable addresses are the same as the local address. Not a NAT system.

The rest of the output was all public IP's, as for local network, we frequently SSH to our servers from private IP's. We are the ISP if that helps any so we have total control routing wise.

Thank you.

cPanelLauren · Feb 14, 2020

That's interesting if you're accessing from a local network it would be the only instance in which it would make sense to see the private IP address, but it still doesn't explain why cPhulk would block it unless it sees the internal IP from originating from a blocked Country. I'd suggest opening a ticket so our analysts can take a look at the server, to be honest - I don't think it'd be a simple task to find out what's occurring without being able to see it.

kcorbin · Feb 18, 2020

Thank you I have created ticket number 93452104.

cPanelLauren · Feb 18, 2020

Hello,

Thank you for that, I added the following note to the ticket as well:

They access SSH with an internal IP 10.140.140.28
They enabled CC blocking with cPHulkd
With CC blocking enabled their internal IP address is being blocked:

Code:

[2020-02-03 14:36:33 -0500] info [cPhulkd] Login Blocked: The country is blacklisted. [Service]=[sshd] [Remote IP Address]=[10.140.140.28] [Authentication Database]=[system] [Username]=[XXXXXXX]

I'll update here when there is more information on this.

cPanelLauren · Feb 20, 2020

Hi @kcorbin

The analyst who spent a good portion of yesterday investigating this noted that she found an internal case which indicates that some IP's in the 10.X.X.X range are considered to be part of "Unknown Region" (ZZ) because of this when you block all except for the US (or another all but X method) it's also including anything that doesn't have a region assigned (unknown). Internal IP's wouldn't have a region assigned to them so the assumption is that it's blocking them for you because of this. She asked if you could un-include the "Unknown Region" (ZZ) selection when blocking countries, and let her know when it was done so that she could continue testing/or let us know if you continue to have the issue.

Thanks!

cPanelLauren · Feb 25, 2020

Just as a final follow up, I wanted to note that the solution for this does as of right now appear to have been to add the unknown country code into the whitelist for cPHulk. I checked in on the ticket and it looks like things have been good since making that change. If you do run into any further issues with this or with cPhulk please don't hesitate to contact us again.

kcorbin · Feb 27, 2020

After working with cPanel support I can confirm that removing country code ZZ from the blacklist has fixed the issue. I would like to thank everyone at cPanel for the assistance and support.

Thanks again.

Thread starter	Similar threads	Forum	Replies	Date
N	cpHulk locked root	Security	1	Aug 12, 2022
E	Backlisted all Countries into cPHulk Brute Force Protection now blocked	Security	4	May 23, 2021
C	CPHulk Blocked	Security	11	Mar 27, 2021
K	Can you whitelist an IP in a blocked country in cPHulk	Security	1	Feb 8, 2019
Z	cPHulk concerns about being locked out	Security	2	Aug 18, 2018

Search

Search

CPHulk locked all access

kcorbin

Member

cPanelLauren

Product Owner II

kcorbin

Member

cPanelLauren

Product Owner II

kcorbin

Member

cPanelLauren

Product Owner II

kcorbin

Member

cPanelLauren

Product Owner II

kcorbin

Member

cPanelLauren

Product Owner II

cPanelLauren

Product Owner II

cPanelLauren

Product Owner II

kcorbin

Member