kcorbin

Member
Nov 19, 2012
5
0
1
cPanel Access Level
DataCenter Provider
We enabled country blocking on our server and CPhulk blocked everything, web and ssh, even blocked access to hosted sites. I tried the following;

Code:
/usr/local/cpanel/bin/cphulk_pam_ctl --disable
/usr/local/cpanel/etc/init/stopcphulkd
but cphulk would restart. I then tried;

Code:
whmapi1 configureservice service=cphulkd enabled=0 monitord=0
iptables -F cphulk && /usr/local/cpanel/3rdparty/bin/sqlite3 /var/cpanel/hulkd/cphulk.sqlite "DELETE FROM login_track;"
and was able to get access to WHM and SSH again, so I have a few questions.

What did I do wrong? I went to Home > Security Center > Cphulk Brute force protection and selected every nation except the US and chose blacklist. I didn't chose anything else, but I'm sure I probably missed something.
Is there a way to reset cphulk through cli back to its defaults? I would like to restart it and use it again.
What can I do to prevent this from happening again?

Thank you all.
 

cPanelLauren

Forums Analyst II
Staff member
Nov 14, 2017
9,366
799
263
Houston
Hello,


As far as managing cPHulk we have the following documentation that might be helpful: cPHulk Management on the Command Line | cPanel & WHM Documentation

In order to know how to prevent the issue from occurring again, we'd have to understand what happened specifically, the log files might be helpful for this which can be found here:
  • /usr/local/cpanel/logs/cphulkd.log
  • /usr/local/cpanel/logs/cphulkd_errors.log

To manage cPHulk's config ove the CLI you'd need to use the API which can be found here: WHM API 1 Functions - set_cphulk_config_key - Developer Documentation - cPanel Documentation

You can still make changes to the configuration in WHM with cPHulk's IP management disabled though and it might be easiest to make those modifications there.
 

kcorbin

Member
Nov 19, 2012
5
0
1
cPanel Access Level
DataCenter Provider
Lauren thank you for the reply. Here is the last few entries in the cphulkd_errors.log

Code:
[2020-02-01 21:30:38 -0500] die [cPhulkd] Timeout while waiting for response at /usr/local/cpanel/Cpanel/Hulkd.pm line 487.
        Cpanel::Hulkd::die(Cpanel::Hulkd=HASH(0xb02130), "Timeout while waiting for response") called at /usr/local/cpanel/Cpanel/Hulkd.pm line 417
        Cpanel::Hulkd::__ANON__(__CPANEL_HIDDEN__) called at /usr/local/cpanel/Cpanel/Hulkd/Processor.pm line 289
        eval {...} called at /usr/local/cpanel/Cpanel/Hulkd/Processor.pm line 289
        Cpanel::Hulkd::Processor::run(Cpanel::Hulkd::Processor=HASH(0xe00cf8), undef) called at /usr/local/cpanel/Cpanel/Hulkd.pm line 423
        Cpanel::Hulkd::__ANON__() called at /usr/local/cpanel/3rdparty/perl/528/lib/perl5/cpanel_lib/Try/Tiny.pm line 97
        eval {...} called at /usr/local/cpanel/3rdparty/perl/528/lib/perl5/cpanel_lib/Try/Tiny.pm line 88
        Try::Tiny::try(CODE(0xd50888), Try::Tiny::Catch=REF(0xc655a8)) called at /usr/local/cpanel/Cpanel/Hulkd.pm line 427
        Cpanel::Hulkd::handle_one_connection(Cpanel::Hulkd=HASH(0xb02130), Cpanel::Socket::INET=GLOB(0xc65b18), undef) called at /usr/local/cpanel/Cpanel/Hulkd.pm line 594
        Cpanel::Hulkd::_handle_accepted_socket_and_reset_idleloops(Cpanel::Hulkd=HASH(0xb02130), Cpanel::Socket::INET=GLOB(0xc65b18)) called at /usr/local/cpanel/Cpanel/Hulkd.pm line 341
        Cpanel::Hulkd::main_loop(Cpanel::Hulkd=HASH(0xb02130), Cpanel::Socket::UNIX=GLOB(0xb3a718), Cpanel::Socket::INET=GLOB(0xb3aa30)) called at /usr/local/cpanel/Cpanel/Hulkd.pm line 244
        Cpanel::Hulkd::processor_run(Cpanel::Hulkd=HASH(0xb02130)) called at /usr/local/cpanel/Cpanel/Hulkd.pm line 159
        Cpanel::Hulkd::__ANON__(__CPANEL_HIDDEN__) called at /usr/local/cpanel/Cpanel/Hulkd.pm line 170
        Cpanel::Hulkd::launcher(Cpanel::Hulkd=HASH(0xb02130), 0) called at /usr/local/cpanel/Cpanel/Hulkd.pm line 139
        Cpanel::Hulkd::start_daemon(Cpanel::Hulkd=HASH(0xb02130), 0) called at /usr/local/cpanel/Cpanel/Hulkd.pm line 93
        Cpanel::Hulkd::run_daemon(Cpanel::Hulkd=HASH(0xb02130)) called at libexec/cphulkd.pl line 32
[2020-02-01 21:30:38 -0500] info [cPhulkd] The system encountered an error while processing a request: exit level [die] [pid=20503] (Timeout while waiting for response)

[2020-02-03 14:38:31 -0500] info [cPhulkd] The service:[login] unexpectedly sent the invalid remote IP address:[0.0.0.0]. (Consider disabling DNS resolution for this service)
[2020-02-03 14:38:31 -0500] info [cPhulkd] The service:[login] unexpectedly sent the invalid remote IP address:[0.0.0.0]. (Consider disabling DNS resolution for this service)
Here is some of cphulkd.log around the time the first user blocks happened.

Code:
[2020-02-03 14:02:18 -0500] info [cPhulkd] DB processor shutdown via SIGTERM with pid 11073
[2020-02-03 14:07:10 -0500] info [cPhulkd] processor startup with pid 9091
[2020-02-03 14:07:10 -0500] info [cPhulkd] DB processor startup with pid 13789
[2020-02-03 14:08:04 -0500] info [cPhulkd] DB processor shutdown via SIGTERM with pid 13789
[2020-02-03 14:08:04 -0500] info [cPhulkd] processor startup with pid 9091
[2020-02-03 14:08:04 -0500] info [cPhulkd] DB processor startup with pid 13889
[2020-02-03 14:12:33 -0500] info [cPhulkd] Login Blocked: IP reached maximum auth failures for a one day block [Service]=[cpaneld] [Local IP Address]=[74.50.46.249] [Local Port]=[2083] [Remote IP Address]=[195.24.207.249] [Remote Port]=[33544] [Authentication Database]=[system] [Username]=[testsite] (1/30 failures) (blocked until [Tue Feb  4 19:12:33 2020 UTC/Tue Feb  4 14:12:33 2020 LOCAL])
[2020-02-03 14:12:33 -0500] info [cPhulkd] Login Blocked: The country is blacklisted. [Service]=[cpaneld] [Local IP Address]=[74.50.46.249] [Local Port]=[2083] [Remote IP Address]=[195.24.207.249] [Remote Port]=[33544] [Authentication Database]=[system] [Username]=[testsite]
[2020-02-03 14:23:04 -0500] info [cPhulkd] DB processor shutdown via SIGTERM with pid 13889
[2020-02-03 14:36:33 -0500] info [cPhulkd] processor startup with pid 9091
[2020-02-03 14:36:33 -0500] info [cPhulkd] DB processor startup with pid 17495
[2020-02-03 14:36:33 -0500] info [cPhulkd] Login Blocked: IP reached maximum auth failures for a one day block [Service]=[sshd] [Remote IP Address]=[10.140.140.28] [Authentication Database]=[system] [Username]=[cruckrie] (1/30 failures) (blocked until [Tue Feb  4 19:36:33 2020 UTC/Tue Feb  4 14:36:33 2020 LOCAL])
[2020-02-03 14:36:33 -0500] info [cPhulkd] Login Blocked: The country is blacklisted. [Service]=[sshd] [Remote IP Address]=[10.140.140.28] [Authentication Database]=[system] [Username]=[XXXXXXX]
[2020-02-03 14:37:15 -0500] info [cPhulkd] Login Blocked: IP reached maximum auth failures for a one day block [Service]=[sshd] [Remote IP Address]=[10.140.140.5] [Authentication Database]=[system] [Username]=[root] (1/30 failures) (blocked until [Tue Feb  4 19:37:15 2020 UTC/Tue Feb  4 14:37:15 2020 LOCAL])
[2020-02-03 14:37:15 -0500] info [cPhulkd] Login Blocked: The country is blacklisted. [Service]=[sshd] [Remote IP Address]=[10.140.140.5] [Authentication Database]=[system] [Username]=[root]
[2020-02-03 14:37:26 -0500] info [cPhulkd] Login Blocked: IP reached maximum auth failures for a one day block [Service]=[sshd] [Remote IP Address]=[10.140.140.5] [Authentication Database]=[system] [Username]=[root] (1/30 failures) (blocked until [Tue Feb  4 19:37:26 2020 UTC/Tue Feb  4 14:37:26 2020 LOCAL])
[2020-02-03 14:37:26 -0500] info [cPhulkd] Login Blocked: The country is blacklisted. [Service]=[sshd] [Remote IP Address]=[10.140.140.5] [Authentication Database]=[system] [Username]=[root]
[2020-02-03 14:37:27 -0500] info [cPhulkd] Login Blocked: IP reached maximum auth failures for a one day block [Service]=[sshd] [Remote IP Address]=[10.140.140.31] [Authentication Database]=[system] [Username]=[cluescher] (1/30 failures) (blocked until [Tue Feb  4 19:37:27 2020 UTC/Tue Feb  4 14:37:27 2020 LOCAL])
[2020-02-03 14:37:27 -0500] info [cPhulkd] Login Blocked: The country is blacklisted. [Service]=[sshd] [Remote IP Address]=[10.140.140.31] [Authentication Database]=[system] [Username]=[XXXXXXX]
[2020-02-03 14:38:09 -0500] info [cPhulkd] Login Blocked: IP reached maximum auth failures for a one day block [Service]=[sshd] [Remote IP Address]=[10.140.140.5] [Authentication Database]=[system] [Username]=[root] (1/30 failures) (blocked until [Tue Feb  4 19:38:09 2020 UTC/Tue Feb  4 14:38:09 2020 LOCAL])
[2020-02-03 14:38:09 -0500] info [cPhulkd] Login Blocked: The country is blacklisted. [Service]=[sshd] [Remote IP Address]=[10.140.140.5] [Authentication Database]=[system] [Username]=[root]
[2020-02-03 14:43:16 -0500] info [cPhulkd] Login Blocked: IP reached maximum auth failures for a one day block [Service]=[pure-ftpd] [Local IP Address]=[74.50.46.253] [Local Port]=[21] [Remote IP Address]=[78.131.193.167] [Authentication Database]=[system] [Username]=[bluegrasswireless.com] (1/30 failures) (blocked until [Tue Feb  4 19:43:16 2020 UTC/Tue Feb  4 14:43:16 2020 LOCAL])
[2020-02-03 14:43:16 -0500] info [cPhulkd] Login Blocked: The country is blacklisted. [Service]=[pure-ftpd] [Local IP Address]=[74.50.46.253] [Local Port]=[21] [Remote IP Address]=[78.131.193.167] [Authentication Database]=[system] [Username]=[bluegrasswireless.com]
This is about the time I turned on country blocking, I have blocked out some usernames.

Thank you.
 

cPanelLauren

Forums Analyst II
Staff member
Nov 14, 2017
9,366
799
263
Houston
Hello,


This is interesting because it's seeing your private IP address - 10.140.140.31 is not a public IP address. This means it's an IP that should/would only be accessible within your network. I assume your server is NAT routed, based on that I wonder if it's being properly recognized.

Without listing your public IP address can you tell me if this file exists and properly maps your Private IPs to your Public IPs?

Code:
cat /var/cpanel/cpnat
 

kcorbin

Member
Nov 19, 2012
5
0
1
cPanel Access Level
DataCenter Provider
There are no private IP's in the cpnat file only the public IP's assigned to the server. The main reason I came to the forums with the issue is that it blocked private IPs too. No matter were anyone tried to login in from it would block the IP and maybe the account too, I couldn't tell at the time. If it wasn't for the vSphere console we would have lost all access to the server. Let me add these were not repeated failed logins, accounts were blocked even after a successful login. Their IP's were also blocked from hosted websites too.

Thank you for your help.
 

cPanelLauren

Forums Analyst II
Staff member
Nov 14, 2017
9,366
799
263
Houston
Hello,

I believe there's a bit of a misunderstanding and to clarify: on 1:1 NAT Routed systems (the only NAT configuration we support) the private IP address like 10.140.140.31 is mapped to a public IP address. The file at /var/cpanel/cpnat shows this mapping and is a reference for cPanel to recognize whether or not the system is NAT routed.

cPHulk should not be seeing the private IP address to block and this is why I asked the question, in that output you provided the IP is a private IP and should never be listed. What I believe is happening is one of two things:

1. The NAT routing on the server is misconfigured

2. cPanel isn't seeing that the system is NAT routed - you could confirm if this is the case fairly easily by running the following:
Code:
/scripts/build_cpnat
If you're not NAT routed at all, this becomes a bit confusing as it would even further the point that something is misconfigured as private IP's aren't able to connect outside of the local network.
 

kcorbin

Member
Nov 19, 2012
5
0
1
cPanel Access Level
DataCenter Provider
The server is not NAT routed and only has public IP's assigned to it, I believe we sys admins are to public IP's. I did run the script you provided and got the following.

Code:
info [build_cpnat] All publicly routeable addresses are the same as the local address. Not a NAT system.
The rest of the output was all public IP's, as for local network, we frequently SSH to our servers from private IP's. We are the ISP if that helps any so we have total control routing wise.

Thank you.
 

cPanelLauren

Forums Analyst II
Staff member
Nov 14, 2017
9,366
799
263
Houston
That's interesting if you're accessing from a local network it would be the only instance in which it would make sense to see the private IP address, but it still doesn't explain why cPhulk would block it unless it sees the internal IP from originating from a blocked Country. I'd suggest opening a ticket so our analysts can take a look at the server, to be honest - I don't think it'd be a simple task to find out what's occurring without being able to see it.
 

cPanelLauren

Forums Analyst II
Staff member
Nov 14, 2017
9,366
799
263
Houston
Hello,

Thank you for that, I added the following note to the ticket as well:

  • They access SSH with an internal IP 10.140.140.28
  • They enabled CC blocking with cPHulkd
  • With CC blocking enabled their internal IP address is being blocked:

Code:
[2020-02-03 14:36:33 -0500] info [cPhulkd] Login Blocked: The country is blacklisted. [Service]=[sshd] [Remote IP Address]=[10.140.140.28] [Authentication Database]=[system] [Username]=[XXXXXXX]
I'll update here when there is more information on this.