Possible bug with cPHulk Brute Force Protection

[AngelNo3]

Over the weekend, I received two messages from cPHulk that said, "Root was logged into pure-ftpd using following authentication service: system". Each was from a different country and different IP address.

This seemed strange, because in my WHM "FTP Server Configuration" the option "Allow Logins with Root Password" was set to "No" (disabled). Besides that, we use a strong password that would be difficult to guess, especially with cPHulk limiting all brute-force attempts.

I contacted my host, InMotion Hosting, and they looked into it. They said, "I'm seeing that there were attempted logins using the root user but no successful logins." I asked if it was a bug then, and they said, "I would have to say that would be a bug."

So ... my reason for posting here is to see if there is a known issue with this, if there is anything I should do to verify or confirm the issue, and how I might go about submitting a bug report. (I Googled for it, but the pages that came up seemed to indicate that the bug report links were for the license holder, which I guess would be InMotion Hosting in this case.)

 

[cPanelMichael]

Hello

The issue you are reporting sounds similar to an issue that was addressed in cPanel version 11.38.1.2:

"Fixed case 60703: Don't falsely alert the admin about root logins."

Could you check and let us know which version of cPanel is installed on your system? You can use this command:

cat /usr/local/cpanel/version

Thank you.

 

[madsere]

I have the same problem, with Cpanel 11.38.2.7.

Every couple of weeks I get a notification about "Root was logged into pure-ftpd using following authentication service: system", but

1) we use very strong passwords
2) root access is set to no for ftp in cpanel.
3) there are no signs in any logfile or otherwise of any successful root login.

Which process sends this mail? Where does it get it's information?

 

[cPanelMichael]

Please open a support ticket so we can take a closer look:

Submit A Ticket

Let us know the ticket number so we can update this thread with the outcome.

Thank you.

 

[madsere]

Long story, if you could just answer the question I'd be able to check it myself, thanks.

Which process sends this mail? Where does it get it's information?

 

[cPanelMichael]

Long story, if you could just answer the question I'd be able to check it myself, thanks.

Which process sends this mail? Where does it get it's information?

The email notification is sent out by cPhulkd process itself when conditionals are hit. It's getting the information from cPhulkd itself. The alert notification should not be occurring if the IP address is not blacklisted by cPhulkd. A ticket should be opened so we can investigate and determine if a new bug report related to case 60703 is necessary.

Thank you.

 

[madsere]

Since you insist I will explain why I am not bothered to report it.

After a minor hacking problem probably over a year ago now, we were told by your support that you would no longer support this particular server until we reinstall the software. I accept I'm not a Cpanel expert but I've worked with Linux internals for 20 years and know my way around Linux well enough that I would be able to pick up clues if it had been compromised. As I do not think the server has been compromised and don't have any other problems we are not going to waste time and inconvenience our customers with this until we receive some (any) sort of proof that it has been compromised. This is just one of a dozen or so Cpanel servers we have, and it's frankly not a big problem so we're just leaving it as it is.

At this point it has become a "policy" issue so we are not going to get anywhere, and I'm sure if I reported this problem in a ticket it would somehow be construed to support that there was a vulnerability issue even though I think it is quite clear that these are different issues.

Now rather than being so secretive about what "conditionals" are hit, could you please let me know so I can investigate myself, how cphulkd gets the idea that root has been logged in. As I have explained already, there are no signs in any logs, or the cphulk page, of any entry. If you'd rather not publish the information on the forum you're welcome to use email/PM.

 

[cPanelMichael]

The specific entry that sends out the email can be found on line 293 of /usr/local/cpanel/Cpanel/Hulkd/Processor.pm:

            else {
                if ( !$ip_is_whitelisted && !$ip_is_blacklisted && $op eq 'LOGIN' && $user eq 'root' && $conf_ref->{'notify_on_root_login'} && _login_is_new( $self->{'dbh'}, $login_service, $user, $ip ) ) {
                    $self->_report( 'type' => 'root', 'login_service' => $login_service, 'service' => $service, 'ip' => $ip, 'logintime' => $logintime );
                    _notify(
                        'application' => 'cPHulk',
                        'subject'     => 'Root Login from ' . ( $ip ? "IP $ip" : 'Local Machine' ),
                        'ip'          => $ip,
                        'hostname'    => 1,
                        'message'     => "Root was logged into $login_service using following authentication service: $service ",
                        'localuser'   => $ruser,
                    );
                }

                $self->warn("NOT registering [IP:$ip] [USER:$user] [SERVICE:$service]\n") if $debug;
            }
            last if $quit_after;

You should only get the alert if a blacklisted IP address has logged in as "root" with the service. Note that you are welcome to open a ticket and request the issue is reproduced on a test machine so that you do not have to provide direct access to your system (in the event it's marked as "hacked" by our support team).

Thank you.

 

[madsere]

Thank you for the update. I checked, and /usr/local/cpanel/Cpanel/Hulkd/Processor.pm on my server actually have this exact message twice, once after a test for blacklisted IP (line 300)

            else {
                if ( !$ip_is_whitelisted && !$ip_is_blacklisted && $op eq 'LOGIN' && $user eq 'root' && $conf_ref->{'notify_on_root_login'} && _login_is_new( $self->{'dbh'}, $login_service, $user, $ip ) ) {
                    $self->_report( 'type' => 'root', 'login_service' => $login_service, 'service' => $service, 'ip' => $ip, 'logintime' => $logintime );
                    _notify(
                        'application' => 'cPHulk',
                        'subject'     => 'Root Login from ' . ( $ip ? "IP $ip" : 'Local Machine' ),
                        'ip'          => $ip,
                        'hostname'    => 1,
                        'message'     => "Root was logged into $login_service using following authentication service: $service ",
                        'localuser'   => $ruser,
                    );
                }

                $self->warn("NOT registering [IP:$ip] [USER:$user] [SERVICE:$service]\n") if $debug;
            }

and once where there is no such test (line 149).

                if ( !$ip_is_whitelisted && $user eq 'root' && $conf_ref->{'notify_on_root_login'} && _login_is_new( $self->{'dbh'}, $login_service, $user, $ip ) ) {
                    $self->_report( 'type' => 'root', 'login_service' => $login_service, 'service' => $service, 'ip' => $ip, 'logintime' => $logintime );
                    _notify(
                        'application' => 'cPHulk',
                        'subject'     => 'Root Login from ' . ( $ip ? "IP $ip" : 'Local Machine' ),
                        'ip'          => $ip,
                        'message'     => "Root was logged into $login_service using following authentication service: $service ",
                        'hostname'    => 1,
                        'localuser'   => $ruser,
                    );
                }

That suggest to me that it's possible to get the message even if the IP is not blacklisted.

Moreover, it seems cphulk is just passing on values, where does it get $service and $login_service from?

 

[cPanelMichael]

That data is populated directly from the cphulkd backend which obtains that information from the service itself (E.g. /usr/local/cpanel/bin/pureauth).

Thank you.

 

[madsere]

I count 901 failed login attempts in /var/log/messages from the same IP address starting Sep 23 03:57:13 and ending Sep 23 04:17:23.

In addition to my prior 3 points, I'd like to add the following:

4) The timestamp on the cphulk mail is 23 Sep 04:16:09. From this time until 04:17:23 there are still 61 failed root login attempts. If someone had actually managed to login at 04:16:09 I can't see why they would keep attempting to login. Surely the simplest hacking script has an "until" clause that gets it to break off when it gets lucky.

5) If I login as root through pure-ftpd with the correct password I correctly get "530 Login authentication failed". There ought to be no way root can login with pure-ftpd.

I can't say how, but there must be a bug somewhere that cause cphulk to send out a false error message when certain conditions are met.

You write "You should only get the alert if a blacklisted IP address has logged in as "root" with the service. " - how would it ever be possible for someone to login with a blacklisted IP address? I thought the whole idea with blacklisting is they are not allowed to login.

 

[cPanelMichael]

I can't say how, but there must be a bug somewhere that cause cphulk to send out a false error message when certain conditions are met. You write "You should only get the alert if a blacklisted IP address has logged in as "root" with the service. " - how would it ever be possible for someone to login with a blacklisted IP address? I thought the whole idea with blacklisting is they are not allowed to login.

We will need you to open a support ticket so we can attempt to reproduce the issue and open an internal case (we can use a test machine to attempt this to avoid access requirements on your machine that is marked as "hacked"). It's true that a blacklisted IP address can not authenticate. To clarify, the previous internal case was regarding the misleading email claiming that a blacklisted address has logged in as root, when in-fact it had not.

Thank you.

 

[madsere]

Ok, ticket opened, ID# 4355801

 

[cPanelMichael]

To update, it's highly likely the email report you received from cPhulkd was a false positive. There is an open case (#65253) to add more verbose logging information to help the administrator determine if there was actual access.

Thank you.

 

[ottdev]

I am a reseller with a VPS with WHM/cPanel, therefore I cannot submit a case or bug report. I don't know where I Can go to look at the case your referenced 65253 to see if this is already included...
But I wanted to report this kind of line is triggering "root login from IP" messages - though they are not actually successful attempts:

109.201.152.7 - root [09/30/2013:14:17:29 -0000] "GET /cpsess5900450496/ HTTP/1.1" FAILED LOGIN whostmgrd: cookie ip check: IP address has changed

 

[cPanelMichael]

You are welcome to open a ticket if you operate a VPS (root access is available). Internal cases are not viewable to the public. You can monitor the change log to determine when a case has been included:

cPanel - Change Log

Thank you.

 

[webmasternyit]

I just wanted to add something to this thread. I am also running cPanel version 11.38.2.7 and we are a a government agency that gets hit a lot with brute force attempts. Last night, we received an email saying that root was logged in successfully with pure-ftpd but the logs did not support that same outcome. It seemed to be a false positive. We just wanted to inform the cPanel community that the same problem has happened to us as well and a fix for this bug would be greatly appreciated.

 

[cPDavidL]

Greetings everyone! This matter is still under investigation with internal case 86549.

 

[kalsta]

How's that investigation going?? We're on 11.42.0, and it looks like it just happened to us. Same warning, but hosting company says they can only see failed login attempts in the logs.

 

[cPanelMichael]

A resolution stemming from internal case 86549 is scheduled for inclusion with cPanel version 11.42.1. There is currently no exact time frame available for the release of this version.

Thank you.