cphulkd 500 error

[robhooper]

Hello,
We are seeing the below errors when dovecot talks to cPhulkd, this is a new server so I am wondering if something has been misconfigured at some point.

I have attached the error we are seeing in the cPHulkd logs.

Has anyone seen this before?

/var/log/maillog
```
Mar 14 05:29:22 violet dovecot: auth: Error: policy(__cpanel__service__auth__imap__odydfljfiwmg23mgc__...,127.0.0.1,<jCZjNAeEcsd/AAAB>): Policy server HTTP error: 500 Internal Error
Mar 14 05:29:22 violet dovecot: auth: Error: policy(__cpanel__service__auth__imap__odydfljfiwmg23mgc__...,127.0.0.1,<jCZjNAeEcsd/AAAB>): Policy server HTTP error: 500 Internal Error
...
Mar 15 15:13:51 hostname dovecot: auth: Error: policy(__cpanel__service__auth__exim__xhso7zsqxkpfm...,127.0.0.1): Policy server HTTP error: 500 Internal Error
Mar 15 15:13:51 hostname dovecot: auth: Error: policy(__cpanel__service__auth__exim__xhso7zsqxkpfm...,127.0.0.1): Policy server HTTP error: 500 Internal Error
Mar 15 15:13:51 hostname dovecot: auth: Error: policy(__cpanel__service__auth__exim__xhso7zsqxkpfm...,127.0.0.1): Policy server HTTP error: 500 Internal Error
```

/usr/local/cpanel/logs/cphulkd_errors.log
[2019-03-15 14:12:37 +0000] warn [cPhulkd] The socket is not setup in the Cpanel::Hulk object.  Do you need to call connect() first? at /usr/local/cpanel/Cpanel/Hulk.pm line 118, <GEN35> line 4.
        Cpanel::Hulk::_error_with_stack_trace("The socket is not setup in the Cpanel::Hulk object.  Do you n"...) called at /usr/local/cpanel/Cpanel/Hulk.pm line 127
        Cpanel::Hulk::_check_socket(Cpanel::Hulk=HASH(0x1f35d10)) called at /usr/local/cpanel/Cpanel/Hulk.pm line 140
        Cpanel::Hulk::_send_cmd(Cpanel::Hulk=HASH(0x1f35d10), "DBWRITE {\"query_parameters\":[[\"010110110110001000100100101000"..., "ignore_reply") called at /usr/local/cpanel/Cpanel/Hulk.pm line 405
        Cpanel::Hulk::dbwrite(Cpanel::Hulk=HASH(0x1f35d10), HASH(0x1f6ff08)) called at /usr/local/cpanel/Cpanel/Hulkd/Processor.pm line 1417
        Cpanel::Hulkd::Processor::_send_dbwrite_cmd(HASH(0x1f6ff08)) called at /usr/local/cpanel/Cpanel/Hulkd/Processor.pm line 1093
        Cpanel::Hulkd::Processor::_register_failed_login(Cpanel::Hulkd::Processor=HASH(0x1f7ef30)) called at /usr/local/cpanel/Cpanel/Hulkd/Processor.pm line 645
        Cpanel::Hulkd::Processor::_handle_authentication_request(Cpanel::Hulkd::Processor=HASH(0x1f7ef30)) called at /usr/local/cpanel/Cpanel/Hulkd/Processor.pm line 554
        Cpanel::Hulkd::Processor::_handle_input(Cpanel::Hulkd::Processor=HASH(0x1f7ef30), HASH(0x1f7f6f8)) called at /usr/local/cpanel/Cpanel/Hulkd/Processor.pm line 402
        Cpanel::Hulkd::Processor::_handle_action(Cpanel::Hulkd::Processor=HASH(0x1f7ef30), "{\"auth_database\":\"mail\",\"authtoken_hash\":\"\$0\$0\$a48cd8632ebe43"..., HASH(0x95fec8)) called at /usr/local/cpanel/Cpanel/Hulkd/Processor.pm line 159
        Cpanel::Hulkd::Processor::__ANON__() called at /usr/local/cpanel/3rdparty/perl/526/lib64/perl5/cpanel_lib/Try/Tiny.pm line 99
        eval {...} called at /usr/local/cpanel/3rdparty/perl/526/lib64/perl5/cpanel_lib/Try/Tiny.pm line 90
        Try::Tiny::try(CODE(0x1f47660), Try::Tiny::Catch=REF(0x1f36430)) called at /usr/local/cpanel/Cpanel/Hulkd/Processor.pm line 185
        Cpanel::Hulkd::Processor::_run_http(Cpanel::Hulkd::Processor=HASH(0x1f7ef30)) called at /usr/local/cpanel/Cpanel/Hulkd/Processor.pm line 271
        Cpanel::Hulkd::Processor::__ANON__() called at /usr/local/cpanel/3rdparty/perl/526/lib64/perl5/cpanel_lib/Try/Tiny.pm line 99
        eval {...} called at /usr/local/cpanel/3rdparty/perl/526/lib64/perl5/cpanel_lib/Try/Tiny.pm line 90
        Try::Tiny::try(CODE(0x1f36790), Try::Tiny::Catch=REF(0x1f362b0)) called at /usr/local/cpanel/Cpanel/Hulkd/Processor.pm line 284
        Cpanel::Hulkd::Processor::run(Cpanel::Hulkd::Processor=HASH(0x1f7ef30), undef) called at /usr/local/cpanel/Cpanel/Hulkd.pm line 415
        Cpanel::Hulkd::__ANON__() called at /usr/local/cpanel/3rdparty/perl/526/lib64/perl5/cpanel_lib/Try/Tiny.pm line 99
        eval {...} called at /usr/local/cpanel/3rdparty/perl/526/lib64/perl5/cpanel_lib/Try/Tiny.pm line 90
        Try::Tiny::try(CODE(0x1f36148), Try::Tiny::Catch=REF(0x1e42c88)) called at /usr/local/cpanel/Cpanel/Hulkd.pm line 419
        Cpanel::Hulkd::handle_one_connection(Cpanel::Hulkd=HASH(0x1bcccf0), Cpanel::Socket::INET=GLOB(0x1f36160), undef) called at /usr/local/cpanel/Cpanel/Hulkd.pm line 586
        Cpanel::Hulkd::_handle_accepted_socket_and_reset_idleloops(Cpanel::Hulkd=HASH(0x1bcccf0), Cpanel::Socket::INET=GLOB(0x1f36160)) called at /usr/local/cpanel/Cpanel/Hulkd.pm line 333
        Cpanel::Hulkd::main_loop(Cpanel::Hulkd=HASH(0x1bcccf0), Cpanel::Socket::UNIX=GLOB(0x1c00da0), Cpanel::Socket::INET=GLOB(0x1c00ff8)) called at /usr/local/cpanel/Cpanel/Hulkd.pm line 245
        Cpanel::Hulkd::processor_run(Cpanel::Hulkd=HASH(0x1bcccf0)) called at /usr/local/cpanel/Cpanel/Hulkd.pm line 160
        Cpanel::Hulkd::__ANON__(__CPANEL_HIDDEN__) called at /usr/local/cpanel/Cpanel/Hulkd.pm line 171
        Cpanel::Hulkd::launcher(Cpanel::Hulkd=HASH(0x1bcccf0), 0) called at /usr/local/cpanel/Cpanel/Hulkd.pm line 139
        Cpanel::Hulkd::start_daemon(Cpanel::Hulkd=HASH(0x1bcccf0), 0) called at /usr/local/cpanel/Cpanel/Hulkd.pm line 93
        Cpanel::Hulkd::run_daemon(Cpanel::Hulkd=HASH(0x1bcccf0)) called at libexec/cphulkd.pl line 32

 

[cPanelLauren]

Hi @robhooper


Can you try the following:

stop the queueprocd service, restart cphulkd, and start queueprocd once more


You can do this with the following commands:

/scripts/restartsrv_queueprocd --stop
/scripts/restartsrv_cphulkd
/scripts/restartsrv_queueprocd

 

[robhooper]

Thanks for the reply. I have ran those commands and will confirm in the next few days if the problem continues.

This does seem like a strange fix if it does work... why would a running queueprocd service change how cphulkd restart??

 

[cPanelLauren]

Thanks for the reply. I have ran those commands and will confirm in the next few days if the problem continues.

This does seem like a strange fix if it does work... why would a running queueprocd service change how cphulkd restart??

queueprocd handles task scheduling - including graceful restarts - this issue seems to commonly occur when the taskqueue is overloaded/stalled.

Thanks!

 

[robhooper]

I am afraid this hasn't fixed the issue. We saw some more "500 internal error" log's logged this morning

 

[cPanelLauren]

Can you please open a ticket using the link in my signature? Once open please reply with the Ticket ID here so that we can update this thread with the resolution once the ticket is resolved.


Thanks!

 

[robhooper]

Hi,

Just to update this forum post in case it affects anyone else.

We're no longer seeing this issue after the dormant setting has been disabled for cphulkd.

The explanation given for this was:
> It appears that this issue may be related to an internal case that we have, CPANEL-26599. This issue is caused by the cphulkd service going dormant and the socket not existing due to this. This is then incorrectly detected as the process stopping. cphulkd would then automatically restart upon the first login to the server.

The ticket ID is/was 11713357