The whois query for the address x took longer than 30 seconds

[bloatedstoat]

I'm having some trouble today with consistent issues related to cpHulk and Whois lookups.

I did read another thread "SOLVED - Whois query took longer than 30 seconds" with a similar issue to mine but it did not address my problem.

The problem started today, prior to today there have been no issues.
My firewall rules are accurate with 43 open and I have no country blocks in place at the firewall level.

[2018-08-07 16:39:44 +1000] warn [queueprocd] (XID 5n8ayj) The whois query for the address 'xxx.xxx.xxx.xxx' took longer than 30 seconds. at /usr/local/cpanel/Cpanel/Net/Whois/IP/Cached.pm line 96.
Cpanel::Net::Whois::IP::Cached::__ANON__(Cpanel::Exception::Timeout=HASH(0x116c6a0)) called at /usr/local/cpanel/3rdparty/perl/526/lib64/perl5/cpanel_lib/Try/Tiny.pm line 120
Try::Tiny::try(CODE(0x102c558), Try::Tiny::Catch=REF(0xa8b780)) called at /usr/local/cpanel/Cpanel/Net/Whois/IP/Cached.pm line 97
Cpanel::Net::Whois::IP::Cached::lookup_address(Cpanel::Net::Whois::IP::Cached=HASH(0xe58580), "xxx.xxx.xxx.xxx") called at /usr/local/cpanel/Cpanel/TaskProcessors/cPHulkTasks.pm line 40
Cpanel::TaskProcessors::cPHulkTasks::KnownNetblocks::add_known_ip_for_user(Cpanel::TaskProcessors::cPHulkTasks::KnownNetblocks::AddIPForUser=HASH(0xad5f88), HASH(0xcabf10)) called at /usr/local/cpanel/Cpanel/TaskProcessors/cPHulkTasks.pm line 105
Cpanel::TaskProcessors::cPHulkTasks::KnownNetblocks::AddIPForUser::__ANON__(HASH(0xcabf10)) called at /usr/local/cpanel/Cpanel/Hulkd/QueuedTasks/Harvester.pm line 102
eval {...} called at /usr/local/cpanel/Cpanel/Hulkd/QueuedTasks/Harvester.pm line 95
Cpanel::Hulkd::QueuedTasks::Harvester::harvest("Cpanel::Hulkd::QueuedTasks::AddKnownIPForUser::Harvester", CODE(0xde0148)) called at /usr/local/cpanel/Cpanel/TaskProcessors/cPHulkTasks.pm line 107
Cpanel::TaskProcessors::cPHulkTasks::KnownNetblocks::AddIPForUser::_do_child_task(Cpanel::TaskProcessors::cPHulkTasks::KnownNetblocks::AddIPForUser=HASH(0xad5f88), cPanel::TaskQueue::Task=HASH(0xf74998), Cpanel::LoggerAdapter=HASH(0xa8bc60)) called at /usr/local/cpanel/3rdparty/perl/526/lib64/perl5/cpanel_lib/cPanel/TaskQueue/ChildProcessor.pm line 63
eval {...} called at /usr/local/cpanel/3rdparty/perl/526/lib64/perl5/cpanel_lib/cPanel/TaskQueue/ChildProcessor.pm line 66
cPanel::TaskQueue::ChildProcessor::process_task(Cpanel::TaskProcessors::cPHulkTasks::KnownNetblocks::AddIPForUser=HASH(0xad5f88), cPanel::TaskQueue::Task=HASH(0xf74998), Cpanel::LoggerAdapter=HASH(0xa8bc60), cPanel::StateFile::Guard=HASH(0xf74c38)) called at /usr/local/cpanel/3rdparty/perl/526/lib64/perl5/cpanel_lib/cPanel/TaskQueue.pm line 608
eval {...} called at /usr/local/cpanel/3rdparty/perl/526/lib64/perl5/cpanel_lib/cPanel/TaskQueue.pm line 608
cPanel::TaskQueue::process_next_task(cPanel::TaskQueue=HASH(0xd1d3c0)) called at /usr/local/cpanel/Cpanel/QueueProcd/Queueing.pm line 49
eval {...} called at /usr/local/cpanel/Cpanel/QueueProcd/Queueing.pm line 49
Cpanel::QueueProcd::Queueing::queue_process_next_task(cPanel::TaskQueue=HASH(0xd1d3c0), Cpanel::LoggerAdapter=HASH(0xa8bc60)) called at libexec/queueprocd.pl line 351
libexec::queueprocd::process_tasks(cPanel::TaskQueue=HASH(0xd1d3c0), cPanel::TaskQueue::Scheduler=HASH(0xd1d378)) called at libexec/queueprocd.pl line 368
libexec::queueprocd::_process_tasks_until_no_more_are_ready(cPanel::TaskQueue=HASH(0xd1d3c0), cPanel::TaskQueue::Scheduler=HASH(0xd1d378)) called at libexec/queueprocd.pl line 191
libexec::queueprocd::script("libexec::queueprocd") called at libexec/queueprocd.pl line 111

Can anyone offer some tests I can perform to try and troubleshoot this please? Or any advice at the very least.

Thank you.

 

[cPanelLauren]

Hi @bloatedstoat


Are you having issues with anything else such as mail? Or is it just cPhulkd? What is in the resolv.conf?

 

[bloatedstoat]

Thanks @cPanelLauren

In the 24 hours ish this has been happening I have not noticed anything unusual with mail delivery and no clients have reached out to support with mail issues so I'll say at this stage mail delivery is behaving normally.

resolv.conf

domain our_main_server_domain.com
nameserver 127.0.0.1
nameserver 101.0.101.10
nameserver 101.0.101.11

Nameservers are provided to us by our data centre and in the case of the local address, blacklist lookups fail to work in SA without it. That said, the file has remained unmodified for some time now.

There are no visible updates to the system by way of upcp.

I've flushed out #donotdelete entries in our csf.deny file without any effect.

Are we heading into support ticket territory?

Thanks.

 

[cPanelLauren]

Hi @bloatedstoat


It's possible we may be heading that way but I'd like to try something first. Does your datacenter require you to use their DNS resolvers? If they do not I'd like to see if the issue is related to that. Could you comment out everything in /etc/resolv.conf and replace with google's resolvers temporarily? I'd like to see if you're still encountering the cPhulkd error after they're changed.


Thanks!

 

[bloatedstoat]

Thanks @cPanelLauren

Here's what I've found.

When UPCP ran early this morning this message appeared in our log file, at this point in time I had "disabled" cpHulk entirely and made no changes to resolv.conf.

[2018-08-09 04:26:42 +1000] warn [cPanel] (XID 5wdrys) The whois query for the address 'OUR_OWN_SERVER_IP' took longer than 30 seconds. at /usr/local/cpanel/Cpanel/Net/Whois/IP/Cached.pm line 96.
   Cpanel::Net::Whois::IP::Cached::__ANON__(Cpanel::Exception::Timeout=HASH(0x3c69e68)) called at /usr/local/cpanel/3rdparty/perl/526/lib64/perl5/cpanel_lib/Try/Tiny.pm line 120
   Try::Tiny::try(CODE(0x3b77840), Try::Tiny::Catch=REF(0x3776430)) called at /usr/local/cpanel/Cpanel/Net/Whois/IP/Cached.pm line 97
   Cpanel::Net::Whois::IP::Cached::lookup_address(Cpanel::Net::Whois::IP::Cached=HASH(0x3776610), "OUR_OWN_SERVER_IP") called at /usr/local/cpanel/Cpanel/iContact/Class/FromUserAction.pm line 92
   Cpanel::iContact::Class::FromUserAction::__ANON__() called at /usr/local/cpanel/3rdparty/perl/526/lib64/perl5/cpanel_lib/Try/Tiny.pm line 99
   eval {...} called at /usr/local/cpanel/3rdparty/perl/526/lib64/perl5/cpanel_lib/Try/Tiny.pm line 90
   Try::Tiny::try(CODE(0x3776280), Try::Tiny::Catch=REF(0x3af0d68)) called at /usr/local/cpanel/Cpanel/iContact/Class/FromUserAction.pm line 99
   Cpanel::iContact::Class::FromUserAction::_origin_info_hr(Cpanel::iContact::Class::Update::EndOfLife=HASH(0x355a2e0)) called at /usr/local/cpanel/Cpanel/iContact/Class/FromUserAction.pm line 42
   Cpanel::iContact::Class::FromUserAction::_template_args(Cpanel::iContact::Class::Update::EndOfLife=HASH(0x355a2e0)) called at /usr/local/cpanel/Cpanel/iContact/Class/Update/EndOfLife.pm line 74
   Cpanel::iContact::Class::Update::EndOfLife::_template_args(Cpanel::iContact::Class::Update::EndOfLife=HASH(0x355a2e0)) called at /usr/local/cpanel/Cpanel/iContact/Class/Update/EndOfLife.pm line 100
   Cpanel::iContact::Class::Update::EndOfLife::send(Cpanel::iContact::Class::Update::EndOfLife=HASH(0x355a2e0)) called at /usr/local/cpanel/Cpanel/iContact/Class.pm line 350
   Cpanel::iContact::Class::new("Cpanel::iContact::Class::Update::EndOfLife", "origin", "upcp", "source_ip_address", "OUR_OWN_SERVER_IP") called at /usr/local/cpanel/Cpanel/Notify.pm line 77
   Cpanel::Notify::__ANON__() called at /usr/local/cpanel/Cpanel/Notify.pm line 150
   Cpanel::Notify::_notification_backend("Update::EndOfLife", "No status set", 1, CODE(0x2cee710)) called at /usr/local/cpanel/Cpanel/Notify.pm line 79
   Cpanel::Notify::notification_class("constructor_args", ARRAY(0x2c82280), "class", "Update::EndOfLife", "application", "Update::EndOfLife") called at /usr/local/cpanel/scripts/upcp line 490
   scripts::upcp::__ANON__() called at /usr/local/cpanel/3rdparty/perl/526/lib64/perl5/cpanel_lib/Try/Tiny.pm line 99
   eval {...} called at /usr/local/cpanel/3rdparty/perl/526/lib64/perl5/cpanel_lib/Try/Tiny.pm line 90
   Try::Tiny::try(CODE(0x2606220)) called at /usr/local/cpanel/scripts/upcp line 493
   scripts::upcp::upcp() called at /usr/local/cpanel/scripts/upcp line 74

Later this morning I updated resolv.conf to use solely Google's resolvers:
nameserver 8.8.8.8
nameserver 8.8.4.4

The error messages described in my initial post regards cpHulk above appear to go away, I tailed the log file for a few hours.

I then updated resolv.conf to:
nameserver 127.0.0.1
nameserver 8.8.8.8
nameserver 8.8.4.4

The error messages returned.

I'll throw this in here too and this could quite easily be a red herring, but I did come across a few other interesting entries that have appeared since the cpHulk issue began, the entries on the 7th appear "prior" to any change to resolv.conf, those on the 9th are using solely Google's resolvers.

[2018-08-07 04:14:09 +1000] info [cpsrvd] Internal Server Error: "-" 500 The cPanel Server operation timed out at cpsrvd.pl line 532.
[2018-08-07 12:32:07 +1000] info [cpsrvd] Internal Server Error: "-" 500 The cPanel Server operation timed out at cpsrvd.pl line 532.
[2018-08-07 12:32:07 +1000] info [cpsrvd] Internal Server Error: "-" 500 The cPanel Server operation timed out at cpsrvd.pl line 532.
[2018-08-09 12:04:38 +1000] info [cpsrvd] Internal Server Error: "-" 500 The cPanel Server operation timed out at cpsrvd.pl line 532.
[2018-08-09 12:04:38 +1000] info [cpsrvd] Internal Server Error: "-" 500 The cPanel Server operation timed out at cpsrvd.pl line 532.
[2018-08-09 12:04:43 +1000] info [cpsrvd] Internal Server Error: "-" 500 The cPanel Server operation timed out at cpsrvd.pl line 532.

I will say I'm reluctant to remove 127.0.0.1 as a nameserver purely because the RBL lookups throw the "this query was blocked" error if I do and in our case RBL lookups are high value in terms of spam prevention. I'd sooner disable cpHulk entirely.

I've trawled the cron upcp logs from the 3rd August forward to today, our system remains on the target version of 11.72.0.10 and the only software update within that period was to LVE stats.

Thanks.

 

[cPanelLauren]

Hi @bloatedstoat

Just to clarify, are you saying the whois related error seemed to have resolved itself prior to modifying the resolv.conf?

I'll throw this in here too and this could quite easily be a red herring, but I did come across a few other interesting entries that have appeared since the cpHulk issue began, the entries on the 7th appear "prior" to any change to resolv.conf, those on the 9th are using solely Google's resolvers.

The errors being referenced here I don't think are specifically related to that same issue but without access I'm unsure.

I will say I'm reluctant to remove 127.0.0.1 as a nameserver purely because the RBL lookups throw the "this query was blocked" error if I do and in our case RBL lookups are high value in terms of spam prevention. I'd sooner disable cpHulk entirely.

I don't think that will be necessary, since it's seeing one of the google resolvers as a secondary.

 

[bloatedstoat]

Hello @cPanelMichael

No the issue persists, I was saying that there was an error when the Whois perl module was executed whilst cpHulk was disabled during the UPCP run that bore similarities with the cpHulk issue. The Whois module attempted to resolve our main server ip and timed out throwing the error.

1) cpHulk throws errors with our data centre's IPs and the loopback address in resolv.conf
2) cpHulk does NOT throw errors when solely using Google's resolvers.
3) cphulk throws errors when using Google's resolvers AND the loopback address

So 2 is the only instance where the errors go away - but we want the loopback in resolv.conf for the RBL lookups but that then re-introduces errors.

Thanks.

 

[cPanelLauren]

Hi @bloatedstoat


It does sound like a resolver issue but I'd like to confirm that in terms of the loopback. Can you please open a ticket using the link in my signature? Once open please reply with the Ticket ID here so that we can update this thread with the resolution once the ticket is resolved.


Thanks!

 

[bloatedstoat]

Thanks @cPanelLauren

The issue appears resolved as there have been no further errors logged with the original configuration. I have no idea why the problem presented in the first place and wish I did, always useful to know why.

I disabled cpHulk for a few days and then re-enabled it and since that time the errors have disappeared.

Thank you for your help.

 

[cPanelLauren]

Hi @bloatedstoat

I'm glad to hear that the issue is no longer occurring. I'm not sure exactly why but my assumption is there was some network issues with the resolvers which may now be resolved, though that's just an assumption. If it does occur again I would encourage you to open a ticket so we can look into it more in-depth.

Thanks!