SOLVED queueprocd fails with json parse error

gdimi

Member
Jun 2, 2014
7
2
53
cPanel Access Level
Root Administrator
Hello to all,

after some pool metadata failure and recovery, all seems ok but I discovered that queueprocd fails with some weird JSON data error

here is the log output:
Code:
------------------------------
warn [queueprocd] 1[2017-08-30 22:24:01 +0300] warn [queueprocd] Moving bad state file and retry.
 at /usr/local/cpanel/Cpanel/LoggerAdapter.pm line 27.
        Cpanel::LoggerAdapter::warn(Cpanel::LoggerAdapter=HASH(0x1a24630), "Moving bad state file and retry.\x{a}") called at /usr/local/cpanel/3rdparty/perl/524/lib64/perl5/cpanel_lib/cPanel/StateFile.pm line 90
        cPanel::StateFile::_warn("cPanel::StateFile", "Moving bad state file and retry.\x{a}") called at /usr/local/cpanel/3rdparty/perl/524/lib64/perl5/cpanel_lib/cPanel/TaskQueue.pm line 266
        cPanel::TaskQueue::new("cPanel::TaskQueue", HASH(0x1dd1940)) called at /usr/local/cpanel/libexec/queueprocd line 143
[2017-08-30 22:24:02 +0300] die [queueprocd] 1Cpanel::Exception::JSONParseError/(XID 9v28e6) The system failed to parse the [asis,JSON] stream data “^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@” from the file “GLOB(0x19eb150)” because of an error: malformed JSON string, neither tag, array, object, number, string or atom, at character offset 0 (before "\x{0}\x{0}\x{0}\x{0}...") at /usr/local/cpanel/Cpanel/JSON.pm line 123.

 at /usr/local/cpanel/Cpanel/Exception/CORE.pm line 330.
        Cpanel::Exception::create("JSONParseError", HASH(0x2177d70)) called at /usr/local/cpanel/Cpanel/JSON.pm line 146
        Cpanel::JSON::_throw_json_error("malformed JSON string, neither tag, array, object, number, st"..., GLOB(0x19eb150), SCALAR(0x2028ca8)) called at /usr/local/cpanel/Cpanel/JSON.pm line 124
        Cpanel::JSON::Load("\x{0}\x{0}\x{0}\x{0}\x{0}\x{0}\x{0}\x{0}\x{0}\x{0}\x{0}\x{0}\x{0}\x{0}\x{0}\x{0}\x{0}\x{0}\x{0}\x{0}\x{0}\x{0}\x{0}\x{0}\x{0}\x{0}\x{0}\x{0}\x{0}\x{0}\x{0}\x{0}\x{0}\x{0}\x{0}\x{0}\x{0}\x{0}\x{0}\x{0}\x{0}\x{0}\x{0}\x{0}\x{0}\x{0}\x{0}\x{0}\x{0}\x{0}\x{0}\x{0}\x{0}\x{0}\x{0}\x{0}\x{0}\x{0}\x{0}\x{0}\x{0}"..., GLOB(0x19eb150)) called at /usr/local/cpanel/Cpanel/JSON.pm line 288
        Cpanel::JSON::_LoadFile(GLOB(0x19eb150), 0, 0, undef, 0) called at /usr/local/cpanel/Cpanel/JSON.pm line 222
        Cpanel::JSON::LoadFile(GLOB(0x19eb150)) called at /usr/local/cpanel/Cpanel/TaskQueue/Serializer.pm line 38
        Cpanel::TaskQueue::Serializer::load("Cpanel::TaskQueue::Serializer", GLOB(0x19eb150)) called at /usr/local/cpanel/3rdparty/perl/524/lib64/perl5/cpanel_lib/cPanel/TaskQueue/Scheduler.pm line 232
        cPanel::TaskQueue::Scheduler::load_from_cache(cPanel::TaskQueue::Scheduler=HASH(0x19bb000), GLOB(0x19eb150)) called at /usr/local/cpanel/3rdparty/perl/524/lib64/perl5/cpanel_lib/cPanel/StateFile.pm line 453
        cPanel::StateFile::_resynch(cPanel::StateFile=HASH(0x208d278), undef, 1504119246, 464) called at /usr/local/cpanel/3rdparty/perl/524/lib64/perl5/cpanel_lib/cPanel/StateFile.pm line 431
        cPanel::StateFile::synch(cPanel::StateFile=HASH(0x208d278)) called at /usr/local/cpanel/3rdparty/perl/524/lib64/perl5/cpanel_lib/cPanel/StateFile.pm line 357
        cPanel::StateFile::new("cPanel::StateFile", HASH(0x1a24d68)) called at /usr/local/cpanel/3rdparty/perl/524/lib64/perl5/cpanel_lib/cPanel/TaskQueue/Scheduler.pm line 182
        eval {...} called at /usr/local/cpanel/3rdparty/perl/524/lib64/perl5/cpanel_lib/cPanel/TaskQueue/Scheduler.pm line 184
        cPanel::TaskQueue::Scheduler::new("cPanel::TaskQueue::Scheduler", HASH(0x1dd1a30)) called at /usr/local/cpanel/libexec/queueprocd line 151

----------------------
any ideas on which file may be corrupted?

Thanx in advance, George
 
Last edited by a moderator:

gdimi

Member
Jun 2, 2014
7
2
53
cPanel Access Level
Root Administrator
ok I found it.

I search for recently modified json files in /var/cpanel and /usr/local/cpanel/ and found that
/var/cpanel/taskqueue/servers_sched.json

was corrupted. So I nulled its contents and restarted queueprocd ( /usr/local/cpanel/scripts/restartsrv_queueprocd) and this did the trick!
 

gdimi

Member
Jun 2, 2014
7
2
53
cPanel Access Level
Root Administrator
just a followup, eventually I realised that erasing the contents of servers_sched.json wasn't enough because
a) null content is not valid JSON
b) there has to be a minimum content like ["TaskScheduler","X",{"waiting_queue":[],"nextid":XXX}]

without that content, queueprocd starts ok but then fails afterwards
 
  • Like
Reactions: cPanelMichael

cPanelMichael

Administrator
Staff member
Apr 11, 2011
47,904
2,218
463
Hello,

I'm glad to see you were able to solve the issue. You can also remove the file completely and then restart queueprocd to regenerate the file if it's corrupted. EX:

Code:
# mv  /var/cpanel/taskqueue/servers_sched.json /root/servers_sched.json.backup
# /scripts/restartsrv_queueprocd
# cat /var/cpanel/taskqueue/servers_sched.json
["TaskScheduler","2",{"waiting_queue":[],"nextid":719}]
Thank you.