Munin stopped working after an update

Rodrigo Gomes

Well-Known Member
Apr 6, 2016
128
29
78
Brazil
cPanel Access Level
Root Administrator
After some update (56.xx to 58.xx) Munin stopped work. I tried to solve the problem, but without success.

Acess level: root

Cpanel plugins:
clamavconnector 0.99.2-2.cp1156
munin Version: 2.0.25-5.cp1158
csf v9.13

Code:
/etc/redhat-release:CentOS Linux release 7.2.1511 (Core)
/usr/local/cpanel/version:11.58.0.25
/var/cpanel/envtype:kvm
CPANEL=stable
Server version: Apache/2.4.23 (Unix)
Server built:   Aug 25 2016 17:44:41
Cpanel::Easy::Apache v3.34.4 rev9999
PHP 5.6.25 (cli) (built: Aug 25 2016 17:49:05)
Copyright (c) 1997-2016 The PHP Group
Zend Engine v2.6.0, Copyright (c) 1998-2016 Zend Technologies
    with the ionCube PHP Loader v4.7.5, Copyright (c) 2002-2014, by ionCube Ltd., and
    with Zend OPcache v7.0.6-dev, Copyright (c) 1999-2016, by Zend Technologies
    with Suhosin v0.9.38, Copyright (c) 2007-2015, by SektionEins GmbH
mysql  Ver 14.14 Distrib 5.6.31, for Linux (x86_64) using  EditLine wrapper

/var/log/munin/munin-node.log:
Code:
2016/09/02-19:10:01 CONNECT TCP Peer: "[::ffff:127.0.0.1]:38166" Local: "[::ffff:127.0.0.1]:4949"
2016/09/02-19:10:09 [6821] Error output from http_loadtime:
2016/09/02-19:10:09 [6821]     /etc/munin/plugins/http_loadtime: line 78: syntax error: unexpected end of file
2016/09/02-19:10:09 [6821] Service 'http_loadtime' exited with status 2/0.
2016/09/02-19:10:09 [6821] Error output from http_loadtime:
2016/09/02-19:10:09 [6821]     /etc/munin/plugins/http_loadtime: line 78: syntax error: unexpected end of file
2016/09/02-19:10:09 [6821] Service 'http_loadtime' exited with status 2/0.
2016/09/02-19:15:02 CONNECT TCP Peer: "[::ffff:127.0.0.1]:40546" Local: "[::ffff:127.0.0.1]:4949"
2016/09/02-19:15:09 [8040] Error output from http_loadtime:
2016/09/02-19:15:09 [8040]     /etc/munin/plugins/http_loadtime: line 78: syntax error: unexpected end of file
2016/09/02-19:15:09 [8040] Service 'http_loadtime' exited with status 2/0.
2016/09/02-19:15:09 [8040] Error output from http_loadtime:
2016/09/02-19:15:09 [8040]     /etc/munin/plugins/http_loadtime: line 78: syntax error: unexpected end of file
2016/09/02-19:15:09 [8040] Service 'http_loadtime' exited with status 2/0.
2016/09/02-19:20:02 CONNECT TCP Peer: "[::ffff:127.0.0.1]:42950" Local: "[::ffff:127.0.0.1]:4949"
2016/09/02-19:20:11 [9318] Error output from http_loadtime:
2016/09/02-19:20:11 [9318]     /etc/munin/plugins/http_loadtime: line 78: syntax error: unexpected end of file
2016/09/02-19:20:11 [9318] Service 'http_loadtime' exited with status 2/0.
2016/09/02-19:20:11 [9318] Error output from http_loadtime:
2016/09/02-19:20:11 [9318]     /etc/munin/plugins/http_loadtime: line 78: syntax error: unexpected end of file
2016/09/02-19:20:11 [9318] Service 'http_loadtime' exited with status 2/0.
2016/09/02-19:25:01 CONNECT TCP Peer: "[::ffff:127.0.0.1]:45190" Local: "[::ffff:127.0.0.1]:4949"
2016/09/02-19:25:08 [10493] Error output from http_loadtime:
2016/09/02-19:25:08 [10493]     /etc/munin/plugins/http_loadtime: line 78: syntax error: unexpected end of file
2016/09/02-19:25:08 [10493] Service 'http_loadtime' exited with status 2/0.
2016/09/02-19:25:08 [10493] Error output from http_loadtime:
2016/09/02-19:25:08 [10493]     /etc/munin/plugins/http_loadtime: line 78: syntax error: unexpected end of file
2016/09/02-19:25:08 [10493] Service 'http_loadtime' exited with status 2/0.
print.png

I've tried replacing the 'http_loadtime' for the latest release version of munin, the error disappeared, but still not load the images of munin.
 
Last edited by a moderator:

cPanelMichael

Administrator
Staff member
Apr 11, 2011
47,904
2,237
463
Hello,

It looks like one of the Munin plugins you have installed, http_loadtime, is not yet supported by default. Could you verify if the issue persists when disabling that plugin?

Thank you.
 
  • Like
Reactions: Rodrigo Gomes

Rodrigo Gomes

Well-Known Member
Apr 6, 2016
128
29
78
Brazil
cPanel Access Level
Root Administrator
I have deleted all files in /etc/munin/plugins, and reinstalled munin.
The error is gone, but still does not generate images.

print.png


Code:
2016/09/05-21:56:42 Server closing!
Process Backgrounded
2016/09/05-21:56:42 Munin::Node::Server (type Net::Server::Fork) starting! pid(6496)
Resolved [*]:4949 to [::]:4949, IPv6
Not including resolved host [0.0.0.0] IPv4 because it will be handled by [::] IPv6
Binding to TCP port 4949 on host :: with IPv6
Setting gid to "10 10"
2016/09/05-22:00:02 CONNECT TCP Peer: "[::ffff:127.0.0.1]:55840" Local: "[::ffff:127.0.0.1]:4949"
2016/09/05-22:05:01 CONNECT TCP Peer: "[::ffff:127.0.0.1]:57362" Local: "[::ffff:127.0.0.1]:4949"
 
Last edited by a moderator:

cPanelMichael

Administrator
Staff member
Apr 11, 2011
47,904
2,237
463
The error is gone, but still does not generate images.
Hello,

You can manually initiate the munin update process via the following commands while logged in as "root" via SSH:

Code:
su - munin --shell=/bin/bash
/usr/local/cpanel/3rdparty/perl/522/bin/munin-cron
exit
Let us know if this helps.

Thank you.
 
  • Like
Reactions: Rodrigo Gomes

Rodrigo Gomes

Well-Known Member
Apr 6, 2016
128
29
78
Brazil
cPanel Access Level
Root Administrator
Hello,

You can manually initiate the munin update process via the following commands while logged in as "root" via SSH:

Code:
su - munin --shell=/bin/bash
/usr/local/cpanel/3rdparty/perl/522/bin/munin-cron
exit
Let us know if this helps.

Thank you.

I received this error:

Code:
# /usr/local/cpanel/3rdparty/perl/522/bin/munin-cron
Magic number checking on storable file failed at /usr/local/cpanel/3rdparty/perl/522/lib64/perl5/5.22.1/x86_64-linux-64int/Storable.pm line 410, at /usr/local/cpanel/3rdparty/perl/522/lib64/perl5/cpanel_lib/Munin/Master/Utils.pm line 279.
Magic number checking on storable file failed at /usr/local/cpanel/3rdparty/perl/522/lib64/perl5/5.22.1/x86_64-linux-64int/Storable.pm line 410, at /usr/local/cpanel/3rdparty/perl/522/lib64/perl5/cpanel_lib/Munin/Master/Utils.pm line 279.
 

Rodrigo Gomes

Well-Known Member
Apr 6, 2016
128
29
78
Brazil
cPanel Access Level
Root Administrator
I tryed:
1 - I removed all plug-state files and reinstall = This error: "Magic number checking on storable file failed" not show anymore, but graphs don't work.
2 - I run Munin just with "memory" plugin, and reset plug-state files = Don't work.
3 - I run /usr/local/cpanel/scripts/check_cpanel_rpms --fix and resintalled munin again = Don't work.


The only thing that worked is below:

Geoffrey Stephens said:
Hi,

I ran a command to remove munin while specifying --nodeps and --justdb and ran check_cpanel_rpms --fix to get it reintalled properly.
Code:
[22:44:50 server [email protected] /etc/munin]cPs# rpm -e --nodeps --justdb cpanel-perl-522-munin-2.0.25-5.cp1158.x86_64
[22:45:00 server [email protected] /etc/munin]cPs# /scripts/check_cpanel_rpms --fix
[2016-09-06 22:45:07 -0300]
[2016-09-06 22:45:07 -0300] Problems were detected with cPanel-provided files which are RPM controlled.
[2016-09-06 22:45:07 -0300] If you did not make these changes intentionally, you can correct them by running:
[2016-09-06 22:45:07 -0300]
[2016-09-06 22:45:07 -0300] > /usr/local/cpanel/scripts/check_cpanel_rpms --fix
[2016-09-06 22:45:07 -0300] The following RPMs are missing from your system:
[2016-09-06 22:45:07 -0300] cpanel-perl-522-munin-2.0.25-5.cp1158
[2016-09-06 22:45:31 -0300] Removing 0 broken rpms:
[2016-09-06 22:45:31 -0300] rpm: no packages given for erase
[2016-09-06 22:45:33 -0300] Downloading http://httpupdate.cpanel.net/RPM/11.58/centos/7/x86_64/rpm.sha512
[2016-09-06 22:45:33 -0300] Successfully verified signature for cpanel (key types: release).
[2016-09-06 22:45:33 -0300] Downloading http://httpupdate.cpanel.net/RPM/11.58/centos/7/x86_64/cpanel-perl-522-munin-2.0.25-5.cp1158.x86_64.rpm
[2016-09-06 22:45:34 -0300] Disabling service monitoring.
[2016-09-06 22:45:34 -0300] Hooks system enabled.
[2016-09-06 22:45:34 -0300] Checking for and running RPM::Versions 'pre' hooks for any RPMs about to be installed
[2016-09-06 22:45:34 -0300] All required 'pre' hooks have been run
[2016-09-06 22:45:38 -0300] No RPMS need to be uninstalled
[2016-09-06 22:45:38 -0300] Installing new rpms: cpanel-perl-522-munin-2.0.25-5.cp1158.x86_64.rpm
[2016-09-06 22:45:38 -0300] Preparing packages...
[2016-09-06 22:45:38 -0300] - checking munin user and group
[2016-09-06 22:45:38 -0300] User 'munin' already exists. ( permissions, owner and quota reset )
[2016-09-06 22:45:38 -0300] groupadd: group 'munin' already exists
[2016-09-06 22:45:38 -0300] useradd: user 'munin' already exists
[2016-09-06 22:45:38 -0300] cpanel-perl-522-munin-2.0.25-5.cp1158.x86_64
[2016-09-06 22:45:38 -0300] warning: /etc/munin/plugin-conf.d/cpanel.conf saved as /etc/munin/plugin-conf.d/cpanel.conf.rpmorig
[2016-09-06 22:45:40 -0300] InnoDB access granted ok.
[2016-09-06 22:45:41 -0300] enable was successful
[2016-09-06 22:45:43 -0300] Restarting munin-node (via systemctl): [ OK ]
[2016-09-06 22:45:43 -0300] Restarting apache (gracefully)....
[2016-09-06 22:45:45 -0300] This account is currently not available.
[2016-09-06 22:45:45 -0300] - running munin-cron for the first time
[2016-09-06 22:45:55 -0300] Hooks system enabled.
[2016-09-06 22:45:55 -0300] Checking for and running RPM::Versions 'post' hooks for any RPMs about to be installed
[2016-09-06 22:45:55 -0300] All required 'post' hooks have been run
[2016-09-06 22:45:55 -0300] Restoring service monitoring.
Thanks to the support and everyone who tried to help!
 
Last edited by a moderator:

cPanelMichael

Administrator
Staff member
Apr 11, 2011
47,904
2,237
463
I'm happy to see the issue was resolved. Thank you for updating us with the outcome.
 
  • Like
Reactions: Rodrigo Gomes

Johan Carlsson

Registered
Mar 23, 2017
1
0
1
Sweden
cPanel Access Level
Root Administrator
I'm happy to see the issue was resolved. Thank you for updating us with the outcome.
Hello,
We are running a CENTOS 7.3 x86_64 server with WHM 62.0 (build 17), and I recently detected the same kind of error on our server.
Magic number checking on storable file failed at /usr/local/cpanel/3rdparty/perl/524/lib64/perl5/5.24.1/x86_64-linux-64int/Storable.pm line 410, at /usr/local/cpanel/3rdparty/perl/524/lib64/perl5/cpanel_lib/Munin/Master/Utils.pm line 279.
Magic number checking on storable file failed at /usr/local/cpanel/3rdparty/perl/524/lib64/perl5/5.24.1/x86_64-linux-64int/Storable.pm line 410, at /usr/local/cpanel/3rdparty/perl/524/lib64/perl5/cpanel_lib/Munin/Master/Utils.pm line 279.

I have tried the solution provided by Rodrigo, but to no avail.
 

cPanelMichael

Administrator
Staff member
Apr 11, 2011
47,904
2,237
463
Magic number checking on storable file failed at /usr/local/cpanel/3rdparty/perl/524/lib64/perl5/5.24.1/x86_64-linux-64int/Storable.pm line 410, at /usr/local/cpanel/3rdparty/perl/524/lib64/perl5/cpanel_lib/Munin/Master/Utils.pm line 279.
Hello,

Please post the output from the following command:

Code:
stat /var/lib/munin/limits.storable
I've seen previous reports of this issue when the above file is empty. If so, you can run the following command to move the file out of the way:

Code:
mv /var/lib/munin/limits.storable /root/munin-limits.storable
Then, reinstall the Munin RPM:

Code:
rpm -e --nodeps cpanel-perl-524-munin-2.0.25-4.cp1162.x86_64
/scripts/check_cpanel_rpms --fix
Thank you.