Munin stopped working after an update

[Rodrigo Gomes]

After some update (56.xx to 58.xx) Munin stopped work. I tried to solve the problem, but without success.

Acess level: root

Cpanel plugins:
clamavconnector 0.99.2-2.cp1156
munin Version: 2.0.25-5.cp1158
csf v9.13

/etc/redhat-release:CentOS Linux release 7.2.1511 (Core)
/usr/local/cpanel/version:11.58.0.25
/var/cpanel/envtype:kvm
CPANEL=stable
Server version: Apache/2.4.23 (Unix)
Server built:   Aug 25 2016 17:44:41
Cpanel::Easy::Apache v3.34.4 rev9999
PHP 5.6.25 (cli) (built: Aug 25 2016 17:49:05)
Copyright (c) 1997-2016 The PHP Group
Zend Engine v2.6.0, Copyright (c) 1998-2016 Zend Technologies
    with the ionCube PHP Loader v4.7.5, Copyright (c) 2002-2014, by ionCube Ltd., and
    with Zend OPcache v7.0.6-dev, Copyright (c) 1999-2016, by Zend Technologies
    with Suhosin v0.9.38, Copyright (c) 2007-2015, by SektionEins GmbH
mysql  Ver 14.14 Distrib 5.6.31, for Linux (x86_64) using  EditLine wrapper

/var/log/munin/munin-node.log:

2016/09/02-19:10:01 CONNECT TCP Peer: "[::ffff:127.0.0.1]:38166" Local: "[::ffff:127.0.0.1]:4949"
2016/09/02-19:10:09 [6821] Error output from http_loadtime:
2016/09/02-19:10:09 [6821]     /etc/munin/plugins/http_loadtime: line 78: syntax error: unexpected end of file
2016/09/02-19:10:09 [6821] Service 'http_loadtime' exited with status 2/0.
2016/09/02-19:10:09 [6821] Error output from http_loadtime:
2016/09/02-19:10:09 [6821]     /etc/munin/plugins/http_loadtime: line 78: syntax error: unexpected end of file
2016/09/02-19:10:09 [6821] Service 'http_loadtime' exited with status 2/0.
2016/09/02-19:15:02 CONNECT TCP Peer: "[::ffff:127.0.0.1]:40546" Local: "[::ffff:127.0.0.1]:4949"
2016/09/02-19:15:09 [8040] Error output from http_loadtime:
2016/09/02-19:15:09 [8040]     /etc/munin/plugins/http_loadtime: line 78: syntax error: unexpected end of file
2016/09/02-19:15:09 [8040] Service 'http_loadtime' exited with status 2/0.
2016/09/02-19:15:09 [8040] Error output from http_loadtime:
2016/09/02-19:15:09 [8040]     /etc/munin/plugins/http_loadtime: line 78: syntax error: unexpected end of file
2016/09/02-19:15:09 [8040] Service 'http_loadtime' exited with status 2/0.
2016/09/02-19:20:02 CONNECT TCP Peer: "[::ffff:127.0.0.1]:42950" Local: "[::ffff:127.0.0.1]:4949"
2016/09/02-19:20:11 [9318] Error output from http_loadtime:
2016/09/02-19:20:11 [9318]     /etc/munin/plugins/http_loadtime: line 78: syntax error: unexpected end of file
2016/09/02-19:20:11 [9318] Service 'http_loadtime' exited with status 2/0.
2016/09/02-19:20:11 [9318] Error output from http_loadtime:
2016/09/02-19:20:11 [9318]     /etc/munin/plugins/http_loadtime: line 78: syntax error: unexpected end of file
2016/09/02-19:20:11 [9318] Service 'http_loadtime' exited with status 2/0.
2016/09/02-19:25:01 CONNECT TCP Peer: "[::ffff:127.0.0.1]:45190" Local: "[::ffff:127.0.0.1]:4949"
2016/09/02-19:25:08 [10493] Error output from http_loadtime:
2016/09/02-19:25:08 [10493]     /etc/munin/plugins/http_loadtime: line 78: syntax error: unexpected end of file
2016/09/02-19:25:08 [10493] Service 'http_loadtime' exited with status 2/0.
2016/09/02-19:25:08 [10493] Error output from http_loadtime:
2016/09/02-19:25:08 [10493]     /etc/munin/plugins/http_loadtime: line 78: syntax error: unexpected end of file
2016/09/02-19:25:08 [10493] Service 'http_loadtime' exited with status 2/0.

I've tried replacing the 'http_loadtime' for the latest release version of munin, the error disappeared, but still not load the images of munin.

 

[Rodrigo Gomes]

Uninstalling and reinstalling did not work too.

 

[cPanelMichael]

Hello,

It looks like one of the Munin plugins you have installed, http_loadtime, is not yet supported by default. Could you verify if the issue persists when disabling that plugin?

Thank you.

 

[Rodrigo Gomes]

I have deleted all files in /etc/munin/plugins, and reinstalled munin.
The error is gone, but still does not generate images.

2016/09/05-21:56:42 Server closing!
Process Backgrounded
2016/09/05-21:56:42 Munin::Node::Server (type Net::Server::Fork) starting! pid(6496)
Resolved [*]:4949 to [::]:4949, IPv6
Not including resolved host [0.0.0.0] IPv4 because it will be handled by [::] IPv6
Binding to TCP port 4949 on host :: with IPv6
Setting gid to "10 10"
2016/09/05-22:00:02 CONNECT TCP Peer: "[::ffff:127.0.0.1]:55840" Local: "[::ffff:127.0.0.1]:4949"
2016/09/05-22:05:01 CONNECT TCP Peer: "[::ffff:127.0.0.1]:57362" Local: "[::ffff:127.0.0.1]:4949"

 

[cPanelMichael]

The error is gone, but still does not generate images.

Hello,

You can manually initiate the munin update process via the following commands while logged in as "root" via SSH:

su - munin --shell=/bin/bash
/usr/local/cpanel/3rdparty/perl/522/bin/munin-cron
exit

Let us know if this helps.

Thank you.

 

[Rodrigo Gomes]

Hello,

You can manually initiate the munin update process via the following commands while logged in as "root" via SSH:

su - munin --shell=/bin/bash
/usr/local/cpanel/3rdparty/perl/522/bin/munin-cron
exit

Let us know if this helps.

Thank you.

I received this error:

# /usr/local/cpanel/3rdparty/perl/522/bin/munin-cron
Magic number checking on storable file failed at /usr/local/cpanel/3rdparty/perl/522/lib64/perl5/5.22.1/x86_64-linux-64int/Storable.pm line 410, at /usr/local/cpanel/3rdparty/perl/522/lib64/perl5/cpanel_lib/Munin/Master/Utils.pm line 279.
Magic number checking on storable file failed at /usr/local/cpanel/3rdparty/perl/522/lib64/perl5/5.22.1/x86_64-linux-64int/Storable.pm line 410, at /usr/local/cpanel/3rdparty/perl/522/lib64/perl5/cpanel_lib/Munin/Master/Utils.pm line 279.

 

[Rodrigo Gomes]

I tryed:
1 - I removed all plug-state files and reinstall = This error: "Magic number checking on storable file failed" not show anymore, but graphs don't work.
2 - I run Munin just with "memory" plugin, and reset plug-state files = Don't work.
3 - I run /usr/local/cpanel/scripts/check_cpanel_rpms --fix and resintalled munin again = Don't work.


The only thing that worked is below:

Hi,

I ran a command to remove munin while specifying --nodeps and --justdb and ran check_cpanel_rpms --fix to get it reintalled properly.

[22:44:50 server [email protected] /etc/munin]cPs# rpm -e --nodeps --justdb cpanel-perl-522-munin-2.0.25-5.cp1158.x86_64
[22:45:00 server [email protected] /etc/munin]cPs# /scripts/check_cpanel_rpms --fix
[2016-09-06 22:45:07 -0300]
[2016-09-06 22:45:07 -0300] Problems were detected with cPanel-provided files which are RPM controlled.
[2016-09-06 22:45:07 -0300] If you did not make these changes intentionally, you can correct them by running:
[2016-09-06 22:45:07 -0300]
[2016-09-06 22:45:07 -0300] > /usr/local/cpanel/scripts/check_cpanel_rpms --fix
[2016-09-06 22:45:07 -0300] The following RPMs are missing from your system:
[2016-09-06 22:45:07 -0300] cpanel-perl-522-munin-2.0.25-5.cp1158
[2016-09-06 22:45:31 -0300] Removing 0 broken rpms:
[2016-09-06 22:45:31 -0300] rpm: no packages given for erase
[2016-09-06 22:45:33 -0300] Downloading http://httpupdate.cpanel.net/RPM/11.58/centos/7/x86_64/rpm.sha512
[2016-09-06 22:45:33 -0300] Successfully verified signature for cpanel (key types: release).
[2016-09-06 22:45:33 -0300] Downloading http://httpupdate.cpanel.net/RPM/11.58/centos/7/x86_64/cpanel-perl-522-munin-2.0.25-5.cp1158.x86_64.rpm
[2016-09-06 22:45:34 -0300] Disabling service monitoring.
[2016-09-06 22:45:34 -0300] Hooks system enabled.
[2016-09-06 22:45:34 -0300] Checking for and running RPM::Versions 'pre' hooks for any RPMs about to be installed
[2016-09-06 22:45:34 -0300] All required 'pre' hooks have been run
[2016-09-06 22:45:38 -0300] No RPMS need to be uninstalled
[2016-09-06 22:45:38 -0300] Installing new rpms: cpanel-perl-522-munin-2.0.25-5.cp1158.x86_64.rpm
[2016-09-06 22:45:38 -0300] Preparing packages...
[2016-09-06 22:45:38 -0300] - checking munin user and group
[2016-09-06 22:45:38 -0300] User 'munin' already exists. ( permissions, owner and quota reset )
[2016-09-06 22:45:38 -0300] groupadd: group 'munin' already exists
[2016-09-06 22:45:38 -0300] useradd: user 'munin' already exists
[2016-09-06 22:45:38 -0300] cpanel-perl-522-munin-2.0.25-5.cp1158.x86_64
[2016-09-06 22:45:38 -0300] warning: /etc/munin/plugin-conf.d/cpanel.conf saved as /etc/munin/plugin-conf.d/cpanel.conf.rpmorig
[2016-09-06 22:45:40 -0300] InnoDB access granted ok.
[2016-09-06 22:45:41 -0300] enable was successful
[2016-09-06 22:45:43 -0300] Restarting munin-node (via systemctl): [ OK ]
[2016-09-06 22:45:43 -0300] Restarting apache (gracefully)....
[2016-09-06 22:45:45 -0300] This account is currently not available.
[2016-09-06 22:45:45 -0300] - running munin-cron for the first time
[2016-09-06 22:45:55 -0300] Hooks system enabled.
[2016-09-06 22:45:55 -0300] Checking for and running RPM::Versions 'post' hooks for any RPMs about to be installed
[2016-09-06 22:45:55 -0300] All required 'post' hooks have been run
[2016-09-06 22:45:55 -0300] Restoring service monitoring.

Thanks to the support and everyone who tried to help!

 

[cPanelMichael]

I'm happy to see the issue was resolved. Thank you for updating us with the outcome.

 

[Johan Carlsson]

I'm happy to see the issue was resolved. Thank you for updating us with the outcome.

Hello,
We are running a CENTOS 7.3 x86_64 server with WHM 62.0 (build 17), and I recently detected the same kind of error on our server.
Magic number checking on storable file failed at /usr/local/cpanel/3rdparty/perl/524/lib64/perl5/5.24.1/x86_64-linux-64int/Storable.pm line 410, at /usr/local/cpanel/3rdparty/perl/524/lib64/perl5/cpanel_lib/Munin/Master/Utils.pm line 279.
Magic number checking on storable file failed at /usr/local/cpanel/3rdparty/perl/524/lib64/perl5/5.24.1/x86_64-linux-64int/Storable.pm line 410, at /usr/local/cpanel/3rdparty/perl/524/lib64/perl5/cpanel_lib/Munin/Master/Utils.pm line 279.

I have tried the solution provided by Rodrigo, but to no avail.

 

[cPanelMichael]

Magic number checking on storable file failed at /usr/local/cpanel/3rdparty/perl/524/lib64/perl5/5.24.1/x86_64-linux-64int/Storable.pm line 410, at /usr/local/cpanel/3rdparty/perl/524/lib64/perl5/cpanel_lib/Munin/Master/Utils.pm line 279.

Hello,

Please post the output from the following command:

stat /var/lib/munin/limits.storable

I've seen previous reports of this issue when the above file is empty. If so, you can run the following command to move the file out of the way:

mv /var/lib/munin/limits.storable /root/munin-limits.storable

Then, reinstall the Munin RPM:

rpm -e --nodeps cpanel-perl-524-munin-2.0.25-4.cp1162.x86_64
/scripts/check_cpanel_rpms --fix

Thank you.