SpamAssassin constantly crashing, reinstall script fails

[Smartypants]

Encountering various problems with SpamAssassin on one server recently - it's crashing at LEAST every 2 hours and the "FAILED" notification EMail doesn't provide any useful information as to why, just this message:

"Died at /usr/local/cpanel/Cpanel/ServiceManager/Services/Spamd.pm line 188.
...propagated at /usr/local/cpanel/Cpanel/ServiceManager/Services/Spamd.pm line 200.

The subprocess “/usr/local/cpanel/scripts/restartsrv_spamd” reported error number 69 when it ended."

I haven't been able find any useful info by Google'ing those error messages.

I also have a training script that runs every 24 hours to train SA on a set of folders containing spam messages - and the cron notification EMails now contain more error message than actual useful output, E.g.:

"SPAM (learn_spam):
Checking read messages to learn SPAM: Bareword found where operator expected at /usr/local/cpanel/Cpanel/IP/Collapse.pm line 10, near "tr/://dr"
Unquoted string "r" may clash with future reserved word at /usr/local/cpanel/Cpanel/IP/Collapse.pm line 10.
plugin: failed to parse plugin /usr/local/cpanel/Cpanel/Mail/SpamAssassin/Plugin/P0f.pm: syntax error at /usr/local/cpanel/Cpanel/IP/Collapse.pm line 10, near "tr/://dr "
Compilation failed in require at /usr/local/cpanel/Cpanel/IP/Convert.pm line 11.
BEGIN failed--compilation aborted at /usr/local/cpanel/Cpanel/IP/Convert.pm line 11.
Compilation failed in require at /usr/local/cpanel/Cpanel/Net/P0f.pm line 13.
BEGIN failed--compilation aborted at /usr/local/cpanel/Cpanel/Net/P0f.pm line 13.
Compilation failed in require at /usr/local/cpanel/Cpanel/Mail/SpamAssassin/Plugin/P0f.pm line 52.
BEGIN failed--compilation aborted at /usr/local/cpanel/Cpanel/Mail/SpamAssassin/Plugin/P0f.pm line 53.
Compilation failed in require at /usr/local/lib/perl5/site_perl/5.8.8/Mail/SpamAssassin/PluginHandler.pm line 109."

I found a few references via Google to a "/scripts/fixspamassassinfailedupdate" script - but that command fails, so it appears to no longer be included with CPanel. The only option I could find that looks like it might work is forcibly reinstalling SpamAssassin... but even the command to do that fails with a bunch of cryptic error messages that are likely useless to anyone who isn't a Perl dev:

"Search pattern not terminated at /usr/local/cpanel/Cpanel/LoadFile/ReadFast.pm line 100.
Compilation failed in require at /usr/local/cpanel/Cpanel/LoadFile.pm line 13.
BEGIN failed--compilation aborted at /usr/local/cpanel/Cpanel/LoadFile.pm line 13.
Compilation failed in require at /usr/local/cpanel/Cpanel/CachedCommand.pm line 11.
BEGIN failed--compilation aborted at /usr/local/cpanel/Cpanel/CachedCommand.pm line 11.
Compilation failed in require at /usr/local/cpanel/Cpanel/cPCPAN/Install.pm line 74."

Is there any fix/workaround for this comedy of errors?

 

[cPanelLauren]

Hi @Smartypants

Out of curiosity what's your memory usage at? You can view stats on the last 12 hours with sysstats:

sar -r

The instructions you were looking at for reinstalling seem to be old, we package SA in an RPM now so simply reinstalling the package should do the trick:

First find which version you're running:

rpm -qa |grep SpamAssassin

The reinstall:

yum reinstall packagename

Thanks!

 

[Smartypants]

Hi @Smartypants

Out of curiosity what's your memory usage at? You can view stats on the last 12 hours with sysstats:

sar -r

I assume it's the %memused column that's relevant? It's showing a min of 77.51%, and a max of 94.36%.

The instructions you were looking at for reinstalling seem to be old, we package SA in an RPM now so simply reinstalling the package should do the trick:

First find which version you're running:

rpm -qa |grep SpamAssassin

The reinstall:

yum reinstall packagename

Thanks!

Not working. Output of the first command:

cpanel-perl-526-Mail-SpamAssassin-3.004002-1.cp1170.x86_64

And attempting to run "yum reinstall cpanel-perl-526-Mail-SpamAssassin-3.004002-1.cp1170.x86_64" produces an error message:

Installed package cpanel-perl-526-Mail-SpamAssassin-3.004002-1.cp1170.x86_64 not available.
Error: Nothing to do

 

[cPanelLauren]

Actually that's telling you that the package you have is no longer available. Which makes sense considering it seems to be in an older perl version. What happens when you attempt to update SpamAssassin?

The way I would do this is different than simply running yum.


1. I would run the following:

rpm -e --nodeps cpanel-perl-526-Mail-SpamAssassin-3.004002-1.cp1170.x86_64

2. Then run

/scripts/check_cpanel_rpms --fix

which should find that SpamAssassin is missing from the system and reinstall + resolve any dependencies

Thanks!

 

[cPanelLauren]

I also missed this initially:

I assume it's the %memused column that's relevant? It's showing a min of 77.51%, and a max of 94.36%.

This is important but it's actually the %commit that I was looking for.

 

[Smartypants]

Actually that's telling you that the package you have is no longer available. Which makes sense considering it seems to be in an older perl version.

"Old" as in older than it should be? Is that something that should/can be addressed?

What happens when you attempt to update SpamAssassin?

Running "yum update cpanel-perl-526-Mail-SpamAssassin-3.004002-1.cp1170.x86_64" produces:

No Packages marked for Update

The way I would do this is different than simply running yum.

Yes, I would normally try to do that with rpm as well - I tried it with yum because I was following the instructions you posted.

1. I would run the following:

rpm -e --nodeps cpanel-perl-526-Mail-SpamAssassin-3.004002-1.cp1170.x86_64

2. Then run

/scripts/check_cpanel_rpms --fix

which should find that SpamAssassin is missing from the system and reinstall + resolve any dependencies

Thanks!

This seems to have worked - no crashes in the 2-3 hours since following those steps. I'll follow-up in a day or two if it starts crashing again, or to confirm that the issue is resolved if not. Thanks!

 

[cPanelLauren]

Yes, I would normally try to do that with rpm as well - I tried it with yum because I was following the instructions you posted.

You wouldn't really want to update a package with check_cpanel_rpms - the reason for me having you check/reinstall with yum was to ensure that in the event there was an update available it was handled and to see if the package existed. To install a differently named cPanel package the best way is to use check_cpanel_rpms which is why I had you do that later.

This seems to have worked - no crashes in the 2-3 hours since following those steps. I'll follow-up in a day or two if it starts crashing again, or to confirm that the issue is resolved if not. Thanks!

I'm glad the updated version is working for you and I'm happy to have been able to help. Let us know how it goes and if you have any further issues.


Thanks!

 

[Smartypants]

Update: it was fine for a few hours, but it started crashing again around 3 Eastern.

Also, I noticed that the check_cpanel_rpms command reinstalled the same version of SpamAssassin - was it supposed to install an updated version?

I followed the instructions at Apache SpamAssassin - Version 68 Documentation - cPanel Documentation to check my SpamAssassin version on the server where the issue is occurring, and compared it against another server where SpamAssassin is running without any problems - but both show:

SpamAssassin version 3.4.2
  running on Perl version 5.26.0

Though on the server where the issue is occurring, it looks like there's a different versions of SpamAssassin installed at the system level - if I just run "spamassassin --version," then it reports:

SpamAssassin version 3.4.1
  running on Perl version 5.8.8

Is it possible that the separate copies/versions of SpamAssassin are causing the issue?

 

[cPanelLauren]

It could definitely be the cause but you showed only one installed previously. Can you run the following now?

rpm -qa |grep SpamAssassin

 

[Smartypants]

It could definitely be the cause but you showed only one installed previously.

Yes, and that command still produces the same output:

[email protected] [/scripts]# rpm -qa |grep SpamAssassin
cpanel-perl-526-Mail-SpamAssassin-3.004002-1.cp1170.x86_64

 

[cPanelLauren]

Hi @Smartypants

I actually believe the issue might be related to the Perl version on the server, you don't have two installations of SA what you have is two separate Perl versions and SA seems to be trying to run on both. Can you please open a ticket using the link in my signature? Once open please reply with the Ticket ID here so that we can update this thread with the resolution once the ticket is resolved.


Thanks!

 

[Smartypants]

I actually believe the issue might be related to the Perl version on the server, you don't have two installations of SA what you have is two separate Perl versions and SA seems to be trying to run on both. Can you please open a ticket using the link in my signature? Once open please reply with the Ticket ID here so that we can update this thread with the resolution once the ticket is resolved.

Ticket opened: 11359617

 

[cPanelLauren]

Thanks @Smartypants

I've noted and am following that ticket. I'll update here with more information when it's available.

 

[eugenevdm.host]

I'm curious what the outcome of this was, as we're having a similar problem on a small server with only two sites. So far I suspect lack of RAM.

 

[cPRex]

The original ticket mentioned in this thread was closed without resolution.

@eugenevdm.host - can you check /var/log/messages to see if any processes are being killed due to the lack of memory on the system?

 

[eugenevdm.host]

Hi here @cPRex,

Yeah looks a big dodgy:

[[email protected] ~]# cat /var/log/messages | grep oom-kill
Jan  8 05:19:54 us-xyz kernel: mysqld invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
Jan  8 05:19:58 us-xyz kernel: php-cgi invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
Jan  8 05:20:03 us-xyz kernel: php-cgi invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0

[[email protected] ~]# free -m
              total        used        free      shared  buff/cache   available
Mem:           1838         649         196          59         992         948
Swap:          1023         142         881

Look I don't really like running a server with this little amount of RAM anyway, but had to get it up to start that franchise. There are currently only two smalls sites on there. But I'll be monitoring the logs and if I get daily `ooms` I'm going to upgrade. Thanks a lot for that tip!

 

[cPRex]

I'm glad that helped! With other services being killed due to RAM issues that's almost certainly the problem, but let me know if you need anything else!