Increase in clamd failures after manually updating

Since September 15, 2018, there has been an alarming increase in cpanel monitoring emails related to clamd failures. Since 1:08 AM this morning, there have been nearly 60 notifications related to this. It always seems to recover but I'd like to minimize or eliminate these errors.

What information do you need from me to help troubleshoot this?

 

Here's what I found:
The contents of this file (/usr/local/cpanel/logs/error_log) date all the way back to the incept date of my VPS. In a subset of the log beginning at September 1, 2018, there is nothing related to clamd.

The contents of this file (/var/cpanel/clam-update.log) were actually found at another location (/var/log/clam-update.log). My first question is related to the location ... should it be where you initially suggested or is it okay where it is? Most of what seems notable in this log is a repeated warning about CLAMAV being outdated. I found a how-to link related to updating it manually here:
ClamavNet.

What is strange about this warning regarding outdated CLAMAV is that there have been several notifications dating all the way back to May 2016. I've never had to manually update it before so will this self correct or does it actually require manual intervention?

I never found this file: /var/log/messages, at least not anywhere in /var/log/. Is there something that I need to do to enable messages on my VPS?

I found another article before I posted my inquiry here. Here's the link:
ttps://documentation.cpanel.net/display/74Docs/Configure+ClamAV+Scanner#59c785fb8e6b40ff909c553292031d32

I have a small VPS with only 2GB of memory and 80GB of disk. In spurious searches, I've noted that there could be memory constraints that are solvable to make ClamAV work peacefully on servers with similarly limited configurations. So, my question to you related the last link I dropped is whether there is something from the cpanel.net documentation that I should implement to avoid capping out memory?

I'd like to wait to hear back from you on what I've posted before I try the restart command which was last on your list. Besides, hile I was typing this reply, I noticed that a new version of CENTOS (v74.0.8) was available so I am implementing it now. It is possible that cpanel is rolling out a fix with this upgrade. (fingers crossed)

 

It's very hard to tell if the upgrade solved the problem because it appears that there was a failure within an hour of the upgrade that took the nameserver service down with it. (groan)

It's good to know the log is in the correct location. Here is the most recent entry from that log:

Code:

ClamAV update process started at Mon Sep 17 15:17:31 2018
WARNING: Your ClamAV installation is OUTDATED!
WARNING: Local version: 0.100.0 Recommended version: 0.100.1
DON'T PANIC! Read ClamavNet
main.cld is up to date (version: 58, sigs: 4566249, f-level: 60, builder: sigmgr)
daily.cld is up to date (version: 24948, sigs: 2090217, f-level: 63, builder: neo)
bytecode.cld is up to date (version: 327, sigs: 91, f-level: 63, builder: neo)

Here are the results of the command you told me to use:

Code:

root@server [~]# journalctl |grep clamd
Sep 18 10:46:18 server.example.com systemd[1]: Starting clamd antivirus daemon...
Sep 18 10:47:27 server.example.com systemd[1]: Started clamd antivirus daemon.
Sep 18 11:08:07 server.example.com systemd[1]: clamd.service: main process exited, code=killed, status=9/KILL
Sep 18 11:08:07 server.example.com systemd[1]: Unit clamd.service entered failed state.
Sep 18 11:08:07 server.example.com systemd[1]: clamd.service failed.

This morning, I could not login to my VPS nor collect email. Once I got it rebooted, I downloaded email and found three failure notifications from yesterday evening. Here is the output from those:

spamd notification:

Code:

Raw Output: The subprocess reported error number 69 when it ended.
(XID jzn59x) The service “spamd” failed to send the expected response to host “127.0.0.1” and port “783” because of an error: The service did not pass the built-in GTUBE test.

The subprocess “/usr/local/cpanel/scripts/restartsrv_spamd” reported error number 69 when it ended.

Sep 17 16:51:13 server.example.com spamd[718]: prefork: child states: I
Sep 17 16:51:13 server.example.com spamd[718]: spamd: handled cleanup of child pid [858] due to SIGCHLD: KILLED, signal 9 (0009)
Sep 17 16:51:13 server.example.com spamd[718]: spamd: server successfully spawned child process, pid 5130
Sep 17 16:51:14 server.example.com spamd[718]: prefork: child states: I
Sep 17 16:51:15 server.example.com spamd[718]: prefork: child states: I
Sep 17 16:51:15 server.example.com spamd[718]: spamd: handled cleanup of child pid [5130] due to SIGCHLD: KILLED, signal 9 (0009)
Sep 17 16:51:15 server.example.com spamd[718]: spamd: server successfully spawned child process, pid 5138
Sep 17 16:51:15 server.example.com systemd[1]: spamd.service: main process exited, code=killed, status=9/KILL
Sep 17 16:51:15 server.example.com systemd[1]: Unit spamd.service entered failed state.
Sep 17 16:51:15 server.example.com systemd[1]: spamd.service failed.

The system could not provide log messages for “spamd” because it failed to read all of the potential log files with the following errors: Error while attempting to open “/var/log/maillog”: “No such file or directory”, Error while attempting to open “/var/log/messages”: “No such file or directory”, Error while attempting to open “/var/log/secure”: “No such file or directory”
Memory Information

Used

935 MB

Available

1.72 GB

Installed

2 GB

Load Information

16.18 5.xxx.xx
======================================

nameserver failure (and, of course, this is why I couldn't login to WHM until I rebooted it from customer portal)
Service Check Raw Output

Code:

(XID rg4bza) The “named” service is down.

The subprocess “/usr/local/cpanel/scripts/restartsrv_named” reported error number 255 when it ended.

Startup Log

Sep 17 16:51:15 server.example.com sh[5136]: -q, --queue use sigqueue(2) rather than kill(2)
Sep 17 16:51:15 server.example.com sh[5136]: -p, --pid print pids without signaling them
Sep 17 16:51:15 server.example.com sh[5136]: -l, --list [=] list signal names, or convert one to a name
Sep 17 16:51:15 server.example.com sh[5136]: -L, --table list signal names and numbers
Sep 17 16:51:15 server.example.com sh[5136]: -h, --help display this help and exit
Sep 17 16:51:15 server.example.com sh[5136]: -V, --version output version information and exit
Sep 17 16:51:15 server.example.com sh[5136]: For more details see kill(1).
Sep 17 16:51:15 server.example.com systemd[1]: named.service: control process exited, code=exited status=1
Sep 17 16:51:15 server.example.com systemd[1]: Unit named.service entered failed state.
Sep 17 16:51:15 server.example.com systemd[1]: named.service failed.

Log Messages
The system could not provide log messages for “named” because it failed to read all of the potential log files with the following errors: Error while attempting to open “/var/log/maillog”: “No such file or directory”, Error while attempting to open “/var/log/messages”: “No such file or directory”, Error while attempting to open “/var/log/secure”: “No such file or directory”

Memory Information
Used
935 MB

Available
1.72 GB

Installed
2 GB

Load Information
16.18 5.58 2.47
========================================

Last, but not least, is the clamd failure notification:

(XID rqgk58) The “clamd” service is down.

The subprocess “/usr/local/cpanel/scripts/restartsrv_clamd” reported error number 255 when it ended.

Sep 17 15:31:05 server.example.com systemd[1]: Starting clamd antivirus daemon...
Sep 17 15:31:22 server.example.com systemd[1]: Started clamd antivirus daemon.
Sep 17 16:51:07 server.example.com systemd[1]: clamd.service: main process exited, code=killed, status=9/KILL
Sep 17 16:51:07 server.example.com systemd[1]: Unit clamd.service entered failed state.
Sep 17 16:51:07 server.example.com systemd[1]: clamd.service failed.

The system could not provide log messages for “clamd” because it failed to read all of the potential log files with the following errors: Error while attempting to open “/var/log/secure”: “No such file or directory”, Error while attempting to open “/var/log/messages”: “No such file or directory”, Error while attempting to open “/var/log/maillog”: “No such file or directory”

Used

1.31 GB

Available

1.23 GB

Installed

2 GB

12.89 5.xxx.xxx
=============================

I'm sorry to possibly overwhelm you with all of this information. Maybe you can find a clue in it and point me toward the direction of a solution.

 

Here are the results of the commands you requested me to run:

Code:

Using username "root".
root@198.100.xx.xx's password:
Last login: Tue Sep 18 11:09:05 2018 from ip98-176-224-175.sd.sd.cox.net

root@server [~]# free -m
              total        used        free      shared  buff/cache   available
Mem:           2048         554        1217           9         275        1336
Swap:           512         402         109
root@server [~]# sar -r
Linux 2.6.32-042stab120.19 (server.example.com)       09/18/18       _x86_64_ (24 CPU)

10:46:11          LINUX RESTART

10:50:01    kbmemfree kbmemused  %memused kbbuffers  kbcached  kbcommit   %commit  kbactive   kbinact   kbdirty
11:00:01       950412   1146740     54.68         0    495504         0      0.00    308484    777848       188
11:10:01      1866364    230788     11.00         0     35172         0      0.00    140528     34268        28
11:20:01      1117052    980100     46.73         0    197120         0      0.00    666524    255060        28
11:30:01      1213132    884020     42.15         0    197156         0      0.00    481148    344664        84
11:40:01      1160012    937140     44.69         0    228888         0      0.00    526720    353896       104
11:50:01      1154620    942532     44.94         0    242728         0      0.00    447488    436092       100
12:00:01      1136396    960756     45.81         0    270124         0      0.00    454892    447272        24
Average:      1228284    868868     41.43         0    238099         0      0.00    432255    378443        79
root@server [~]#
==================

I procured the VPS service on May 19, 2016. I did not intentionally delete logging but disk space conservation if a concern of mine so, if you are kind enough to direct me in how to implement logs, I do hope you'll include information about how to limit their growth.

I'll check back soon to see what your reply is.

 

Oh. I forgot to add that clamd failed and recovered again during the cycle since my response. Here are the email notifications.

Code:

Failure:
(XID eewmh3) The “clamd” service is down.

The subprocess “/usr/local/cpanel/scripts/restartsrv_clamd” reported error number 255 when it ended.

Sep 18 10:46:18 server.example.com systemd[1]: Starting clamd antivirus daemon...
Sep 18 10:47:27 server.example.com systemd[1]: Started clamd antivirus daemon.
Sep 18 11:08:07 server.example.com systemd[1]: clamd.service: main process exited, code=killed, status=9/KILL
Sep 18 11:08:07 server.example.com systemd[1]: Unit clamd.service entered failed state.
Sep 18 11:08:07 server.example.com systemd[1]: clamd.service failed.

The system could not provide log messages for “clamd” because it failed to read all of the potential log files with the following errors: Error while attempting to open “/var/log/maillog”: “No such file or directory”, Error while attempting to open “/var/log/secure”: “No such file or directory”, Error while attempting to open “/var/log/messages”: “No such file or directory”

Used

1.5 GB

Available

1.26 GB

Installed

2 GB

0.41 2.03 1.57

Recovery:
The 'clamd' service passed the check: clamd (/usr/local/cpanel/3rdparty/bin/clamd) is running as root with PID 3544 (systemd+/proc check method).

Sep 18 11:12:40 server.example.com systemd[1]: Starting clamd antivirus daemon...
Sep 18 11:12:53 server.example.com systemd[1]: Started clamd antivirus daemon.

The system could not provide log messages for “clamd” because it failed to read all of the potential log files with the following errors: Error while attempting to open “/var/log/messages”: “No such file or directory”, Error while attempting to open “/var/log/secure”: “No such file or directory”, Error while attempting to open “/var/log/maillog”: “No such file or directory”

Used

1.54 GB

Available

1.25 GB

Installed

2 GB

0.25 1.92 1.80

 

Thanks for the links that will help me maintain log growth.

You made this statement:

I imagine you're referring to the logs you noted as missing:

I can't imagine myself intentionally deleting these log files. I'm not fluent with the command line interface. I'm not sure the logs were setup at the time the VPS was provisioned by my hosting company.

The only reference that I could find to enabling logging on WHM was this:
Tweak Settings - Logging - Version 74 Documentation - cPanel Documentation

That documentation does not appear to create the above missing log files. I used the touch command with no options to create empty files as listed above. Was that the proper way to recreate them?

I guess that I need to await another cPanel monitoring to send another email warning to see if that part of the notification message goes away. Did you have any further suggestions regarding memory usage being a concern?

 

As a post script to my most recent reply, it appears that CentOS 7.1 didn't use the missing log files and, since that is what I requested when I had the VPS provisioned, that would explain why those log files were absent. I got that information from a reply to an inquiry on another forum:
lowendtalk.com/discussion/comment/1341701/#Comment_1341701

 

We are running the same version of CentOS. Oh well, I probably deleted them but I honestly don't remember doing that.

There was a point in time when backup transfers to S3 were failing and the disk was filling up. Amazingly, my hosting company realized they had not been granting me all the disk space I signed up for when I originally requested the service so, after fixing what changed on their end to enable transfer of backups to S3 again, they moved my VPS to another server so I could get all the disk I'd been paying for. (lol)

I checked out Tweak Settings. At this time, I don't have any logging enabled that are options on that settings page. It also appears that those are defaults. Would you recommend enabling any of those logs?

The output from the lsof commands was nothing.

I looked into systemd journal and ran a couple of other commands that I found on this link:
systemd-journald.service(8) - Linux manual page

Here are the results:

So, maybe I need to do something else to enable systemd journal?

P.S. I have to say that you've been very responsive and helpful, @cPanelLauren. I truly have appreciated your patience and guidance on this issue.

 

I'm back again with another issue, I guess. Here's the output from my latest putty session:

I guess that indicates that rsyslogd is not running or even installed on my VPS. I found what appears to be a helpful link for installing it but I wanted to run this tutorial past you before I embark on this mission. Here's the link:
tecmint.com/create-centralized-log-server-with-rsyslog-in-centos-7/

 

Okay. I installed it. Here's the output from putty:

Do I need to do any further configuration?

 

I'm not sure what you're referring to when you reference same permissions/ownership. I can't find your response where you specifically noted creating the files with particular permissions/ownership. I created the files using the touch command and, as of this writing, it is a regular empty file.

Code:

Using username "root".
root@198.100.45.196's password:
Last login: Wed Sep 19 12:02:18 2018 from ip98-176-224-175.sd.sd.cox.net

root@server [~]# stat /var/log/messages /var/log/maillog /var/log/secure
  File: '/var/log/messages'
  Size: 0               Blocks: 0          IO Block: 4096   regular empty file
Device: d0h/208d        Inode: 37765784    Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2018-09-19 09:02:06.005057932 -0700
Modify: 2018-09-19 08:38:10.720958008 -0700
Change: 2018-09-19 08:38:10.720958008 -0700
 Birth: -
  File: '/var/log/maillog'
  Size: 0               Blocks: 0          IO Block: 4096   regular empty file
Device: d0h/208d        Inode: 37762735    Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2018-09-19 09:02:06.005057932 -0700
Modify: 2018-09-19 08:38:01.224036439 -0700
Change: 2018-09-19 08:38:01.224036439 -0700
 Birth: -
  File: '/var/log/secure'
  Size: 0               Blocks: 0          IO Block: 4096   regular empty file
Device: d0h/208d        Inode: 37762731    Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2018-09-19 09:02:06.005057932 -0700
Modify: 2018-09-19 08:37:36.176243329 -0700
Change: 2018-09-19 08:37:36.176243329 -0700
 Birth: -
root@server [~]#