minilogd and syslogd

[acenetryan]

This isn't really a cPanel issue, but has helped the performance of one of our servers, so I thought I'd share. Forgive me if I posted this in the wrong place, I'm still a relative n00b in the forums.

We had a server use quite a large amount of swap. After rebooting, the server was again using a large amount within several days.

vmstat showed:

root@XXXX [/etc/init.d]# vmstat
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 1  0 601852  30952  76500 928884    0    0     6     3    3     8 20  7 61 12

600 MB of swap! We have 4 GB of RAM in this box!

After combing ps aux and running top with M option (must be used interactively, top -M and top M didn't seem to work) I noticed that minilogd is using crazy RAM (61%). What is minilogd?

I went through cPanel forums, chirpy says google it. Googled. I couldn't find a definitive explanation, but from the pieces I've found:

minilogd is a logging application that starts prior to syslog on bootups. minilogd sits in queue collecting kernel messages until syslog can get up and running and take over. While it's queueing, it eats up memory and dramatically kills performance.

Upon checking, syslog was not running.

service syslog restart

Shutting down kernel logger:                               [  FAILED  ]
Shutting down system logger:                               [  FAILED  ]
Starting system logger:                                    [  OK  ]
Starting kernel logger:                                    [  OK  ]

As soon as I restarted syslogd, minilogd started handing off all of the messages its been collecting since the last reboot (roughly 12 days worth!).

Once minilogd was done passing off the message, the memory it was using was freed up:

root@XXXX [/etc/init.d]# free -m
             total       used       free     shared    buffers     cached
Mem:          4052       1208       2844          0         67        822
-/+ buffers/cache:        318       3734
Swap:         4094          0       4093

WOW! 3 GB of RAM usage by a single process!

Our average load dropped dramatically once minilogd was no longer running.

Hope this helps someone.

[chirpy]

Thanks for posting that

Do/Did you have syslog checked in both boxes in WHM > Service Manager > syslog?

[acenetryan]

Enabled, but not monitored. Hopefully this will get it started on reboots. I'll repost next time we boot. Thanks chirpy.

[acenetryan]

Yup, looks like that was it. syslogd started up by itself on the next boot.