Lots of FAILED: lfd and RECOVERED: lfd e-mails.

[Spork Schivago]

Hello.

I've been receiving a good number of emails every day that say LFD failed and then 4 or 5 minutes later, it says LFD recovered. For example, on Sunday, I received 14 messages (7 times it failed, 7 times it recovered).

Does anyone have any suggestions on what might be causing this? I noticed one time when I was reading my e-mail, I got a message saying it failed. I was logged into my server, I ran ps aux |grep -i lfd and I saw that it was sleeping. A good 4 or 5 minutes later, I received an e-mail saying it was back up.

I'm not sure if lfd is actually failing or if this might be some sort of bug in cPanel or what. Not really sure how to proceed in diagnosing this and was hoping someone could provide some helpful suggestions. Thanks!

I'd like to also add it's not just lfd but that's the most frequent. I also see mysql and spamd failed / recover e-mails. Thanks!

 

[Infopro]

CSF/LFD is not supported by cPanel of course. If you suspect an issue with it, they've got forums you can post to over there.

Here's a post by Chirpy from 2002 you might find of some use:
lfd - failed

If MySQL, spamd, lfd are all failing over and over, you've got something else going on it seems. Taking a closer look your system logs is suggested.

 

[rpvw]

Do you have any outdated binaries ?

/usr/bin/needs-restarting

I was getting lfd failures and recovered until I ran

/etc/init.d/lfd restart

Maybe the other services need restarting cleanly as well ?

 

[Spork Schivago]

CSF/LFD is not supported by cPanel of course. If you suspect an issue with it, they've got forums you can post to over there.

Here's a post by Chirpy from 2002 you might find of some use:
lfd - failed

If MySQL, spamd, lfd are all failing over and over, you've got something else going on it seems. Taking a closer look your system logs is suggested.

Thank you. It's definitely not just lfd, it's just lfd was the most prevalent. It's almost like clockwork though, like there's a pattern. Which makes me think maybe none of them are failing, just the thing that checks to see if they're down.

I checked my e-mail and there's 140 of these messages. But they all stop on Tuesday. I didn't realize that before. So, it appears something happened on Tuesday that maybe fixed the issue?

I've just recently upgraded to v11.58.0.20 but I was checking the changelog for v58 and noticed on v11.58.0.13...

Fixed case CPANEL-7819: Prevent spurious service down notifications.

This was for v11.58.0.13 released on 2016-08-04. Maybe my system didn't update right away to v11.58.0.13 or maybe something needed restarted after it was upgraded and didn't get restarted until Tuesday.

If I see anymore e-mails related to all the services, I'll post back here. Thanks!

 

[Spork Schivago]

Do you have any outdated binaries ?

/usr/bin/needs-restarting

I was getting lfd failures and recovered until I ran

/etc/init.d/lfd restart

Maybe the other services need restarting cleanly as well ?

Thanks for pointing out /usr/bin/needs-restarting. Nothing shows it needs restarting. As noted in the previous post, I just realized the messages seem to stop on Tuesday. This is when v11.58.0.19 was released. Perhaps one of the fixes in v11.58.0.19 fixed the issue. Maybe it had something to do with the: CPANEL-7486: Set cpanel service to have a KillMode=process fix. I dunno.

 

[rpvw]

58 still makes me very nervous - I wish they would have stopped at 42 !!

 

[Infopro]

But they all stop on Tuesday. I didn't realize that before.

It's been my experience to watch time stamps for most everything, very closely, when issues come up.

For example, if you do a manual update to cPanel at 4Pm in the afternoon, CSF will alert you that system files have changed. You might assume those file changes are from the update and you'll be fine. If you didn't do a manual update, and its 4 PM in the afternoon (cPupdate doesn't run at 4 PM of course) and you get one of those emails, then you might want to go take a closer look.

Speaking of time stamps, taking a closer look at your recent posts, you were quite active here on Tuesday with other issues. Could those emails be related? Maybe, the time stamps on the emails from Tuesday might be helpful to you.

GL!

 

[Spork Schivago]

It's been my experience to watch time stamps for most everything, very closely, when issues come up.

For example, if you do a manual update to cPanel at 4Pm in the afternoon, CSF will alert you that system files have changed. You might assume those file changes are from the update and you'll be fine. If you didn't do a manual update, and its 4 PM in the afternoon (cPupdate doesn't run at 4 PM of course) and you get one of those emails, then you might want to go take a closer look.

Speaking of time stamps, taking a closer look at your recent posts, you were quite active here on Tuesday with other issues. Could those emails be related? Maybe, the time stamps on the emails from Tuesday might be helpful to you.

GL!

Thank you! I looked at the time stamp from the first e-mail and seen Today and just did a quick glance at the ones under it. I saw the T and just mistakenly must have thought they were from today as well! You could be very right about the problems on Tuesday. How can I see my posts and search them by date?

I also just noticed a 1.6GB file in the /root directory. strace.sec_adv

It was created on the 25th of July at 21:54:04 and last accessed on the 30th of July at 19:36:23.

Because it's so large, I cannot open it easily, so I cat it and pipe the output to more. I see it's looking for and not finding a lot of files:

/etc/ld.so.preload
/usr/local/cpanel/3rdparty/perl/522/lib64/tls/x86_64/libgdbm.so.2
/usr/local/cpanel/3rdparty/perl/522/lib64/perl5/5.22.1/x86_64-linux-64int/CORE/tls/x86_64/libgdbm.so.2

Then, I see /usr/lib64/libgdbm.so.2 was successfully opened. It looks like something might have been looking in cPanel places for libraries at first, and not finding them, it defaulted to using the system libraries instead.

Further down, I see /usr/local/cpanel/scripts/check_security_advice_changes gets read and then I see directly after that, a -1 ENOTTY (Inappropriate ioctl for device) error message.

I see a lot of ENOTTY's. Could this file be related to the e-mails about all the various services that went down and restarted? I'm curious as to what caused the creation of this file and if there's still a problem or not. I know when I'm in WHM and I run the security advisor, I get warnings about various things (like clamav not being installed, no symlink protection, etc). I used to get e-mails every day saying stuff like symlink protection wasn't enabled but they just stopped. I figured this was related to the problems with the check_security_advice_changes bug that I think is fixed now. Maybe there's something else going on. What do you think?

Thanks.

 

[Infopro]

How can I see my posts and search them by date?

To the right side of your forum posts, click the number of posts you have.