SOLVED CPANEL-39815 - Not receiving security advisor notifications

[cPSloaneB]

@WorkinOnIt, as the root user, if you rename the /var/cpanel/security_advisor_history.json file to something else and then run /scripts/check_security_advice_changes --notify once again, does it re-issue all, some, or none of the missing notifications you are expecting?

 

[WorkinOnIt]

@cPRex

Thank you for your integrity. I have replied to the ticket to see what they say about a partial refund - although to be honest, the money is not really the issue (although when refunds are involved it does sharpen the focus somewhat, so that might help ;-)

Did you hear anything from your colleagues following the meeting?

I've just discovered another kernel update is ready in the manual security scan (of course, once again, no notification received !)

 

[WorkinOnIt]

@WorkinOnIt, as the root user, if you rename the /var/cpanel/security_advisor_history.json file to something else and then run /scripts/check_security_advice_changes --notify once again, does it re-issue all, some, or none of the missing notifications you are expecting?

Thank you @cPSloaneB - yes, that actually worked! I also received the email notification I updated on four servers and they all responded the same.

Of course, it remains to be seen if this will fix future auto-generated notices - I guess will have to wait and see if I still receive the email the next time there are new security advisor notifications ? Do you think this is now fixed - or does this need further steps?

I have no idea why renaming that file fixed it - would you care to explain further?

 

[cPRex]

Deleting the history file and forcing the run to happen again forces all notifications to be sent again.

I did talk with the development team about this earlier this week, and there is work actively happening on the issue now.

 

[cPSloaneB]

Of course, it remains to be seen if this will fix future auto-generated notices - I guess will have to wait and see if I still receive the email the next time there are new security advisor notifications ? Do you think this is now fixed - or does this need further steps?

I have no idea why renaming that file fixed it - would you care to explain further?

This by itself will not fix future notifications, unless the file is removed again continually. The Security Advisor currently remembers the internal names of each advice item, a hash of the text of the advice, and the severity of the item in that file, and it carries over this information across runs of the script. Unless the severity of the item was increased since the script last saw the item, it will not issue the same notification twice. Removing the history file makes the script forget about all previous notifications, which may work for you but would be too noisy for other users. (In principle, removing just the information pertaining to the notification you expect to be re-issued will also work.)

At this time, I have submitted a fix which changes this behavior so that the history file only remembers the notices which appeared the last time the script ran. This way, it should notify whenever the state of the item degrades since the last notification; if it stays the same, improves, or disappears, then this should still not trigger a notification, which I am hoping is not too noisy of a policy. Furthermore, it doesn't require changing the data format of the history file.

This change is being targeted for a 104 release. A backport to 102 may be possible, but we would likely want to see whether we get negative feedback for this change in 104, which might make necessary further changes to the kind of data recorded in the history file, so that the Security Advisor could apply more complicated hysteresis to the policy of when to issue notifications.

 

[WorkinOnIt]

At this time, I have submitted a fix which changes this behavior so that the history file only remembers the notices which appeared the last time the script ran. This way, it should notify whenever the state of the item degrades since the last notification; if it stays the same, improves, or disappears, then this should still not trigger a notification, which I am hoping is not too noisy of a policy. Furthermore, it doesn't require changing the data format of the history file.

This change is being targeted for a 104 release. A backport to 102 may be possible, but we would likely want to see whether we get negative feedback for this change in 104, which might make necessary further changes to the kind of data recorded in the history file, so that the Security Advisor could apply more complicated hysteresis to the policy of when to issue notifications.

Good one thanks. I am rather surprised that I am the only person who seems to have noticed / reported this matter..... I guess I must be old school !

I think another useful way to handle this is some kind of dashboard notification / flashing icon that says "hey something needs attention!" There is already a notification menu icon in the new layout at top right - I would suggest showing new security state change notices in there would be sensible?

 

[cPRex]

Update - this is resolved in version 104.0.4

 

[WorkinOnIt]

Hi there

I am now upgraded to 104.0.4 and will report back when / if I receive the notifications

 

[WorkinOnIt]

Hi there @cPRex and @cPSloaneB

Today I ran a manual check in the WHM security advisor and it came up with :

Detected 1 service that is running outdated executables: httpd.service
You must take one of the following actions to ensure the system is up-to-date:

• Restart the listed service using “systemctl restart httpd.service”; then click “Scan Again” to check non-service processes.
• Reboot the server.

But sadly - no email.... this has not been sent to me as an email notification... In previous emails that I used to receive years ago, this type of notification was indeed sent under the subject line of: "New Security Advisor notifications with High importance" - but I did not receive this - so what now....

[edit] Further more, once I had rebooted the server and re-run the security advisor, it now says a kernel update is available - but once again - no email.....

I am in WHM version 104.0.7

 

[WorkinOnIt]

@cPRex @cPSloaneB, just checking you saw this as your usernames did not seem to get tagged?

 

[cPRex]

We've got it - Sloane will check this out later today.

 

[cPSloaneB]

Today I ran a manual check in the WHM security advisor and it came up with :

Detected 1 service that is running outdated executables: httpd.service
You must take one of the following actions to ensure the system is up-to-date:

• Restart the listed service using “systemctl restart httpd.service”; then click “Scan Again” to check non-service processes.
• Reboot the server.

But sadly - no email.... this has not been sent to me as an email notification... In previous emails that I used to receive years ago, this type of notification was indeed sent under the subject line of: "New Security Advisor notifications with High importance" - but I did not receive this - so what now....

[edit] Further more, once I had rebooted the server and re-run the security advisor, it now says a kernel update is available - but once again - no email.....

I am in WHM version 104.0.7

There are still several possible reasons why you may not have received a notification.

As one example, note that the /var/cpanel/security_advisor_history.json file is not updated by the Security Advisor itself but rather by the /scripts/check_security_advice_changes script run during the maintenance phase of the nightly cPanel update. Furthermore, the new behavior only makes an advice item eligible for a new notification if that script notices that the condition went away: if the file says that it saw an item last time, but it's not seeing it this time, it removes it from the file, as if the previous notification never happened. Thus, it's possible that Apache was restarted, but an update to the EasyApache packages returned the issue of Apache needing a restart just prior to the next running of the script. In this case, it would not have noticed a change, so that item would still not be eligible for a new notification.

This is not to say that this is definitely what is happening. We would have to know the specifics from your server. It may be time for another support ticket.

(Something I'm considering as a further improvement is to have /scripts/check_security_advice_changes keep a copy of the previous history file before it writes the new one. Comparing these would be an easier way to know whether the script saw a change (and genuinely failed to notify) or not.)

 

[WorkinOnIt]

@cPSloaneB @cPRex

Hi both

I am still not receiving the update email notifications at all. I did a cPanel Security Advisor manual check on one of my 5 servers. It showed the
The system kernel is at version...... message - basically - never received the notification..... and nothing for the other servers either.

I have re-opened the previous ticket.

Request #94427824

 

[cPRex]

Thanks for that - I'm following along with that ticket now.

 

[cPRex]

I checked the ticket this morning, and it looks like we recommended renaming or moving the /var/cpanel/security_advisor_history.json file to see if this allows future notifications to be sent. Have you tried doing that yet?

 

[cPSloaneB]

I just looked at one of the the servers attached to your ticket. What I see makes me believe that the Security Advisor is for certain handing its request for a contact event off to the iContact system, as the messages being printed to the error log are from the iContact template belonging to the "change in Security Advisor state" notification. However, if iContact is generating an error, it is not being recorded anywhere. I'm going to have someone investigate further with this in mind.

 

[WorkinOnIt]

I just looked at one of the the servers attached to your ticket... I'm going to have someone investigate further with this in mind.

Hello @cPSloaneB

and @cPRex

I have today opened a further ticket: 94487970

I am still not getting any notifications on my 5 servers. I did a test "manual run" on one of the servers and all the notifications are there - just not getting anything on the emails... This is getting a bit tedious, but I do appreciate your assistance. I am surprised I am the only one that's brought it up! I assume I am not the only one affected !!

Could you change the status of this thread to something other than solved? It's a bit inaccurate.

Thank you

 

[cPRex]

The original issue we thought this may be, CPANEL-39815, has been solved in version 104. However, there is likely something else going on here, and we'll get that tracked down through the ticket. I'm following along with that now as well.

 

[cPRex]

As I expected, our team created a new case at CPANEL-41626 which can be followed here:

https://support.cpanel.net/hc/en-us/articles/9212696818583

 

[WorkinOnIt]

As I have been saying all along, cPanel notifications are broken - and this has been the case for almost a year.

It is frustrating to have to raise the matter several times, then be told "it's solved" when it clearly isn't.