SOLVED Cron Jobs and /usr/lib/systemd/systemd --user in Almalinux

[mkhuder]

Hello,
I have multiple websites with PHP cron jobs, on cron job run, the following command run too:
/usr/lib/systemd/systemd --user which is cauding server load spikes and the server becomes unstable.
(Screenshots attached)

It happens with noshell or jailshell, it is only gone when I give shell access to the users.
this started after migrating from CentOs to Almalinux.

and CSF started to catch those with alerts like:

Time: Mon Aug 14 12:28:13 2023 +0200
Account: xxxxx
Resource: Process Time
Exceeded: 3908 > 1800 (seconds)
Executable: /usr/lib/systemd/systemd
Command Line: /usr/lib/systemd/systemd --user
PID: 48066 (Parent PID:48066)
Killed: No

I am running PHP 7.4 with DSO and Apache prefork.
I already checked those thread who have the same problem, but no fix helped so far.

systemd processes keep spawning

I started getting these systemd processes for every account on the system. I found this : Excessive resource usage Executable: /usr/lib/systemd/systemd Command Line: (sd-pam) where someone at the end said it could be jail shell related. But I tried setting shell to jailed and disabled and that...

forums.cpanel.net

Excessive resource usage Executable: /usr/lib/systemd/systemd Command Line: (sd-pam)

Hello, I am aware that CSF reports these via email, but I have never seen this with one of my websites . usually its /usr/bin/curl or /opt/cpanel/ea-php7.4/root/usr/sbin/php-fpm or /opt/cpanel/ea-php8/root/usr/sbin/php-fpm but never systemd or command line (sd-pam) Any reason you can think of...

forums.cpanel.net

I would really appreciate any help with this.
Thanks

 

[cPRex]

Hey there! Can you let me know specifically where those screenshots are from? If that is from WHM >> Process Manager, that's showing very little CPU usage and almost no Memory usage, so it would be odd if that alone was making the server unstable.

The CSF notice is normal as it's just telling you the process has run longer than 1800 seconds, which is the default for that software.

 

[mkhuder]

Thanks for you answer,
yes, the screenshot is from WHM >> Process Manager
The numbers sometimes go as high as 70-80
I am not worried about CSF as I can whitelist those processes, I am worried about the spikes.
here is another screenshot

and this is the log from /var/log/messages
which are logged for every website which has a cron job

Aug 15 16:56:14 server systemd[2311719]: Stopped target Default.
Aug 15 16:56:14 server systemd[2311719]: Stopped target Basic System.
Aug 15 16:56:14 server systemd[2311719]: Stopped target Timers.
Aug 15 16:56:14 server systemd[2311719]: Stopped target Paths.
Aug 15 16:56:14 server systemd[2311719]: Stopped target Sockets.
Aug 15 16:56:14 server systemd[2311719]: Stopped Mark boot as successful after the user session has run 2 minutes.
Aug 15 16:56:14 server systemd[2311719]: Closed D-Bus User Message Bus Socket.
Aug 15 16:56:14 server systemd[2311719]: Reached target Shutdown.
Aug 15 16:56:14 server systemd[2311719]: Started Exit the Session.
Aug 15 16:56:14 server systemd[2311719]: Reached target Exit the Session.
Aug 15 16:56:14 server systemd[1]: [email protected]: Succeeded.
Aug 15 16:56:14 server systemd[1]: Stopped User Manager for UID 1046.
Aug 15 16:56:14 server systemd[1]: Stopping User runtime directory /run/user/1046...
Aug 15 16:56:14 server systemd[1]: run-user-1046.mount: Succeeded.
Aug 15 16:56:14 server systemd[1]: [email protected]: Succeeded.
Aug 15 16:56:14 server systemd[1]: Stopped User runtime directory /run/user/1046.
Aug 15 16:56:14 server systemd[1]: Removed slice User Slice of UID 1046.
Aug 15 16:56:15 server systemd[1]: Stopping User Manager for UID 1016...
Aug 15 16:56:15 server systemd[2311732]: Stopped target Default.
Aug 15 16:56:15 server systemd[2311732]: Stopped target Basic System.
Aug 15 16:56:15 server systemd[2311732]: Stopped target Sockets.
Aug 15 16:56:15 server systemd[2311732]: Stopped target Timers.
Aug 15 16:56:15 server systemd[2311732]: Stopped Mark boot as successful after the user session has run 2 minutes.
Aug 15 16:56:15 server systemd[2311732]: Stopped target Paths.
Aug 15 16:56:15 server systemd[2311732]: Closed D-Bus User Message Bus Socket.
Aug 15 16:56:15 server systemd[2311732]: Reached target Shutdown.
Aug 15 16:56:15 server systemd[2311732]: Started Exit the Session.
Aug 15 16:56:15 server systemd[2311732]: Reached target Exit the Session.
Aug 15 16:56:15 server systemd[1]: [email protected]: Succeeded.
Aug 15 16:56:15 server systemd[1]: Stopped User Manager for UID 1016.
Aug 15 16:56:15 server systemd[1]: Stopping User runtime directory /run/user/1016...
Aug 15 16:56:15 server systemd[1]: run-user-1016.mount: Succeeded.
Aug 15 16:56:15 server systemd[1]: [email protected]: Succeeded.
Aug 15 16:56:15 server systemd[1]: Stopped User runtime directory /run/user/1016.
Aug 15 16:56:15 server systemd[1]: Removed slice User Slice of UID 1016.
Aug 15 16:56:17 server systemd[1]: session-81270.scope: Succeeded.

 

[cPRex]

Interesting - we may need to see this in action, so could you open a ticket so we can check the server directly?

 

[mkhuder]

Thanks cPRex, Ticket ID: 95106429
I thought the problem was with sd-pam, which also fired along with the cron jobs.
but sd-pam are not taking that much process and I am more concerned about /usr/lib/systemd/systemd --user

 

[cPRex]

Thanks - I'm following along with that ticket now.

 

[cPRex]

Our team was able to provide the following details in the ticket:

On the RHEL 8 family of operating systems (probably ubuntu too?) the pam_systemd module is enabled in /etc/pam.d/system-auth. As a result systemd spawns a user level process to manage services, crons, etc upon login, or any time a user cron is executed.
On systems with a large number of users, this can amount to a staggering level of additional %sys or %si level CPU usage. This new user level process management is not required on cPanel servers. Systems running a graphical user environment where gdm is in use will not work with this disabled.

This can be disabled by default by running the following as the root user:

systemctl mask [email protected]

 
Then any existing user sessions would need to be stopped or restarted.
 
To then enable this on a per user basis, use the following command, replacing the myuser portion with the desired cPanel username:
 

ln -s /usr/lib/systemd/system/[email protected] /etc/systemd/system/user@$(id -u myuser).service

 

[mkhuder]

Thanks cPRex,

I really appreciate your efforts along with cPanel team.
I just want to say that this worked for me, in case someone has the same issue.
The server went from a load average of 6 to 2!

Thanks a lot

 

[cPRex]

I'm glad to hear it!!!