cpanel uses jailshell for cron (problem)

[MaRiOsGR66]

Hello,

I have for all my users disabled Shell Access

but I see jailshell activity on the server

like the folders been created :

# df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/simfs 734003200 86170816 647832384 12% /home/virtfs/livecomg/lib
/dev/simfs 734003200 86170816 647832384 12% /home/virtfs/livecomg/bin
/dev/simfs 734003200 86170816 647832384 12% /home/virtfs/livecomg/sbin
/dev/simfs 734003200 86170816 647832384 12% /home/virtfs/livecomg/opt
/dev/simfs 734003200 86170816 647832384 12% /home/virtfs/livecomg/lib64
/dev/simfs 734003200 86170816 647832384 12% /home/virtfs/livecomg/usr
/dev/simfs 734003200 86170816 647832384 12% /home/virtfs/livecomg/var
/dev/simfs 734003200 86170816 647832384 12% /home/virtfs/livecomg/var/spool
/dev/simfs 734003200 86170816 647832384 12% /home/virtfs/livecomg/var/log
/dev/simfs 734003200 86170816 647832384 12% /home/virtfs/livecomg/etc/mail
/dev/simfs 734003200 86170816 647832384 12% /home/virtfs/livecomg/tmp
/dev/simfs 734003200 86170816 647832384 12% /home/virtfs/livecomg/var/tmp
none 4169728 4 4169724 1% /home/virtfs/livecomg/dev
/dev/simfs 734003200 86170816 647832384 12% /home/virtfs/livecomg/usr/sbin
/dev/simfs 734003200 86170816 647832384 12% /home/virtfs/livecomg/home/livecomg

so I do a further check

root@[/var/log]# cat /etc/passwd | grep livecomg
livecomg:x:***:***::/home/livecomg:/usr/local/cpanel/bin/noshell

again there is no shell or jail shell installed for that user

and so I removed tha virtfs folders and later I saw what was executed to create them :

at top I saw something called via cron: /usr/local/cpanel/bin/jailshell -c php -f /home/livecomg/public_html/domain.tld/cronjobs/cron_address.php

I did check the client's cron settings inside cpanel and I just saw:

Minute Hour Day Month Weekday Command Actions
*/10 * * * * php -f /home/livecomg/public_html/domain.tld/cronjobs/cron_address.php

so it seems cPanel it self uses jailshell for cron,
how can I stop that ?

 

[cPanelMichael]

Hello

This is normal, and the intended behavior with cPanel 11.38. It's documented at:

Jail System Update

Thank you.

 

[MaRiOsGR66]

Hello Michael,

is this something that came with WHM 11.38.0 (build 17) ?
I never had this problem again.

Today I had to reboot the server because of the /usr/local/cpanel/bin/jailshell using 100% of the cpu and kill -9 proc couldn't stop it.
I had to reboot the server (trying hard for this also)

can I disable jailshell for cron ?

 

[cPanelMichael]

Cron jobs are now run with jailshell. The recent changes to the jail system were part of the cPanel 11.38 release. However, if a jailshell process is utilizing 100% CPU, then it should be investigated further. What specific jailed process was utilizing 100% CPU usage?

 

[AllianceOne]

Hello Michael,

is this something that came with WHM 11.38.0 (build 17) ?
I never had this problem again.

Today I had to reboot the server because of the /usr/local/cpanel/bin/jailshell using 100% of the cpu and kill -9 proc couldn't stop it.
I had to reboot the server (trying hard for this also)

can I disable jailshell for cron ?

same problem/error here

jailshell are using 100% of the cpu, CSF can´t stop the cron and can´t be killed with kill -9
before the change on cronjobs all was working fine...

 

[MaRiOsGR66]

so it wasn't only me

 

[rlshosting]

I enabled Users of mod_ruid2 can now enable “Jailed apache” support which will chroot() each virtual host into their virtfs. for the server and it increased load big time and I had to reboot server and it made pages time out.

 

[ThinIce]

If anyone has opened tickets it would be worth posting the numbers here so that any issues can be crosschecked - I've not run into any problems but would be interested if these processes that can't be killed can be straced and if so whether this shows anything interesting...

 

[gu1lle]

this is useful as it sound, from jailshell many process return a error due insufficient and cant connect mysql engine neither....

at least , should would have the possibility change and select a default shell for cron jobs.

*not a good feature*

 

[cPanelMichael]

this is useful as it sound, from jailshell many process return a error due insufficient and cant connect mysql engine neither....

Could you let us know the specific error messages you have received? Also, please post the ticket number if you have opened a ticket to report this issue.

Thank you.

 

[gruvin]

I too have just experienced 100% CPU load from cPanel cron task running a PHP script (php -q ...).

Additionally, this task ran for at least 25 minutes longer than it should have and required manual killing.

The only extra information I can offer right now is that this one particular script was attempting a connection to an external POP3 server, to collect mail. (The script was pop.php, from the WHMCS package -- and is closed source [ionCube]).

I have not experienced any crashes from virtfs mounts so far, thankfully.

Running cPanel/WHM 11.38.2 (build 6) on Centos 5, Kernel is 2.6.18-028stab095.1 (x86_64), as a VPS under Virtuoso. (I'm just a VPS customer on this box, without full system access.)

Gruvin.

 

[cPanelMichael]

I too have just experienced 100% CPU load from cPanel cron task running a PHP script (php -q ...).. Additionally, this task ran for at least 25 minutes longer than it should have and required manual killing.

Please open a support ticket so we can investigate the specific problem you are experiencing if this continues.

Thank you.

 

[nasir5]

What solution was proposed by cPanel support for this issue? I have seen this issue many times but each time on VPS and the solution I know is nothing logical but reboot the VPS.

 

[cPanelMichael]

What solution was proposed by cPanel support for this issue? I have seen this issue many times but each time on VPS and the solution I know is nothing logical but reboot the VPS.

It's recommended that you open a support ticket and provide the ticket number here so we can investigate the issue and update this thread with the outcome.

 

[Johnson]

Please add the possibility to run crontab via normal shell while keeping jailshell for SSH access ASAP. We had to reboot servers on a daily basis until we switched shell access for all users to "bash". Here is an example from a random server where users still have "jailshell":

# wc -l /proc/mounts 
10795 /proc/mounts

/proc/mounts consists of lines like the following:

/dev/sda3 /home/virtfs/USER1/usr/sbin ext3 rw,noatime,nodiratime,errors=continue,barrier=1,data=writeback,usrquota 0 0
/dev/sda3 /home/virtfs/USER2/usr/sbin ext3 rw,noatime,nodiratime,errors=continue,barrier=1,data=writeback,usrquota 0 0
/dev/sda3 /usr/sbin ext3 rw,noatime,nodiratime,errors=continue,barrier=1,data=writeback,usrquota 0 0

/scripts/clear_orphaned_virtfs_mounts does nothing. So now any command that deals with /proc/mounts takes much more time to execute, for example "ifconfig" takes 6 seconds to execute. This is NOT NORMAL and should be taken care of ASAP.

 

[Johnson]

Another server:

# wc -l /proc/mounts 
594 /proc/mounts

and another:

# wc -l /proc/mounts
1011 /proc/mounts

 

[Aaron.Edwards]

Hi Johnson,

Did you try the below script to remove all bind mounts for a particular user,

/usr/local/cpanel/3rdparty/bin/perl -MCpanel::Filesys::Virtfs -e 'Cpanel::Filesys::Virtfs::clean_user_virtfs("username");'

Docs from http://docs.cpanel.net/twiki/bin/view/AllDocumentation/WHMDocs/VirtFS

You would need to replace the "username" with your actual cPanel username.

 

[gruvin]

Please open a support ticket so we can investigate the specific problem you are experiencing if this continues.

I raised a support ticket with the cPanel reseller/partner relevant to my cPanel installation (PowerVPS). They in turn said they were aware of this problem and had contacted cPanel about it already but received no solution as yet. However, they did change my set-up (I haven't investigated how, yet) so that it no longer uses jailshell, but /bin/bash instead.

The 100% CPU utilisation problems went away after that and have not returned.

I am not aware of any tie-up with mount points. In my case it was always php scripts running under jailshell that were maxing out. Others have said that increasing the memory available to each instance of jailshell can or might help. I have no idea how to do that.

Sorry, but that is about the sum total of what I know about the solution they provided -- except that I'm running on Centos v5.5, if that helps?

- - -
Obviously, it would be nice to have user cron jobs run in a more secure environment, as was cPanel's clear intention. Fortunately for me, none of my current users know what a cron or sadly, even a cPanel is. So I should be fairly safe, for a while anyway.

 

[Johnson]

Hi Johnson,

Did you try the below script to remove all bind mounts for a particular user,

It just gives a bunch of warnings like:

warn [-e] Virtfs mountpoint failed to umount: /home/virtfs/USER/var
warn [-e] Virtfs directory "/home/virtfs/USER/var" was not umounted properly, not removing

 

[Aaron.Edwards]

However, they did change my set-up (I haven't investigated how, yet) so that it no longer uses jailshell, but /bin/bash instead. The 100% CPU utilisation problems went away after that and have not returned.

I hope they should have followed this URL =======cPanel Docs Regarding VirtFS =======

This document states

Before you can safely remove the jailed shell environment, you must switch the user's shell to a normal shell via the Manage Shell Access interface