Random Problems, error 500, ruby errors executing php cli

[ITGabs]

I am trying to find what happened probably was a cPanel Update or a Security Breach

First I found this very weird doing a print_r($_SERVER); I got this

[PATH] => /usr/local/jdk/bin:/usr/kerberos/sbin:/usr/kerberos/bin:/usr/lib/courier-imap/sbin:/usr/lib/courier-imap/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/X11R6/bin:/usr/local/bin:/usr/X11R6/bin:/root/bin:/opt/bin

This started to happen around Fri, 22 Jan 2016 10:45:41 GMT

Before the PATH was

"PATH":"sbin:/usr/sbin:/bin:/usr/bin"
Sun, 17 Jan 2016 14:40:27 GMT


2) The error 500 have something to do with a Ruby problem when running php cli / bash from command line, many scripts are showing this error, the problem was that fcgid is detecting the error and close the connection by FcgidErrorScanInterval 3 . I changed to a higher value and now the scripts end properly but all the time showing this error at the end

/usr/lib/ruby/1.8/thread.rb:5:in `require': no such file to load -- thread.so (LoadError)
from /usr/lib/ruby/1.8/thread.rb:5
from /usr/lib/ruby/site_ruby/1.8/rubygems.rb:11:in `require'
from /usr/lib/ruby/site_ruby/1.8/rubygems.rb:11
from /usr/bin/gem:8:in `require'
from /usr/bin/gem:8
/usr/lib/ruby/1.8/thread.rb:5:in `require': no such file to load -- thread.so (LoadError)
from /usr/lib/ruby/1.8/thread.rb:5
from /usr/lib/ruby/site_ruby/1.8/rubygems.rb:11:in `require'
from /usr/lib/ruby/site_ruby/1.8/rubygems.rb:11
from /usr/bin/gem:8:in `require'
from /usr/bin/gem:8

3) I was using the cPanel API v1 also around the same date stopped to work and also give random error messages, I updated the scripts to use the cPanel API v2 (related to database, user creation) and that fixed the issue but I am thinking everything is related.

Linux Centos 6.7
cPanel 54 build 14 updated from 11.52 weeks ago
modsecurity disabled
apache 2.2 php 5.5.x fcgid

Also notice that I have a new reseller (someone that I trust from years ago) and I gave access to the transfer tool too

What could be the main problem or the next steps to fix or prevent this issues

 

[cPanelMichael]

Hello

It's important to open separate threads for each individual issue, and we can help determine if the issues are related. Could you let us know which type of shell access is assigned to this user? Do you experience similar results with normal shell access compared to jailed shell access? Also, please let us know if this troubleshooting guide is helpful:

Ruby, Ruby Gem, Rails and Mongrel Troubleshooting Guide | cPanel Blog

Thank you.

 

[ITGabs]

Hi Michael,

Yes I know but really I don't know what is causing the error 500 or where these ruby errors come from (I am not using ruby at all in any of those scripts)

More Data
I think everything is related to php-fpm, I don't know where are the settings of php-fpm but I see a lot of new logs and process also errors with few information.

The scripts run only as root, in bash, probably in some point also use sh, these scripts do a lot of tasks, incremental websites backups, clone restore, websites, move sites between servers, updates servers also connect with the hyper-visor and a lot of things using ncat expert, almost 90% shell commands or C, 9% php/perl 1% the cpanel API2 (only about the database privileges)

I also created different wrappers or ports to trigger some of these tasks.

About the Path
The problem with the path I have no idea what was causing it, to me looks like the enviroment data of some developer that was included in the release of v 54 I never installed or used kerberos or jdk or X11R6, I changed to apache 2.4 and now the path is ok

About Ruby
Apache 2.4 broken the configuration that I had for redmine, Anyway I have another server with redmine so i will completely remove ruby.

About the cPanel API 2 I opened a new thread Set database user privileges from bash or API1 or API2

About error 500
I did a lot of test and this is not about timeout limitations, I think is a new limitations from php-fpm since the script take less than 30 seconds, run many tasks, git calls, bash commands, one php script and later the API2 I think the API2 generate many subprocess or because the json error and the ruby warning the task is somewhat blacklisted and dropped, using similar code I can have tasks running per hours like filtering and showing security/error logs or converting a stream of webcams into images etc. where I can find more info about the new integration of cpanel with php-fpm, where are these settings?

Thanks

 

[ITGabs]

in sudoers I added the user and the script

cpuser ALL=(root) NOPASSWD: /path/to/the/script.sh
Defaults!/path/to/the/script.sh !requiretty

the script start with #!/bin/bash
the "cpuser" actually doesn't have any shell
echo $SHELL
print /usr/local/cpanel/bin/noshell

the cpuser has only access to php exec and fopen so read the output of the command in real time, in that way finally i can run commands and see the output in a browser, the ruby error is present in many scripts and the behavior is exactly the same when i run the script from command line or from exec call the only difference is that the script stops near the API2 calls, both calls show the ruby error at the end

OK I found where the PATH was coming from and also why the the ruby errors

in /etc/bashrc

I found this
...
export JAVA_HOME=/usr/local/jdk
export PATH=$JAVA_HOME/bin:$PATH
export CLASSPATH=.:$JAVA_HOME/lib/classes.zip
export PATH="$PATH:/usr/local/bin:/usr/X11R6/bin"

#cPanel Added Limit Protections -- BEGIN
...
in /etc/bash.bashrc
just
type rvm >/dev/null 2>/dev/null || echo ${PATH} | __rvm_grep "/usr/local/rvm/bin" > /dev/null || export PATH="${PATH}:/usr/local/rvm/bin"

I installed rvm anyway that java path I don't rememeber
where i can find the original files /etc/bashrc /etc/profile for cpanel 54/centos 6.x ?

 

[ITGabs]

Following the same process to install redmine ruby and gems, I found exactly the same error when
#gem install bundle

/usr/lib/ruby/site_ruby/1.8/rubygems.rb:11:in `require': no such file to load -- thread (LoadError)
from /usr/lib/ruby/site_ruby/1.8/rubygems.rb:11
from /usr/bin/gem:8:in `require'
from /usr/bin/gem:8
Seems it is already too old (Ruby MRI 1.8.7 support has reached its EOL and its use is discouraged) Important: Ruby 1.8.7 out of support - Redmine

Also I found an issue with ulimit, I am still testing

 

[cPanelMichael]

Could you open a support ticket using the link in my signature so we can take a closer look? You can post the ticket number in this thread so we can update this thread with the outcome.

Thank you.