Shell Fork Bomb Protection causing segfaults?

[bls24]

For the past few months I've been experiencing General Protection Errors.
Recently, they have began coupling with Segmentation Faults.
They seem to come the most, when I work on my site (I use WinSCP for ftp transfers & putty for command line... both connect under the SSH port.)

Initially, I suspected these errors were due to my php installation. Php has been reinstalled/recompiled with no success.

I've had my data center do quick & extended hard disk scans; came up clean.

Here is an example of the error:

segfaults:
Jun 28 00:42:53 host sshd(pam_unix)[16319]: session opened for user myusername by (uid=0)
Jun 28 00:44:24 host kernel: php-cgi[16560] general protection rip:7800d8 rsp:7fbfffede0 error:0
Jun 28 00:44:28 host kernel: php-cgi[16584] general protection rip:7800d8 rsp:7fbfffee20 error:0
Jun 28 00:53:38 host kernel: php-cgi[18870]: segfault at 0000002ab5357920 rip 00000000007800d8 rsp 0000007fbfffee00 error 4
Jun 28 00:54:24 host kernel: php-cgi[18932] general protection rip:7800d8 rsp:7fbfffee40 error:0
Jun 28 00:57:14 host sshd(pam_unix)[14600]: session closed for user myusername
Jun 28 00:57:58 host sshd(pam_unix)[19223]: session opened for user myusername by (uid=0)
Jun 28 05:46:21 host kernel: php-cgi[30423]: segfault at 0000002ab5357920 rip 00000000007800d8 rsp 0000007fbfffee30 error 4
Jun 28 05:49:22 host sshd(pam_unix)[30539]: session opened for user root by (uid=0)
Jun 28 05:49:28 host sshd(pam_unix)[30571]: session opened for user root by (uid=0)



protection faults:
Jun 27 02:09:35 host named[2992]: lame server resolving 'backup-blabla.net' (in 'blabla.net'?): ipstuffbla#53
Jun 27 02:11:01 host kernel: php-cgi[6825] general protection rip:7800d8 rsp:7fbfffee00 error:0
Jun 27 02:11:06 host kernel: php-cgi[6839] general protection rip:7800d8 rsp:7fbfffee50 error:0
Jun 27 02:12:45 host named[2992]: lame server resolving 'ns2.blabla.net' (in 'blabla.net'?): ipstuffbla#53
Jun 28 00:42:53 host sshd(pam_unix)[16319]: session opened for user myusername by (uid=0)
Jun 28 00:44:24 host kernel: php-cgi[16560] general protection rip:7800d8 rsp:7fbfffede0 error:0
Jun 28 00:44:28 host kernel: php-cgi[16584] general protection rip:7800d8 rsp:7fbfffee20 error:0
Jun 28 00:53:38 host kernel: php-cgi[18870]: segfault at 0000002ab5357920 rip 00000000007800d8 rsp 0000007fbfffee00 error 4
Jun 28 00:54:24 host kernel: php-cgi[18932] general protection rip:7800d8 rsp:7fbfffee40 error:0
Jun 28 00:57:14 host sshd(pam_unix)[14600]: session closed for user myusername
Jun 28 00:57:58 host sshd(pam_unix)[19223]: session opened for user myusername by (uid=0)
Jun 28 05:28:39 host sshd(pam_unix)[27139]: session closed for user myusername
Jun 28 05:28:52 host kernel: php-cgi[25720] general protection rip:7800d8 rsp:7fbfffee40 error:0

I'm continuously analyzing my access logs and timestamps, trying to locate a connection to a php script.. nothing matches. Different script everytime and sometimes at the very second of the timestamp, there isn't a click on my site.
My site loads pretty fast.. it's on a dedicated server with only 1 other site. Average memory usage (according to cpanel) is 20-25% any time of the day. Load averages at 0.05-0.40
99% of the time these errors occur, I am online doing something on the site. Be it working on it, browsing, etc. whatever... I never notice any service interruptions whatsoever.

Apache Version: 1.3.41
OS: Linux
Kernel: 2.6.9-67.0.15.ELsmp
Architecture: x86_64
PHP: 5.2.6
Cpanel: 11.23.3-RELEASE

Last night I upgraded to Apache 2.2.9 to see if they persist.
So far, not a single error since upgrading... but I haven't logged into SSH since 5am (until now, about 15 minutes ago).

Wondering if there is a possible connection between the shell bomb protection.
Could my ssh client (WinSCP) be requesting too much memory, and the protection is limiting it? I do frequently receive timeouts & errors from the WinSCP client after leaving it idle.

 

[bls24]

Tonight when I was working in WinSCP I wrote down the times of login, logout & any errors that appeared.

From my logs this evening:

Jun 30 22:26:02 host kernel: php-cgi[30844]: segfault at 0000002ab5357920 rip 00000000007800d8 rsp 0000007fbfffee10 error 4
Jun 30 22:26:49 host sshd(pam_unix)[30868]: session opened for user myusername by (uid=0)
Jun 30 22:27:30 host sshd(pam_unix)[30898]: session opened for user myusername by (uid=0)

I wrote down the following events (some of my times may be a few minutes off, as I forgot to write them down.. so I estimated the nearest time):
"Terminated by User" Error @ 10:21
Restarted my SSH session at 10:26
(as well, sometimes my client lies idle in my task bar before I notice an error has occured.)

Jun 30 21:44:35 host sshd(pam_unix)[21493]: session opened for user myusername by (uid=0)
Jun 30 21:46:06 host sshd(pam_unix)[21493]: session closed for user myusername
Jun 30 21:47:14 host kernel: php-cgi[22315] general protection rip:7800d8 rsp:7fbfffede0 error:0
Jun 30 21:48:04 host sshd(pam_unix)[22424]: session opened for user myusername by (uid=0)
Jun 30 21:48:37 host sshd(pam_unix)[22424]: session closed for user myusername
Jun 30 21:49:01 host kernel: php-cgi[22552] general protection rip:7800d8 rsp:7fbfffee40 error:0
Jun 30 21:56:04 host sshd(pam_unix)[24121]: session opened for user myusername by (uid=0)
Jun 30 21:57:45 host sshd(pam_unix)[24355]: session opened for user myusername by (uid=0)

Around 10:30 I got sidetracked and stopped keeping track of my errors.
But seeing here, as I am reconnecting so many times, I again receive the "Terminated by User error and attempt to tick "Reconnect" on the error window.


When I gracefully logout (hit the X), no general protection errors or segfaults were logged.


So it seems to me that WinSCP / SSH is causing my errors. Wondering if there is anything that can be done about it?
How can I check if my allotted memory in the server config is maybe too low? What would be a recommended setting?

 

[chirpy]

Is this on a real server or a VPS? There is a known issue with the "Shell Fork Bomb Protection" feature and the Virtuozzo VPS and you shouldn not enable it on such systems.

 

[bls24]

Dedicated server using cpanel & APF.

 

[bls24]

I disabled shell bomb protection yesterday afternoon, refrained from logging into WinSCP, but I logged into SSH a few times to email myself logwatch results (which initially caught the faults)

No errors.
If using WinSCP in place of an ftp client causes segfaults, I assume it should probably not be used. Is the only alternative ftp?

If so, is there any secure way to login to ftp?

 

[bls24]

Haven't had a single error since only ssh for command-line (refrained from using WinSCP).

However, got a protection error this morning when changing an account's password via cpanel. I tried to replicate it several times to no avail. . assuming this one was just a hiccup. The rest of my logs this week have been completely clean!

 

[bls24]

First night of using ftp heavily, general protection faults are back! I'm starting to think there is a misconfig somewhere... not sure where to look?

 

[bls24]

Logged back into Winscp tonight for more testing.

NO errors. BUT i only edited files in private directories that are not accessible to my visitors.

My theory:
the kernel error is triggered as I save a file, and in the same moment it is attempted to be viewed on the website.

(The ftp/winscp clients delete the file and replace it with my edited version after I hit save)
To the viewer it would be a 404 if the client disconnects in the middle of transfer before the new file was completely replaced, but is there something going on in the server memory that could possibly be the culprit?

if this is possible, it would explain the php-cgi warnings and why this only happens during the ftp sessions.


This is the only thing that would make any sort of sense in this situation. Can anyone confirm or deny my assumptions?

 

[blargman]

I've noticed the same when browsing anything using php-cgi. Only thing running cgi is cpanel php. Tried /scripts/makecpphp didn't do much to help. I can reproduce it by refreshing phpMyAdmin and it instantly shows up in dmesg. I'll let ya know if I find anything.

 

[andywest]

also getting GPF for php-cgi, and use ssh

I have also been getting the following error of great concern in my logwatch log files this last month or so (not before that). I have CPanel provide automatic updates to the latest stable builds. I use REDHAT ENTERPRISE 5.2 x86_64, Apache Webserver xxx, and have WHM 11.23.2 and cPanel 11.23.4.

from yesterday's log file:
--------------------- Kernel Begin ------------------------
WARNING: General Protection Faults in these executables
php-cgi : 6 Time(s)


Thx for the tips on a possible SSH connection!

I use SSH regularly for file transfers, including yesterday extensively (above log file), but to a non-public, development web site that the public does not see or use (I highly doubt, therefore, that anyone is trying to concurrently view files on the site while I'm uploading the new file(s)). I use the "SSH Secure Shell" product, from Secure Shell Communications. I don't use WinSCP. My SSH regularly times out and I wonder if that is connected to the issue??

I will now try to correlate my SSH activity, as well as timeouts (I'll let some occur and note times), with the timing of these GPF's.

Andy

 

[raysolomon]

--------------------- Kernel Begin ------------------------
WARNING: General Protection Faults in these executables
php-cgi : 6 Time(s)

I have been getting this error on a particular server too.
But I don't use any php-cgi scripts, only cpanel does.
I use php as an apache module, not cgi.

Therefore it is cpanel that has a cgi-based script that is not logically functioning.

My advice is to put in a support request to cpanel.