php 5.3.3 & apache 2.2 w/ eaccel, zend, fastcgi Super high cpu spikes

[sfraise]

It was set to forward to oohyane, I set it to discard.
I never really thought much about the cpanel settings as I have it set in whm to fail.

The tmp files your seeing are from caching, eaccelerator and memcached both use the tmp directory.

[sfraise]

Ok, not going to jinx myself or anything here, but I was able to track down an issue in sh404sef which is the url rewriting component I use, makes sense as I recently upgraded to a newer version shortly after I moved to this new box. I upgraded to the latest version that just came out and so far I'm running between 0.22 - 1.5 with no high cpu spikes yet (knock on wood).

I'll keep an eye on things for a while and see if this load holds or if it ends up spiking again, hopefully I got a handle on it finally though.

[sfraise]

Well, still spiking but doesn't seem to be nearly as severe. Where it was spiking to as high as 100, it now seems to stay well under 20 during the spikes and then go back down under 1 fairly quickly. I'm still getting excessive processes under oohyane.

I've posted in the sh404sef forum and they seem to think the main issue lies with community builder. I've posted on their forum as well but have yet to hear anything back yet. Here's an excerpt from todays csf output, what causes the system integrity notices?

Sep 8 00:51:25 server lfd[12497]: 5 (sshd) login failures from 118.125.243.7 (CN/China/-) in the last 300 secs - *Blocked in csf*
Sep 8 00:57:51 server lfd[13791]: 10 (pop3d) login failures from 92.63.14.18 (TR/Turkey/storage.nethouse.net) in the last 300 secs - *Blocked in csf*
Sep 8 01:11:53 server lfd[22887]: *LOAD* 5 minute load average is 6.05, threshold is 6 - email sent
Sep 8 01:28:40 server lfd[24279]: Directory Watching terminated after 22 seconds
Sep 8 01:28:40 server lfd[24279]: LF_DIRWATCH taking 22 seconds, temporarily throttled to run every 360 seconds
Sep 8 02:00:49 server lfd[27043]: *System Integrity* has detected modified file(s): /usr/bin/imgsize
Sep 8 02:22:10 server lfd[29419]: *LOAD* 5 minute load average is 7.63, threshold is 6 - email sent
Sep 8 04:21:06 server lfd[14189]: 5 (sshd) login failures from 117.41.229.178 (CN/China/-) in the last 300 secs - *Blocked in csf*
Sep 8 04:43:05 server lfd[16260]: *LOAD* 5 minute load average is 6.31, threshold is 6 - email sent
Sep 8 05:00:17 server lfd[17849]: *System Integrity* has detected modified file(s): /usr/bin/pure-pw /usr/bin/pure-pwconvert /usr/bin/pure-statsdecode /usr/sbin/exim /usr/sbin/exim_dbmbuild /usr/sbin/exim_dumpdb /usr/sbin/exim_fixdb /usr/sbin/exim_lock /usr/sbin/exim_tidydb /usr/sbin/pure-authd /usr/sbin/pure-ftpd /usr/sbin/pure-ftpwho /usr/sbin/pure-mrtginfo /usr/sbin/pure-quotacheck /usr/sbin/pure-uploadscript /usr/sbin/runq /usr/sbin/sendmail
Sep 8 05:26:32 server lfd[20671]: 10 (pop3d) login failures from 196.25.159.89 (ZA/South Africa/159.25.196.in-addr.arpa) in the last 300 secs - *Blocked in csf*
Sep 8 06:04:52 server lfd[24409]: *LOAD* 5 minute load average is 7.91, threshold is 6 - email sent
Sep 8 06:06:09 server lfd[24424]: Directory Watching terminated after 46 seconds
Sep 8 06:06:09 server lfd[24424]: LF_DIRWATCH taking 46 seconds, temporarily throttled to run every 1080 seconds
Sep 8 06:25:32 server lfd[26294]: Directory Watching terminated after 118 seconds
Sep 8 06:25:32 server lfd[26294]: LF_DIRWATCH taking 118 seconds, temporarily throttled to run every 3240 seconds
Sep 8 07:07:36 server lfd[30066]: *LOAD* 5 minute load average is 12.16, threshold is 6 - email sent
Sep 8 08:07:47 server lfd[3594]: *LOAD* 5 minute load average is 16.13, threshold is 6 - email sent
Sep 8 08:57:18 server lfd[7926]: *Email Queue* Unable to obtain exim queue length within 30 seconds - Timed out
Sep 8 09:02:53 server lfd[8207]: *Excessive Processes* Userohyane Kill:0 Process Count:16
Sep 8 09:08:00 server lfd[9214]: *LOAD* 5 minute load average is 15.31, threshold is 6 - email sent
Sep 8 10:13:47 server lfd[15400]: *LOAD* 5 minute load average is 7.15, threshold is 6 - email sent
Sep 8 10:36:18 server lfd[17166]: *Excessive Processes* Userohyane Kill:0 Process Count:16
Sep 8 11:23:31 server lfd[21851]: 5 (sshd) login failures from 200.111.39.250 (CL/Chile/-) in the last 300 secs - *Blocked in csf*
Sep 8 11:29:12 server lfd[22270]: *LOAD* 5 minute load average is 6.59, threshold is 6 - email sent
Sep 8 13:05:06 server lfd[31804]: *WHM root access* from 216.51.193.200
Sep 8 13:35:36 server lfd[2450]: *Skipped File* /tmp/#sql_a6e_0.MYD - Too large to scan
Sep 8 13:35:46 server lfd[2502]: *LOAD* 5 minute load average is 10.12, threshold is 6 - email sent
Sep 8 13:42:17 server lfd[3036]: *Excessive Processes* Userohyane Kill:0 Process Count:16

***Scratch that, just spiked to 72.

[Infopro]

what causes the system integrity notices?

Upgrades to cPanel will update files that CSF is monitoring. If you've just completed a cPanel update, you should restart CSF. Every time.

Also, problems can generate that email.

There is some sort of problem with your caching somewhere. I'd start looking closer there.

[phinsup]

I dunno if this was the issue or not, i'm still playing around, but I was having some strange server loads with the exact same config, fastcgi, zend and eaccel. since the most recent change i could think of was adding and using eaccelerator....

I recompiled without eaccelerator and the loads have been sitting normal for a day and a half.

[sfraise]

Thanks phinsup, so you think there may be an issue with eaccelerator running with fastcgi and zend?
I think I've gotten my problems pretty much solved by going back to prefork instead of worker.
I'm guessing there was something that just couldn't use multithreading and it was jacking up the whole thing, I've been running around .45 on the server load since switching back to prefork for a couple of hours now with no spikes.

I also turned off gzip in the joomla backend and went to using deflate instead, it seems to have knocked a couple of seconds off of the page load speed.

The site just "feels" smoother and quicker after making these two changes.
We'll see if this completely solves my load spikes or not over the next day or two but so far all seems well.

*** It's been several hours now since recompiling with prefork instead of worker and using deflate instead of gzip and there's still been no spikes what so ever. In fact I just checked the load a little bit ago and it was at .09 which I didn't even know was possible on this with everything it has going on lol. I really think this was using worker instead of prefork was the main issue. Between probably OVER optimizing everything from mysql, to apache, to the code it's self over this issue I think I ended up with a much faster and smoother site than I had before I started having load problems.

[phinsup]

Recompiling without eaccelerator lowered my overall load, but I'm still seeing what I would consider "odd" spikes, that don't make much sense. I'm thinking I may give a recompile with prefork a try just to see how it acts.

I've done a great deal of tweaking on mysql with very little effect on the spikes.

[phinsup]

Are you using mysql 5.0 or 5.1?

[sfraise]

I'm using 5.0.
One other thing I might mention is I noticed a load spike just a little bit ago, it ran really low for several hours then bam, hit a spike of around 20 for a couple of minutes, then back down to normal.

After a little investigation I noticed the /tmp /var/tmp were 100% full.
Once the tmp folder cleared the cpu load went back down.
Since eaccelerator is set to write to the /tmp folder by default it's obvious that I need to change it's cache path to another place with more room. However....
My problem now seems to be that I can't seem to change any of the settings in eaccelerator like I normal. I compiled this through a whm package but the ini file is in an odd place and changing the values don't appear to do any good, and there are no settings in php.ini for eaccelerator.

I'm going to post another thread to see if anyone can make heads or tales out of this. I think once I prevent the tmp folder from filling up every few hours that should be the last of my load issues.

[phinsup]

My temp directory is 4 gigs so I don't think I'm having that issue.

I'm still watching mine, but I upgraded to mysql 5.1 and I am not seeing any loads over 1 for the last 4 hours, but it's too soon to tell.

[fujipadam]

I had the same issue after I updated from php 5.3.1 to php 5.3.3 - The load just spiked up to 100s... I tried moving to php 5.3.2 but that didnt help either. After trying a bunch of stuff I at last gave up and moved back to php 5.2.x

After I moved to php 5.2.x the load settled down

Considering I had DSO and didnt have fastcgi etc I strongly suspect eaccelerator but couldnt really pin-point the issue (I had a similar experience with eaccelerator 0.9.6 and waited for 0.9.6.1 to fix the issue

I really need eaccelerator and love php 5.3's improved performance so hope this is resolved soon

Fuji

[boonjackmedia]

Zend Optimizer will only work with PHP 5.214 version or lower, that was your problem.

Mike