php 5.3.3 & apache 2.2 w/ eaccel, zend, fastcgi Super high cpu spikes

[sfraise]

I'm trying to get this thing under control but so far am having no luck.
I'm running a Joomla site under apache 2.2, php 5.3.3, eaccelerator, zend, memcached/mod_memcache, and using fastcgi with worker.
Server specs are:
Intel core 2 quad q9400 2.66ghz
8gb ddr2 ram
64bit
1tb hdd

Everything will be going smooth and fine, cpu load 0.5-3 depending, then all of a sudden out of nowhere it spikes to 40,50, or even 70. Looking at top I don't see anything eating that much cpu, looking at error logs I don't see any scripts with fatal errors on an endless loop or anything.

Anyone have any idea what is causing this? I'm going to try backing the php version down and see if that gives me any better luck.

[sfraise]

I backed php down to 5.3.2 but still getting high cpu spikes with php hogging the majority. Here's what a snapshot from top looks like:

Tasks: 191 total, 2 running, 189 sleeping, 0 stopped, 0 zombie
Cpu(s): 34.4%us, 3.3%sy, 0.0%ni, 60.6%id, 0.9%wa, 0.1%hi, 0.8%si, 0.0%st
Mem: 8177492k total, 5500940k used, 2676552k free, 52472k buffers
Swap: 2096472k total, 49036k used, 2047436k free, 2965196k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
24164 oohyane 16 0 215m 63m 22m R 63.6 0.8 0:11.39 php
24150 oohyane 16 0 220m 68m 22m S 29.6 0.9 0:17.10 php
24170 oohyane 16 0 203m 51m 22m S 27.6 0.6 0:09.79 php
5185 nobody 18 0 441m 42m 2692 S 9.0 0.5 4:49.70 httpd
2718 mysql 15 0 611m 222m 4100 S 5.0 2.8 100:47.11 mysqld
5310 nobody 18 0 436m 38m 2708 S 2.3 0.5 4:56.47 httpd
5186 nobody 18 0 1249m 38m 2728 S 2.0 0.5 5:03.47 httpd
5272 nobody 18 0 432m 37m 2692 S 2.0 0.5 5:21.10 httpd
17024 nobody 18 0 431m 31m 2644 S 2.0 0.4 1:08.55 httpd
19580 nobody 18 0 364m 27m 2624 S 1.3 0.3 0:37.67 httpd
9978 nobody 18 0 436m 36m 2732 S 1.0 0.5 3:08.89 httpd
17059 nobody 18 0 363m 26m 2344 S 1.0 0.3 1:03.24 httpd
19841 nobody 18 0 358m 21m 2640 S 1.0 0.3 0:35.11 httpd
2056 root 10 -5 0 0 0 S 0.7 0.0 35:31.76 kondemand/2
17135 root 18 0 370m 69m 9.9m S 0.7 0.9 8:00.47 java
566 root 11 -5 0 0 0 S 0.3 0.0 3:08.29 kjournald
18221 root 20 0 118m 16m 1592 S 0.3 0.2 0:46.87 lfd

I really need to get this figured out, I can't afford to keep spending the majority of my day messing with this instead of actually developing sites.

[Infopro]

Is this the account with the Joomla site?

24164 oohyane 16 0 215m 63m 22m R 63.6 0.8 0:11.39 php
24150 oohyane 16 0 220m 68m 22m S 29.6 0.9 0:17.10 php
24170 oohyane 16 0 203m 51m 22m S 27.6 0.6 0:09.79 php

[sfraise]

Yeah, it's a dedicated server just for myself.
It only has 2 accounts, the oohyane which has the main production Joomla site, and another account for a very low traffic Joomla site built for a friend. Pretty much any taxation on the machine will all come from the oohyane account.

I've currently got everything set to chown nobody and chgrp oohyane on that account as with this php/apache config that seems to be the only thing that works. I was running suphp prior to moving to the fastcgi set up for performance issues where I had everything chown and chgrp oohyane.

One other thing I might mention, when looking in the whm server status during the cpu spikes I start to use swap which usually shows as 0 before the cpu spikes. The memory seems to stay under 35% so I have plenty of usable ram if I could just find a way to utilize it instead of the cpu. I'm caching as much as I can, like I said I'm running eaccelerator, memcached/memcache, zend optimizer, I am letting memcached handle my sessions and I have memcache set in Joomla as the cache option. I also have q-cache installed in my Joomla build but I'm not currently running it as I haven't really seen any difference and wanted to rule that out as an offender. I'm also using a script/css compressor and minfier.

As Joomla is heavy on php and mysql I've spent quite a bit of time tuning my.conf and I think I've got that optimized pretty well. One thing I had considered doing was upping the eaccelerater shm size but as I compiled it through whm I don't seem to be able to change the size. I've tried tweaking the eaccelerater.ini, moving it to phpd, as well as trying to set it in the php.ini, but nothing seems to work. I've searched all over but can't seem to find a way to up the value without uninstalling it and compiling it manually. As I'm not sure this will really have any impact I hate to go through the hassle.

Serverbeach is switching out the server chassis to help rule out hardware issues so we'll see if it still spikes afterwards.

[Infopro]

Tell 'em Infopro says hello, they know me very well over there.

I used to run a Joomla site, and have hosted many more. I've never had problems that needed this kind of tweakage you're working with to make one run very fast. And, SuPHP worked very well.

Leaving the server tweaking aside that you've got going on for a moment, what sorts of addon modules are you using for the Joomla site?

This sounds like a bad script running out of control to me. If you could turn off extras you've got enabled on the Joomla it might be helpful to see how it goes overnight.

[sfraise]

Thanks for the reply infopro.
I've got several components and modules loaded up on this site, it's fairly large.
I've been going through and disabling modules trying to pinpoint the bad script but I have yet to find it. It seems no matter what I disable the cpu still eventually runs wild.

One last thing that I'm going to try is rebuilding the site in a different template. This site was migrated over to Joomla 1.5 from 1.0 and I had to go through and change some of the template code to get it to work in a 1.5 legacy environment, as well as change some of the deprecated functions to make it more php 5.3 friendly.

Maybe there's something in the template it's self that is causing the runoff, but I'm just not sure as I didn't really have this problem right after the migration.

I went to bed last night with several modules disabled and the cpu was running around 0.5, I wake up this morning and it's at 22 and I can't get the site to load. I restart apache and it's back down to 0.7. I'm guessing it will run that way for a couple of hours before spiking again. I'll try and get the rebuild in a new template done later and see if it solves it, I wish there was a way to find exactly what script was causing the runoff, running in debug mode doesn't seem to lead me to the problem and the error logs aren't showing me any fatal errors that point towards any one script.

[Infopro]

IMHO, if the server is going nuts, I'd personally disable every. single. addon. I had running on my site right out of the gate. A feature you like or your users like a lot, is still worthless if the site goes down. Take away all added features down to a core site and then wait and see how it goes.

I used to have a link to a really nice long list of vulnerable extensions for Joomla to stay away from, but I can't dig it up to share here. I did however do some searching for it online, problem there is, a few of the Joomla sites I visited, are also down or having problems.

So, instead I'll link you a site that has a comment about the new? list on doc.joomla.org which does not come up for me here, ATM.

/http://www.alltogetherasawhole.org/profiles/blogs/joomla-vulnerable-extensions

Switch to a default theme, turn off everything extra you added, disable an addon forum even if that's what it takes to keep the site up.

Then of course once you know it's staying up, turn only one on that you are sure of, an have looked into to be sure it's the newest and safest version available.

Lather, rinse, repeat.

I have not played much with Joomla in the past year or so, but I have done many upgrades from 1 to 1.5 and most times, problems I came across were out of date features the user had added and refused to give up.


Give them all "up" (turn 'em off) for a bit at least, until the server is under control. Best advise I can give you. Aside from mentioning if you don't have CSF installed (and are root and can install it) get it installed, it can help you out with many things, like alerting you to a script gone wild.

A snip from the features list:

Alert when end-user scripts sending excessive emails per hour - for identifying spamming scripts
Suspicious process reporting - reports potential exploits running on the server
Excessive user processes reporting
Excessive user process usage reporting and optional termination
Suspicious file reporting - reports potential exploit files in /tmp and similar directories

ConfigServer Security & Firewall

Honestly, I think you're going to have more luck over on the Joomla forums or in examining everything you've added for vulnerabilities.

Good Luck!

[sfraise]

Thanks again, I've disabled all but the absolute core functions such as community builder and login module, still it just runs for a couple of hours fine then spikes. As soon as I get a chance I'm going to rebuild the site using a template built for 1.5 instead of the modified one I've been using from the old 1.0 version. I have a strong hunch at this point that there is something in that template that isn't jiving right.

I do have configserver lsf on there with mod_security, and run mailscanner with spamassasin. I also use maldet to scan the public directory once a day, and use chkrootkit once in a while just to make sure things are as they should be.

One other thing I might mention here that has me a bit unsure, when the cpu spikes I usually get a system email stating that spamd, exim, pop, and imap failed, not always all of them, but at least one or two of them. Are these failing simply due to the cpu being ran up by php from bad code, or could there be something in the email settings that are actually to blame for the cpu run up?

[Infopro]

Are these failing simply due to the cpu being ran up by php from bad code

Could be. I'm curious why CSF isn't telling you more though. It should be.

[sfraise]

Here's a bit from the csf log, maybe there's something in there that catches your eye. The excessive processes under oohyane I'm guessing is a good indicator but I'm not sure exactly how to track down what is causing them.

Sep 5 14:33:26 server lfd[676]: *Email Queue* The exim delivery queue size is 198882
Sep 5 14:41:32 server lfd[1456]: *Suspicious Process* PID:2640 User:ntp Uptime:18323 secs EXE:/usr/sbin/ntpd CMD:ntpd -u ntp:ntp -p /var/run/ntpd.pid -g
Sep 5 14:41:32 server lfd[1456]: *Suspicious Process* PID:2653 User:clamav Uptime:18312 secs EXE:/usr/sbin/clamd CMD:clamd
Sep 5 14:41:32 server lfd[1456]: *User Processing* PID:2653 Kill:0 User:clamav Time:18312 EXE:/usr/sbin/clamd CMD:clamd
Sep 5 14:41:33 server lfd[1456]: *User Processing* PID:2640 Kill:0 User:ntp Time:18323 EXE:/usr/sbin/ntpd CMD:ntpd -u ntp:ntp -p /var/run/ntpd.pid -g
Sep 5 14:44:28 server lfd[1835]: 5 (mod_security) rule triggers from 109.93.1.104 (EU/-/109-93-1-104.dynamic.isp.telekom.rs) in the last 300 secs - *Blocked in csf*
Sep 5 14:45:29 server lfd[1904]: *Skipped File* /tmp/eaccelerator/c/9/eaccelerator-177.67723412 - Too large to scan
Sep 5 15:03:32 server lfd[3831]: *Skipped File* /tmp/eaccelerator/c/9/eaccelerator-177.67723412 - Too large to scan
Sep 5 15:21:35 server lfd[6388]: *Skipped File* /tmp/eaccelerator/c/9/eaccelerator-177.67723412 - Too large to scan
Sep 5 15:33:38 server lfd[7468]: *Email Queue* The exim delivery queue size is 198604
Sep 5 15:39:38 server lfd[8020]: *Skipped File* /tmp/eaccelerator/c/9/eaccelerator-177.67723412 - Too large to scan
Sep 5 15:41:44 server lfd[8137]: *Suspicious Process* PID:2640 User:ntp Uptime:21934 secs EXE:/usr/sbin/ntpd CMD:ntpd -u ntp:ntp -p /var/run/ntpd.pid -g
Sep 5 15:41:44 server lfd[8137]: *Suspicious Process* PID:2653 User:clamav Uptime:21923 secs EXE:/usr/sbin/clamd CMD:clamd
Sep 5 15:41:44 server lfd[8137]: *User Processing* PID:2653 Kill:0 User:clamav Time:21923 EXE:/usr/sbin/clamd CMD:clamd
Sep 5 15:41:44 server lfd[8137]: *User Processing* PID:2640 Kill:0 User:ntp Time:21934 EXE:/usr/sbin/ntpd CMD:ntpd -u ntp:ntp -p /var/run/ntpd.pid -g
Sep 5 15:57:41 server lfd[9751]: *Skipped File* /tmp/eaccelerator/c/9/eaccelerator-177.67723412 - Too large to scan
Sep 5 16:15:45 server lfd[11747]: *Skipped File* /tmp/eaccelerator/c/9/eaccelerator-177.67723412 - Too large to scan
Sep 5 16:33:49 server lfd[13231]: *Skipped File* /tmp/eaccelerator/c/9/eaccelerator-177.67723412 - Too large to scan
Sep 5 16:33:50 server lfd[13228]: *Email Queue* The exim delivery queue size is 198525
Sep 5 16:40:47 server lfd[13948]: 5 (mod_security) rule triggers from 77.120.115.211 (UA/Ukraine/211.115.120.77.colo.static.dc.volia.com) in the last 300 secs - *Blocked in csf*
Sep 5 16:41:57 server lfd[14025]: *Suspicious Process* PID:2640 User:ntp Uptime:25548 secs EXE:/usr/sbin/ntpd CMD:ntpd -u ntp:ntp -p /var/run/ntpd.pid -g
Sep 5 16:41:57 server lfd[14025]: *Suspicious Process* PID:2653 User:clamav Uptime:25536 secs EXE:/usr/sbin/clamd CMD:clamd
Sep 5 16:41:57 server lfd[14025]: *User Processing* PID:2653 Kill:0 User:clamav Time:25536 EXE:/usr/sbin/clamd CMD:clamd
Sep 5 16:41:57 server lfd[14025]: *User Processing* PID:2640 Kill:0 User:ntp Time:25548 EXE:/usr/sbin/ntpd CMD:ntpd -u ntp:ntp -p /var/run/ntpd.pid -g
Sep 5 16:51:49 server lfd[14872]: *Skipped File* /tmp/eaccelerator/c/9/eaccelerator-177.67723412 - Too large to scan
Sep 5 16:55:05 server lfd[15185]: *LOAD* 5 minute load average is 7.14, threshold is 6 - email sent
Sep 5 17:02:05 server lfd[15515]: *Excessive Processes* Userohyane Kill:0 Process Count:16
Sep 5 17:09:54 server lfd[16951]: *Skipped File* /tmp/eaccelerator/c/9/eaccelerator-177.67723412 - Too large to scan
Sep 5 17:27:58 server lfd[18616]: *Skipped File* /tmp/eaccelerator/c/9/eaccelerator-177.67723412 - Too large to scan
Sep 5 17:34:04 server lfd[19239]: *Email Queue* The exim delivery queue size is 198492
Sep 5 17:42:15 server lfd[19971]: *Suspicious Process* PID:2640 User:ntp Uptime:29166 secs EXE:/usr/sbin/ntpd CMD:ntpd -u ntp:ntp -p /var/run/ntpd.pid -g
Sep 5 17:42:15 server lfd[19971]: *Suspicious Process* PID:2653 User:clamav Uptime:29155 secs EXE:/usr/sbin/clamd CMD:clamd
Sep 5 17:42:16 server lfd[19971]: *User Processing* PID:2653 Kill:0 User:clamav Time:29155 EXE:/usr/sbin/clamd CMD:clamd
Sep 5 17:42:16 server lfd[19971]: *User Processing* PID:2640 Kill:0 User:ntp Time:29166 EXE:/usr/sbin/ntpd CMD:ntpd -u ntp:ntp -p /var/run/ntpd.pid -g
Sep 5 17:46:02 server lfd[20342]: *Skipped File* /tmp/eaccelerator/c/9/eaccelerator-177.67723412 - Too large to scan
Sep 5 18:04:06 server lfd[21921]: *Skipped File* /tmp/eaccelerator/c/9/eaccelerator-177.67723412 - Too large to scan
Sep 5 18:05:02 server lfd[21941]: *LOAD* 5 minute load average is 7.69, threshold is 6 - email sent
Sep 5 18:11:27 server lfd[22172]: *Excessive Processes* Userohyane Kill:0 Process Count:16
Sep 5 18:22:09 server lfd[23873]: *Skipped File* /tmp/eaccelerator/c/9/eaccelerator-177.67723412 - Too large to scan
Sep 5 18:34:21 server lfd[24539]: *Email Queue* The exim delivery queue size is 198477
Sep 5 18:40:22 server lfd[25240]: *Skipped File* /tmp/eaccelerator/c/9/eaccelerator-177.67723412 - Too large to scan
Sep 5 18:42:47 server lfd[25257]: *Suspicious Process* PID:2640 User:ntp Uptime:32798 secs EXE:/usr/sbin/ntpd CMD:ntpd -u ntp:ntp -p /var/run/ntpd.pid -g
Sep 5 18:42:53 server lfd[25257]: *Suspicious Process* PID:2653 User:clamav Uptime:32786 secs EXE:/usr/sbin/clamd CMD:clamd
Sep 5 18:42:53 server lfd[25257]: *User Processing* PID:2640 Kill:0 User:ntp Time:32798 EXE:/usr/sbin/ntpd CMD:ntpd -u ntp:ntp -p /var/run/ntpd.pid -g
Sep 5 18:42:53 server lfd[25257]: *User Processing* PID:2653 Kill:0 User:clamav Time:32786 EXE:/usr/sbin/clamd CMD:clamd
Sep 5 18:52:18 server lfd[26591]: *WHM root access* from 216.51.193.200
Sep 5 18:57:14 server lfd[27045]: *SSH login* from 216.51.193.200 into the root account using password authentication
Sep 5 18:58:14 server lfd[27102]: *Skipped File* /tmp/phpRE2pPI - Too large to scan
Sep 5 18:58:14 server lfd[27102]: *Skipped File* /tmp/eaccelerator/c/9/eaccelerator-177.67723412 - Too large to scan
Sep 5 19:06:16 server lfd[28128]: 5 (sshd) login failures from 118.123.15.100 (CN/China/-) in the last 300 secs - *Blocked in csf*
Sep 5 19:16:19 server lfd[28959]: *Skipped File* /tmp/eaccelerator/c/9/eaccelerator-177.67723412 - Too large to scan
Sep 5 19:34:22 server lfd[30554]: *Skipped File* /tmp/eaccelerator/c/9/eaccelerator-177.67723412 - Too large to scan
Sep 5 19:34:37 server lfd[30570]: *Email Queue* The exim delivery queue size is 198486
Sep 5 19:42:58 server lfd[31392]: *Suspicious Process* PID:2640 User:ntp Uptime:36409 secs EXE:/usr/sbin/ntpd CMD:ntpd -u ntp:ntp -p /var/run/ntpd.pid -g
Sep 5 19:42:58 server lfd[31392]: *Suspicious Process* PID:2653 User:clamav Uptime:36398 secs EXE:/usr/sbin/clamd CMD:clamd
Sep 5 19:42:58 server lfd[31392]: *User Processing* PID:2653 Kill:0 User:clamav Time:36398 EXE:/usr/sbin/clamd CMD:clamd
Sep 5 19:42:59 server lfd[31392]: *User Processing* PID:2640 Kill:0 User:ntp Time:36409 EXE:/usr/sbin/ntpd CMD:ntpd -u ntp:ntp -p /var/run/ntpd.pid -g
Sep 5 19:52:25 server lfd[32309]: *Skipped File* /tmp/eaccelerator/c/9/eaccelerator-177.67723412 - Too large to scan
Sep 5 19:58:22 server lfd[447]: *LOAD* 5 minute load average is 7.63, threshold is 6 - email sent
Sep 5 20:05:10 server lfd[676]: *Excessive Processes* Userohyane Kill:0 Process Count:16
Sep 5 20:10:38 server lfd[1816]: *Skipped File* /tmp/eaccelerator/c/9/eaccelerator-177.67723412 - Too large to scan
Sep 5 20:28:36 server lfd[3446]: *Skipped File* /tmp/eaccelerator/c/9/eaccelerator-177.67723412 - Too large to scan
Sep 5 20:35:20 server lfd[3823]: *Email Queue* Unable to obtain exim_outgoing.conf queue length within 30 seconds - Timed out
Sep 5 20:43:35 server lfd[4799]: *Suspicious Process* PID:2640 User:ntp Uptime:40046 secs EXE:/usr/sbin/ntpd CMD:ntpd -u ntp:ntp -p /var/run/ntpd.pid -g
Sep 5 20:43:35 server lfd[4799]: *Suspicious Process* PID:2653 User:clamav Uptime:40034 secs EXE:/usr/sbin/clamd CMD:clamd
Sep 5 20:43:35 server lfd[4799]: *User Processing* PID:2653 Kill:0 User:clamav Time:40034 EXE:/usr/sbin/clamd CMD:clamd
Sep 5 20:43:35 server lfd[4799]: *User Processing* PID:2640 Kill:0 User:ntp Time:40046 EXE:/usr/sbin/ntpd CMD:ntpd -u ntp:ntp -p /var/run/ntpd.pid -g
Sep 5 20:46:30 server lfd[5103]: *Skipped File* /tmp/eaccelerator/c/9/eaccelerator-177.67723412 - Too large to scan
Sep 5 21:04:34 server lfd[6799]: *Skipped File* /tmp/eaccelerator/c/9/eaccelerator-177.67723412 - Too large to scan
Sep 5 21:08:23 server lfd[7285]: *LOAD* 5 minute load average is 6.00, threshold is 6 - email sent
Sep 5 21:22:36 server lfd[8608]: *Skipped File* /tmp/eaccelerator/c/9/eaccelerator-177.67723412 - Too large to scan
Sep 5 21:40:01 server lfd[10354]: *Email Queue* The exim delivery queue size is 198403
Sep 5 21:40:36 server lfd[10404]: *Skipped File* /tmp/eaccelerator/c/9/eaccelerator-177.67723412 - Too large to scan
Sep 5 21:44:01 server lfd[10582]: *Suspicious Process* PID:2640 User:ntp Uptime:43672 secs EXE:/usr/sbin/ntpd CMD:ntpd -u ntp:ntp -p /var/run/ntpd.pid -g
Sep 5 21:44:02 server lfd[10582]: *Suspicious Process* PID:2653 User:clamav Uptime:43661 secs EXE:/usr/sbin/clamd CMD:clamd
Sep 5 21:44:02 server lfd[10582]: *User Processing* PID:2653 Kill:0 User:clamav Time:43661 EXE:/usr/sbin/clamd CMD:clamd
Sep 5 21:44:02 server lfd[10582]: *User Processing* PID:2640 Kill:0 User:ntp Time:43672 EXE:/usr/sbin/ntpd CMD:ntpd -u ntp:ntp -p /var/run/ntpd.pid -g
Sep 5 21:58:40 server lfd[11687]: *Skipped File* /tmp/eaccelerator/c/9/eaccelerator-177.67723412 - Too large to scan
Sep 5 22:08:52 server lfd[12484]: *LOAD* 5 minute load average is 23.34, threshold is 6 - email sent
Sep 5 22:16:44 server lfd[13526]: *Skipped File* /tmp/eaccelerator/c/9/eaccelerator-177.67723412 - Too large to scan
Sep 5 22:22:05 server lfd[13935]: 5 (sshd) login failures from 62.193.62.134 (FR/France/62-193-62-134.stella-net.net) in the last 300 secs - *Blocked in csf*
Sep 5 22:34:49 server lfd[15049]: *Skipped File* /tmp/eaccelerator/c/9/eaccelerator-177.67723412 - Too large to scan
Sep 5 22:40:20 server lfd[15654]: *Email Queue* The exim delivery queue size is 198411
Sep 5 22:44:41 server lfd[15849]: *Suspicious Process* PID:2640 User:ntp Uptime:47311 secs EXE:/usr/sbin/ntpd CMD:ntpd -u ntp:ntp -p /var/run/ntpd.pid -g
Sep 5 22:45:41 server lfd[15986]: *Suspicious Process* PID:2653 User:clamav Uptime:47360 secs EXE:/usr/sbin/clamd CMD:clamd
Sep 5 22:52:18 server lfd[16172]: *WHM root access* from 216.51.193.200
Sep 5 22:52:49 server lfd[16195]: *Excessive Processes* Userohyane Kill:0 Process Count:16

[Infopro]

These can be added to your csf.pignore

exe:/usr/sbin/clamd
exe:/usr/sbin/ntpd

--
I'm no expert on eaccelerator but I would think these could be killed from tmp;


*Skipped File* /tmp/phpRE2pPI - Too large to scan
*Skipped File* /tmp/eaccelerator/c/9/eaccelerator-177.67723412 - Too large to scan

As right after those, you are getting these.

*LOAD* 5 minute load average is 23.34, threshold is 6 - email sent

--

Definitely something going on with this account, caching?

*Excessive Processes* User oohyane Kill:0 Process Count:16

[sfraise]

Ok, quick question, since the LOAD notice is coming right after the *Skipped File* /tmp/eaccelerator/c/9/eaccelerator-177.67723412 - Too large to scan notice, could the eaccelerator compression be causing the run away load, or at least compounding the load issue?

I believe that the /c/9/ directories relate to the eaccelerator compression setting being set to 9, and I know compression can increase the cpu load, it's a long shot here but just one thing that popped into my head.

I set csf to ignore the ntp and clamd processes, it's something I kept meaning to do but hadn't gotten around to yet. I went ahead and set csf to ignore the /tmp/eaccelerator/ directory as well.

The thing that really has me the most concerned is the *Excessive Processes* User oohyane Kill:0 Process Count:16. I'm caching as much as I can for this site, I'm using memcached/memcache and have Joomla set to use memcache as it's cache method instead of file, which I've always done without any problem until now. I also set the php sessions in php.ini to be handled by memcached. I also started testing with a component called qcache but am going to remove it all together as I don't know that its doing any good and it could very well be harming as I'm running in legacy mode at the moment.

As I'm writing this my load just spiked and here's what csf just shot out:
Sep 6 16:51:25 server lfd[23375]: *Email Queue* The exim delivery queue size is 196092
Sep 6 16:58:50 server lfd[24042]: Directory Watching terminated after 22 seconds
Sep 6 16:58:50 server lfd[24042]: LF_DIRWATCH taking 22 seconds, temporarily throttled to run every 360 seconds
Sep 6 17:01:59 server lfd[24153]: *LOAD* 5 minute load average is 17.57, threshold is 6 - email sent
Sep 6 17:04:29 server lfd[24235]: *Skipped File* /tmp/#sql_ab1_0.MYD - Too large to scan
Sep 6 17:06:31 server lfd[24272]: *Excessive Processes* Userohyane Kill:0 Process Count:16
Sep 6 17:07:30 server lfd[24235]: Directory Watching terminated after 46 seconds
Sep 6 17:07:30 server lfd[24235]: LF_DIRWATCH taking 46 seconds, temporarily throttled to run every 1080 seconds
Sep 6 17:13:49 server lfd[25900]: 5 (sshd) login failures from 201.38.138.2 (BR/Brazil/-) in the last 300 secs - *Blocked in csf*
Sep 6 17:14:34 server lfd[25981]: *SSH login* from 216.51.193.200 into the root account using password authentication
Sep 6 17:51:39 server lfd[29370]: *Email Queue* The exim delivery queue size is 196099
Sep 6 18:02:04 server lfd[30624]: *LOAD* 5 minute load average is 13.37, threshold is 6 - email sent


I really have a hunch that the real issue here lies somewhere in the mail handling under the oohyane account. I wonder if there is an issue with mailscanner/spamassassin/exim, could that cause the excessive processes if they weren't configured correctly and cause the cpu load to spike?

The email queue of 196,099 is outrageous and I'm sure that is a leftover result of the malware that made it's way on to the machine a couple of months ago. I went ahead and deleted everything in the queue as I'm sure it's all junk spam that a malware script was sending at the time. I'll keep an eye on that to make sure it doesn't shoot back up, could just having this much in the mail queue put a load on the server?

[sfraise]

Here's a bit more out of csf, notice the one notice of 93 processes for oohyane, cpu spike to over 100 at this point.

Sep 7 00:37:59 server lfd[6525]: 5 (mod_security) rule triggers from 67.83.75.157 (US/United States/ool-43534b9d.dyn.optonline.net) in the last 300 secs - *Blocked in csf*
Sep 7 00:40:46 server lfd[6689]: *Email Queue* Unable to obtain exim_outgoing.conf queue length within 30 seconds - Timed out
Sep 7 00:42:16 server lfd[6707]: *Skipped File* /tmp/#sql_ab1_0.MYD - Too large to scan
Sep 7 00:46:19 server lfd[6881]: *Excessive Processes* Userohyane Kill:0 Process Count:16
Sep 7 00:49:04 server lfd[7825]: *LOAD* 5 minute load average is 23.80, threshold is 6 - email sent
Sep 7 00:54:20 server lfd[8203]: *Skipped File* /tmp/#sql_ab1_0.MYD - Too large to scan
Sep 7 00:58:24 server lfd[8515]: 5 (sshd) login failures from 122.72.31.130 (CN/China/-) in the last 300 secs - *Blocked in csf*
Sep 7 01:49:33 server lfd[13940]: *LOAD* 5 minute load average is 11.07, threshold is 6 - email sent
Sep 7 02:00:08 server lfd[14733]: *System Integrity* has detected modified file(s): /usr/bin/pure-pw /usr/bin/pure-pwconvert /usr/bin/pure-statsdecode /usr/sbin/exim /usr/sbin/exim_dbmbuild /usr/sbin/exim_dumpdb /usr/sbin/exim_fixdb /usr/sbin/exim_lock /usr/sbin/exim_tidydb /usr/sbin/pure-authd /usr/sbin/pure-ftpd /usr/sbin/pure-ftpwho /usr/sbin/pure-mrtginfo /usr/sbin/pure-quotacheck /usr/sbin/pure-uploadscript /usr/sbin/runq /usr/sbin/sendmail
Sep 7 02:26:52 server lfd[16805]: *Excessive Processes* Userohyane Kill:0 Process Count:16
Sep 7 02:40:14 server lfd[18280]: *WHM root access* from 216.51.193.200
Sep 7 03:24:26 server lfd[21998]: *LOAD* 5 minute load average is 7.65, threshold is 6 - email sent
Sep 7 03:51:26 server lfd[24104]: *Email Queue* Unable to obtain exim queue length within 30 seconds - Timed out
Sep 7 03:53:11 server lfd[24171]: *Excessive Processes* Userohyane Kill:0 Process Count:93
Sep 7 03:54:11 server lfd[24188]: *User Processing* PID:23666 Kill:0 Userohyane VM:219(MB) EXE:/usr/bin/php CMD:/usr/bin/php
Sep 7 03:54:40 server lfd[23996]: Directory Watching terminated after 46 seconds
Sep 7 03:54:40 server lfd[23996]: LF_DIRWATCH taking 46 seconds, temporarily throttled to run every 1080 seconds
Sep 7 03:55:11 server lfd[24361]: *User Processing* PID:22040 Kill:0 Userohyane VM:221(MB) EXE:/usr/bin/php CMD:/usr/bin/php

[sfraise]

The mail queue is now back up to over 300 since cleaning it out, do I still have a malware script running somewhere that I can't find hitting the mail on this account?

Looking through some of the notices from the root account here's something that might be related to this:
1Ot2T0-0001tY-8g-D
Time: Tue Sep 7 13:00:22 2010 -0500
PID: 5731
Account: nobody
Uptime: 21602 seconds


Executable:

/usr/bin/perl


Command Line (often faked in exploits):

spamd child


Network connections by the process (if any):

tcp: 127.0.0.1:783 -> 0.0.0.0:0
tcp: 127.0.0.1:783 -> 127.0.0.1:54786


Files open by the process (if any):

/dev/null
/dev/null
/dev/null
/usr/bin/spamd
/usr/lib/perl5/site_perl/5.8.8/Mail/SpamAssassin/Plugin/VBounce.pm


Memory maps by the process (if any):

00400000-00403000 r-xp 00000000 03:03 35300188 /usr/bin/perl
00602000-00604000 rw-p 00002000 03:03 35300188 /usr/bin/perl
02cc7000-05589000 rw-p 02cc7000 00:00 0 [heap]
3a33c00000-3a33c1c000 r-xp 00000000 03:03 105480238 /lib64/ld-2.5.so
3a33e1b000-3a33e1c000 r--p 0001b000 03:03 105480238 /lib64/ld-2.5.so
3a33e1c000-3a33e1d000 rw-p 0001c000 03:03 105480238 /lib64/ld-2.5.so
3a34000000-3a3414e000 r-xp 00000000 03:03 105480259 /lib64/libc-2.5.so
3a3414e000-3a3434d000 ---p 0014e000 03:03 105480259 /lib64/libc-2.5.so
3a3434d000-3a34351000 r--p 0014d000 03:03 105480259 /lib64/libc-2.5.so
3a34351000-3a34352000 rw-p 00151000 03:03 105480259 /lib64/libc-2.5.so
3a34352000-3a34357000 rw-p 3a34352000 00:00 0
3a34400000-3a34402000 r-xp 00000000 03:03 105480261 /lib64/libdl-2.5.so
3a34402000-3a34602000 ---p 00002000 03:03 105480261 /lib64/libdl-2.5.so
3a34602000-3a34603000 r--p 00002000 03:03 105480261 /lib64/libdl-2.5.so
3a34603000-3a34604000 rw-p 00003000 03:03 105480261 /lib64/libdl-2.5.so
3a34800000-3a34882000 r-xp 00000000 03:03 105480263 /lib64/libm-2.5.so
3a34882000-3a34a81000 ---p 00082000 03:03 105480263 /lib64/libm-2.5.so
3a34a81000-3a34a82000 r--p 00081000 03:03 105480263 /lib64/libm-2.5.so
3a34a82000-3a34a83000 rw-p 00082000 03:03 105480263 /lib64/libm-2.5.so
3a34c00000-3a34c16000 r-xp 00000000 03:03 105480266 /lib64/libpthread-2.5.so
3a34c16000-3a34e15000 ---p 00016000 03:03 105480266 /lib64/libpthread-2.5.so
3a34e15000-3a34e16000 r--p 00015000 03:03 105480266 /lib64/libpthread-2.5.so
3a34e16000-3a34e17000 rw-p 00016000 03:03 105480266 /lib64/libpthread-2.5.so
3a34e17000-3a34e1b000 rw-p 3a34e17000 00:00 0
3a35400000-3a35407000 r-xp 00000000 03:03 105480340 /lib64/librt-2.5.so
3a35407000-3a35607000 ---p 00007000 03:03 105480340 /lib64/librt-2.5.so
3a35607000-3a35608000 r--p 00007000 03:03 105480340 /lib64/librt-2.5.so
3a35608000-3a35609000 rw-p 00008000 03:03 105480340 /lib64/librt-2.5.so
3a36000000-3a36015000 r-xp 00000000 03:03 105480381 /lib64/libnsl-2.5.so
3a36015000-3a36214000 ---p 00015000 03:03 105480381 /lib64/libnsl-2.5.so
3a36214000-3a36215000 r--p 00014000 03:03 105480381 /lib64/libnsl-2.5.so
3a36215000-3a36216000 rw-p 00015000 03:03 105480381 /lib64/libnsl-2.5.so
3a36216000-3a36218000 rw-p 3a36216000 00:00 0
3a36400000-3a36409000 r-xp 00000000 03:03 105480392 /lib64/libcrypt-2.5.so
3a36409000-3a36608000 ---p 00009000 03:03 105480392 /lib64/libcrypt-2.5.so
3a36608000-3a36609000 r--p 00008000 03:03 105480392 /lib64/libcrypt-2.5.so
3a36609000-3a3660a000 rw-p 00009000 03:03 105480392 /lib64/libcrypt-2.5.so
3a3660a000-3a36638000 rw-p 3a3660a000 00:00 0
3a36c00000-3a36d2b000 r-xp 00000000 03:03 35423580 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so
3a36d2b000-3a36f2b000 ---p 0012b000 03:03 35423580 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so
3a36f2b000-3a36f34000 rw-p 0012b000 03:03 35423580 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so
3a36f34000-3a36f36000 rw-p 3a36f34000 00:00 0
3a37800000-3a37811000 r-xp 00000000 03:03 105480385 /lib64/libresolv-2.5.so
3a37811000-3a37a11000 ---p 00011000 03:03 105480385 /lib64/libresolv-2.5.so
3a37a11000-3a37a12000 r--p 00011000 03:03 105480385 /lib64/libresolv-2.5.so
3a37a12000-3a37a13000 rw-p 00012000 03:03 105480385 /lib64/libresolv-2.5.so
3a37a13000-3a37a15000 rw-p 3a37a13000 00:00 0
3a43a00000-3a43a02000 r-xp 00000000 03:03 105480393 /lib64/libutil-2.5.so
3a43a02000-3a43c01000 ---p 00002000 03:03 105480393 /lib64/libutil-2.5.so
3a43c01000-3a43c02000 r--p 00001000 03:03 105480393 /lib64/libutil-2.5.so
3a43c02000-3a43c03000 rw-p 00002000 03:03 105480393 /lib64/libutil-2.5.so
2aad551a5000-2aad551a7000 rw-p 2aad551a5000 00:00 0
2aad551be000-2aad551e3000 rw-p 2aad551be000 00:00 0
2aad551e3000-2aad551e8000 r-xp 00000000 03:03 35455182 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Socket/Socket.so
2aad551e8000-2aad553e7000 ---p 00005000 03:03 35455182 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Socket/Socket.so
2aad553e7000-2aad553e8000 rw-p 00004000 03:03 35455182 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Socket/Socket.so
2aad553e8000-2aad553ec000 r-xp 00000000 03:03 35455212 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/IO/IO.so
2aad553ec000-2aad555eb000 ---p 00004000 03:03 35455212 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/IO/IO.so
2aad555eb000-2aad555ec000 rw-p 00003000 03:03 35455212 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/IO/IO.so
2aad555ec000-2aad555f2000 r-xp 00000000 03:03 35487848 /usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi/auto/Socket6/Socket6.so
2aad555f2000-2aad557f1000 ---p 00006000 03:03 35487848 /usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi/auto/Socket6/Socket6.so
2aad557f1000-2aad557f2000 rw-p 00005000 03:03 35487848 /usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi/auto/Socket6/Socket6.so
2aad557f2000-2aad557f4000 r-xp 00000000 03:03 35455209 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Sys/Hostname/Hostname.so
2aad557f4000-2aad559f3000 ---p 00002000 03:03 35455209 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Sys/Hostname/Hostname.so
2aad559f3000-2aad559f4000 rw-p 00001000 03:03 35455209 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Sys/Hostname/Hostname.so
2aad559f4000-2aad559f7000 r-xp 00000000 03:03 35454992 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Fcntl/Fcntl.so
2aad559f7000-2aad55bf6000 ---p 00003000 03:03 35454992 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Fcntl/Fcntl.so
2aad55bf6000-2aad55bf7000 rw-p 00002000 03:03 35454992 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Fcntl/Fcntl.so
2aad55bf7000-2aad55c13000 r-xp 00000000 03:03 35455020 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/POSIX/POSIX.so
2aad55c13000-2aad55e13000 ---p 0001c000 03:03 35455020 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/POSIX/POSIX.so
2aad55e13000-2aad55e14000 rw-p 0001c000 03:03 35455020 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/POSIX/POSIX.so
2aad55e14000-2aad55e17000 r-xp 00000000 03:03 35455185 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/MIME/Base64/Base64.so
2aad55e17000-2aad56016000 ---p 00003000 03:03 35455185 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/MIME/Base64/Base64.so
2aad56016000-2aad56017000 rw-p 00002000 03:03 35455185 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/MIME/Base64/Base64.so
2aad56017000-2aad5601c000 r-xp 00000000 03:03 35454995 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/File/Glob/Glob.so
2aad5601c000-2aad5621b000 ---p 00005000 03:03 35454995 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/File/Glob/Glob.so
2aad5621b000-2aad5621c000 rw-p 00004000 03:03 35454995 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/File/Glob/Glob.so
2aad5621c000-2aad56222000 r-xp 00000000 03:03 35455505 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Time/HiRes/HiRes.so
2aad56222000-2aad56421000 ---p 00006000 03:03 35455505 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Time/HiRes/HiRes.so
2aad56421000-2aad56422000 rw-p 00005000 03:03 35455505 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Time/HiRes/HiRes.so
2aad56422000-2aad5642d000 r-xp 00000000 03:03 36605256 /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/HTML/Parser/Parser.so
2aad5642d000-2aad5662c000 ---p 0000b000 03:03 36605256 /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/HTML/Parser/Parser.so
2aad5662c000-2aad5662d000 rw-p 0000a000 03:03 36605256 /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/HTML/Parser/Parser.so
2aad5662d000-2aad5662f000 r-xp 00000000 03:03 36602631 /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/Net/DNS/DNS.so
2aad5662f000-2aad5682e000 ---p 00002000 03:03 36602631 /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/Net/DNS/DNS.so
2aad5682e000-2aad5682f000 rw-p 00001000 03:03 36602631 /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/Net/DNS/DNS.so
2aad5682f000-2aad56837000 r-xp 00000000 03:03 35424031 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Data/Dumper/Dumper.so
2aad56837000-2aad56a36000 ---p 00008000 03:03 35424031 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Data/Dumper/Dumper.so
2aad56a36000-2aad56a37000 rw-p 00007000 03:03 35424031 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Data/Dumper/Dumper.so
2aad56a37000-2aad56a3a000 r-xp 00000000 03:03 35425014 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Cwd/Cwd.so
2aad56a3a000-2aad56c39000 ---p 00003000 03:03 35425014 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Cwd/Cwd.so
2aad56c39000-2aad56c3a000 rw-p 00002000 03:03 35425014 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Cwd/Cwd.so
2aad56c3a000-2aad56c41000 r-xp 00000000 03:03 35455368 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/List/Util/Util.so
2aad56c41000-2aad56e40000 ---p 00007000 03:03 35455368 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/List/Util/Util.so
2aad56e40000-2aad56e41000 rw-p 00006000 03:03 35455368 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/List/Util/Util.so
2aad56e41000-2aad56e45000 r-xp 00000000 03:03 35455506 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Sys/Syslog/Syslog.so
2aad56e45000-2aad57045000 ---p 00004000 03:03 35455506 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Sys/Syslog/Syslog.so
2aad57045000-2aad57046000 rw-p 00004000 03:03 35455506 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Sys/Syslog/Syslog.so
2aad5705d000-2aad57067000 r-xp 00000000 03:03 105480226 /lib64/libnss_files-2.5.so
2aad57067000-2aad57266000 ---p 0000a000 03:03 105480226 /lib64/libnss_files-2.5.so
2aad57266000-2aad57267000 r--p 00009000 03:03 105480226 /lib64/libnss_files-2.5.so
2aad57267000-2aad57268000 rw-p 0000a000 03:03 105480226 /lib64/libnss_files-2.5.so
2aad57268000-2aad5726d000 r-xp 00000000 03:03 36602559 /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/Digest/SHA1/SHA1.so
2aad5726d000-2aad5746c000 ---p 00005000 03:03 36602559 /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/Digest/SHA1/SHA1.so
2aad5746c000-2aad5746d000 rw-p 00004000 03:03 36602559 /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/Digest/SHA1/SHA1.so
2aad5746d000-2aad5747a000 r-xp 00000000 03:03 35423909 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/DB_File/DB_File.so
2aad5747a000-2aad57679000 ---p 0000d000 03:03 35423909 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/DB_File/DB_File.so
2aad57679000-2aad5767a000 rw-p 0000c000 03:03 35423909 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/DB_File/DB_File.so
2aad5767a000-2aad5776b000 r-xp 00000000 03:03 105480405 /lib64/libdb-4.3.so
2aad5776b000-2aad5796b000 ---p 000f1000 03:03 105480405 /lib64/libdb-4.3.so
2aad5796b000-2aad57970000 rw-p 000f1000 03:03 105480405 /lib64/libdb-4.3.so
2aad57970000-2aad57991000 rw-p 2aad57970000 00:00 0
2aad57991000-2aad57995000 r-xp 00000000 03:03 36570779 /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/BSD/Resource/Resource.so
2aad57995000-2aad57b94000 ---p 00004000 03:03 36570779 /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/BSD/Resource/Resource.so
2aad57b94000-2aad57b95000 rw-p 00003000 03:03 36570779 /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/BSD/Resource/Resource.so
2aad57bcd000-2aad57c3d000 rw-p 2aad57bcd000 00:00 0
2aad57c3d000-2aad57c45000 r-xp 00000000 03:03 36570770 /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/version/vxs/vxs.so
2aad57c45000-2aad57e44000 ---p 00008000 03:03 36570770 /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/version/vxs/vxs.so
2aad57e44000-2aad57e45000 rw-p 00007000 03:03 36570770 /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/version/vxs/vxs.so
2aad57e45000-2aad57e4c000 r-xp 00000000 03:03 36603381 /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/NetAddr/IP/Util/Util.so
2aad57e4c000-2aad5804c000 ---p 00007000 03:03 36603381 /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/NetAddr/IP/Util/Util.so
2aad5804c000-2aad5804d000 rw-p 00007000 03:03 36603381 /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/NetAddr/IP/Util/Util.so
7fff558d5000-7fff55905000 rw-p 7ffffffcf000 00:00 0 [stack]
ffffffffff600000-ffffffffffe00000 ---p 00000000 00:00 0 [vdso]

[Infopro]

As with the others previously mentioned, you might add this one to the ignore as well.

exe:/usr/bin/perl

In your cPanel for this account, find the Default Address icon and click. What are your settings here exactly?

*Skipped File* /tmp/#sql_ab1_0.MYD - Too large to scan

Why are things being run from your tmp directory? Is this the caching you're using?