The Community Forums

Interact with an entire community of cPanel & WHM users!
  1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

php 5.3.3 & apache 2.2 w/ eaccel, zend, fastcgi Super high cpu spikes

Discussion in 'Workarounds and Optimization' started by sfraise, Sep 3, 2010.

  1. sfraise

    sfraise Member

    Joined:
    Aug 5, 2010
    Messages:
    20
    Likes Received:
    0
    Trophy Points:
    1
    I'm trying to get this thing under control but so far am having no luck.
    I'm running a Joomla site under apache 2.2, php 5.3.3, eaccelerator, zend, memcached/mod_memcache, and using fastcgi with worker.
    Server specs are:
    Intel core 2 quad q9400 2.66ghz
    8gb ddr2 ram
    64bit
    1tb hdd

    Everything will be going smooth and fine, cpu load 0.5-3 depending, then all of a sudden out of nowhere it spikes to 40,50, or even 70. Looking at top I don't see anything eating that much cpu, looking at error logs I don't see any scripts with fatal errors on an endless loop or anything.

    Anyone have any idea what is causing this? I'm going to try backing the php version down and see if that gives me any better luck.
     
    #1 sfraise, Sep 3, 2010
    Last edited: Sep 4, 2010
  2. sfraise

    sfraise Member

    Joined:
    Aug 5, 2010
    Messages:
    20
    Likes Received:
    0
    Trophy Points:
    1
    I backed php down to 5.3.2 but still getting high cpu spikes with php hogging the majority. Here's what a snapshot from top looks like:

    Tasks: 191 total, 2 running, 189 sleeping, 0 stopped, 0 zombie
    Cpu(s): 34.4%us, 3.3%sy, 0.0%ni, 60.6%id, 0.9%wa, 0.1%hi, 0.8%si, 0.0%st
    Mem: 8177492k total, 5500940k used, 2676552k free, 52472k buffers
    Swap: 2096472k total, 49036k used, 2047436k free, 2965196k cached

    PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
    24164 oohyane 16 0 215m 63m 22m R 63.6 0.8 0:11.39 php
    24150 oohyane 16 0 220m 68m 22m S 29.6 0.9 0:17.10 php
    24170 oohyane 16 0 203m 51m 22m S 27.6 0.6 0:09.79 php
    5185 nobody 18 0 441m 42m 2692 S 9.0 0.5 4:49.70 httpd
    2718 mysql 15 0 611m 222m 4100 S 5.0 2.8 100:47.11 mysqld
    5310 nobody 18 0 436m 38m 2708 S 2.3 0.5 4:56.47 httpd
    5186 nobody 18 0 1249m 38m 2728 S 2.0 0.5 5:03.47 httpd
    5272 nobody 18 0 432m 37m 2692 S 2.0 0.5 5:21.10 httpd
    17024 nobody 18 0 431m 31m 2644 S 2.0 0.4 1:08.55 httpd
    19580 nobody 18 0 364m 27m 2624 S 1.3 0.3 0:37.67 httpd
    9978 nobody 18 0 436m 36m 2732 S 1.0 0.5 3:08.89 httpd
    17059 nobody 18 0 363m 26m 2344 S 1.0 0.3 1:03.24 httpd
    19841 nobody 18 0 358m 21m 2640 S 1.0 0.3 0:35.11 httpd
    2056 root 10 -5 0 0 0 S 0.7 0.0 35:31.76 kondemand/2
    17135 root 18 0 370m 69m 9.9m S 0.7 0.9 8:00.47 java
    566 root 11 -5 0 0 0 S 0.3 0.0 3:08.29 kjournald
    18221 root 20 0 118m 16m 1592 S 0.3 0.2 0:46.87 lfd

    I really need to get this figured out, I can't afford to keep spending the majority of my day messing with this instead of actually developing sites.
     
  3. Infopro

    Infopro cPanel Sr. Product Evangelist
    Staff Member

    Joined:
    May 20, 2003
    Messages:
    14,460
    Likes Received:
    195
    Trophy Points:
    63
    Location:
    Pennsylvania
    cPanel Access Level:
    Root Administrator
    Twitter:
    Is this the account with the Joomla site?

    24164 oohyane 16 0 215m 63m 22m R 63.6 0.8 0:11.39 php
    24150 oohyane 16 0 220m 68m 22m S 29.6 0.9 0:17.10 php
    24170 oohyane 16 0 203m 51m 22m S 27.6 0.6 0:09.79 php
     
  4. sfraise

    sfraise Member

    Joined:
    Aug 5, 2010
    Messages:
    20
    Likes Received:
    0
    Trophy Points:
    1
    Yeah, it's a dedicated server just for myself.
    It only has 2 accounts, the oohyane which has the main production Joomla site, and another account for a very low traffic Joomla site built for a friend. Pretty much any taxation on the machine will all come from the oohyane account.

    I've currently got everything set to chown nobody and chgrp oohyane on that account as with this php/apache config that seems to be the only thing that works. I was running suphp prior to moving to the fastcgi set up for performance issues where I had everything chown and chgrp oohyane.

    One other thing I might mention, when looking in the whm server status during the cpu spikes I start to use swap which usually shows as 0 before the cpu spikes. The memory seems to stay under 35% so I have plenty of usable ram if I could just find a way to utilize it instead of the cpu. I'm caching as much as I can, like I said I'm running eaccelerator, memcached/memcache, zend optimizer, I am letting memcached handle my sessions and I have memcache set in Joomla as the cache option. I also have q-cache installed in my Joomla build but I'm not currently running it as I haven't really seen any difference and wanted to rule that out as an offender. I'm also using a script/css compressor and minfier.

    As Joomla is heavy on php and mysql I've spent quite a bit of time tuning my.conf and I think I've got that optimized pretty well. One thing I had considered doing was upping the eaccelerater shm size but as I compiled it through whm I don't seem to be able to change the size. I've tried tweaking the eaccelerater.ini, moving it to phpd, as well as trying to set it in the php.ini, but nothing seems to work. I've searched all over but can't seem to find a way to up the value without uninstalling it and compiling it manually. As I'm not sure this will really have any impact I hate to go through the hassle.

    Serverbeach is switching out the server chassis to help rule out hardware issues so we'll see if it still spikes afterwards.
     
    #4 sfraise, Sep 4, 2010
    Last edited: Sep 4, 2010
  5. Infopro

    Infopro cPanel Sr. Product Evangelist
    Staff Member

    Joined:
    May 20, 2003
    Messages:
    14,460
    Likes Received:
    195
    Trophy Points:
    63
    Location:
    Pennsylvania
    cPanel Access Level:
    Root Administrator
    Twitter:
    Tell 'em Infopro says hello, they know me very well over there. ;)

    I used to run a Joomla site, and have hosted many more. I've never had problems that needed this kind of tweakage you're working with to make one run very fast. And, SuPHP worked very well.

    Leaving the server tweaking aside that you've got going on for a moment, what sorts of addon modules are you using for the Joomla site?

    This sounds like a bad script running out of control to me. If you could turn off extras you've got enabled on the Joomla it might be helpful to see how it goes overnight.
     
  6. sfraise

    sfraise Member

    Joined:
    Aug 5, 2010
    Messages:
    20
    Likes Received:
    0
    Trophy Points:
    1
    Thanks for the reply infopro.
    I've got several components and modules loaded up on this site, it's fairly large.
    I've been going through and disabling modules trying to pinpoint the bad script but I have yet to find it. It seems no matter what I disable the cpu still eventually runs wild.

    One last thing that I'm going to try is rebuilding the site in a different template. This site was migrated over to Joomla 1.5 from 1.0 and I had to go through and change some of the template code to get it to work in a 1.5 legacy environment, as well as change some of the deprecated functions to make it more php 5.3 friendly.

    Maybe there's something in the template it's self that is causing the runoff, but I'm just not sure as I didn't really have this problem right after the migration.

    I went to bed last night with several modules disabled and the cpu was running around 0.5, I wake up this morning and it's at 22 and I can't get the site to load. I restart apache and it's back down to 0.7. I'm guessing it will run that way for a couple of hours before spiking again. I'll try and get the rebuild in a new template done later and see if it solves it, I wish there was a way to find exactly what script was causing the runoff, running in debug mode doesn't seem to lead me to the problem and the error logs aren't showing me any fatal errors that point towards any one script.
     
  7. Infopro

    Infopro cPanel Sr. Product Evangelist
    Staff Member

    Joined:
    May 20, 2003
    Messages:
    14,460
    Likes Received:
    195
    Trophy Points:
    63
    Location:
    Pennsylvania
    cPanel Access Level:
    Root Administrator
    Twitter:
    IMHO, if the server is going nuts, I'd personally disable every. single. addon. I had running on my site right out of the gate. A feature you like or your users like a lot, is still worthless if the site goes down. Take away all added features down to a core site and then wait and see how it goes.

    I used to have a link to a really nice long list of vulnerable extensions for Joomla to stay away from, but I can't dig it up to share here. I did however do some searching for it online, problem there is, a few of the Joomla sites I visited, are also down or having problems.

    So, instead I'll link you a site that has a comment about the new? list on doc.joomla.org which does not come up for me here, ATM.

    /http://www.alltogetherasawhole.org/profiles/blogs/joomla-vulnerable-extensions

    Switch to a default theme, turn off everything extra you added, disable an addon forum even if that's what it takes to keep the site up.

    Then of course once you know it's staying up, turn only one on that you are sure of, an have looked into to be sure it's the newest and safest version available.

    Lather, rinse, repeat.

    I have not played much with Joomla in the past year or so, but I have done many upgrades from 1 to 1.5 and most times, problems I came across were out of date features the user had added and refused to give up.


    Give them all "up" (turn 'em off) for a bit at least, until the server is under control. Best advise I can give you. Aside from mentioning if you don't have CSF installed (and are root and can install it) get it installed, it can help you out with many things, like alerting you to a script gone wild.

    A snip from the features list:

    ConfigServer Security & Firewall

    Honestly, I think you're going to have more luck over on the Joomla forums or in examining everything you've added for vulnerabilities.

    Good Luck!
     
  8. sfraise

    sfraise Member

    Joined:
    Aug 5, 2010
    Messages:
    20
    Likes Received:
    0
    Trophy Points:
    1
    Thanks again, I've disabled all but the absolute core functions such as community builder and login module, still it just runs for a couple of hours fine then spikes. As soon as I get a chance I'm going to rebuild the site using a template built for 1.5 instead of the modified one I've been using from the old 1.0 version. I have a strong hunch at this point that there is something in that template that isn't jiving right.

    I do have configserver lsf on there with mod_security, and run mailscanner with spamassasin. I also use maldet to scan the public directory once a day, and use chkrootkit once in a while just to make sure things are as they should be.

    One other thing I might mention here that has me a bit unsure, when the cpu spikes I usually get a system email stating that spamd, exim, pop, and imap failed, not always all of them, but at least one or two of them. Are these failing simply due to the cpu being ran up by php from bad code, or could there be something in the email settings that are actually to blame for the cpu run up?
     
  9. Infopro

    Infopro cPanel Sr. Product Evangelist
    Staff Member

    Joined:
    May 20, 2003
    Messages:
    14,460
    Likes Received:
    195
    Trophy Points:
    63
    Location:
    Pennsylvania
    cPanel Access Level:
    Root Administrator
    Twitter:
    Could be. I'm curious why CSF isn't telling you more though. It should be.
     
  10. sfraise

    sfraise Member

    Joined:
    Aug 5, 2010
    Messages:
    20
    Likes Received:
    0
    Trophy Points:
    1
    Here's a bit from the csf log, maybe there's something in there that catches your eye. The excessive processes under oohyane I'm guessing is a good indicator but I'm not sure exactly how to track down what is causing them.

    Sep 5 14:33:26 server lfd[676]: *Email Queue* The exim delivery queue size is 198882
    Sep 5 14:41:32 server lfd[1456]: *Suspicious Process* PID:2640 User:ntp Uptime:18323 secs EXE:/usr/sbin/ntpd CMD:ntpd -u ntp:ntp -p /var/run/ntpd.pid -g
    Sep 5 14:41:32 server lfd[1456]: *Suspicious Process* PID:2653 User:clamav Uptime:18312 secs EXE:/usr/sbin/clamd CMD:clamd
    Sep 5 14:41:32 server lfd[1456]: *User Processing* PID:2653 Kill:0 User:clamav Time:18312 EXE:/usr/sbin/clamd CMD:clamd
    Sep 5 14:41:33 server lfd[1456]: *User Processing* PID:2640 Kill:0 User:ntp Time:18323 EXE:/usr/sbin/ntpd CMD:ntpd -u ntp:ntp -p /var/run/ntpd.pid -g
    Sep 5 14:44:28 server lfd[1835]: 5 (mod_security) rule triggers from 109.93.1.104 (EU/-/109-93-1-104.dynamic.isp.telekom.rs) in the last 300 secs - *Blocked in csf*
    Sep 5 14:45:29 server lfd[1904]: *Skipped File* /tmp/eaccelerator/c/9/eaccelerator-177.67723412 - Too large to scan
    Sep 5 15:03:32 server lfd[3831]: *Skipped File* /tmp/eaccelerator/c/9/eaccelerator-177.67723412 - Too large to scan
    Sep 5 15:21:35 server lfd[6388]: *Skipped File* /tmp/eaccelerator/c/9/eaccelerator-177.67723412 - Too large to scan
    Sep 5 15:33:38 server lfd[7468]: *Email Queue* The exim delivery queue size is 198604
    Sep 5 15:39:38 server lfd[8020]: *Skipped File* /tmp/eaccelerator/c/9/eaccelerator-177.67723412 - Too large to scan
    Sep 5 15:41:44 server lfd[8137]: *Suspicious Process* PID:2640 User:ntp Uptime:21934 secs EXE:/usr/sbin/ntpd CMD:ntpd -u ntp:ntp -p /var/run/ntpd.pid -g
    Sep 5 15:41:44 server lfd[8137]: *Suspicious Process* PID:2653 User:clamav Uptime:21923 secs EXE:/usr/sbin/clamd CMD:clamd
    Sep 5 15:41:44 server lfd[8137]: *User Processing* PID:2653 Kill:0 User:clamav Time:21923 EXE:/usr/sbin/clamd CMD:clamd
    Sep 5 15:41:44 server lfd[8137]: *User Processing* PID:2640 Kill:0 User:ntp Time:21934 EXE:/usr/sbin/ntpd CMD:ntpd -u ntp:ntp -p /var/run/ntpd.pid -g
    Sep 5 15:57:41 server lfd[9751]: *Skipped File* /tmp/eaccelerator/c/9/eaccelerator-177.67723412 - Too large to scan
    Sep 5 16:15:45 server lfd[11747]: *Skipped File* /tmp/eaccelerator/c/9/eaccelerator-177.67723412 - Too large to scan
    Sep 5 16:33:49 server lfd[13231]: *Skipped File* /tmp/eaccelerator/c/9/eaccelerator-177.67723412 - Too large to scan
    Sep 5 16:33:50 server lfd[13228]: *Email Queue* The exim delivery queue size is 198525
    Sep 5 16:40:47 server lfd[13948]: 5 (mod_security) rule triggers from 77.120.115.211 (UA/Ukraine/211.115.120.77.colo.static.dc.volia.com) in the last 300 secs - *Blocked in csf*
    Sep 5 16:41:57 server lfd[14025]: *Suspicious Process* PID:2640 User:ntp Uptime:25548 secs EXE:/usr/sbin/ntpd CMD:ntpd -u ntp:ntp -p /var/run/ntpd.pid -g
    Sep 5 16:41:57 server lfd[14025]: *Suspicious Process* PID:2653 User:clamav Uptime:25536 secs EXE:/usr/sbin/clamd CMD:clamd
    Sep 5 16:41:57 server lfd[14025]: *User Processing* PID:2653 Kill:0 User:clamav Time:25536 EXE:/usr/sbin/clamd CMD:clamd
    Sep 5 16:41:57 server lfd[14025]: *User Processing* PID:2640 Kill:0 User:ntp Time:25548 EXE:/usr/sbin/ntpd CMD:ntpd -u ntp:ntp -p /var/run/ntpd.pid -g
    Sep 5 16:51:49 server lfd[14872]: *Skipped File* /tmp/eaccelerator/c/9/eaccelerator-177.67723412 - Too large to scan
    Sep 5 16:55:05 server lfd[15185]: *LOAD* 5 minute load average is 7.14, threshold is 6 - email sent
    Sep 5 17:02:05 server lfd[15515]: *Excessive Processes* User:eek:ohyane Kill:0 Process Count:16
    Sep 5 17:09:54 server lfd[16951]: *Skipped File* /tmp/eaccelerator/c/9/eaccelerator-177.67723412 - Too large to scan
    Sep 5 17:27:58 server lfd[18616]: *Skipped File* /tmp/eaccelerator/c/9/eaccelerator-177.67723412 - Too large to scan
    Sep 5 17:34:04 server lfd[19239]: *Email Queue* The exim delivery queue size is 198492
    Sep 5 17:42:15 server lfd[19971]: *Suspicious Process* PID:2640 User:ntp Uptime:29166 secs EXE:/usr/sbin/ntpd CMD:ntpd -u ntp:ntp -p /var/run/ntpd.pid -g
    Sep 5 17:42:15 server lfd[19971]: *Suspicious Process* PID:2653 User:clamav Uptime:29155 secs EXE:/usr/sbin/clamd CMD:clamd
    Sep 5 17:42:16 server lfd[19971]: *User Processing* PID:2653 Kill:0 User:clamav Time:29155 EXE:/usr/sbin/clamd CMD:clamd
    Sep 5 17:42:16 server lfd[19971]: *User Processing* PID:2640 Kill:0 User:ntp Time:29166 EXE:/usr/sbin/ntpd CMD:ntpd -u ntp:ntp -p /var/run/ntpd.pid -g
    Sep 5 17:46:02 server lfd[20342]: *Skipped File* /tmp/eaccelerator/c/9/eaccelerator-177.67723412 - Too large to scan
    Sep 5 18:04:06 server lfd[21921]: *Skipped File* /tmp/eaccelerator/c/9/eaccelerator-177.67723412 - Too large to scan
    Sep 5 18:05:02 server lfd[21941]: *LOAD* 5 minute load average is 7.69, threshold is 6 - email sent
    Sep 5 18:11:27 server lfd[22172]: *Excessive Processes* User:eek:ohyane Kill:0 Process Count:16
    Sep 5 18:22:09 server lfd[23873]: *Skipped File* /tmp/eaccelerator/c/9/eaccelerator-177.67723412 - Too large to scan
    Sep 5 18:34:21 server lfd[24539]: *Email Queue* The exim delivery queue size is 198477
    Sep 5 18:40:22 server lfd[25240]: *Skipped File* /tmp/eaccelerator/c/9/eaccelerator-177.67723412 - Too large to scan
    Sep 5 18:42:47 server lfd[25257]: *Suspicious Process* PID:2640 User:ntp Uptime:32798 secs EXE:/usr/sbin/ntpd CMD:ntpd -u ntp:ntp -p /var/run/ntpd.pid -g
    Sep 5 18:42:53 server lfd[25257]: *Suspicious Process* PID:2653 User:clamav Uptime:32786 secs EXE:/usr/sbin/clamd CMD:clamd
    Sep 5 18:42:53 server lfd[25257]: *User Processing* PID:2640 Kill:0 User:ntp Time:32798 EXE:/usr/sbin/ntpd CMD:ntpd -u ntp:ntp -p /var/run/ntpd.pid -g
    Sep 5 18:42:53 server lfd[25257]: *User Processing* PID:2653 Kill:0 User:clamav Time:32786 EXE:/usr/sbin/clamd CMD:clamd
    Sep 5 18:52:18 server lfd[26591]: *WHM root access* from 216.51.193.200
    Sep 5 18:57:14 server lfd[27045]: *SSH login* from 216.51.193.200 into the root account using password authentication
    Sep 5 18:58:14 server lfd[27102]: *Skipped File* /tmp/phpRE2pPI - Too large to scan
    Sep 5 18:58:14 server lfd[27102]: *Skipped File* /tmp/eaccelerator/c/9/eaccelerator-177.67723412 - Too large to scan
    Sep 5 19:06:16 server lfd[28128]: 5 (sshd) login failures from 118.123.15.100 (CN/China/-) in the last 300 secs - *Blocked in csf*
    Sep 5 19:16:19 server lfd[28959]: *Skipped File* /tmp/eaccelerator/c/9/eaccelerator-177.67723412 - Too large to scan
    Sep 5 19:34:22 server lfd[30554]: *Skipped File* /tmp/eaccelerator/c/9/eaccelerator-177.67723412 - Too large to scan
    Sep 5 19:34:37 server lfd[30570]: *Email Queue* The exim delivery queue size is 198486
    Sep 5 19:42:58 server lfd[31392]: *Suspicious Process* PID:2640 User:ntp Uptime:36409 secs EXE:/usr/sbin/ntpd CMD:ntpd -u ntp:ntp -p /var/run/ntpd.pid -g
    Sep 5 19:42:58 server lfd[31392]: *Suspicious Process* PID:2653 User:clamav Uptime:36398 secs EXE:/usr/sbin/clamd CMD:clamd
    Sep 5 19:42:58 server lfd[31392]: *User Processing* PID:2653 Kill:0 User:clamav Time:36398 EXE:/usr/sbin/clamd CMD:clamd
    Sep 5 19:42:59 server lfd[31392]: *User Processing* PID:2640 Kill:0 User:ntp Time:36409 EXE:/usr/sbin/ntpd CMD:ntpd -u ntp:ntp -p /var/run/ntpd.pid -g
    Sep 5 19:52:25 server lfd[32309]: *Skipped File* /tmp/eaccelerator/c/9/eaccelerator-177.67723412 - Too large to scan
    Sep 5 19:58:22 server lfd[447]: *LOAD* 5 minute load average is 7.63, threshold is 6 - email sent
    Sep 5 20:05:10 server lfd[676]: *Excessive Processes* User:eek:ohyane Kill:0 Process Count:16
    Sep 5 20:10:38 server lfd[1816]: *Skipped File* /tmp/eaccelerator/c/9/eaccelerator-177.67723412 - Too large to scan
    Sep 5 20:28:36 server lfd[3446]: *Skipped File* /tmp/eaccelerator/c/9/eaccelerator-177.67723412 - Too large to scan
    Sep 5 20:35:20 server lfd[3823]: *Email Queue* Unable to obtain exim_outgoing.conf queue length within 30 seconds - Timed out
    Sep 5 20:43:35 server lfd[4799]: *Suspicious Process* PID:2640 User:ntp Uptime:40046 secs EXE:/usr/sbin/ntpd CMD:ntpd -u ntp:ntp -p /var/run/ntpd.pid -g
    Sep 5 20:43:35 server lfd[4799]: *Suspicious Process* PID:2653 User:clamav Uptime:40034 secs EXE:/usr/sbin/clamd CMD:clamd
    Sep 5 20:43:35 server lfd[4799]: *User Processing* PID:2653 Kill:0 User:clamav Time:40034 EXE:/usr/sbin/clamd CMD:clamd
    Sep 5 20:43:35 server lfd[4799]: *User Processing* PID:2640 Kill:0 User:ntp Time:40046 EXE:/usr/sbin/ntpd CMD:ntpd -u ntp:ntp -p /var/run/ntpd.pid -g
    Sep 5 20:46:30 server lfd[5103]: *Skipped File* /tmp/eaccelerator/c/9/eaccelerator-177.67723412 - Too large to scan
    Sep 5 21:04:34 server lfd[6799]: *Skipped File* /tmp/eaccelerator/c/9/eaccelerator-177.67723412 - Too large to scan
    Sep 5 21:08:23 server lfd[7285]: *LOAD* 5 minute load average is 6.00, threshold is 6 - email sent
    Sep 5 21:22:36 server lfd[8608]: *Skipped File* /tmp/eaccelerator/c/9/eaccelerator-177.67723412 - Too large to scan
    Sep 5 21:40:01 server lfd[10354]: *Email Queue* The exim delivery queue size is 198403
    Sep 5 21:40:36 server lfd[10404]: *Skipped File* /tmp/eaccelerator/c/9/eaccelerator-177.67723412 - Too large to scan
    Sep 5 21:44:01 server lfd[10582]: *Suspicious Process* PID:2640 User:ntp Uptime:43672 secs EXE:/usr/sbin/ntpd CMD:ntpd -u ntp:ntp -p /var/run/ntpd.pid -g
    Sep 5 21:44:02 server lfd[10582]: *Suspicious Process* PID:2653 User:clamav Uptime:43661 secs EXE:/usr/sbin/clamd CMD:clamd
    Sep 5 21:44:02 server lfd[10582]: *User Processing* PID:2653 Kill:0 User:clamav Time:43661 EXE:/usr/sbin/clamd CMD:clamd
    Sep 5 21:44:02 server lfd[10582]: *User Processing* PID:2640 Kill:0 User:ntp Time:43672 EXE:/usr/sbin/ntpd CMD:ntpd -u ntp:ntp -p /var/run/ntpd.pid -g
    Sep 5 21:58:40 server lfd[11687]: *Skipped File* /tmp/eaccelerator/c/9/eaccelerator-177.67723412 - Too large to scan
    Sep 5 22:08:52 server lfd[12484]: *LOAD* 5 minute load average is 23.34, threshold is 6 - email sent
    Sep 5 22:16:44 server lfd[13526]: *Skipped File* /tmp/eaccelerator/c/9/eaccelerator-177.67723412 - Too large to scan
    Sep 5 22:22:05 server lfd[13935]: 5 (sshd) login failures from 62.193.62.134 (FR/France/62-193-62-134.stella-net.net) in the last 300 secs - *Blocked in csf*
    Sep 5 22:34:49 server lfd[15049]: *Skipped File* /tmp/eaccelerator/c/9/eaccelerator-177.67723412 - Too large to scan
    Sep 5 22:40:20 server lfd[15654]: *Email Queue* The exim delivery queue size is 198411
    Sep 5 22:44:41 server lfd[15849]: *Suspicious Process* PID:2640 User:ntp Uptime:47311 secs EXE:/usr/sbin/ntpd CMD:ntpd -u ntp:ntp -p /var/run/ntpd.pid -g
    Sep 5 22:45:41 server lfd[15986]: *Suspicious Process* PID:2653 User:clamav Uptime:47360 secs EXE:/usr/sbin/clamd CMD:clamd
    Sep 5 22:52:18 server lfd[16172]: *WHM root access* from 216.51.193.200
    Sep 5 22:52:49 server lfd[16195]: *Excessive Processes* User:eek:ohyane Kill:0 Process Count:16
     
  11. Infopro

    Infopro cPanel Sr. Product Evangelist
    Staff Member

    Joined:
    May 20, 2003
    Messages:
    14,460
    Likes Received:
    195
    Trophy Points:
    63
    Location:
    Pennsylvania
    cPanel Access Level:
    Root Administrator
    Twitter:
    These can be added to your csf.pignore

    exe:/usr/sbin/clamd
    exe:/usr/sbin/ntpd

    --
    I'm no expert on eaccelerator but I would think these could be killed from tmp;


    *Skipped File* /tmp/phpRE2pPI - Too large to scan
    *Skipped File* /tmp/eaccelerator/c/9/eaccelerator-177.67723412 - Too large to scan

    As right after those, you are getting these.

    *LOAD* 5 minute load average is 23.34, threshold is 6 - email sent

    --

    Definitely something going on with this account, caching?

    *Excessive Processes* User oohyane Kill:0 Process Count:16
     
  12. sfraise

    sfraise Member

    Joined:
    Aug 5, 2010
    Messages:
    20
    Likes Received:
    0
    Trophy Points:
    1
    Ok, quick question, since the LOAD notice is coming right after the *Skipped File* /tmp/eaccelerator/c/9/eaccelerator-177.67723412 - Too large to scan notice, could the eaccelerator compression be causing the run away load, or at least compounding the load issue?

    I believe that the /c/9/ directories relate to the eaccelerator compression setting being set to 9, and I know compression can increase the cpu load, it's a long shot here but just one thing that popped into my head.

    I set csf to ignore the ntp and clamd processes, it's something I kept meaning to do but hadn't gotten around to yet. I went ahead and set csf to ignore the /tmp/eaccelerator/ directory as well.

    The thing that really has me the most concerned is the *Excessive Processes* User oohyane Kill:0 Process Count:16. I'm caching as much as I can for this site, I'm using memcached/memcache and have Joomla set to use memcache as it's cache method instead of file, which I've always done without any problem until now. I also set the php sessions in php.ini to be handled by memcached. I also started testing with a component called qcache but am going to remove it all together as I don't know that its doing any good and it could very well be harming as I'm running in legacy mode at the moment.

    As I'm writing this my load just spiked and here's what csf just shot out:
    Sep 6 16:51:25 server lfd[23375]: *Email Queue* The exim delivery queue size is 196092
    Sep 6 16:58:50 server lfd[24042]: Directory Watching terminated after 22 seconds
    Sep 6 16:58:50 server lfd[24042]: LF_DIRWATCH taking 22 seconds, temporarily throttled to run every 360 seconds
    Sep 6 17:01:59 server lfd[24153]: *LOAD* 5 minute load average is 17.57, threshold is 6 - email sent
    Sep 6 17:04:29 server lfd[24235]: *Skipped File* /tmp/#sql_ab1_0.MYD - Too large to scan
    Sep 6 17:06:31 server lfd[24272]: *Excessive Processes* User:eek:ohyane Kill:0 Process Count:16
    Sep 6 17:07:30 server lfd[24235]: Directory Watching terminated after 46 seconds
    Sep 6 17:07:30 server lfd[24235]: LF_DIRWATCH taking 46 seconds, temporarily throttled to run every 1080 seconds
    Sep 6 17:13:49 server lfd[25900]: 5 (sshd) login failures from 201.38.138.2 (BR/Brazil/-) in the last 300 secs - *Blocked in csf*
    Sep 6 17:14:34 server lfd[25981]: *SSH login* from 216.51.193.200 into the root account using password authentication
    Sep 6 17:51:39 server lfd[29370]: *Email Queue* The exim delivery queue size is 196099
    Sep 6 18:02:04 server lfd[30624]: *LOAD* 5 minute load average is 13.37, threshold is 6 - email sent


    I really have a hunch that the real issue here lies somewhere in the mail handling under the oohyane account. I wonder if there is an issue with mailscanner/spamassassin/exim, could that cause the excessive processes if they weren't configured correctly and cause the cpu load to spike?

    The email queue of 196,099 is outrageous and I'm sure that is a leftover result of the malware that made it's way on to the machine a couple of months ago. I went ahead and deleted everything in the queue as I'm sure it's all junk spam that a malware script was sending at the time. I'll keep an eye on that to make sure it doesn't shoot back up, could just having this much in the mail queue put a load on the server?
     
    #12 sfraise, Sep 6, 2010
    Last edited: Sep 6, 2010
  13. sfraise

    sfraise Member

    Joined:
    Aug 5, 2010
    Messages:
    20
    Likes Received:
    0
    Trophy Points:
    1
    Here's a bit more out of csf, notice the one notice of 93 processes for oohyane, cpu spike to over 100 at this point.

    Sep 7 00:37:59 server lfd[6525]: 5 (mod_security) rule triggers from 67.83.75.157 (US/United States/ool-43534b9d.dyn.optonline.net) in the last 300 secs - *Blocked in csf*
    Sep 7 00:40:46 server lfd[6689]: *Email Queue* Unable to obtain exim_outgoing.conf queue length within 30 seconds - Timed out
    Sep 7 00:42:16 server lfd[6707]: *Skipped File* /tmp/#sql_ab1_0.MYD - Too large to scan
    Sep 7 00:46:19 server lfd[6881]: *Excessive Processes* User:eek:ohyane Kill:0 Process Count:16
    Sep 7 00:49:04 server lfd[7825]: *LOAD* 5 minute load average is 23.80, threshold is 6 - email sent
    Sep 7 00:54:20 server lfd[8203]: *Skipped File* /tmp/#sql_ab1_0.MYD - Too large to scan
    Sep 7 00:58:24 server lfd[8515]: 5 (sshd) login failures from 122.72.31.130 (CN/China/-) in the last 300 secs - *Blocked in csf*
    Sep 7 01:49:33 server lfd[13940]: *LOAD* 5 minute load average is 11.07, threshold is 6 - email sent
    Sep 7 02:00:08 server lfd[14733]: *System Integrity* has detected modified file(s): /usr/bin/pure-pw /usr/bin/pure-pwconvert /usr/bin/pure-statsdecode /usr/sbin/exim /usr/sbin/exim_dbmbuild /usr/sbin/exim_dumpdb /usr/sbin/exim_fixdb /usr/sbin/exim_lock /usr/sbin/exim_tidydb /usr/sbin/pure-authd /usr/sbin/pure-ftpd /usr/sbin/pure-ftpwho /usr/sbin/pure-mrtginfo /usr/sbin/pure-quotacheck /usr/sbin/pure-uploadscript /usr/sbin/runq /usr/sbin/sendmail
    Sep 7 02:26:52 server lfd[16805]: *Excessive Processes* User:eek:ohyane Kill:0 Process Count:16
    Sep 7 02:40:14 server lfd[18280]: *WHM root access* from 216.51.193.200
    Sep 7 03:24:26 server lfd[21998]: *LOAD* 5 minute load average is 7.65, threshold is 6 - email sent
    Sep 7 03:51:26 server lfd[24104]: *Email Queue* Unable to obtain exim queue length within 30 seconds - Timed out
    Sep 7 03:53:11 server lfd[24171]: *Excessive Processes* User:eek:ohyane Kill:0 Process Count:93
    Sep 7 03:54:11 server lfd[24188]: *User Processing* PID:23666 Kill:0 User:eek:ohyane VM:219(MB) EXE:/usr/bin/php CMD:/usr/bin/php
    Sep 7 03:54:40 server lfd[23996]: Directory Watching terminated after 46 seconds
    Sep 7 03:54:40 server lfd[23996]: LF_DIRWATCH taking 46 seconds, temporarily throttled to run every 1080 seconds
    Sep 7 03:55:11 server lfd[24361]: *User Processing* PID:22040 Kill:0 User:eek:ohyane VM:221(MB) EXE:/usr/bin/php CMD:/usr/bin/php
     
  14. sfraise

    sfraise Member

    Joined:
    Aug 5, 2010
    Messages:
    20
    Likes Received:
    0
    Trophy Points:
    1
    The mail queue is now back up to over 300 since cleaning it out, do I still have a malware script running somewhere that I can't find hitting the mail on this account?

    Looking through some of the notices from the root account here's something that might be related to this:
    1Ot2T0-0001tY-8g-D
    Time: Tue Sep 7 13:00:22 2010 -0500
    PID: 5731
    Account: nobody
    Uptime: 21602 seconds


    Executable:

    /usr/bin/perl


    Command Line (often faked in exploits):

    spamd child


    Network connections by the process (if any):

    tcp: 127.0.0.1:783 -> 0.0.0.0:0
    tcp: 127.0.0.1:783 -> 127.0.0.1:54786


    Files open by the process (if any):

    /dev/null
    /dev/null
    /dev/null
    /usr/bin/spamd
    /usr/lib/perl5/site_perl/5.8.8/Mail/SpamAssassin/Plugin/VBounce.pm


    Memory maps by the process (if any):

    00400000-00403000 r-xp 00000000 03:03 35300188 /usr/bin/perl
    00602000-00604000 rw-p 00002000 03:03 35300188 /usr/bin/perl
    02cc7000-05589000 rw-p 02cc7000 00:00 0 [heap]
    3a33c00000-3a33c1c000 r-xp 00000000 03:03 105480238 /lib64/ld-2.5.so
    3a33e1b000-3a33e1c000 r--p 0001b000 03:03 105480238 /lib64/ld-2.5.so
    3a33e1c000-3a33e1d000 rw-p 0001c000 03:03 105480238 /lib64/ld-2.5.so
    3a34000000-3a3414e000 r-xp 00000000 03:03 105480259 /lib64/libc-2.5.so
    3a3414e000-3a3434d000 ---p 0014e000 03:03 105480259 /lib64/libc-2.5.so
    3a3434d000-3a34351000 r--p 0014d000 03:03 105480259 /lib64/libc-2.5.so
    3a34351000-3a34352000 rw-p 00151000 03:03 105480259 /lib64/libc-2.5.so
    3a34352000-3a34357000 rw-p 3a34352000 00:00 0
    3a34400000-3a34402000 r-xp 00000000 03:03 105480261 /lib64/libdl-2.5.so
    3a34402000-3a34602000 ---p 00002000 03:03 105480261 /lib64/libdl-2.5.so
    3a34602000-3a34603000 r--p 00002000 03:03 105480261 /lib64/libdl-2.5.so
    3a34603000-3a34604000 rw-p 00003000 03:03 105480261 /lib64/libdl-2.5.so
    3a34800000-3a34882000 r-xp 00000000 03:03 105480263 /lib64/libm-2.5.so
    3a34882000-3a34a81000 ---p 00082000 03:03 105480263 /lib64/libm-2.5.so
    3a34a81000-3a34a82000 r--p 00081000 03:03 105480263 /lib64/libm-2.5.so
    3a34a82000-3a34a83000 rw-p 00082000 03:03 105480263 /lib64/libm-2.5.so
    3a34c00000-3a34c16000 r-xp 00000000 03:03 105480266 /lib64/libpthread-2.5.so
    3a34c16000-3a34e15000 ---p 00016000 03:03 105480266 /lib64/libpthread-2.5.so
    3a34e15000-3a34e16000 r--p 00015000 03:03 105480266 /lib64/libpthread-2.5.so
    3a34e16000-3a34e17000 rw-p 00016000 03:03 105480266 /lib64/libpthread-2.5.so
    3a34e17000-3a34e1b000 rw-p 3a34e17000 00:00 0
    3a35400000-3a35407000 r-xp 00000000 03:03 105480340 /lib64/librt-2.5.so
    3a35407000-3a35607000 ---p 00007000 03:03 105480340 /lib64/librt-2.5.so
    3a35607000-3a35608000 r--p 00007000 03:03 105480340 /lib64/librt-2.5.so
    3a35608000-3a35609000 rw-p 00008000 03:03 105480340 /lib64/librt-2.5.so
    3a36000000-3a36015000 r-xp 00000000 03:03 105480381 /lib64/libnsl-2.5.so
    3a36015000-3a36214000 ---p 00015000 03:03 105480381 /lib64/libnsl-2.5.so
    3a36214000-3a36215000 r--p 00014000 03:03 105480381 /lib64/libnsl-2.5.so
    3a36215000-3a36216000 rw-p 00015000 03:03 105480381 /lib64/libnsl-2.5.so
    3a36216000-3a36218000 rw-p 3a36216000 00:00 0
    3a36400000-3a36409000 r-xp 00000000 03:03 105480392 /lib64/libcrypt-2.5.so
    3a36409000-3a36608000 ---p 00009000 03:03 105480392 /lib64/libcrypt-2.5.so
    3a36608000-3a36609000 r--p 00008000 03:03 105480392 /lib64/libcrypt-2.5.so
    3a36609000-3a3660a000 rw-p 00009000 03:03 105480392 /lib64/libcrypt-2.5.so
    3a3660a000-3a36638000 rw-p 3a3660a000 00:00 0
    3a36c00000-3a36d2b000 r-xp 00000000 03:03 35423580 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so
    3a36d2b000-3a36f2b000 ---p 0012b000 03:03 35423580 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so
    3a36f2b000-3a36f34000 rw-p 0012b000 03:03 35423580 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so
    3a36f34000-3a36f36000 rw-p 3a36f34000 00:00 0
    3a37800000-3a37811000 r-xp 00000000 03:03 105480385 /lib64/libresolv-2.5.so
    3a37811000-3a37a11000 ---p 00011000 03:03 105480385 /lib64/libresolv-2.5.so
    3a37a11000-3a37a12000 r--p 00011000 03:03 105480385 /lib64/libresolv-2.5.so
    3a37a12000-3a37a13000 rw-p 00012000 03:03 105480385 /lib64/libresolv-2.5.so
    3a37a13000-3a37a15000 rw-p 3a37a13000 00:00 0
    3a43a00000-3a43a02000 r-xp 00000000 03:03 105480393 /lib64/libutil-2.5.so
    3a43a02000-3a43c01000 ---p 00002000 03:03 105480393 /lib64/libutil-2.5.so
    3a43c01000-3a43c02000 r--p 00001000 03:03 105480393 /lib64/libutil-2.5.so
    3a43c02000-3a43c03000 rw-p 00002000 03:03 105480393 /lib64/libutil-2.5.so
    2aad551a5000-2aad551a7000 rw-p 2aad551a5000 00:00 0
    2aad551be000-2aad551e3000 rw-p 2aad551be000 00:00 0
    2aad551e3000-2aad551e8000 r-xp 00000000 03:03 35455182 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Socket/Socket.so
    2aad551e8000-2aad553e7000 ---p 00005000 03:03 35455182 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Socket/Socket.so
    2aad553e7000-2aad553e8000 rw-p 00004000 03:03 35455182 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Socket/Socket.so
    2aad553e8000-2aad553ec000 r-xp 00000000 03:03 35455212 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/IO/IO.so
    2aad553ec000-2aad555eb000 ---p 00004000 03:03 35455212 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/IO/IO.so
    2aad555eb000-2aad555ec000 rw-p 00003000 03:03 35455212 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/IO/IO.so
    2aad555ec000-2aad555f2000 r-xp 00000000 03:03 35487848 /usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi/auto/Socket6/Socket6.so
    2aad555f2000-2aad557f1000 ---p 00006000 03:03 35487848 /usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi/auto/Socket6/Socket6.so
    2aad557f1000-2aad557f2000 rw-p 00005000 03:03 35487848 /usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi/auto/Socket6/Socket6.so
    2aad557f2000-2aad557f4000 r-xp 00000000 03:03 35455209 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Sys/Hostname/Hostname.so
    2aad557f4000-2aad559f3000 ---p 00002000 03:03 35455209 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Sys/Hostname/Hostname.so
    2aad559f3000-2aad559f4000 rw-p 00001000 03:03 35455209 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Sys/Hostname/Hostname.so
    2aad559f4000-2aad559f7000 r-xp 00000000 03:03 35454992 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Fcntl/Fcntl.so
    2aad559f7000-2aad55bf6000 ---p 00003000 03:03 35454992 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Fcntl/Fcntl.so
    2aad55bf6000-2aad55bf7000 rw-p 00002000 03:03 35454992 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Fcntl/Fcntl.so
    2aad55bf7000-2aad55c13000 r-xp 00000000 03:03 35455020 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/POSIX/POSIX.so
    2aad55c13000-2aad55e13000 ---p 0001c000 03:03 35455020 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/POSIX/POSIX.so
    2aad55e13000-2aad55e14000 rw-p 0001c000 03:03 35455020 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/POSIX/POSIX.so
    2aad55e14000-2aad55e17000 r-xp 00000000 03:03 35455185 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/MIME/Base64/Base64.so
    2aad55e17000-2aad56016000 ---p 00003000 03:03 35455185 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/MIME/Base64/Base64.so
    2aad56016000-2aad56017000 rw-p 00002000 03:03 35455185 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/MIME/Base64/Base64.so
    2aad56017000-2aad5601c000 r-xp 00000000 03:03 35454995 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/File/Glob/Glob.so
    2aad5601c000-2aad5621b000 ---p 00005000 03:03 35454995 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/File/Glob/Glob.so
    2aad5621b000-2aad5621c000 rw-p 00004000 03:03 35454995 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/File/Glob/Glob.so
    2aad5621c000-2aad56222000 r-xp 00000000 03:03 35455505 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Time/HiRes/HiRes.so
    2aad56222000-2aad56421000 ---p 00006000 03:03 35455505 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Time/HiRes/HiRes.so
    2aad56421000-2aad56422000 rw-p 00005000 03:03 35455505 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Time/HiRes/HiRes.so
    2aad56422000-2aad5642d000 r-xp 00000000 03:03 36605256 /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/HTML/Parser/Parser.so
    2aad5642d000-2aad5662c000 ---p 0000b000 03:03 36605256 /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/HTML/Parser/Parser.so
    2aad5662c000-2aad5662d000 rw-p 0000a000 03:03 36605256 /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/HTML/Parser/Parser.so
    2aad5662d000-2aad5662f000 r-xp 00000000 03:03 36602631 /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/Net/DNS/DNS.so
    2aad5662f000-2aad5682e000 ---p 00002000 03:03 36602631 /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/Net/DNS/DNS.so
    2aad5682e000-2aad5682f000 rw-p 00001000 03:03 36602631 /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/Net/DNS/DNS.so
    2aad5682f000-2aad56837000 r-xp 00000000 03:03 35424031 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Data/Dumper/Dumper.so
    2aad56837000-2aad56a36000 ---p 00008000 03:03 35424031 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Data/Dumper/Dumper.so
    2aad56a36000-2aad56a37000 rw-p 00007000 03:03 35424031 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Data/Dumper/Dumper.so
    2aad56a37000-2aad56a3a000 r-xp 00000000 03:03 35425014 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Cwd/Cwd.so
    2aad56a3a000-2aad56c39000 ---p 00003000 03:03 35425014 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Cwd/Cwd.so
    2aad56c39000-2aad56c3a000 rw-p 00002000 03:03 35425014 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Cwd/Cwd.so
    2aad56c3a000-2aad56c41000 r-xp 00000000 03:03 35455368 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/List/Util/Util.so
    2aad56c41000-2aad56e40000 ---p 00007000 03:03 35455368 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/List/Util/Util.so
    2aad56e40000-2aad56e41000 rw-p 00006000 03:03 35455368 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/List/Util/Util.so
    2aad56e41000-2aad56e45000 r-xp 00000000 03:03 35455506 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Sys/Syslog/Syslog.so
    2aad56e45000-2aad57045000 ---p 00004000 03:03 35455506 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Sys/Syslog/Syslog.so
    2aad57045000-2aad57046000 rw-p 00004000 03:03 35455506 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Sys/Syslog/Syslog.so
    2aad5705d000-2aad57067000 r-xp 00000000 03:03 105480226 /lib64/libnss_files-2.5.so
    2aad57067000-2aad57266000 ---p 0000a000 03:03 105480226 /lib64/libnss_files-2.5.so
    2aad57266000-2aad57267000 r--p 00009000 03:03 105480226 /lib64/libnss_files-2.5.so
    2aad57267000-2aad57268000 rw-p 0000a000 03:03 105480226 /lib64/libnss_files-2.5.so
    2aad57268000-2aad5726d000 r-xp 00000000 03:03 36602559 /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/Digest/SHA1/SHA1.so
    2aad5726d000-2aad5746c000 ---p 00005000 03:03 36602559 /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/Digest/SHA1/SHA1.so
    2aad5746c000-2aad5746d000 rw-p 00004000 03:03 36602559 /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/Digest/SHA1/SHA1.so
    2aad5746d000-2aad5747a000 r-xp 00000000 03:03 35423909 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/DB_File/DB_File.so
    2aad5747a000-2aad57679000 ---p 0000d000 03:03 35423909 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/DB_File/DB_File.so
    2aad57679000-2aad5767a000 rw-p 0000c000 03:03 35423909 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/DB_File/DB_File.so
    2aad5767a000-2aad5776b000 r-xp 00000000 03:03 105480405 /lib64/libdb-4.3.so
    2aad5776b000-2aad5796b000 ---p 000f1000 03:03 105480405 /lib64/libdb-4.3.so
    2aad5796b000-2aad57970000 rw-p 000f1000 03:03 105480405 /lib64/libdb-4.3.so
    2aad57970000-2aad57991000 rw-p 2aad57970000 00:00 0
    2aad57991000-2aad57995000 r-xp 00000000 03:03 36570779 /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/BSD/Resource/Resource.so
    2aad57995000-2aad57b94000 ---p 00004000 03:03 36570779 /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/BSD/Resource/Resource.so
    2aad57b94000-2aad57b95000 rw-p 00003000 03:03 36570779 /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/BSD/Resource/Resource.so
    2aad57bcd000-2aad57c3d000 rw-p 2aad57bcd000 00:00 0
    2aad57c3d000-2aad57c45000 r-xp 00000000 03:03 36570770 /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/version/vxs/vxs.so
    2aad57c45000-2aad57e44000 ---p 00008000 03:03 36570770 /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/version/vxs/vxs.so
    2aad57e44000-2aad57e45000 rw-p 00007000 03:03 36570770 /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/version/vxs/vxs.so
    2aad57e45000-2aad57e4c000 r-xp 00000000 03:03 36603381 /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/NetAddr/IP/Util/Util.so
    2aad57e4c000-2aad5804c000 ---p 00007000 03:03 36603381 /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/NetAddr/IP/Util/Util.so
    2aad5804c000-2aad5804d000 rw-p 00007000 03:03 36603381 /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/NetAddr/IP/Util/Util.so
    7fff558d5000-7fff55905000 rw-p 7ffffffcf000 00:00 0 [stack]
    ffffffffff600000-ffffffffffe00000 ---p 00000000 00:00 0 [vdso]
     
    #14 sfraise, Sep 7, 2010
    Last edited: Sep 7, 2010
  15. Infopro

    Infopro cPanel Sr. Product Evangelist
    Staff Member

    Joined:
    May 20, 2003
    Messages:
    14,460
    Likes Received:
    195
    Trophy Points:
    63
    Location:
    Pennsylvania
    cPanel Access Level:
    Root Administrator
    Twitter:
    As with the others previously mentioned, you might add this one to the ignore as well.

    exe:/usr/bin/perl

    In your cPanel for this account, find the Default Address icon and click. What are your settings here exactly?

    Why are things being run from your tmp directory? Is this the caching you're using?
     
  16. sfraise

    sfraise Member

    Joined:
    Aug 5, 2010
    Messages:
    20
    Likes Received:
    0
    Trophy Points:
    1
    It was set to forward to oohyane, I set it to discard.
    I never really thought much about the cpanel settings as I have it set in whm to fail.

    The tmp files your seeing are from caching, eaccelerator and memcached both use the tmp directory.
     
  17. sfraise

    sfraise Member

    Joined:
    Aug 5, 2010
    Messages:
    20
    Likes Received:
    0
    Trophy Points:
    1
    Ok, not going to jinx myself or anything here, but I was able to track down an issue in sh404sef which is the url rewriting component I use, makes sense as I recently upgraded to a newer version shortly after I moved to this new box. I upgraded to the latest version that just came out and so far I'm running between 0.22 - 1.5 with no high cpu spikes yet (knock on wood).

    I'll keep an eye on things for a while and see if this load holds or if it ends up spiking again, hopefully I got a handle on it finally though.
     
  18. sfraise

    sfraise Member

    Joined:
    Aug 5, 2010
    Messages:
    20
    Likes Received:
    0
    Trophy Points:
    1
    Well, still spiking but doesn't seem to be nearly as severe. Where it was spiking to as high as 100, it now seems to stay well under 20 during the spikes and then go back down under 1 fairly quickly. I'm still getting excessive processes under oohyane.

    I've posted in the sh404sef forum and they seem to think the main issue lies with community builder. I've posted on their forum as well but have yet to hear anything back yet. Here's an excerpt from todays csf output, what causes the system integrity notices?

    Sep 8 00:51:25 server lfd[12497]: 5 (sshd) login failures from 118.125.243.7 (CN/China/-) in the last 300 secs - *Blocked in csf*
    Sep 8 00:57:51 server lfd[13791]: 10 (pop3d) login failures from 92.63.14.18 (TR/Turkey/storage.nethouse.net) in the last 300 secs - *Blocked in csf*
    Sep 8 01:11:53 server lfd[22887]: *LOAD* 5 minute load average is 6.05, threshold is 6 - email sent
    Sep 8 01:28:40 server lfd[24279]: Directory Watching terminated after 22 seconds
    Sep 8 01:28:40 server lfd[24279]: LF_DIRWATCH taking 22 seconds, temporarily throttled to run every 360 seconds
    Sep 8 02:00:49 server lfd[27043]: *System Integrity* has detected modified file(s): /usr/bin/imgsize
    Sep 8 02:22:10 server lfd[29419]: *LOAD* 5 minute load average is 7.63, threshold is 6 - email sent
    Sep 8 04:21:06 server lfd[14189]: 5 (sshd) login failures from 117.41.229.178 (CN/China/-) in the last 300 secs - *Blocked in csf*
    Sep 8 04:43:05 server lfd[16260]: *LOAD* 5 minute load average is 6.31, threshold is 6 - email sent
    Sep 8 05:00:17 server lfd[17849]: *System Integrity* has detected modified file(s): /usr/bin/pure-pw /usr/bin/pure-pwconvert /usr/bin/pure-statsdecode /usr/sbin/exim /usr/sbin/exim_dbmbuild /usr/sbin/exim_dumpdb /usr/sbin/exim_fixdb /usr/sbin/exim_lock /usr/sbin/exim_tidydb /usr/sbin/pure-authd /usr/sbin/pure-ftpd /usr/sbin/pure-ftpwho /usr/sbin/pure-mrtginfo /usr/sbin/pure-quotacheck /usr/sbin/pure-uploadscript /usr/sbin/runq /usr/sbin/sendmail
    Sep 8 05:26:32 server lfd[20671]: 10 (pop3d) login failures from 196.25.159.89 (ZA/South Africa/159.25.196.in-addr.arpa) in the last 300 secs - *Blocked in csf*
    Sep 8 06:04:52 server lfd[24409]: *LOAD* 5 minute load average is 7.91, threshold is 6 - email sent
    Sep 8 06:06:09 server lfd[24424]: Directory Watching terminated after 46 seconds
    Sep 8 06:06:09 server lfd[24424]: LF_DIRWATCH taking 46 seconds, temporarily throttled to run every 1080 seconds
    Sep 8 06:25:32 server lfd[26294]: Directory Watching terminated after 118 seconds
    Sep 8 06:25:32 server lfd[26294]: LF_DIRWATCH taking 118 seconds, temporarily throttled to run every 3240 seconds
    Sep 8 07:07:36 server lfd[30066]: *LOAD* 5 minute load average is 12.16, threshold is 6 - email sent
    Sep 8 08:07:47 server lfd[3594]: *LOAD* 5 minute load average is 16.13, threshold is 6 - email sent
    Sep 8 08:57:18 server lfd[7926]: *Email Queue* Unable to obtain exim queue length within 30 seconds - Timed out
    Sep 8 09:02:53 server lfd[8207]: *Excessive Processes* User:eek:ohyane Kill:0 Process Count:16
    Sep 8 09:08:00 server lfd[9214]: *LOAD* 5 minute load average is 15.31, threshold is 6 - email sent
    Sep 8 10:13:47 server lfd[15400]: *LOAD* 5 minute load average is 7.15, threshold is 6 - email sent
    Sep 8 10:36:18 server lfd[17166]: *Excessive Processes* User:eek:ohyane Kill:0 Process Count:16
    Sep 8 11:23:31 server lfd[21851]: 5 (sshd) login failures from 200.111.39.250 (CL/Chile/-) in the last 300 secs - *Blocked in csf*
    Sep 8 11:29:12 server lfd[22270]: *LOAD* 5 minute load average is 6.59, threshold is 6 - email sent
    Sep 8 13:05:06 server lfd[31804]: *WHM root access* from 216.51.193.200
    Sep 8 13:35:36 server lfd[2450]: *Skipped File* /tmp/#sql_a6e_0.MYD - Too large to scan
    Sep 8 13:35:46 server lfd[2502]: *LOAD* 5 minute load average is 10.12, threshold is 6 - email sent
    Sep 8 13:42:17 server lfd[3036]: *Excessive Processes* User:eek:ohyane Kill:0 Process Count:16

    ***Scratch that, just spiked to 72.
     
    #18 sfraise, Sep 8, 2010
    Last edited: Sep 8, 2010
  19. Infopro

    Infopro cPanel Sr. Product Evangelist
    Staff Member

    Joined:
    May 20, 2003
    Messages:
    14,460
    Likes Received:
    195
    Trophy Points:
    63
    Location:
    Pennsylvania
    cPanel Access Level:
    Root Administrator
    Twitter:
    Upgrades to cPanel will update files that CSF is monitoring. If you've just completed a cPanel update, you should restart CSF. Every time.

    Also, problems can generate that email.

    There is some sort of problem with your caching somewhere. I'd start looking closer there.
     
  20. phinsup

    phinsup Active Member

    Joined:
    Jun 25, 2003
    Messages:
    35
    Likes Received:
    0
    Trophy Points:
    6
    I dunno if this was the issue or not, i'm still playing around, but I was having some strange server loads with the exact same config, fastcgi, zend and eaccel. since the most recent change i could think of was adding and using eaccelerator....

    I recompiled without eaccelerator and the loads have been sitting normal for a day and a half.
     
Loading...

Share This Page