server load problems SMF and joomla

[abubin]

Recently we upgraded one of our website to using joomla and SMF.

Things start to go bad from there. Server load became extremely high until more than 120 in top.

After days of testing and checking, we manage to narrow down the problem to SMF.

If we disable SMF, everything will run nicely. Once we started SMF, when users logon and start using the forum. The load will start going up up up and up until the server stop responding.

We see httpd to be causing this. When we run SMF, httpd starts to use lots and lots of memory. In top, it shows httpd to be using like 90mb for 1 process.

Here is a snapshot of top :

17:16:59 up 1 day, 19:39, 2 users, load average: 19.62, 8.62, 3.94
325 processes: 315 sleeping, 6 running, 3 zombie, 1 stopped
CPU states: cpu user nice system irq softirq iowait idle
total 89.4% 0.0% 9.0% 0.3% 1.1% 0.0% 0.0%
cpu00 85.7% 0.0% 12.7% 0.7% 0.7% 0.0% 0.0%
cpu01 93.1% 0.0% 5.3% 0.0% 1.5% 0.0% 0.0%
Mem: 1015908k av, 995980k used, 19928k free, 0k shrd, 6812k buff
758620k actv, 142600k in_d, 12616k in_c
Swap: 2048276k av, 687008k used, 1361268k free 78396k cached

PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME CPU COMMAND
8649 nobody 21 0 93260 84M 5616 D 0.0 8.5 0:30 1 httpd
1685 nobody 25 0 96544 81M 8844 D 0.7 8.2 0:34 0 httpd
1683 nobody 15 0 83460 78M 8844 D 0.3 7.8 1:08 0 httpd
8652 nobody 15 0 79900 76M 5840 S 0.3 7.7 0:05 1 httpd
7185 nobody 25 0 93648 71M 5980 R 22.2 7.1 1:00 0 httpd
1677 nobody 25 0 92068 68M 8848 R 21.8 6.9 0:30 1 httpd
1674 nobody 25 0 92072 66M 9172 D 14.3 6.7 0:36 0 httpd
1668 nobody 15 0 77744 61M 7964 S 0.0 6.1 0:35 0 httpd
1680 nobody 15 0 91332 38M 8496 D 0.0 3.8 1:36 0 httpd
1230 mysql 15 0 76584 32M 2740 S 0.0 3.3 0:00 0 mysqld
1233 mysql 25 0 76584 32M 2740 S 0.0 3.3 0:00 0 mysqld
1234 mysql 15 0 76584 32M 2740 S 0.0 3.3 0:00 0 mysqld
1235 mysql 15 0 76584 32M 2740 S 0.7 3.3 0:15 0 mysqld
1236 mysql 15 0 76584 32M 2740 S 0.0 3.3 0:00 0 mysqld
1237 mysql 15 0 76584 32M 2740 S 0.0 3.3 0:00 0 mysqld
1238 mysql 15 0 76584 32M 2740 S 0.0 3.3 0:00 1 mysqld
1239 mysql 15 0 76584 32M 2740 S 0.0 3.3 0:00 0 mysqld
1240 mysql 15 0 76584 32M 2740 S 0.0 3.3 0:00 1 mysqld
1241 mysql 15 0 76584 32M 2740 S 0.0 3.3 0:01 1 mysqld
1243 mysql 15 0 76584 32M 2740 S 0.0 3.3 0:00 1 mysqld
1247 mysql 15 0 76584 32M 2740 S 0.0 3.3 0:02 1 mysqld
1253 mysql 15 0 76584 32M 2740 S 0.0 3.3 0:12 0 mysqld
1254 mysql 15 0 76584 32M 2740 S 0.0 3.3 0:08 0 mysqld
1255 mysql 15 0 76584 32M 2740 S 0.0 3.3 0:15 1 mysqld
1259 mysql 15 0 76584 32M 2740 S 0.0 3.3 0:02 1 mysqld


I have to tried doing mysql optimization, httpd optimization, eaccelerator, convert some tables to innodb (as suggested from smf website) but still have this load problem.

My server is running P4 2.4c with 1gb RAM.
WHM 10.8.0 cPanel 10.9.0-R34
Apache 1.3.x
PHP 4.4.4
mysql 4.1.12
mailscanner

Running only about 10 websites. All other websites are quite idle. Only this main website is the one with lots of visitors. Around 5,000 unique visitors per day.

Any other suggestions? Should I upgrade the RAM?

 

[Infopro]

Custom theme or other hacks by chance?

 

[abubin]

Custom theme or other hacks by chance?

not really our own customization but downloaded third party theme from smf site called dilbert MC. No other mods.

Roughly around 100,000 members in the db.

Only mod is bridge script for connecting between joomla and smf.

 

[abubin]

one thing I noticed is that when the load starts getting very high, as mentioned, there will be some httpd process started using lots of memory as well. In one case, I see three httpd at around 80mb in size. Then to avoid system crashing, I changed the smf forum to maintenance mode. Once I set to maintenance mode, load will start to go down. It will go down back to less than 1.0 as usual.

But I still see that few httpd process which is around 80mb still hovering in top. Only when I restart apache that I see these httpd process gone.

Any idea?

 

[Infopro]

Easy test. Switch to the default theme and do not allow that custom theme to be used by anyone for a few hours. Worth a try.

 

[abubin]

Here is what I tried, I created another SMF board with plain installation. No mods and default themes. Copied the database into this standalone board.

I can access this board no problem. Once I start accessing the board, I do see this big httpd file with 80mb memory usage hovering around. When I stop accessing the board, I still see this big file hanging around. It doesn't release those memory. Normal? Strange?

 

[Infopro]

I hope I'm not leading you down the wrong road here but this sounds to me like there is a custom hack of some type, gone wrong. But I did google a bit for you and found this thread. Not sure if that's your problem but have a peek.

 

[abubin]

thanks for trying to help. I checked using IE and problem still exist. So, most probably not related to that problem.

 

[madmac]

When I stop accessing the board, I still see this big file hanging around. It doesn't release those memory. Normal? Strange?

No it is normal. Apache processes do not release memory until killed, so especially with PHP they can easily stack up some hefty memory usage.

In httpd.conf you may want to lower the number of requests a child server will serve before dying. It is the MaxRequestsPerChild entry. 50 is usually cited as a good number in cases like this.

 

[abubin]

anyone can help? this problem still exist despite upgrading RAM to 2GB.

That dreaded 90mb httpd process still come back whenever I try to access the forum from joomla.

 

[brianoz]

How are you running your PHP? 90mb might be normal if you're running PHP inside your apache (httpd). It's probably worth asking on the SMF forums rather than here though. SMF forums are under www.simplemachines.org (here).

 

[gorilla]

The bridemodule for joomla and SMF might be insecure

Full Name: SMF Bridge
Short Name: com_smf
Version: <= 1.1.4
Fix: For SMF version 1.1RC2 only. Upgrade available. See this thread.
References: http://secunia.com/advisories/21079/
http://www.simplemachines.org/community/index.php?topic=100140.0
http://forum.joomla.org/index.php/topic,78313.0.html
http://forum.joomla.org/index.php/topic,77716.0.html
http://forum.joomla.org/index.php/topic,78359.0.html
http://forum.joomla.org/index.php/topic,76609.0.html

Full Name: SimpleBoard
Short Name: com_simpleboard
Version: All Versions.
Fix: Upgrade to JoomlaBoard 1.1.2. JoomlaBoard is compatible with SimpleBoard. Download it here.
References: http://secunia.com/advisories/20981/
http://secunia.com/advisories/20409/
http://forum.joomla.org/index.php/topic,75668.0.html

and here are additional links for your Joomla security upgrades
http://forum.joomla.org/index.php/topic,79477.0.html

 

[celliott]

Have you optimized your server configuration?

I would reccomend modifying your Apache Configuration, My.cnf and installing eAccellerator as a Zend extension. Eaccellerator will cache Php requests and therefore lower server load and improve performance at the same time.

How to Install eAccellerator:

First of all run /scripts/installzendopt to ensure you have the latest version of Zend Optimizer Installed.

PHP eAccelerator can be found at the following URL

http://eaccelerator.sourceforge.net/
or
http://sourceforge.net/projects/eaccelerator/

Installation if fairly straight forward..

Download the file to your server from http://prdownloads.sourceforge.net/e...ar.gz?download

i.e ...
# wget http://switch.dl.sourceforge.net/sou...-0.9.2a.tar.gz

extract the archive
# tar zxvf eaccelerator-0.9.2a.tar.gz

change to the source dir
# cd eaccelerator

export the following system var
# export PHP_PREFIX="/usr/local"
(this is the corect location on cpanel servers, some servers it may be in "/usr")

then phpize...
# $PHP_PREFIX/bin/phpize

configure the program...
# ./configure --enable-eaccelerator=shared --with-php-config=$PHP_PREFIX/bin/php-config

compile the program..
# make

install the program...
# make install

create a temp directory for cached scripts and set the perms..
# mkdir /tmp/eaccelerator
# chmod 0777 /tmp/eaccelerator

edit your php.ini file
# pico -w /usr/local/lib/php.ini
may be located in "/usr/lib/php.ini" on some machines

add the following to the end of your php.ini to install it as a Zend extension:

zend_extension="/usr/local/lib/php/extensions/no-debug-non-zts-20020429/eaccelerator.so"
eaccelerator.shm_size="16"
eaccelerator.cache_dir="/tmp/eaccelerator"
eaccelerator.enable="1"
eaccelerator.optimizer="1"
eaccelerator.check_mtime="1"
eaccelerator.debug="0"
eaccelerator.filter=""
eaccelerator.shm_max="0"
eaccelerator.shm_ttl="0"
eaccelerator.shm_prune_period="0"
eaccelerator.shm_only="0"
eaccelerator.compress="1"
eaccelerator.compress_level="7"

You may want to change the cache directory to something other than /tmp/eaccelerator as it may fill up quickly if you have a small /tmp. Its a good idea to create a cron which will empty the folder every so often.

Chris.

 

[abubin]

like I said in my first post, I already installed eaccelerator and still having this problem.

As for the security of the joomla-smf bridge, I am not using this one. There is another one written by wolverine from joomlahacks.

Anyway, my problem is not security as the problem occurred even before we really launch the new forum. We also did a fresh install using same database on another domain which yielded same problem.

Thanks for all the advices and keep them coming...maybe we missed something.

 

[katmai]

installed eaccelerator as per tutorial, but it gave me error:

client denied by server configuration. any clue ?