migrating from phpsuexec to module

[dchepishev]

Hi guys,

My server is really loaded when running phpsuexec so we will try to switch to mod_php.
For the guys who are wondering what the server is: Xeon 3G, 2G Memory. And it has only 150 accounts on it. However there are 3 accounts which are killing the machine. In the loaded hours the load average goes ot 5-6 and sometimes 8-9. This is not acceptable for me because we intend to have 500 accounts on this machine. I cant even imagine what the load will be with 500 accounts.

Ok I started talking too much . The real question is: Can someone tell me what problems could occur during the switch and what should I be careful about in order to have painless switching for me and the clients?

 

[chirpy]

The only thing likely to be of issue is file permissions/ownerships as without phpsuexec scripts will run as the nobody:nobody user which needs at least read access to all files associated with the php scripts. /scripts/chownpublichtmls can go some way in helping with that, otherwise it's probably a matter of fixing things if they break by making them less secure until they do work.

 

[DigitalN]

dchepishev]However there are 3 accounts which are killing the machine

And you want to get rid of phpsuexec (and its benefits) so that 3 accounts can still kill your server after you have gone through the migration to mod_php ... would it not be better to have the 3 accounts, either fix their abusive scripts or let them find another host or dedicated server? I don't know all the ins and outs of your situation, but I have been in it a few thousand times.

 

[chirpy]

True. Although phpsuexec does at load to the server, it doesn't add that much load to it. Tweaking php/mysql/apache may help in the short-term, but ultimately, if the server is already loaded you may not get much more out of it to add that much more load with new accounts.

 

[DigitalN]

True. Although phpsuexec does at load to the server, it doesn't add that much load to it. Tweaking php/mysql/apache may help in the short-term, but ultimately, if the server is already loaded you may not get much more out of it to add that much more load with new accounts.

Indeed, I migrated around 20 servers to phpsuexec, some time ago now, and I didn't see anything significant in load increase with phpsuexec.. some of those servers had 1500+ domains on them (I won't mention the company involved

In this case, it doesn't sound as though the poster is looking in the right direction, moving to mod_php might not improve anything, cause a lot of support for a while AND make it harder to find the problem scripts .. just my take

 

[dchepishev]

Indeed, I migrated around 20 servers to phpsuexec, some time ago now, and I didn't see anything significant in load increase with phpsuexec.. some of those servers had 1500+ domains on them (I won't mention the company involved

In this case, it doesn't sound as though the poster is looking in the right direction, moving to mod_php might not improve anything, cause a lot of support for a while AND make it harder to find the problem scripts .. just my take

Guys. The problems are not the scripts itself. They are not the problem. The problem is the number of hits they get. The stupid CGI architecture simply can not handle high loads. I will give you a simple example
I have made some tests using my laptop as server (p3 1G with 384MB memory). The results are
when using phpsuexec i get at most 30 hits/sec
when i use mod_php i get 1500 hits/sec. Huge difference.

the php filre requested is just a simple echo "something"

Huge part of my CPU time goes as system time which is probabably the overhead coming from creating processes My load average for 24 hour period is increasing and increasing when we add new and new accounts.

One of the accounts which is generating a load is just a simple forum running IPB 2.0. Nothing unusual. But it has lots of accesses some times and this kills the machine. However the traffic generated is not that much ~ 10G/month

I am running CentOS 4.2 with custom kernel 2.6.14.6-grsec. In the loaded hours the system time is about 40-50%. I think this is way too much. I dont want to migrate to module too, but no choice so far.

If someone has and Idea how I can decrease the load I would be really happy. I have seen hosting server with 700 account which load during high loads is around 2-3 I really dont understand how they do this.

 

[chirpy]

2.6.14.6-grsec

IMX, grsec protected kernels produce far more problems, greater server overheads and instability than using the CentOSv4 kernels. I would suggest booting off the latest CentOS kernel and see if that improves things. I find that over 90% of the time it does.

Few other things:

1. Have you tuned MySQL and enabled query caching as a minimum?

2. Do you have a fast disk setup? If you're using IDE disks, have they had DMA enabled? Most of your overhead is likely to be I/O related - check top when the load is high to try and identify the bottlneck.

3. Check your swapfile usage at high load to make sure you're not memory thrashing.

4. Set a lowish value in httpd.conf (e.g. 250) for MaxRequestsPerChild incase of memory leaks

5. Install Zend Optimizer

 

[DigitalN]

The number of domains, while not insignificant, isn't the only factor.
Just 1 script can kill a server.

Is your server being attacked or something? How many connections is it running and what are they? Often forums etc are subject to automated scripts, bots and the like, trying to exploit them, look at the requests that the forums etc are getting - do you use mod_security?

You really need to do some diagnostics and determine how much memory and cpu time are the cgi processes eating up? Is it apache or mysql where you are seeing high load?

What do you have set in apache for

MaxRequestsPerChild

Try setting that to say, 100 instead of 0 (If it is)
There are many tweaks that you can do and should do to apache, to minimse load, you really need some figures as benchmarks to start with.

I just can't see that the install of php (phpsuexec) is the cause for such extreme load.

 

[dchepishev]

IMX, grsec protected kernels produce far more problems, greater server overheads and instability than using the CentOSv4 kernels. I would suggest booting off the latest CentOS kernel and see if that improves things. I find that over 90% of the time it does.

Few other things:

1. Have you tuned MySQL and enabled query caching as a minimum?

2. Do you have a fast disk setup? If you're using IDE disks, have they had DMA enabled? Most of your overhead is likely to be I/O related - check top when the load is high to try and identify the bottlneck.

3. Check your swapfile usage at high load to make sure you're not memory thrashing.

4. Set a lowish value in httpd.conf (e.g. 250) for MaxRequestsPerChild incase of memory leaks

5. Install Zend Optimizer

Regarding GRSEC stuff:
I have not booted grsec disabled kernel on this machine but i tried in the office what is the difference in speed between grsec and non grsec kernel. I found no difference. The test was 10000 times to start /usr/bin/php p.php > /dev/null
p.php is just echo "something"
And measure the time taken to do this. No difference.
I will really feel bad if I run with no grsec kernel

1. MySQL. I havent done any optimization but mysql is not consuming any cpu. For uptime 12 days all mysql intances have used ~ 10 min. time. I dont think this is the problem. When I say to top to sort by CPU usage, Mysql does not even shows
2. Disks: I use sata disks, The IO/WAIT time is 0% almost all the time. This could not be the issue
3. SWAP file usage: 0%
4. MaxRequestsPerChild: just set it to 100. Will wait for the results
5. Zend optimizer slows the things down if I am not wrong. This is just another one .so to load and according to my simple echo "something" tests, when i load zend the hits per second goes down. Probably this is wrong measurement but I could give it a try on the server if you say I should .

 

[DigitalN]

I would be interested to see a post of the full ps output when the server load was up.

Can you post that and maybe the memory infornation too,

# ps auxf

# free -m

# top

# netstat -a -n -p

# mysqladmin processlist

are a few things to start with, to see where exactly the load is coming from.

I would agree with Chirpy about grsec too, depending upon your options set in grsec, you can actually cause more issues than it solves. I found the openwall patches much nicer, but then you run 2.6.

 

[dchepishev]

The number of domains, while not insignificant, isn't the only factor.
Just 1 script can kill a server.

Is your server being attacked or something? How many connections is it running and what are they? Often forums etc are subject to automated scripts, bots and the like, trying to exploit them, look at the requests that the forums etc are getting - do you use mod_security?

You really need to do some diagnostics and determine how much memory and cpu time are the cgi processes eating up? Is it apache or mysql where you are seeing high load?

What do you have set in apache for

MaxRequestsPerChild

Try setting that to say, 100 instead of 0 (If it is)
There are many tweaks that you can do and should do to apache, to minimse load, you really need some figures as benchmarks to start with.

I just can't see that the install of php (phpsuexec) is the cause for such extreme load.

High load is coming from creating php instances. No specific process is on the top. The are changing really fast and no specific process on the top of cpu usage.
MaxRequestsPerChild - just fixed it.

I think the load is comming from the number of hits i get. They are not coming from spceific ip but from different ones and seems like legal requests. In the time i am writing this i have 130 connections to the server (including time_wait) ones. And load average is 1.82 which is not loaded time.
Every morning until 10 AM everything is fine. And load is 0.6 - 0.8 and in the aftenoon start the load

 

[DigitalN]

Oh and you need to upgrade your kernel, it is subject to a dos vulnerability

 

[dchepishev]

I would be interested to see a post of the full ps output when the server load was up.

Can you post that and maybe the memory infornation too,

# ps auxf

# free -m

# top

# netstat -a -n -p

# mysqladmin processlist

are a few things to start with, to see where exactly the load is coming from.

I would agree with Chirpy about grsec too, depending upon your options set in grsec, you can actually cause more issues than it solves. I found the openwall patches much nicer, but then you run 2.6.

you will get the info. I will post it once the load starts which will be in around 3 hours from now

 

[dchepishev]

Oh and you need to upgrade your kernel, it is subject to a dos vulnerability

As soon as grsec releases patch for later kernel

 

[DigitalN]

As soon as grsec releases patch for later kernel

There's another reason to not use grsec

Maybe you want to block icmp until they do release one, to be safe.

I'll be in bed in 3 hours, but I look forward to seeing the output and helping tosolve the case.

 

[dchepishev]

I just found what the problem is

Hi guys. I just found why the server was so fuckin slow. It is good to share this with other people which face the same problem.
The problem was the mm library compiled in php. After 3 weeks of testing and recompiling php i noticed that cpanel installation is much slower than mine installs. After another day of recompiling php i managed to find that when i compile mm library in php the speed decreases by 8 times yes you read correct it is EIGHT times slower when you compile mm. When I removed it from my server the load dropped from ~3 to 0.5-1 which is realy great improvement . Thanks for the help.

 

[chirpy]

That's interesting indeed, well done for finding it

This is not aimed at you at all - but it does illustrate to those that simply compile in all modules in php just because they're there on offer, not because they know that that must have them. Which is something that I've seen people do.

 

[DigitalN]

Hi guys. I just found why the server was so fuckin slow. It is good to share this with other people which face the same problem.
The problem was the mm library compiled in php. After 3 weeks of testing and recompiling php i noticed that cpanel installation is much slower than mine installs. After another day of recompiling php i managed to find that when i compile mm library in php the speed decreases by 8 times yes you read correct it is EIGHT times slower when you compile mm. When I removed it from my server the load dropped from ~3 to 0.5-1 which is realy great improvement . Thanks for the help.

I have found that too in the past as well - Top marks to you for turning it up, as something like that can be a bind to find.

phpsuexec is still all good then ?

 

[dchepishev]

I have found that too in the past as well .

You could have told me this yesterday, couldnt you ?

phpsuexec is still all good then ?

With the performance i have now, phpsuexec is just great. I didnt even imagine that something that stupid which actually is used for speeding up php can kill the machine. On machine P3 733 Mhz it takes 200 ms to start php and exit and the script just do echo "something". This is insane time man . I told this to cpanel support and I hope they will be able to help to other users with simalr problem, because they advised me to migrate to module and I really didnt wanted to

 

[heavypredator]

well phpsuexec is slwing down server

i recently moved from phpsuexec to mod and with phpsuexec it had almost constant load ~2, with mod it has below 1

dual xeon 2.4 , 2gb ram, scsi drives