Optimize High-Traffic Servers

[anamip]

Hello,

I'm an international member and right now, I need to manage two high traffic servers.

With my poor english skill, I cant fully understand what does

Scoreboard Key:

"_" Waiting for Connection
"S" Starting up
"R" Reading Request
"W" Sending Reply
"K" Keepalive (read)
"." Open slot with no current process

And what does httpd.conf effect to this

Here is some of my httpd.conf lines

ServerType standalone
KeepAlive On
MaxKeepAliveRequests 100
KeepAliveTimeout 10
MinSpareServers 5
MaxSpareServers 15
StartServers 200
MaxClients 2500


The server status





My server hardware config

Single Processor Quad Core Xeon 5420 - 2.55GHz (Harpertown) - 1 x 12MB cache

With 4GB OF RAM, CentOS 4.6-32

I saw the server alway red but the Memory Used very low ( Just 25% of ram like the image above showed )

My site has over 100.000 visitor per day with nearly 2000.000 ( 2mil ) requests. All has dynamic pages and some static files.


Could you please give me some advices?

Thank you very much for your time,

Anamip

[WireNine]

Are you noticing any performance degrading with the red status? Load of 2.x on a powerful server should not affect anything negatively. You can change the status turning red at the load of 2 from tweak settings to something like 5 or 10.

[GoWilkes]

My site is high traffic, too. I don't usually see a performance problem until the server load gets to around 15 or 20, although I've seen it as high as 100 before. Visitors usually start to complain when it's around 30.

When it gets over 50, I usually replace the most popular program with a "please wait" page for 2 or 3 minutes until it cools off.

To address your original question, though, I don't think your poor English skill is the problem; English is my native language, and I don't understand the text any better than you do

[raysolomon]

Just a few words of advice.

You say that you are using dynamic pages.

If those dynamic pages are communicating with a database, then part of your problem appear to originate from that.
Also, the programming code that you are using most likely need to be optimized too.

I have a server that appears to be doing more that your server is doing and I am able to keep the entire load of the server down to about 5-6% usage at peak times. Your server will be able to handle more than you think if optimized well.

Reason is, when people learn to write code, the code will work well and fast and appear efficient with little traffic, but when you start getting millions of unique users a day, that is when you have to pay special attention to optimize the programming code and database queries. For example, having a query that takes 0.5 seconds to pull a result versus 0.0001/sec will make a huge difference.

This is also the point where you first start to become a better programmer and server admin, no matter how many years you have been programming. Because you will end up having to troubleshoot performance issues more often than you will be writing code sometimes until you learn it well. It takes time and lots of patience to learn these things and it will make your boss happier, because then you will be more valuable.

Unfortunately, this kind of programming skill is not taught well enough online or in schools, but instead it will come with experience overtime when you have a chance to really get involved with a high traffic server. This is really the point where it will make you a good programmer and server administrator. We all learn the hard way. But it is fun too. :-)
It would pay to join forums and ask questions to specific problems where there are more experienced programmers. A great place I recommend is experts-exchange.com.
Also, mysqlperformanceblog.com as reference info.

You may end up rewriting your programming code and your database queries and database structure to reach much better performance. I'm positive you can attain higher performance this way. I had to learn it myself.

High traffic sites needs better hardware too. Some advice.
I will suggest you get 2+ Xeon's instead of 1.
Upgrade the mysql database to 5.0.
Use the 64-bit CentOS 5 instead of the 32-bit.
This will dramatically help too, but first, optimize code and database queries.

[brianoz]

Some great advice from a number of people here. You can't go past application tuning in a high performance scenario - that is always where the biggest improvements are to be found. If you can get an expert to watch the database when the system is under heavy load they should be able to come up with some improvements that could give you a factor of 10 to 1000 times improvement. Seriously, the impact can be that great.

There are other wins - make sure you are running your PHP or Perl (whatever the site is written in) as a module and not a CGI. This will give you a huge improvement in itself.

If possible, convert the site home page, and maybe other common pages, to static by caching the dynamic pages and regenerating the cache regularly. This isn't difficult in PHP and if those pages are hit often and do a few queries, will make a huge difference.

It's probably a good idea to make sure you run a good solid firewall like CSF. This will kill off minor performance hits (such as accidental denial-of-service hits, and some bots) without you even noticing.

Finally, if you are new at system admin, I'd recommend hiring someone knowledgeable and trustworthy to help you get started. A few hours from a top admin will save you many months of research. Don't kid yourself - system admin is a rich and complex field that takes years to gain real solid competency in. You'll learn a lot from using one of these guys. Don't expect them to know everything though, you may need input from several people to get really solid gains. www.configserver.com is one place to start although there are many good people around if you do the research.

Finally, do lots of backups, both on-server and off-server. Both are important, and you won't need them until you need them suddenly!