Virtual server advice

[maever]

We recently switched from a dedicated machine to a virtual one.
Now our Dedicated server had 1 Intel Xeon 3.2ghz CPU with 4 GB ram and normal 7200RPM WD harddisks.
The average load was around 0.8~2.

Now since yesterday we have moved our business to a Virtual Server environment (VMWare ESX) for better redundancy.
We have freshly installed a new Centos Enviroment on this VPS server.
We have 2GB ram, 80GB space (enough) and 1 virtual CPU.

The load now averages between 3 and 30.
It has become quite unstable and the websites on it respond too slow (3s for a pageload).
Now I'm wondering what causes this odd behavior.


Here is a copy of the "top" command

top - 13:07:05 up 9 days, 23:25, 3 users, load average: 8.91, 5.95, 8.38
Tasks: 167 total, 27 running, 137 sleeping, 1 stopped, 2 zombie
Cpu(s): 86.5%us, 12.5%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 1.0%si, 0.0%st
Mem: 2059620k total, 1933192k used, 126428k free, 74532k buffers
Swap: 4095992k total, 145344k used, 3950648k free, 900708k cached

There are no processes which consistantly hog the CPU, rather just normal HTTPD threads like you'd see in normal operation. I also don't get the idea that RAM shortage is an issue here as there is little to no swap.
We are also not being limited in CPU usage by our VPS provider. (the virtual CPU should automatically adapt to how much we use and according to the hosting company the current server we run on has 70% idle cpu at almost any given time.)

Now one of the discussions I'm currently having with our hosting provider is IF enabling multiple virtual CPU's would increase performance and fix our load issues?and why)

so my questions are:

1. Are there any special steps one would need to take on a WMware hosted CentOS virtual environment which 'once neglected' could cause such issues as I describe?

2. Would enabling more virtual CPU's offer a solution (and more importantly .. why ?)

3. Are there any other things I should consider when running in a virtual environment?

 

[madaboutlinux]

Looking at the output of 'top' command, your VPS is swapping, though less but still it is swapping. The CPU usage is almost 100% and 0.0% idle which will increase load average and in turn will make the websites to repond slowly.

The performance may look good if virtual CPUs are increased but after all a VPS is a VPS and your VPS performance will also depend on the neighborhoods you are have on the host machine.

You have cut down the resources (CPU + RAM) to half by switching from Dedicated to VPS + resource usage of other VPS'es on the host server, both will make a difference.

 

[Spiral]

Now since yesterday we have moved our business to a Virtual Server environment (VMWare ESX) for better redundancy.

Well your reasoning for moving is a bit of an oxymoron since it would seem you are trying to do exactly the opposite of your intentions ...

We have freshly installed a new Centos Enviroment on this VPS server.
We have 2GB ram, 80GB space (enough) and 1 virtual CPU.

The load now averages between 3 and 30.
It has become quite unstable and the websites on it respond too slow (3s for a pageload).
Now I'm wondering what causes this odd behavior.

Do you own the entire physical server and setup a VPS on it or did you go puchase a "VPS" server from some host someplace?

VPS servers are exactly the same as regular shared hosting account resource usage wise and when you purchase a VPS server, you are going to be charing cpu clock cycles and memory resources with all the other VPS servers setup on the same physical machine.

Unusually high loads and slow performance is the trademark of traditional VPS servers.

Also since resource usage settings can be overlapped using the various bursting limit options and even "oversold" (setup beyond physical limits) this can and often does impact server performance especially if there is a higher number of servers running from the same physical machine.

We are also not being limited in CPU usage by our VPS provider. (the virtual CPU should automatically adapt to how much we use and according to the hosting company the current server we run on has 70% idle cpu at almost any given time.)

Well that statement alone just told me a bit more on what's going on ----

I'd be willing to bet your VPS provider has got the settings all wrong and minimizing isolation of resources and those fitting that profile have a tendency to overcrowd the servers which is why I mentioned it earlier.
There may be budget constraints else a better platform would have been selected already than vmware and used as production VPS deployment also makes me suspect a few other things I won't necessarily mention.

Now one of the discussions I'm currently having with our hosting provider is IF enabling multiple virtual CPU's would increase performance and fix our load issues?and why)

Nope --- I suspect your issues are at a much lower and basic level.

Your issue here is going to do partly with the VPS settings of the VPS server itself (not your CentOS installation on top of it) along with the physical hardware and the current resource usage and loads from the other VPS servers neighboring along side yours on this machine.

1. Are there any special steps one would need to take on a WMware hosted CentOS virtual environment which 'once neglected' could cause such issues as I describe?

You just said the "V" world (though spelled it "W" hehe :D ) ...

Not the best or most ideal of choices for commercial VPS hosting ...

Is this what your VPS provider is using? This also says a lot too!

2. Would enabling more virtual CPU's offer a solution (and more importantly .. why ?)

I very seriously doubt it ....

3. Are there any other things I should consider when running in a virtual environment?

Actually quite a few things --- most are security and performance related.

Moving away a bit from that technical side of things for a moment though, Cpanel also has options to help try to compensate for the inherent performance issues with VPS servers with a "VPS" version of Cpanel, SQLite for modules, and functions to reduce memory consumption in "Tweak Settings" though in your case I have strong suspicions any of that would do anything to help your situation.

Given what you have said and reading "between the lines", I would wager there is little you could do to improve things on your end and you would be best suited finding a better solution and probably a better host just based on what I profile from various comments you made in your posts.

There are some traditional VPS providers that keep well tuned and lower VPS ratios (I have doubts that your host is really one of those) but even given that, I really don't advise or recommend traditional VPS servers for ironically much of the same reasons which you are now complaining regarding loading and resource issues.

If not dedicated, then the one other hosting technology I would take a hard look at is cloud server technology which is ironically and somewhat confusingly is sometimes marketed as "VPS" since it can address the same market in the same price ranges but is an entirely different creature all together.

Hope that helps

 

[maever]

Thank you both for your useful comments,
This does help get a better view on things.

I myself am familiar with Linux Environments on dedicated hardware solutions.
Though have worked a bit with VPS solutions i'm not familiar with the details.
What I do know is that a lot of companies do oversell their servers, I however do not think this is the case with the mentioned company.

The reason for the switch was mainly a decision from higher up.
I do know this new hosting company offers professional business solutions and their servers are equiped to handle high CPU usage (VMWare ESX cluster servers that are each 2x Quad Core XEON with around 32/48 GB of ram and a dedicated hardware RAID controllers).

One main advantage is increased redundancy, the setup the hosting company has is claimed to be far more redundant then our single-server setup. Since we have only a few servers we want to avoid having to go to the datacenter ourselves to fix server hardware issues.

I do not believe the company is overselling, they however have limited linux knowledge and the problem could perhaps lie in this fact. Do linux environments require special configuration on a VMWare setup ?

Our first intention is getting this issues resolved with the current hosting company before looking at alternatives.

I am however interested in knowing what (if they exist) redundant dedicated Hardware solutions there are out there that don't cost an arm and five legs.

 

[Spiral]

Though have worked a bit with VPS solutions i'm not familiar with the details.

I typically don't recommend traditional VPS servers at all ...

There is just far too many performance, security, and reliability issues even among the "best" of them (which VMware is not) and is not a solution I would personally consider for any real production use where these any of these things were actually mission critical.

What I do know is that a lot of companies do oversell their servers, I however do not think this is the case with the mentioned company.

Overselling servers is certainly one major aspect but it is actually not at all what I was referring to. Rather, the physical technical limitations of VPS servers just the nature of being what they are and the various underlying complications that go along with all of that.

The reason for the switch was mainly a decision from higher up.

One main advantage is increased redundancy, the setup the hosting company has is claimed to be far more redundant then our single-server setup. Since we have only a few servers we want to avoid having to go to the datacenter ourselves to fix server hardware issues.

Well from what I've heard thus far, I get the very strong impression that someone up your food chain was a little bit misguided in their thinking as I really do think you managed to do precisely the exact opposite of everything you just said was all your reasons and intentions and missed your target objective by a fair distance.

I do not believe the company is overselling, they however have limited linux knowledge and the problem could perhaps lie in this fact. Do linux environments require special configuration on a VMWare setup ?

Yes but more than that think about what you just said ---

"They have limited Linux knowledge" yet setting up, configuring, and managing a VPS server requires a very substantial amount of experience and knowledge in the operating system and underlying hardware and experience understanding what impact each configuration setting has on all of the above. If they can't manage the operating system, how can you expect them to manage VPS layers which is infinitely more complicated than a basic server without that technology.

Our first intention is getting this issues resolved with the current hosting company before looking at alternatives.

I would certainly be glad to give you a hand to try to get the most out your current servers and I can probably push that threshold far more than most but I can tell you right now you are probably pushing up against a brick wall of physical technical limits.

I am however interested in knowing what (if they exist) redundant dedicated Hardware solutions there are out there that don't cost an arm and five legs.

How about hard drives that can't fail and multiple backup redundancies stored all over the world that you can bring to live status in a heartbeat?

(Ironically --- probably cost you less than what you are trying to do now)

Anyway the point being made here is "yes" there are much better alternatives out there and "no" they do not cost "an arm and five legs"

 

[maever]

A small update concerning the primary issues:

We have increased the number of virtual CPU's (total of 2) and added 1 GB ram.
It has been running a bit smoother but the load still tends to jump around. (though now not going above 8 at peak times)

typically don't recommend traditional VPS servers at all ...

There is just far too many performance, security, and reliability issues even among the "best" of them (which VMware is not) and is not a solution I would personally consider for any real production use where these any of these things were actually mission critical.

I did ofcourse investigate beforehand came across various sources but none made the clear statement that it was strongly discouraged as a webhosting solution.

We have established a business relationship with this company (they actually reside 2 blocks over from here) and there is room for almost any improvement so if a dedicated server is truly the better option it is what we will end up getting.

Well from what I've read and thus far, I get the very strong impression that someone up your food chain was a little bit misguided in their thinking as I really do think you managed to do precisely the exact opposite of everything you just said was all your reasons and intentions and missed your target objective by a fair distance.

I disagree, the previous environment on hardware failure would require us to go to the datacenter ourself and in case of motherboard failure tamper with the hardware risking hours of downtime (we owned the machine ourself).
Recovering cpanel backups (or rebuilding a RAID array) takes hours and having a machine on standby with all latest data is not an option either, nor is DNS-based solutions as almost all internet providers in The Netherlands ignore set TTLs and will still cache results for DAYS.

The current setup covers hardware and network failure where another VMWare ESX server would take over in case of 1 failing. So this would be an improvement, that is except for the load issues we are having.

"They have limited Linux knowledge" yet setting up, configuring, and managing a VPS server requires a very substantial amount of experience and knowledge in the operating system and underlying hardware and experience understanding what impact each configuration setting has on all of the above. If they can't manage the operating system, how can you expect them to manage VPS layers which is infinitely more complicated than a basic server without that technology.

They specialize in windows solutions, my personal impression has always been that installing a linux VPS only required a few OS optimizations specific to the Virtualization software the VPS runs on (Cpanel support claims something along the lines of this as well). As for management of the system I myself manage the OS environment, they manage the Virtual layers.

I would certainly be glad to give you a hand to try to get the most out your current servers and I can probably push that threshold far more than most but I can tell you right now you are probably pushing up against a brick wall of physical technical limits.

Thank you for the offer, though I cannot give you access to a production environment server, I would like to know what kind of optimizations you would consider specific to the VPS environment would these optimizations be more general OS-specifc or more VPS-specific ? (maybe an example?)

How about hard drives that can't fail and multiple backup redundancies stored all over the world that you can bring to live status in a heartbeat?

(Ironically --- probably cost you less than what you are trying to do now)

Anyway the point being made here is "yes" there are much better alternatives out there and "no" they do not cost "an arm and five legs"

Always interesting to hear about various hosting solutions!
We have experimented with various methods but all seemed to have one or more drawbacks.
for example:

• DNS-based solutions are not an option since DNS-caching done by providers completely kills it.
• Server Rsyncing leads to strange and unwanted issues with different software versions (and sometimes fails to copy apply permissions).

Could you give a more concrete example of how the described setup would work?

Thanks again for the reply,
appreciate it!

 

[maever]

Small update,
We installed 'VMWare tools' on the OS and enabled multiple virtual CPU's.
This does seem to take care of most of the issues!

It seems that VMWare has come along way in the past years and if your provider takes care with distributing the virtual servers then you should be okay with the virtual setup. (though a lot of providers WILL infact oversell their servers, but in terms of scalability and redundancy, virtual servers ARE the better option that is, IF it is configured correctly)

It would seem that linux needs to manage its own Threaded core balancing instead of VMWare (so you will need to enable 2~4 virtual cpus)