Server crash due to high RAM usage

[marcochiesi]

Hello everyone,
on peak hours we are experiencing a severe high RAM usage that bring our server to a total hang, such that we are forced to reboot it.

Server environment
Server: Virtual machine based on OnApp / Xen technology
RAM: 6GB
OS: Cloudlinux 5.8 x86_64
Panel: Cpanel /WHM 11.32.3 (build 21)
Total accounts: ~125 accounts.
Services: websites (all sites are made by us for our clients) + mail services, etc.

What we tried to do until now (also with the help of an expert server admin)

• We added 2GB of RAM (from 4GB to 6GB)
• Installed the Cloudlinux environment (at the moment we kept the memory limit disabled as default)
• Installed the mysql db governor tool provided by Cloudlinux
• Installed nginx (but then we reverted to Apache, since it didn't help).
• Switched from SuPHP to FastCGI
• Tuned many MySQL settings in my.cnf in order to improve performance. We also used the mysqltuner to analyze and config MySQL server.
• On the most busy websites we installed cache plugins in order to reduce the time spent on PHP execution.
• We forced websites not to use mysql persistent connections.
• We distribute cron jobs over time so that they overlap as few as possible.
• In the end we suspended the most single heavy website to reduce load, and it looks this is the only way we found to prevent the server from crashing again.

I expected that Cloudlinux (even with default settings: 25% CPU, 20 Max processes and memory limits disabled) would have been able to prevent the server from crashing due to a single "heavy" account.

Any ideas?

 

[Infopro]

Services: websites (all sites are made by us for our clients) + mail services, etc.

What are these sites made with? Wordpress or similar?

 

[cPanelTristan]

If the reason the site was causing it to crash was due to memory, how would it be prevented from crashing with CloudLinux if the issue were caused by memory rather than CPU usage? I'm just asking as I don't know what in that site caused it to crash, but if it were related to memory, having the memory limits disabled would mean it wouldn't be doing anything to limit memory usage.

 

[marcochiesi]

What are these sites made with? Wordpress or similar?

Yes, ~40 websites are based on Wordpress. On the busiest ones we installed a caching plugin. Other ~50 sites are PHP+MySQL based on a framework, there are a few phpBB and Joomla instances, and the remaining are static sites.

 

[marcochiesi]

If the reason the site was causing it to crash was due to memory, how would it be prevented from crashing with CloudLinux if the issue were caused by memory rather than CPU usage? I'm just asking as I don't know what in that site caused it to crash, but if it were related to memory, having the memory limits disabled would mean it wouldn't be doing anything to limit memory usage.

Thanks for your answer. I did not have enough time to make accurate checks, but it looks like the system went out of memory mainly because of MySQL server overload, so I am not sure that Cloudlinux memory limits would have take effect.
I did not enable memory limits initially, because I've read that "Memory limits are disabled by default, as memory limit bugs are somewhat confusing for new CloudLinux users" (I've also read this article on Cloudlinux website). Anyway after that, I tried to enable memory limits and I made also some other tweaks at different levels (also installed other Cloudlinux tools like mysql governor, memmonitor, ...).
The server didn't crash any more, but I think that's mainly because we moved away the busiest website that was on it. As for the memory limits I am keeping them enabled now, but they are giving problem since the default settings I've set (500MB) is often reached and so sites are not fully working because they're blocked. Now I put it again to 1GB (which is the default value) but I fear that it's not very useful to have a 1GB limit on a machine with 4 or 6 GB and 125 accounts.

 

[Infopro]

If it was me, I'd start there to be honest.

WHM > Server Status > Daily Process Log

Top of that page, you see dates to go forward or backward with a click. Clicking back a few days you might get some indication here as to which sites are spiking resources. If you're lucky enough to see something worthwhile here, then you can take a close look at site(s) that may be causing the problems. From there you'd be checking for up to date software, plugins used, etc.

One poorly coded out of date script. mod or plugin being poked for security holes by someone pounding the site with a hacking script could cause problems similar to this in some cases, I would think.

Do you have CSF installed? This might be helpful in directing you to a specific account causing or having issues as well.