Server crash due to high RAM usage

Hello everyone,
on peak hours we are experiencing a severe high RAM usage that bring our server to a total hang, such that we are forced to reboot it.

Server environment
Server: Virtual machine based on OnApp / Xen technology
RAM: 6GB
OS: Cloudlinux 5.8 x86_64
Panel: Cpanel /WHM 11.32.3 (build 21)
Total accounts: ~125 accounts.
Services: websites (all sites are made by us for our clients) + mail services, etc.

What we tried to do until now (also with the help of an expert server admin)

• We added 2GB of RAM (from 4GB to 6GB)
• Installed the Cloudlinux environment (at the moment we kept the memory limit disabled as default)
• Installed the mysql db governor tool provided by Cloudlinux
• Installed nginx (but then we reverted to Apache, since it didn't help).
• Switched from SuPHP to FastCGI
• Tuned many MySQL settings in my.cnf in order to improve performance. We also used the mysqltuner to analyze and config MySQL server.
• On the most busy websites we installed cache plugins in order to reduce the time spent on PHP execution.
• We forced websites not to use mysql persistent connections.
• We distribute cron jobs over time so that they overlap as few as possible.
• In the end we suspended the most single heavy website to reduce load, and it looks this is the only way we found to prevent the server from crashing again.

I expected that Cloudlinux (even with default settings: 25% CPU, 20 Max processes and memory limits disabled) would have been able to prevent the server from crashing due to a single "heavy" account.

Any ideas?

 

Yes, ~40 websites are based on Wordpress. On the busiest ones we installed a caching plugin. Other ~50 sites are PHP+MySQL based on a framework, there are a few phpBB and Joomla instances, and the remaining are static sites.

 

Thanks for your answer. I did not have enough time to make accurate checks, but it looks like the system went out of memory mainly because of MySQL server overload, so I am not sure that Cloudlinux memory limits would have take effect.
I did not enable memory limits initially, because I've read that "Memory limits are disabled by default, as memory limit bugs are somewhat confusing for new CloudLinux users" (I've also read this article on Cloudlinux website). Anyway after that, I tried to enable memory limits and I made also some other tweaks at different levels (also installed other Cloudlinux tools like mysql governor, memmonitor, ...).
The server didn't crash any more, but I think that's mainly because we moved away the busiest website that was on it. As for the memory limits I am keeping them enabled now, but they are giving problem since the default settings I've set (500MB) is often reached and so sites are not fully working because they're blocked. Now I put it again to 1GB (which is the default value) but I fear that it's not very useful to have a 1GB limit on a machine with 4 or 6 GB and 125 accounts.