High RAM usage

[GoWilkes]

I recently discovered in my Munin report that my RAM usage has been very high since around April 1. I'm not sure of the exact date, but it looks to be the week of the 1st, anyway.

On my end, the only thing that's changed is that I updated PHP to version 5.6.23 on April 9. I also had to modify several scripts to use cURL instead of file_get_contents(), since file_get_contents() is (for some reason) deprecated in 5.6.23.

My server has 4G of RAM, but during peak hours it's trying to use close to 6G, so this is a concern!

I spoke to my server management company, and their only suggestion was that they were seeing a lot of TIME_WAIT connections, with the last IP showing 168 connections:

# The last 5 in the list
xxx.xxx.xxx.xxx is connected 66 in state: TIME_WAIT
xxx.xxx.xxx.xxx is connected 69 in state: TIME_WAIT
xxx.xxx.xxx.xxx is connected 71 in state: TIME_WAIT
xxx.xxx.xxx.xxx is connected 103 in state: TIME_WAIT
xxx.xxx.xxx.xxx is connected 168 in state: TIME_WAIT

The IPs all looked local, though, so it's not an issue of an attack or anything; it's apparently a setting, somewhere. But why it only started to have a negative impact on 4/1/17, I don't know.

After doing some research, I modified /etc/sysctl.conf to:

# Added these
net.ipv4.tcp_tw_recycle = 0
net.ipv4.tcp_tw_reuse = 0
net.ipv4.tcp_orphan_retries = 1
net.ipv4.tcp_fin_timeout = 25
net.ipv4.tcp_max_orphans = 8192

# Changed from 65536
net.netfilter.nf_conntrack_max = 32768

# Changed from 432000
net.netfilter.nf_conntrack_tcp_timeout_established = 86400

Then in WHM's Apache Configuration I lowered Server Limit from 400 to 256 (Munin showed the highest in the last week was about 120), and Max Clients from 300 to 150.

Then in /etc/my.cnf, I changed max_connections from 300 to 100.

Can you guys suggest anything else I should check... possibly related to the PHP update?

TIA!

 

[cPanelMichael]

Hello,

You may also find some of the troubleshooting steps on the following thread helpful when investigating high CPU or memory usage:

Troubleshooting high server loads on Linux servers

Thank you.

 

[GoWilkes]

Thanks, Michael I went through the thread, but most of it was geared toward a high CPU load, not RAM. I installed the monitor.sh script (for which I had to use dos2unix, fwiw), but the results didn't tell me a whole lot:

1 minute load avg: 0
Swap used: 193180 kB
Mem used: 3728488 kB
packets received (2 secs): 459
packets sent (2 secs): 235

So you can see that, at almost 11pm, I'm using 3.7G of my 4.0G of RAM, but that doesn't get me any closer to finding out why.

Using top c, then sorting by M (that's a capital, not a lowercase... took me a hot minute to figure THAT out), ClamAV was the highest user at 12.0%. MySQL was next in line at 7.9%, then at 3.2%:

/usr/lib/jvm/jre-1.8.0/bin/java -server -Xms512m -Xmx512m -XX:NewRatio=3 -XX:SurvivorRatio=4 -XX:TargetSurvivorRatio
2

"named" and "cpsrvd" are next, then I have a lot of httpd processes showing around 19m under RES.

I turned off ClamAV for testing purposes (before running monitor.sh) and RAM went down from around 4.5G to just under 4G, but that's all. I can't turn off MySQL since all of my sites use it, and I have no clue what XX:SurvivorRatio is.

The tuning I did to sysctl.conf and my.cnf last night seemed to have no real impact, as today's load was just as high as the day before.

I just now changed Server Limit from 400 to 150, Max Clients from 300 to 100, Max Requests Per Child from 0 (unlimited?) to 10000, and Timeout from 120 to 60. This seemed to have no impact at all, though; my RAM usage is still at 3.70G, even with ClamAV turned off.

I'm not sure whether I need to be tweaking MySQL, or if maybe the number of httpd processes is the problem? The monitor.sh email showed 100 of those processes, and at 19m of RES each that's about 2G of RAM, isn't it?

 

[cPanelMichael]

Hello,

The following thread will help you to tune your MySQL configuration to see if that makes a difference:

[Case # 90317] - Warning on using mysqltuner and MySQL 5.6!

However, you should also note the information on the following post regarding high memory usage, as it's not always a bad thing:

Memory usage apparently higher than expected

Thank you.

 

[GoWilkes]

Softlayer mentioned that Java has a very large Virtual Size on the server, close to 3GB.

None of my sites intentionally use Java; I built them all, and I know that I don't code in Java. Is there any reason for it to be running, then? If not, how do I turn it off? Or at least modify the settings to minimize it?

It's also notable that ClamAV seems to be taking up a lot of RAM for no reason. I had turned off the option to scan anything, but had left it installed. I just now uninstalled it, and RAM usage dropped from 5G to 3.5G.

 

[Anoop P Alias]

the Java process should be Solr used for full-text search in IMAP. it came with new CPanel version which might explain why you see an increase in ram recently and yes java is notorious for RAM requirement

 

[GoWilkes]

Thanks, Anoop, that was definitely the culprit!!! I uninstalled it, and immediately my RAM load dropped by 1.5G!

 

[jonnyydeoliveira]

How you uninstalled solr ?

 

[Infopro]

Have a look here:
WebHost Manager »cPanel »Manage Plugins
Documentation for this area is here:
Manage Plugins - Version 64 Documentation - cPanel Documentation