How to utilize full potential of server memory?

[johnburk]

Perhaps not fully a cPanel question and I hope someone is willing to help

The server is an X3440 with 8GB of ram

Currently the server utilizes an average of 1,2GB of ram an at max 1,8GB or ram according tot Munin, although the server has 8GB.

The average apache process uses 30% of CPU and 2% Ram, there is no iowait and Typically the top -c command gives:

 PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
4831 nobody    25   0  632m 147m  94m R 40.2  1.9   0:58.31 /usr/local/apache/bin/httpd -k start -DSSL
6754 nobody    15   0  629m  85m  34m S 19.9  1.1   0:00.60 /usr/local/apache/bin/httpd -k start -DSSL

I have tried to optimize the mysql as much as possible and this is the my.cnf

[mysqld]
innodb_buffer_pool_size=1M
interactive_timeout=600
join_buffer_size=1M
key_buffer_size=256M
local-infile=0
log-slow-queries
max_allowed_packet=12M
max_heap_table_size=64M
myisam_sort_buffer_size=8M
open_files_limit=8K
query_cache_limit=24M
query_cache_min_res_unit=1k
query_cache_size=128M
query_cache_type=1
query-cache-type = 1
read_buffer_size=2M
read_rnd_buffer_size=128K
safe-show-database
safe-show-database
set-variable = max_connections=300
sort_buffer_size=2M
table_cache=16000
thread_cache_size=64
tmp_table_size=64M
wait_timeout=300

[isamchk]
key_buffer_size = 128M
sort_buffer_size = 20M
read_buffer_size = 6M
write_buffer_size = 6M

[myisamchk]
key_buffer_size = 128M
sort_buffer_size = 20M
read_buffer_size = 6M
write_buffer_size = 6M

Mysqltuner give the following advice

-------- Performance Metrics -------------------------------------------------
[--] Up for: 2d 14h 47m 30s (15M q [66.967 qps], 148K conn, TX: 56B, RX: 6B)
[--] Reads / Writes: 73% / 27%
[--] Total buffers: 451.0M global + 5.4M per thread (300 max threads)
[OK] Maximum possible memory usage: 2.0G (25% of installed RAM)
[OK] Slow queries: 0% (0/15M)
[OK] Highest usage of available connections: 50% (151/300)
[OK] Key buffer size / total MyISAM indexes: 256.0M/127.5M
[OK] Key buffer hit rate: 99.8% (243M cached / 474K reads)
[OK] Query cache efficiency: 86.1% (11M cached / 13M selects)
[!!] Query cache prunes per day: 101694
[OK] Sorts requiring temporary tables: 0% (4 temp sorts / 224K sorts)
[!!] Joins performed without indexes: 1248
[OK] Temporary tables created on disk: 15% (81K on disk / 513K total)
[OK] Thread cache hit rate: 99% (249 created / 148K connections)
[!!] Table cache hit rate: 3% (244 open / 6K opened)
[OK] Open file limit used: 1% (457/32K)
[OK] Table locks acquired immediately: 99% (2M immediate / 2M locks)
[OK] InnoDB data size / buffer pool: 208.0K/1.0M

-------- Recommendations -----------------------------------------------------
General recommendations:
    Adjust your join queries to always utilize indexes
    Increase table_cache gradually to avoid file descriptor limits
Variables to adjust:
    query_cache_size (> 128M)
    join_buffer_size (> 1.0M, or always use indexes with joins)
    table_cache (> 16000)

Even when i set the query_cache_size to 512M it still advises me to increase the value. The same goes for table_cache, even set at 40000 it still advises me to increase it, so I assume that something is not right.

I haven't done much Apache tweaking and the settings are as followed:

Timeout 300
TraceEnable Off
ServerSignature Off
ServerTokens ProductOnly
FileETag None
StartServers 10
<IfModule prefork.c>
MinSpareServers 10
MaxSpareServers 20
</IfModule>
ServerLimit 300
MaxClients 300
MaxRequestsPerChild 2000
KeepAlive On
KeepAliveTimeout 2
MaxKeepAliveRequests 50

What can I do to optimize the server so that it:
1. Utilizes the full potential of the server memory?
2. Reduces that CPU usage per httpd request?

 

[Infopro]

What are the results of this command?

free -m

 

[johnburk]

             total       used       free     shared    buffers     cached
Mem:          7975       7329        645          0        341       5260
-/+ buffers/cache:       1727       6248
Swap:         8189          0       8189

As you can see alot is cached, but not utilized.

 

[cPanelTristan]

I don't understand the request for the first question. Why would you want more memory to be used? Normally, people are trying to reduce the memory usage, and caching but not utilizing the memory is a good thing. Your server's memory usage is optimal at this point from what I can see.

The CPU usage for each Apache request is 30% or only some of the Apache requests? You could run "lsof -p PID#" where PID# is the PID number of the Apache process with the high CPU usage. It's probably a site that has large images that are being processed by Apache or something causing the high usage. Otherwise, it might be the parent process for a lot of child processes. If you are using suPHP, it will show higher load and usage over DSO or FCGI for Apache processes.

Does the server even have a high load at any point? Load and latency are the important considerations for making changes to the server's configuration.

 

[johnburk]

Thank you for taking time to look at this.

If there are 20+ apache processes the load reaches 7+ and the server reaches huge loads when having 50+ connections. I have tried looking are the process, both from command as from WHM (Process Manage -> Trace) and the problem is that I get a few hundred thousand lines. In WHM eventually my browser cannot handle it (FF consuming 3Gb or ram!) and usually crashes.

The problem is that with 50+ connections, the load gets 20+, while the memory consumption is still less than 2Gb of the 8Gb. Also looking at the msqltuner results:

Even when i set the query_cache_size to 512M it still advises me to increase the value. The same goes for table_cache, even set at 40000 it still advises me to increase it, so I assume that something is not right.

I was under the assumption that my system is nut utilizing its full potential of memory.

 

[cPanelTristan]

You do not want to use more memory on the system. The goal isn't to use more memory, so I cannot assist in trying to utilize more memory for the system, since my experience suggests otherwise versus your request to increase the memory usage.

For checking the process, I wouldn't personally check how a process is running in WHM when the server has high load. My suggestion was to use root SSH and run "lsof -p PID#" for checking the processes. One of the sysadmins best tool is lsof and the flags that you can use with it. You might want to read the man page on it by running "man lsof" or view some online guides on using it. Once you start using lsof, you'll find it invaluable in getting output details on what is happening for a process.

 

[johnburk]

Thank you for pointing me in the right direction!

I think it is a log rotation misconfiguration.

With each httpd process it loads all the domain logs from /usr/local/apache/domlogs/

Some of the logs are around 1Gb and the whole directory is 4.2Gb and I think that is causing 25% to 35% cpu usage for each httpd process.

I am going to do more reading and learn more about the problem and solutions. Again thank you.

 

[johnnyN05]

I'm running into the same issues here...

I too have apache processes averaging between 15-40% CPU. I ran a "lsof -p PID#", and this too loads all the domain logs from different VirtualHosts.

Has there been an update to this?

 

[cPanelTristan]

Hello johnnyN05,

The domlogs will show up for all lsof returns as part of the process, so that is part of what it normally looks like. Please open a new thread to ask for assistance with your server's configuration if possible, since these threads are typically very server-specific.

Thanks!

 

[SoftDux]

Memory isn't your problem, but rather CPU (either too few cores, or too slow to process the requests), and / or probably IO (too high) as well. You should first look at improving your IO (move to faster HDD's, and / or RAID1 / RAID10 if you haven't done so. Lower IO will use less CPU, which in turn will lower your server's resources