How to utilize full potential of server memory?

Perhaps not fully a cPanel question and I hope someone is willing to help

The server is an X3440 with 8GB of ram

Currently the server utilizes an average of 1,2GB of ram an at max 1,8GB or ram according tot Munin, although the server has 8GB.

The average apache process uses 30% of CPU and 2% Ram, there is no iowait and Typically the top -c command gives:

Code:

 PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
4831 nobody    25   0  632m 147m  94m R 40.2  1.9   0:58.31 /usr/local/apache/bin/httpd -k start -DSSL
6754 nobody    15   0  629m  85m  34m S 19.9  1.1   0:00.60 /usr/local/apache/bin/httpd -k start -DSSL

I have tried to optimize the mysql as much as possible and this is the my.cnf

Code:

[mysqld]
innodb_buffer_pool_size=1M
interactive_timeout=600
join_buffer_size=1M
key_buffer_size=256M
local-infile=0
log-slow-queries
max_allowed_packet=12M
max_heap_table_size=64M
myisam_sort_buffer_size=8M
open_files_limit=8K
query_cache_limit=24M
query_cache_min_res_unit=1k
query_cache_size=128M
query_cache_type=1
query-cache-type = 1
read_buffer_size=2M
read_rnd_buffer_size=128K
safe-show-database
safe-show-database
set-variable = max_connections=300
sort_buffer_size=2M
table_cache=16000
thread_cache_size=64
tmp_table_size=64M
wait_timeout=300

[isamchk]
key_buffer_size = 128M
sort_buffer_size = 20M
read_buffer_size = 6M
write_buffer_size = 6M

[myisamchk]
key_buffer_size = 128M
sort_buffer_size = 20M
read_buffer_size = 6M
write_buffer_size = 6M

Mysqltuner give the following advice

Code:

-------- Performance Metrics -------------------------------------------------
[--] Up for: 2d 14h 47m 30s (15M q [66.967 qps], 148K conn, TX: 56B, RX: 6B)
[--] Reads / Writes: 73% / 27%
[--] Total buffers: 451.0M global + 5.4M per thread (300 max threads)
[OK] Maximum possible memory usage: 2.0G (25% of installed RAM)
[OK] Slow queries: 0% (0/15M)
[OK] Highest usage of available connections: 50% (151/300)
[OK] Key buffer size / total MyISAM indexes: 256.0M/127.5M
[OK] Key buffer hit rate: 99.8% (243M cached / 474K reads)
[OK] Query cache efficiency: 86.1% (11M cached / 13M selects)
[!!] Query cache prunes per day: 101694
[OK] Sorts requiring temporary tables: 0% (4 temp sorts / 224K sorts)
[!!] Joins performed without indexes: 1248
[OK] Temporary tables created on disk: 15% (81K on disk / 513K total)
[OK] Thread cache hit rate: 99% (249 created / 148K connections)
[!!] Table cache hit rate: 3% (244 open / 6K opened)
[OK] Open file limit used: 1% (457/32K)
[OK] Table locks acquired immediately: 99% (2M immediate / 2M locks)
[OK] InnoDB data size / buffer pool: 208.0K/1.0M

-------- Recommendations -----------------------------------------------------
General recommendations:
    Adjust your join queries to always utilize indexes
    Increase table_cache gradually to avoid file descriptor limits
Variables to adjust:
    query_cache_size (> 128M)
    join_buffer_size (> 1.0M, or always use indexes with joins)
    table_cache (> 16000)

Even when i set the query_cache_size to 512M it still advises me to increase the value. The same goes for table_cache, even set at 40000 it still advises me to increase it, so I assume that something is not right.

I haven't done much Apache tweaking and the settings are as followed:

Code:

Timeout 300
TraceEnable Off
ServerSignature Off
ServerTokens ProductOnly
FileETag None
StartServers 10
<IfModule prefork.c>
MinSpareServers 10
MaxSpareServers 20
</IfModule>
ServerLimit 300
MaxClients 300
MaxRequestsPerChild 2000
KeepAlive On
KeepAliveTimeout 2
MaxKeepAliveRequests 50

What can I do to optimize the server so that it:
1. Utilizes the full potential of the server memory?
2. Reduces that CPU usage per httpd request?

 

Code:

             total       used       free     shared    buffers     cached
Mem:          7975       7329        645          0        341       5260
-/+ buffers/cache:       1727       6248
Swap:         8189          0       8189

As you can see alot is cached, but not utilized.

 

Thank you for taking time to look at this.

If there are 20+ apache processes the load reaches 7+ and the server reaches huge loads when having 50+ connections. I have tried looking are the process, both from command as from WHM (Process Manage -> Trace) and the problem is that I get a few hundred thousand lines. In WHM eventually my browser cannot handle it (FF consuming 3Gb or ram!) and usually crashes.

The problem is that with 50+ connections, the load gets 20+, while the memory consumption is still less than 2Gb of the 8Gb. Also looking at the msqltuner results:

I was under the assumption that my system is nut utilizing its full potential of memory.

 

Thank you for pointing me in the right direction!

I think it is a log rotation misconfiguration.

With each httpd process it loads all the domain logs from /usr/local/apache/domlogs/

Some of the logs are around 1Gb and the whole directory is 4.2Gb and I think that is causing 25% to 35% cpu usage for each httpd process.

I am going to do more reading and learn more about the problem and solutions. Again thank you.