HTTPd and PHP OOM - Out of Memory

[MrMatt]

Appealing to some cpanel + WHM experts.

We experienced Out of Memory and MYsql restarts PHP and httpd on our Centos 6 platform of WHM version 60 and Centos 6.8 with server memory 8GB

/var/log/messages

Jan 12 Restart 1
php invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0

Jan 30 Restart 2
httpd invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0


After Restart 1
We ran the mysqltuner.pl everyone talks about for tweaking mysql.cnf
Also we increase server memory from 8 - 12GB

Then added these settings as recommended.


After Restart 1
We ran the mysqltuner.pl it now recommends this ?

-------- InnoDB Metrics ----------------------------------------------------------------------------
[--] InnoDB is enabled.
[--] InnoDB Thread Concurrency: 8
[OK] InnoDB File per table is activated
[OK] InnoDB buffer pool / data size: 9.0G/2.1G
[!!] Ratio InnoDB log file size / InnoDB Buffer pool size (2.77777777777778 %): 128.0M * 2/9.0G should be equal 25%
[!!] InnoDB buffer pool instances: 8
[--] InnoDB Buffer Pool Chunk Size not used or defined in your version
[OK] InnoDB Read buffer efficiency: 99.98% (345174626 hits/ 345244918 total)
[!!] InnoDB Write Log efficiency: 65.86% (109652 hits/ 166496 total)
[OK] InnoDB log waits: 0.00% (0 waits / 56844 writes)


Variables to adjust:
table_open_cache (> 2000)
key_buffer_size (> 88.1M)
innodb_log_file_size should be equals to 1/4 of buffer pool size (=2G) if possible.
innodb_buffer_pool_instances(=9)

Has anyone experience enough to verify the above configuration or experience with sar analysis for server and memory optimisation ?

Architecturally should we stick with this mix of Apache 2.4 / Mysql 5.6 or considering cloud Linux trialling it on a dev server found the migration to be painful. We would also consider a dedicated mysql server and webserver load balance if thats the better answer than struggling inside one server ?

BTW Apache logs the Restart 2 was triggered by a script coming from a GCP hosted server.

Kind Regards

Matt

 

[cPanelMichael]

Hello,

You may find the following threads helpful when troubleshooting this issue:

SOLVED - The process “php-cgi” was terminated because the system is low on memory
Hang, Failed, Recovered and Out of Memory

Let us know if this helps.

Thank you.

 

[MrMatt]

Thanks Michael for the links I appreciated this command.

ps -e -o pid,user,cpu,size,rss,cmd --sort -size,-rss | head

Notable Results : Size in (kb)

4113 mysql - 8500944 /usr/sbin/mysqld
10511 named - 56472 /usr/sbin/named -u named
5300 root - 24000 /usr/bin/newrelic-daemon -c
4624 root - 442460 /usr/local/cpanel/3rdparty/bin/clamd
7798 nobody - 138296 /usr/local/apache/bin/httpd -k start
8002 nobody - 136508 /usr/local/apache/bin/httpd -k start

What I am tracking is the mysqld service isn't good at releasing memory and is on a constant upward trend until oom killer is invoked.

Do you have any information on MQL memory handling to reclaim this space back ?

We have considered writing a memory monitor which restarts a service when it gets to a threshold level similar to oom killer yet with more variables and settings. Unless you'd consider this a waste if there is configuration for oom Killer ?

Also want to carefully consider the impacts of implementing cloud Linux vs lightspeed as we are using some of the Apache modules.

 

[MrMatt]

Used the 'atop' -m for memory stats its very good.
According to this tool we have 512MB swap and 0 free.

For a server with 16 GB the recommended memory is higher does Cpanel have any guidelines ?

Linux: Should You Use Twice the Amount of Ram as Swap Space?

How do we safely increase the swap size ?

our /etc/fstab :

dev/xvda / ext4 discard,usrjquota=quota.user,jqfmt=vfsv0 1 1
/dev/xvdb none swap sw 0 0
none /dev/pts devpts gid=5,mode=620 0 0

 

[cPanelMichael]

Hello,

Have you considered switching to MariaDB via "WHM >> MySQL/MariaDB Upgrade"? I've seen reports of improved memory performance when switching to MariaDB.

Thank you.