Mysqld terminates, system low on memory

[svgr]

I've been getting this error repeatedly: "The process “mysqld” was terminated because the system is low on memory".

I have a blog running on WordPress and a forum running on XenForo

In the log, I see that too many php processes are there. This usually happens once or twice in the day, and the load spikes so much that the server goes out of memory and mysqld terminates.

As suggested in the famous thread here, I tried checking the access logs in case this increased load is caused by increased traffic, but that does not seem to be the case as traffic is not high when this occurs. The load is very high, there are too many php processes open, there's the WordPress's index.php with a high CPU usage in the WHM's daily process log.

Even the mysql tables report some tables as crashed, but I do have a weekly cron job that repairs them, apart from the InnoDB ones I think.

Can anyone suggest what could be the problem?


Have attached some logs.

 

[cPanelMichael]

Hello,

Could you let us know the contents of your /etc/my.cnf file?

Thank you.

 

[svgr]

Hello,

Could you let us know the contents of your /etc/my.cnf file?

Thank you.

Hello,

Yes, my.cnf contents:

[mysqld]
innodb=on
default-storage-engine=MyISAM
interactive_timeout=600
key_buffer_size=624M
key_cache_block_size=4096
max_heap_table_size=96M
max_join_size=1000000000
max_allowed_packet=268435456
open_files_limit=10000
query_cache_size=96M
thread_cache_size=100
tmp_table_size=96M
wait_timeout=7800
max_user_connections=200
myisam_recover_options=FORCE
innodb_file_per_table=1
innodb_flush_log_at_trx_commit=0
innodb_purge_threads=1
innodb_support_xa=0
innodb_force_recovery=0
innodb_thread_concurrency=8
pid-file="/var/lib/mysql/mysqld.pid"
innodb_buffer_pool_size=256M
innodb_log_file_size=32M
table_open_cache=450

I did change some parameters here after running the mysqltuner.pl script.

Recent PHP changes were also made from mod_fcgid to suphp as the php handler because a script wasn't executing with fcgid. Though the error was occurring on mod_fcgid too.

The ouf of memory error can be viewed here: - Removed -
(I wasn't able to attach the file here).

 

[svgr]

Out of memory log:

[350298.620131] php invoked oom-killer: gfp_mask=0x280da, order=0, oom_adj=0, oom_score_adj=0
[350298.624705] php cpuset=/ mems_allowed=0
[350298.625787] Pid: 17274, comm: php Not tainted 2.6.32-642.1.1.el6.x86_64 #1
[350298.649328] Call Trace:
[350298.650646]  [<ffffffff81131640>] ? dump_header+0x90/0x1b0
[350298.651862]  [<ffffffff8123c20c>] ? security_real_capable_noaudit+0x3c/0x70
[350298.653133]  [<ffffffff81131ac2>] ? oom_kill_process+0x82/0x2a0
[350298.654555]  [<ffffffff81131a01>] ? select_bad_process+0xe1/0x120
[350298.655656]  [<ffffffff81131f00>] ? out_of_memory+0x220/0x3c0
[350298.656867]  [<ffffffff8113e8dc>] ? __alloc_pages_nodemask+0x93c/0x950
[350298.658083]  [<ffffffff81177c2a>] ? alloc_pages_vma+0x9a/0x150
[350298.659325]  [<ffffffff81159fad>] ? handle_pte_fault+0x73d/0xb20
[350298.660488]  [<ffffffff81130170>] ? generic_file_aio_read+0x380/0x700
[350298.661615]  [<ffffffff8115a629>] ? handle_mm_fault+0x299/0x3d0
[350298.662762]  [<ffffffff8106b4c3>] ? perf_event_task_sched_out+0x33/0x70
[350298.664097]  [<ffffffff81052156>] ? __do_page_fault+0x146/0x500
[350298.665289]  [<ffffffff815475be>] ? schedule+0x3ee/0xb70
[350298.666395]  [<ffffffff81548a7e>] ? mutex_lock+0x1e/0x50
[350298.667892]  [<ffffffff8119a0cc>] ? generic_file_llseek_size+0x8c/0xd0
[350298.669136]  [<ffffffff8154dc8e>] ? do_page_fault+0x3e/0xa0
[350298.670292]  [<ffffffff8154af95>] ? page_fault+0x25/0x30
[350298.671394] Mem-Info:
[350298.672543] Node 0 DMA per-cpu:
[350298.674122] CPU    0: hi:    0, btch:   1 usd:   0
[350298.675239] CPU    1: hi:    0, btch:   1 usd:   0
[350298.676308] Node 0 DMA32 per-cpu:
[350298.677517] CPU    0: hi:  186, btch:  31 usd:  81
[350298.678545] CPU    1: hi:  186, btch:  31 usd: 166
[350298.679583] Node 0 Normal per-cpu:
[350298.680669] CPU    0: hi:  186, btch:  31 usd: 143
[350298.681790] CPU    1: hi:  186, btch:  31 usd: 150
[350298.682882] active_anon:713777 inactive_anon:181639 isolated_anon:1504
[350298.682883]  active_file:203 inactive_file:880 isolated_file:261
[350298.682884]  unevictable:0 dirty:0 writeback:53 unstable:0
[350298.682885]  free:22157 slab_reclaimable:5543 slab_unreclaimable:12270
[350298.682886]  mapped:456 shmem:65 pagetables:25031 bounce:0
[350298.688560] Node 0 DMA free:15724kB min:248kB low:308kB high:372kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15312kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
[350298.694818] lowmem_reserve[]: 0 3512 4017 4017
[350298.696525] Node 0 DMA32 free:63932kB min:58868kB low:73584kB high:88300kB active_anon:2658944kB inactive_anon:531784kB active_file:672kB inactive_file:3356kB unevictable:0kB isolated(anon):2560kB isolated(file):768kB present:3596500kB mlocked:0kB dirty:0kB writeback:168kB mapped:1388kB shmem:236kB slab_reclaimable:13132kB slab_unreclaimable:21136kB kernel_stack:10896kB pagetables:66728kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:66400 all_unreclaimable? no
[350298.703573] lowmem_reserve[]: 0 0 505 505
[350298.705278] Node 0 Normal free:8352kB min:8464kB low:10580kB high:12696kB active_anon:196104kB inactive_anon:195128kB active_file:144kB inactive_file:1092kB unevictable:0kB isolated(anon):3328kB isolated(file):276kB present:517120kB mlocked:0kB dirty:0kB writeback:72kB mapped:400kB shmem:24kB slab_reclaimable:9040kB slab_unreclaimable:27944kB kernel_stack:5152kB pagetables:33396kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:60069 all_unreclaimable? no
[350298.711828] lowmem_reserve[]: 0 0 0 0
[350298.713873] Node 0 DMA: 3*4kB 0*8kB 0*16kB 1*32kB 1*64kB 0*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15724kB
[350298.724363] Node 0 DMA32: 952*4kB 580*8kB 327*16kB 146*32kB 553*64kB 47*128kB 2*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB = 62320kB
[350298.730353] Node 0 Normal: 1374*4kB 6*8kB 1*16kB 0*32kB 1*64kB 0*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 0*4096kB = 8952kB
[350298.736946] 20887 total pagecache pages
[350298.738348] 18243 pages in swap cache
[350298.739916] Swap cache stats: add 17697337, delete 17679094, find 10809983/12511189
[350298.742325] Free swap  = 0kB
[350298.743395] Total swap = 4194252kB
[350298.777644] 1048575 pages RAM
[350298.780166] 67960 pages reserved
[350298.781357] 36048 pages shared
[350298.782784] 947214 pages non-shared
[350298.784523] [ pid ]   uid  tgid total_vm      rss cpu oom_adj oom_score_adj name
[350298.786654] [  490]     0   490     2680        1   1     -17         -1000 udevd
[350298.789306] [  790]     0   790     2662        1   1     -17         -1000 udevd
[350298.791688] [ 1342]     0  1342    62320       64   0       0             0 rsyslogd
[350298.794406] [ 1357]     0  1357     4563       72   1       0             0 irqbalance
[350298.796445] [ 1376]    25  1376    59678      161   0       0             0 named
[350298.798547] [ 1394]    81  1394     5359       13   0       0             0 dbus-daemon
[350298.800875] [ 1432]     0  1432     1020        1   0       0             0 acpid
[350298.803071] [ 1451]    28  1451   336791      222   1       0             0 nscd
[350298.806086] [ 1569]     0  1569    16086       36   0     -17         -1000 sshd
[350298.808433] [ 1580]    38  1580     6262       39   1       0             0 ntpd
[350298.811308] [ 2053]     0  2053     4662       11   1       0             0 dovecot
[350298.814059] [ 2065]   401  2065    10639        2   0       0             0 pop3-login
[350298.816606] [ 2066]   401  2066    10637       16   0       0             0 imap-login
[350298.819023] [ 2067]    97  2067     3368       13   1       0             0 anvil
[350298.821178] [ 2068]     0  2068     3401       14   1       0             0 log
[350298.823629] [ 2070]   401  2070    10655        2   1       0             0 pop3-login
[350298.826213] [ 2071]   401  2071    10637       16   1       0             0 imap-login
[350298.828541] [ 2072]     0  2072     3659       16   1       0             0 config
[350298.830667] [ 2082]    47  2082    18238       29   1       0             0 exim
[350298.832887] [ 2100]     0  2100    49742      402   0       0             0 /usr/local/cpan
[350298.835547] [ 2102]     0  2102    52925      184   0       0             0 spamd child
[350298.837546] [ 2129]     0  2129    45352        2   1       0             0 abrtd
[350298.840235] [ 2139]     0  2139    45162       12   1       0             0 abrt-dump-oops
[350298.842453] [ 2150]     0  2150    19189      164   1       0             0 httpd
[350298.844936] [ 2165]     0  2165    34191        5   0       0             0 pure-ftpd
[350298.847513] [ 2167]     0  2167    34036        3   1       0             0 pure-authd
[350298.849849] [ 2260]     0  2260    29271       34   1       0             0 crond
[350298.852149] [ 2275]     0  2275     4857        1   1       0             0 atd
[350298.854527] [ 2375]     0  2375    26952      158   1       0             0 cpsrvd (SSL) -
[350298.857220] [ 2381]     0  2381     8172       72   1       0             0 queueprocd - wa
[350298.859662] [ 2391]     0  2391    27506       32   1       0             0 dnsadmin - serv
[350298.861835] [ 2419]     0  2419    20808      236   1       0             0 tailwatchd
[350298.864235] [ 2433]     0  2433    19586       37   0       0             0 cPhulkd - proce
[350298.866265] [ 2476]     0  2476    33028       74   1       0             0 cpdavd - accept
[350298.869020] [ 2512]     0  2512     5915       80   0       0             0 cpanellogd - sl
[350298.871438] [ 2540]     0  2540     1016        2   1       0             0 mingetty
[350298.873466] [ 2542]     0  2542     1016        2   0       0             0 mingetty
[350298.876566] [ 2545]     0  2545     1016        2   1       0             0 mingetty
[350298.878716] [ 2547]     0  2547     2679        1   0     -17         -1000 udevd
[350298.881694] [ 2548]     0  2548     1016        2   1       0             0 mingetty
[350298.884167] [ 2550]     0  2550     1016        2   1       0             0 mingetty
[350298.886919] [ 2552]     0  2552     1016        2   1       0             0 mingetty
[350298.889212] [13473]     0 13473     3367        2   0       0             0 ssl-params
[350298.891793] [ 9484]     0  9484     2394       15   0       0             0 mysqld_safe
[350298.894148] [10131]     0 10131   146212       16   0       0             0 clamd
[350298.897015] [10833]   402 10833   565594    43704   1       0             0 mysqld
[350298.899710] [10101]    99 10101    19189       91   0       0             0 httpd
[350298.903219] [10102]     0 10102    19061       16   0       0             0 leechprotect
[350298.905757] [10105]    99 10105   334311     1433   0       0             0 httpd
[350298.908375] [10106]    99 10106   334148     1290   1       0             0 httpd
[350298.910609] [10107]    99 10107   334132     1629   0       0             0 httpd
[350298.913125] [10193]    99 10193   334447     1283   0       0             0 httpd
[350298.915317] [10572]    99 10572   334296     1454   1       0             0 httpd
[350298.917957] [11640]    99 11640   334172     1098   0       0             0 httpd
[350298.940859] [17266]   500 17266    67391     4405   0       0             0 php
[350298.943043] [17267]   500 17267    68176     5563   0       0             0 php
[350298.945111] [17268]   500 17268    67258     3739   0       0             0 php
[350298.947643] [17269]   500 17269    67207     3737   0       0             0 php
[350298.950179] [17270]   500 17270    69232     7128   0       0             0 php
[350298.952471] [17271]   500 17271    67207     4163   1       0             0 php
[350298.954654] [17272]   500 17272    66872     4234   0       0             0 php
[350298.957165] [17273]   500 17273    67201     4464   0       0             0 php
[350298.959682] [17274]   500 17274    67006     4889   1       0             0 php
[350298.962353] [17275]   500 17275    67564     5368   0       0             0 php
[350298.965809] [17276]   500 17276    67207     3914   0       0             0 php
[350298.968943] [17277]   500 17277    67385     5173   1       0             0 php
[350298.971422] [17278]   500 17278    67258     3936   1       0             0 php
[350298.973697] [17279]   500 17279    67194     4690   0       0             0 php
[350298.976801] [17280]   500 17280    67194     4433   0       0             0 php
[350298.979837] [17281]   500 17281    67391     4686   0       0             0 php
[350298.982474] [17282]   500 17282    67271     4478   0       0             0 php
[350298.984817] [17283]   500 17283    66195     3007   0       0             0 php
[350298.987409] [17284]   500 17284    67455     5214   1       0             0 php
[350298.990438] [17285]   500 17285    67271     5281   0       0             0 php
[350298.992784] [17286]   500 17286    67650     5195   0       0             0 php
[350298.995364] [17287]   500 17287    66324     3838   0       0             0 php
[350298.997480] [17288]   500 17288    69394     7827   0       0             0 php
[350298.999829] [17289]   500 17289    67207     4614   0       0             0 php
[350299.001774] [17290]   500 17290    66997     4676   1       0             0 php
[350299.003670] [17291]   500 17291    67194     4353   0       0             0 php
[350299.006868] [17292]   500 17292    67450     4823   0       0             0 php
[350299.009223] [17293]   500 17293    67391     5577   0       0             0 php
[350299.011355] [17294]   500 17294    67452     4386   0       0             0 php
[350299.014289] [17295]   500 17295    69600     7275   0       0             0 php
[350299.016437] [17296]   500 17296    69496     7707   0       0             0 php
[350299.018945] [17297]   500 17297    67391     4847   0       0             0 php
[350299.021336] [17298]   500 17298    67383     5149   0       0             0 php
[350299.023280] [17299]   500 17299    66809     4182   0       0             0 php
[350299.025469] [17300]   500 17300    67207     4029   0       0             0 php
[350299.027403] [17301]   500 17301    67207     3717   1       0             0 php
[350299.029556] [17302]   500 17302    67391     4422   0       0             0 php
[350299.031995] [17303]   500 17303    67459     4402   0       0             0 php
[350299.034431] [17304]   500 17304    69411     7612   0       0             0 php
[350299.036560] [17305]   500 17305    67455     4531   0       0             0 php
[350299.039460] [17306]   500 17306    66547     3994   1       0             0 php
[350299.041403] [17307]   500 17307    69406     7885   0       0             0 php
[350299.043566] [17308]   500 17308    68052     5806   0       0             0 php
[350299.045967] [17309]   500 17309    67207     5305   1       0             0 php
[350299.049183] [17310]   500 17310    67455     5329   0       0             0 php
[350299.051991] [17311]   500 17311    67455     4668   0       0             0 php
[350299.054280] [17312]   500 17312    67391     4033   0       0             0 php
[350299.056262] [17313]   500 17313    67384     4708   0       0             0 php
[350299.059027] [17314]   500 17314    67391     5346   0       0             0 php
[350299.061119] [17315]   500 17315    67387     4926   0       0             0 php
[350299.063469] [17316]   500 17316    67391     4523   0       0             0 php
[350299.066010] [17317]   500 17317    67391     5182   0       0             0 php
[350299.068024] [17318]   500 17318    67455     5264   0       0             0 php
[350299.070347] [17319]   500 17319    67391     4299   0       0             0 php
[350299.072337] [17320]   500 17320    69012     7721   0       0             0 php
[350299.074420] [17321]   500 17321    67520     5352   0       0             0 php
[350299.076530] [17357]   500 17357    67455     5333   0       0             0 php
[350299.078552] [17360]   500 17360    68754     6829   1       0             0 php
[350299.080401] [17363]   500 17363    67391     5124   0       0             0 php
[350299.082341] [17364]   500 17364    69600     8627   0       0             0 php
[350299.084885] [17365]   500 17365    68199     7222   0       0             0 php
[350299.087587] [17368]   500 17368    68754     7460   0       0             0 php
[350299.089945] [17369]   500 17369    67391     5297   1       0             0 php
[350299.091945] [17375]   500 17375    69672     8672   0       0             0 php
[350299.094660] [17377]   500 17377    67395     5649   0       0             0 php
[350299.096602] [17380]   500 17380    69492     8154   0       0             0 php
[350299.098716] [17382]   500 17382    69536     8141   0       0             0 php
[350299.100626] [17383]   500 17383    69664     8606   0       0             0 php
[350299.102990] [17422]   500 17422    70186    11312   1       0             0 php
[350299.105022] [17423]    99 17423   334211     2205   0       0             0 httpd
[350299.107096] [17453]   500 17453    70056    12985   0       0             0 php
[350299.109248] [17454]   500 17454    69995    12786   0       0             0 php
[350299.112064] [17457]   500 17457    71827    15152   0       0             0 php
[350299.114144] [17468]   500 17468    60674     8959   1       0             0 php
[350299.116146] [17469]   500 17469    60738     9081   1       0             0 php
[350299.118310] [17470]   500 17470    54427     3423   0       0             0 php
[350299.120502] [17476]   500 17476    60738     9171   1       0             0 php
[350299.122413] [17477]   500 17477    59667     8199   0       0             0 php
[350299.124641] [17482]   500 17482    54259     3346   1       0             0 php
[350299.126649] [17484]     0 17484    35567       84   0       0             0 crond
[350299.129146] [17485]   500 17485    54007     3355   0       0             0 php
[350299.131367] [17487]    99 17487   334166     1209   0       0             0 httpd
[350299.133445] [17491]    99 17491   334201     1362   1       0             0 httpd
[350299.135701] [17494]   500 17494    53333     2409   0       0             0 php
[350299.137930] [17496]   500 17496    58559     7913   0       0             0 php
[350299.140207] [17497]    99 17497   334138     1328   0       0             0 httpd
[350299.142168] [17505]   500 17505    57589     6978   0       0             0 php
[350299.144212] [17506]   500 17506    54215     3280   0       0             0 php
[350299.146347] [17507]   500 17507    54215     3433   0       0             0 php
[350299.148410] [17508]   500 17508    55350     4696   0       0             0 php
[350299.170789] [17509]   500 17509    56907     6248   0       0             0 php
[350299.173387] [17511]   500 17511    54280     3159   1       0             0 php
[350299.175506] [17590]   500 17590    59788     9075   0       0             0 php
[350299.177513] [17591]   500 17591    60735    10096   0       0             0 php
[350299.179530] [17593]   500 17593    58943     8408   0       0             0 php
[350299.181537] [17595]   500 17595    54378     3372   1       0             0 php
[350299.183874] [17596]   500 17596    58943     8353   1       0             0 php
[350299.185941] [17598]   500 17598    60671     9975   1       0             0 php
[350299.187915] [17599]   500 17599    55435     4792   1       0             0 php
[350299.189908] [17600]   500 17600    49128     1241   0       0             0 php
[350299.192339] [17605]   500 17605    54115     2900   0       0             0 php
[350299.194391] [17607]   500 17607    55834     5115   0       0             0 php
[350299.196391] [17608]   500 17608    54215     3350   1       0             0 php
[350299.198412] [17609]   500 17609    54280     3444   0       0             0 php
[350299.200637] [17610]   500 17610    58943     8282   0       0             0 php
[350299.203100] [17612]   500 17612    55480     4790   0       0             0 php
[350299.205249] [17614]   500 17614    59724     9067   0       0             0 php
[350299.207379] [17615]   500 17615    54797     4082   0       0             0 php
[350299.209676] [17616]   500 17616    58369     7816   0       0             0 php
[350299.211744] [17619]   500 17619    56842     6144   0       0             0 php
[350299.213619] [17620]   500 17620    56782     6085   0       0             0 php
[350299.215524] [17622]   500 17622    56486     5757   1       0             0 php
[350299.217405] [17625]   500 17625    55687     4968   0       0             0 php
[350299.219521] [17626]   500 17626    54045     3036   0       0             0 php
[350299.221424] [17629]   500 17629    59072     8347   0       0             0 php
[350299.223704] [17631]   500 17631    55020     4084   0       0             0 php
[350299.226940] [17633]   500 17633    54422     3939   0       0             0 php
[350299.229236] [17636]   500 17636    54376     3714   1       0             0 php
[350299.232666] [17637]   500 17637    60735    10040   1       0             0 php
[350299.234645] [17638]   500 17638    55552     4865   0       0             0 php
[350299.236683] [17641]     0 17641    26526       50   0       0             0 sh
[350299.238738] [17642]   500 17642    54743     4099   0       0             0 php
[350299.240741] [17643]   500 17643    54010     3313   1       0             0 php
[350299.242787] [17644]   500 17644    55676     4917   0       0             0 php
[350299.244824] [17646]   500 17646    54749     3935   1       0             0 php
[350299.247230] [17647]   500 17647    60031     9405   0       0             0 php
[350299.249504] [17649]   500 17649    54742     4040   0       0             0 php
[350299.251636] [17650]   500 17650    50192     1162   0       0             0 php
[350299.253612] [17651]   500 17651    57490     6852   1       0             0 php
[350299.256050] [17652]   500 17652    54428     3704   1       0             0 php
[350299.258169] [17653]   500 17653    60096     9409   0       0             0 php
[350299.260601] [17654]     0 17654    35506      953   1       0             0 /usr/local/cpan
[350299.262765] [17655]   500 17655    54742     4065   0       0             0 php
[350299.265208] [17658]   500 17658    54180     3133   1       0             0 php
[350299.267654] [17659]   500 17659    54202     3720   0       0             0 php
[350299.269827] [17661]   500 17661    57069     6365   0       0             0 php
[350299.272417] [17662]   500 17662    54200     3733   0       0             0 php
[350299.275127] [17663]   500 17663    54002     3318   0       0             0 php
[350299.277601] [17664]   500 17664    54490     3957   0       0             0 php
[350299.280010] [17667]     0 17667    20908      730   0       0             0 tailwatchd - ch
[350299.282374] [17668]   500 17668    54294     3806   0       0             0 php
[350299.284645] [17671]   500 17671    55103     4433   0       0             0 php
[350299.286942] [17672]   500 17672    58943     8258   0       0             0 php
[350299.288975] [17673]   500 17673    55088     4527   1       0             0 php
[350299.291023] [17700]   500 17700    54552     4162   0       0             0 php
[350299.292977] [17706]   500 17706    54078     3218   1       0             0 php
[350299.294971] [17719]   500 17719    53672     3173   1       0             0 php
[350299.296938] [17772]    99 17772   252176      846   1       0             0 httpd
[350299.299512] [17788]    99 17788   219408      664   0       0             0 httpd
[350299.301524] [17789]    99 17789   235792     1147   0       0             0 httpd
[350299.303549] [17791]    99 17791   235792      666   1       0             0 httpd
[350299.305550] [17809]   500 17809    50976     2203   0       0             0 php
[350299.307718] [17810]   500 17810    54202     3819   1       0             0 php
[350299.310657] [17813]    99 17813   219408      749   1       0             0 httpd
[350299.313010] [17814]    99 17814   235792      692   1       0             0 httpd
[350299.315186] [17966]   500 17966    54422     3964   1       0             0 php
[350299.317185] [17967]   500 17967    32182      645   0       0             0 php
[350299.319346] [17968]   500 17968    54486     4054   1       0             0 php
[350299.321413] [17969]   500 17969    55334     4740   1       0             0 php
[350299.323495] [17999]   500 17999    54550     4063   0       0             0 php
[350299.325636] [18000]   500 18000    54422     3871   0       0             0 php
[350299.327666] [18002]   500 18002    54567     4004   1       0             0 php
[350299.330300] [18023]   500 18023    54422     3911   0       0             0 php
[350299.332789] [18025]   500 18025    54488     4079   1       0             0 php
[350299.335951] [18026]   500 18026    54142     3743   0       0             0 php
[350299.338709] [18029]   500 18029    53767     3292   1       0             0 php
[350299.341568] [18030]   500 18030    54294     3896   1       0             0 php
[350299.344237] [18063]   500 18063    32182      636   0       0             0 php
[350299.346566] [18065]   500 18065    32347      827   0       0             0 php
[350299.349040] [18066]   500 18066    32149      563   0       0             0 php
[350299.351194] [18067]   500 18067    32116      521   0       0             0 php
[350299.353195] [18069]   500 18069    32347      827   0       0             0 php
[350299.355329] [18070]   500 18070    32281      756   0       0             0 php
[350299.357368] [18072]   500 18072    52608     1915   1       0             0 php
[350299.359529] [18074]   500 18074    32347      820   0       0             0 php
[350299.361566] [18075]   500 18075    30393      519   0       0             0 php
[350299.363873] [18076]     0 18076    26952      264   0       0             0 cpsrvd (SSL) -
[350299.365889] [18078]   500 18078     2754       21   0       0             0 php
[350299.368501] [18079]   500 18079    32248      693   1       0             0 php
[350299.370901] [18080]   500 18080    32182      647   1       0             0 php
[350299.398974] [18082]   500 18082    32182      606   0       0             0 php
[350299.401369] [18084]   500 18084    32347      808   0       0             0 php
[350299.403551] [18085]   500 18085     2754       20   0       0             0 php
[350299.405727] [18105]   500 18105    52351     1829   0       0             0 php
[350299.408946] [18106]   500 18106    32347      801   0       0             0 php
[350299.411218] [18107]     0 18107    35567      105   0       0             0 crond
[350299.413639] [18109]   500 18109    32347      821   0       0             0 php
[350299.417419] [18113]     0 18113    31058      138   0       0             0 clean_techproxy
[350299.419412] [18114]    99 18114   334148     1297   0       0             0 httpd
[350299.421328] [18142]   500 18142    13982       43   0       0             0 php
[350299.423495] [18143]   500 18143     5500       26   1       0             0 php
[350299.425361] [18144]   500 18144     2754       18   0       0             0 php
[350299.428537] [18145]   500 18145     2754       19   0       0             0 php
[350299.431746] [18146]   500 18146     4463       23   0       0             0 php
[350299.433815] [18147]   500 18147     4980       24   0       0             0 php
[350299.435957] [18148]     0 18148    16010      109   1       0             0 sshd
[350299.438044] [18149]   500 18149     5500       23   1       0             0 php
[350299.440161] [18150]     0 18150    35506      123   0       0             0 crond
[350299.443130] [18151]    99 18151     1532       18   1       0             0 suphp
[350299.445118] [18152]   500 18152     3819       21   0       0             0 php
[350299.447159] [18153]   500 18153     5500       26   0       0             0 php
[350299.449159] [18154]   500 18154     5500       25   0       0             0 php
[350299.451606] [18155]   500 18155     5500       26   0       0             0 php
[350299.454054] [18156]    99 18156   334148     1413   0       0             0 httpd
[350299.456683] [18157]   500 18157     2754       18   0       0             0 php
[350299.458958] [18158]    47 18158     5187       29   0       0             0 exim
[350299.460931] [18159]   500 18159     2754       19   0       0             0 php
[350299.464651] [18160]   500 18160     2754       18   0       0             0 php
[350299.466686] [18161]   500 18161     2754       19   0       0             0 php
[350299.468658] [18162]   500 18162     3116       92   1       0             0 suphp
[350299.470853] [18163]   500 18163     2754       18   0       0             0 php
[350299.473249] [18164]    99 18164     2067       19   1       0             0 suphp
[350299.476234] [18165]   500 18165     2754       18   0       0             0 php
[350299.478515] [18166]     0 18166    35506      122   0       0             0 crond
[350299.480537] [18167]     0 18167    35506      121   1       0             0 crond
[350299.483045] [18168]     0 18168    35506      120   1       0             0 crond
[350299.485540] [18169]     0 18169    35506      119   1       0             0 crond
[350299.488173] [18170]     0 18170    35506      119   1       0             0 crond
[350299.491015] [18171]     0 18171    35506      119   0       0             0 crond
[350299.493674] [18172]     0 18172    35506      119   0       0             0 crond
[350299.495627] [18173]     0 18173    35506      119   0       0             0 crond
[350299.497897] [18174]     0 18174    26952      296   0       0             0 cpsrvd (SSL) -
[350299.500545] [18175]    99 18175   334296     1416   0       0             0 httpd
[350299.502775] Out of memory: Kill process 10833 (mysqld) score 38 or sacrifice child
[350299.505430] Killed process 10833, UID 402, (mysqld) total-vm:2262376kB, anon-rss:174524kB, file-rss:172kB

There are a lot of php processes running.

 

[svgr]

Another out of memory log:

[562074.326710] mysqld invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0
[562074.329646] mysqld cpuset=/ mems_allowed=0
[562074.330882] Pid: 21351, comm: mysqld Not tainted 2.6.32-642.1.1.el6.x86_64 #1
[562074.333171] Call Trace:
[562074.334870]  [<ffffffff81131640>] ? dump_header+0x90/0x1b0
[562074.336812]  [<ffffffff8123c20c>] ? security_real_capable_noaudit+0x3c/0x70
[562074.338692]  [<ffffffff81131ac2>] ? oom_kill_process+0x82/0x2a0
[562074.340270]  [<ffffffff81131a01>] ? select_bad_process+0xe1/0x120
[562074.341582]  [<ffffffff81131f00>] ? out_of_memory+0x220/0x3c0
[562074.343229]  [<ffffffff8113e8dc>] ? __alloc_pages_nodemask+0x93c/0x950
[562074.344930]  [<ffffffffa0098350>] ? ext4_get_block+0x0/0x120 [ext4]
[562074.346436]  [<ffffffff81177b2a>] ? alloc_pages_current+0xaa/0x110
[562074.347604]  [<ffffffff8112ea37>] ? __page_cache_alloc+0x87/0x90
[562074.348889]  [<ffffffff8112e41e>] ? find_get_page+0x1e/0xa0
[562074.350660]  [<ffffffff8112f9d7>] ? filemap_fault+0x1a7/0x500
[562074.352207]  [<ffffffff81159394>] ? __do_fault+0x54/0x530
[562074.353872]  [<ffffffff81159967>] ? handle_pte_fault+0xf7/0xb20
[562074.355966]  [<ffffffff8154a705>] ? rwsem_down_failed_common+0x95/0x1d0
[562074.357531]  [<ffffffff8154a896>] ? rwsem_down_read_failed+0x26/0x30
[562074.359503]  [<ffffffff8115a629>] ? handle_mm_fault+0x299/0x3d0
[562074.361105]  [<ffffffff812a6f74>] ? call_rwsem_down_read_failed+0x14/0x30
[562074.362522]  [<ffffffff81052156>] ? __do_page_fault+0x146/0x500
[562074.364590]  [<ffffffff81466a6c>] ? lock_sock_nested+0xac/0xc0
[562074.387996]  [<ffffffff8154ac6b>] ? _spin_unlock_bh+0x1b/0x20
[562074.389187]  [<ffffffff81046f28>] ? pvclock_clocksource_read+0x58/0xd0
[562074.390457]  [<ffffffff81045fbc>] ? kvm_clock_read+0x1c/0x20
[562074.391586]  [<ffffffff81045fc9>] ? kvm_clock_get_cycles+0x9/0x10
[562074.392673]  [<ffffffff810b1b68>] ? getnstimeofday+0x58/0xf0
[562074.394697]  [<ffffffff8154dc8e>] ? do_page_fault+0x3e/0xa0
[562074.395974]  [<ffffffff8154af95>] ? page_fault+0x25/0x30
[562074.397218] Mem-Info:
[562074.399103] Node 0 DMA per-cpu:
[562074.400605] CPU    0: hi:    0, btch:   1 usd:   0
[562074.402129] CPU    1: hi:    0, btch:   1 usd:   0
[562074.403506] Node 0 DMA32 per-cpu:
[562074.405629] CPU    0: hi:  186, btch:  31 usd:   0
[562074.406749] CPU    1: hi:  186, btch:  31 usd:  30
[562074.407914] Node 0 Normal per-cpu:
[562074.409325] CPU    0: hi:  186, btch:  31 usd:   0
[562074.410787] CPU    1: hi:  186, btch:  31 usd:  30
[562074.411941] active_anon:717855 inactive_anon:182000 isolated_anon:5083
[562074.411943]  active_file:32 inactive_file:192 isolated_file:64
[562074.411944]  unevictable:0 dirty:0 writeback:259 unstable:0
[562074.411944]  free:21353 slab_reclaimable:5762 slab_unreclaimable:10906
[562074.411945]  mapped:63 shmem:43 pagetables:21560 bounce:0
[562074.418218] Node 0 DMA free:15724kB min:248kB low:308kB high:372kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15312kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
[562074.424382] lowmem_reserve[]: 0 3512 4017 4017
[562074.426602] Node 0 DMA32 free:61060kB min:58868kB low:73584kB high:88300kB active_anon:2667432kB inactive_anon:524160kB active_file:4kB inactive_file:396kB unevictable:0kB isolated(anon):15980kB isolated(file):384kB present:3596500kB mlocked:0kB dirty:0kB writeback:560kB mapped:232kB shmem:172kB slab_reclaimable:15184kB slab_unreclaimable:17264kB kernel_stack:6736kB pagetables:66680kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:64 all_unreclaimable? no
[562074.432920] lowmem_reserve[]: 0 0 505 505
[562074.434773] Node 0 Normal free:8392kB min:8464kB low:10580kB high:12696kB active_anon:203604kB inactive_anon:203616kB active_file:124kB inactive_file:320kB unevictable:0kB isolated(anon):4992kB isolated(file):128kB present:517120kB mlocked:0kB dirty:0kB writeback:476kB mapped:20kB shmem:0kB slab_reclaimable:7864kB slab_unreclaimable:26360kB kernel_stack:6832kB pagetables:19560kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:32 all_unreclaimable? no
[562074.441562] lowmem_reserve[]: 0 0 0 0
[562074.443251] Node 0 DMA: 3*4kB 0*8kB 0*16kB 1*32kB 1*64kB 0*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15724kB
[562074.449972] Node 0 DMA32: 492*4kB 4530*8kB 511*16kB 140*32kB 89*64kB 3*128kB 1*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 60784kB
[562074.455851] Node 0 Normal: 1455*4kB 68*8kB 3*16kB 2*32kB 0*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 8396kB
[562074.461844] 22462 total pagecache pages
[562074.463320] 21989 pages in swap cache
[562074.464571] Swap cache stats: add 34726956, delete 34704967, find 25405257/28809084
[562074.466666] Free swap  = 0kB
[562074.467793] Total swap = 4194252kB
[562074.508867] 1048575 pages RAM
[562074.510066] 67960 pages reserved
[562074.511918] 7658 pages shared
[562074.513082] 947230 pages non-shared
[562074.514178] [ pid ]   uid  tgid total_vm      rss cpu oom_adj oom_score_adj name
[562074.516236] [  490]     0   490     2680        0   1     -17         -1000 udevd
[562074.518561] [  790]     0   790     2662        0   1     -17         -1000 udevd
[562074.520752] [ 1342]     0  1342    62368       50   1       0             0 rsyslogd
[562074.522842] [ 1357]     0  1357     4563       23   0       0             0 irqbalance
[562074.525130] [ 1376]    25  1376    59678      129   0       0             0 named
[562074.527174] [ 1394]    81  1394     5359        1   0       0             0 dbus-daemon
[562074.529698] [ 1432]     0  1432     1020        0   0       0             0 acpid
[562074.532021] [ 1451]    28  1451   336791      177   0       0             0 nscd
[562074.534053] [ 1569]     0  1569    16086       24   1     -17         -1000 sshd
[562074.536171] [ 1580]    38  1580     6262       27   1       0             0 ntpd
[562074.539282] [ 2053]     0  2053     4662        1   0       0             0 dovecot
[562074.542072] [ 2065]   401  2065    10639        1   0       0             0 pop3-login
[562074.545036] [ 2066]   401  2066    10637        1   0       0             0 imap-login
[562074.547647] [ 2067]    97  2067     3368        1   0       0             0 anvil
[562074.550029] [ 2068]     0  2068     3401        1   1       0             0 log
[562074.552190] [ 2070]   401  2070    10655        1   0       0             0 pop3-login
[562074.554224] [ 2071]   401  2071    10670        1   0       0             0 imap-login
[562074.556886] [ 2072]     0  2072     3659        1   0       0             0 config
[562074.560112] [ 2082]    47  2082    18238        1   1       0             0 exim
[562074.562160] [ 2100]     0  2100    49742      335   1       0             0 /usr/local/cpan
[562074.564197] [ 2102]     0  2102    53119      223   0       0             0 spamd child
[562074.566209] [ 2129]     0  2129    45352        1   0       0             0 abrtd
[562074.568273] [ 2139]     0  2139    45197        1   0       0             0 abrt-dump-oops
[562074.570489] [ 2150]     0  2150    19189      124   0       0             0 httpd
[562074.572607] [ 2165]     0  2165    34191        0   0       0             0 pure-ftpd
[562074.574719] [ 2167]     0  2167    34036        1   0       0             0 pure-authd
[562074.577609] [ 2260]     0  2260    29271       27   1       0             0 crond
[562074.580212] [ 2275]     0  2275     4857        5   1       0             0 atd
[562074.582301] [ 2375]     0  2375    26952      125   0       0             0 cpsrvd (SSL) -
[562074.584483] [ 2381]     0  2381     8172       69   0       0             0 queueprocd - wa
[562074.586500] [ 2391]     0  2391    27506       53   0       0             0 dnsadmin - serv
[562074.589509] [ 2419]     0  2419    20822      258   0       0             0 tailwatchd
[562074.592045] [ 2433]     0  2433    19586        1   0       0             0 cPhulkd - proce
[562074.594295] [ 2476]     0  2476    33028       60   0       0             0 cpdavd - accept
[562074.597210] [ 2512]     0  2512     5915      102   1       0             0 cpanellogd - sl
[562074.600503] [ 2540]     0  2540     1016        1   1       0             0 mingetty
[562074.603689] [ 2542]     0  2542     1016        1   0       0             0 mingetty
[562074.610302] [ 2545]     0  2545     1016        1   1       0             0 mingetty
[562074.615154] [ 2547]     0  2547     2679        0   0     -17         -1000 udevd
[562074.617989] [ 2548]     0  2548     1016        1   1       0             0 mingetty
[562074.622920] [ 2550]     0  2550     1016        1   1       0             0 mingetty
[562074.625853] [ 2552]     0  2552     1016        1   1       0             0 mingetty
[562074.628530] [30499]     0 30499   147473       23   0       0             0 clamd
[562074.631934] [ 8712]     0  8712     2394        1   1       0             0 mysqld_safe
[562074.634865] [ 9408]   402  9408   564397    42943   0       0             0 mysqld
[562074.637620] [24553]     0 24553    27283        1   1       0             0 sshd
[562074.641062] [24562]     0 24562    13893        1   0       0             0 sftp-server
[562074.643785] [25296]     0 25296    27283        1   1       0             0 sshd
[562074.646513] [25312]     0 25312    26649        1   0       0             0 bash
[562074.649571] [25954]    99 25954    19222       60   0       0             0 httpd
[562074.652579] [25955]     0 25955    19061        1   0       0             0 leechprotect
[562074.655407] [25958]    99 25958   334134      795   0       0             0 httpd
[562074.657912] [25959]    99 25959   334193      806   1       0             0 httpd
[562074.660551] [25960]    99 25960   334438      802   0       0             0 httpd
[562074.663110] [26044]    99 26044   334439      803   0       0             0 httpd
[562074.665618] [26206]    99 26206   334191      844   1       0             0 httpd
[562074.668698] [26361]    99 26361   334134      886   0       0             0 httpd
[562074.671241] [28324]     0 28324    27283        1   1       0             0 sshd
[562074.673759] [28333]     0 28333    26649        1   0       0             0 bash
[562074.676257] [28546]    99 28546   334167      821   0       0             0 httpd
[562074.702428] [ 1091]     0  1091    27283        1   1       0             0 sshd
[562074.704538] [ 1130]     0  1130    26649        1   0       0             0 bash
[562074.706652] [ 5276]   500  5276    77474     7801   1       0             0 php
[562074.708866] [ 5278]   500  5278    77582     7649   0       0             0 php
[562074.711662] [ 5281]   500  5281    77580     7259   1       0             0 php
[562074.714235] [ 5290]   500  5290    77601     7474   1       0             0 php
[562074.716490] [ 5295]   500  5295    77538     7540   1       0             0 php
[562074.718824] [ 5299]   500  5299    77414     7551   1       0             0 php
[562074.721384] [ 5302]   500  5302    77518     8104   1       0             0 php
[562074.724114] [ 5304]   500  5304    77414     7522   1       0             0 php
[562074.726653] [ 5308]   500  5308    77582     7241   0       0             0 php
[562074.729482] [ 5309]   500  5309    77542     7294   0       0             0 php
[562074.731758] [ 5367]   500  5367    79215     8223   0       0             0 php
[562074.734948] [ 5369]   500  5369    79230     7073   1       0             0 php
[562074.738000] [ 5389]   500  5389    74670     6863   1       0             0 php
[562074.740609] [ 5407]   500  5407    74996     5832   0       0             0 php
[562074.742809] [ 5415]   500  5415    75850     6233   1       0             0 php
[562074.745813] [ 5416]   500  5416    75061     5948   1       0             0 php
[562074.748265] [ 5418]   500  5418    74992     5796   1       0             0 php
[562074.750694] [ 5420]   500  5420    76190     7332   0       0             0 php
[562074.753004] [ 5430]   500  5430    74932     6113   1       0             0 php
[562074.755384] [ 5434]   500  5434    75460     6480   1       0             0 php
[562074.757639] [ 5439]   500  5439    74928     5879   1       0             0 php
[562074.761562] [ 5440]   500  5440    75125     5701   1       0             0 php
[562074.764306] [ 5442]   500  5442    75914     6693   1       0             0 php
[562074.766600] [ 5446]   500  5446    75214     5475   1       0             0 php
[562074.768955] [ 5448]   500  5448    74705     6309   1       0             0 php
[562074.771372] [ 5450]   500  5450    73976     6564   0       0             0 php
[562074.773778] [ 5451]   500  5451    73831     6459   1       0             0 php
[562074.776178] [ 5458]   500  5458    74611     7603   0       0             0 php
[562074.778666] [ 5459]   500  5459    75781     8240   1       0             0 php
[562074.781649] [ 5465]   500  5465    73772     7310   1       0             0 php
[562074.784412] [ 5470]   500  5470    74414     7797   1       0             0 php
[562074.788349] [ 5476]   500  5476    74864     8214   1       0             0 php
[562074.790625] [ 5480]   500  5480    73510     8366   0       0             0 php
[562074.793613] [ 5482]   500  5482    73708     8698   0       0             0 php
[562074.796689] [ 5492]   500  5492    72444    11027   1       0             0 php
[562074.799184] [ 5496]   500  5496    72784    12415   1       0             0 php
[562074.802301] [ 5499]   500  5499    73768    11388   0       0             0 php
[562074.804993] [ 5500]   500  5500    72582    11302   0       0             0 php
[562074.808597] [ 5503]   500  5503    70752     9687   0       0             0 php
[562074.810941] [ 5513]   500  5513    73447    10543   0       0             0 php
[562074.813474] [ 5519]   500  5519    72728    12005   1       0             0 php
[562074.815596] [ 5520]   500  5520    72710    11134   1       0             0 php
[562074.818435] [ 5523]   500  5523    72246    11313   0       0             0 php
[562074.821278] [ 5533]   500  5533    73576    12705   0       0             0 php
[562074.824227] [ 5536]   500  5536    70305    11674   1       0             0 php
[562074.826292] [ 5541]   500  5541    73378    13119   1       0             0 php
[562074.828638] [ 5545]   500  5545    69769    10932   1       0             0 php
[562074.831461] [ 5548]   500  5548    69382    10296   0       0             0 php
[562074.833803] [ 5549]   500  5549    73458    13978   1       0             0 php
[562074.836589] [ 5564]   500  5564    69708    11383   1       0             0 php
[562074.839962] [ 5568]   500  5568    69380    12267   0       0             0 php
[562074.842304] [ 5570]   500  5570    69580    13324   1       0             0 php
[562074.844533] [ 5574]   500  5574    69057    13993   0       0             0 php
[562074.847082] [ 5577]   500  5577    69450    13875   1       0             0 php
[562074.849309] [ 5578]   500  5578    70048    14829   1       0             0 php
[562074.851653] [ 5587]   500  5587    69514    15274   1       0             0 php
[562074.854140] [ 5589]   500  5589    69382    16196   1       0             0 php
[562074.856537] [ 5593]   500  5593    68012    14763   0       0             0 php
[562074.858820] [ 5597]   500  5597    71343    18344   0       0             0 php
[562074.861408] [ 5598]   500  5598    66115    12838   1       0             0 php
[562074.864285] [ 5603]   500  5603    69319    16732   1       0             0 php
[562074.866510] [ 5612]   500  5612    63573    11969   0       0             0 php
[562074.868753] [ 5613]   500  5613    61574     9964   1       0             0 php
[562074.871009] [ 5616]   500  5616    63637    12150   0       0             0 php
[562074.874032] [ 5624]   500  5624    63726    12707   1       0             0 php
[562074.876563] [ 5625]   500  5625    55276     4013   1       0             0 php
[562074.879156] [ 5630]   500  5630    52951     2211   0       0             0 php
[562074.881476] [ 5631]   500  5631    53736     3070   1       0             0 php
[562074.883968] [ 5633]   500  5633    53245     2474   0       0             0 php
[562074.886305] [ 5634]   500  5634    54533     3868   1       0             0 php
[562074.888585] [ 5635]   500  5635    54650     3839   0       0             0 php
[562074.890872] [ 5636]   500  5636    56802     6104   0       0             0 php
[562074.893228] [ 5637]   500  5637    54189     3375   1       0             0 php
[562074.895530] [ 5638]   500  5638    54563     3901   0       0             0 php
[562074.897784] [ 5639]   500  5639    52667     1958   1       0             0 php
[562074.900122] [ 5640]   500  5640    54533     3711   0       0             0 php
[562074.902475] [ 5641]   500  5641    52922     2197   0       0             0 php
[562074.904697] [ 5642]   500  5642    52667     1954   1       0             0 php
[562074.907013] [ 5643]   500  5643    52958     2226   0       0             0 php
[562074.909381] [ 5644]   500  5644    52667     1927   0       0             0 php
[562074.911628] [ 5645]   500  5645    54015     3402   1       0             0 php
[562074.913914] [ 5646]   500  5646    52951     2165   0       0             0 php
[562074.916614] [ 5647]   500  5647    54635     3863   1       0             0 php
[562074.919109] [ 5648]    99  5648   219408      400   1       0             0 httpd
[562074.921510] [ 5649]   500  5649    53928     3331   1       0             0 php
[562074.924011] [ 5650]   500  5650    56653     5958   0       0             0 php
[562074.926651] [ 5651]   500  5651    52951     2223   0       0             0 php
[562074.929157] [ 5652]   500  5652    52735     1997   1       0             0 php
[562074.954567] [ 5653]   500  5653    52457     1707   1       0             0 php
[562074.957251] [ 5654]    99  5654   235792      481   1       0             0 httpd
[562074.959480] [ 5660]   500  5660    53928     3321   0       0             0 php
[562074.961682] [ 5686]   500  5686    54417     3563   0       0             0 php
[562074.964573] [ 5712]   500  5712    52958     2216   0       0             0 php
[562074.966804] [ 5713]   500  5713    53887     3247   0       0             0 php
[562074.969044] [ 5714]   500  5714    52951     2206   0       0             0 php
[562074.971840] [ 5715]   500  5715    53736     3103   1       0             0 php
[562074.974827] [ 5716]   500  5716    52663     1938   1       0             0 php
[562074.977527] [ 5717]   500  5717    56118     5385   0       0             0 php
[562074.980867] [ 5726]   500  5726    57172     6396   1       0             0 php
[562074.984658] [ 5727]   500  5727    52523     1760   0       0             0 php
[562074.987267] [ 5728]   500  5728    52587     1869   1       0             0 php
[562074.989641] [ 5730]   500  5730    52430     1711   1       0             0 php
[562074.992423] [ 5731]   500  5731    53672     3061   0       0             0 php
[562074.995140] [ 5732]   500  5732    52587     1860   1       0             0 php
[562074.998136] [ 5733]   500  5733    52770     2016   0       0             0 php
[562075.002104] [ 5734]   500  5734    53216     2537   1       0             0 php
[562075.004987] [ 5735]   500  5735    54351     3615   1       0             0 php
[562075.007912] [ 5737]   500  5737    54329     3665   0       0             0 php
[562075.010646] [ 5738]   500  5738    53864     3231   1       0             0 php
[562075.013388] [ 5750]   500  5750    53992     3332   1       0             0 php
[562075.016259] [ 5751]   500  5751    53992     3356   1       0             0 php
[562075.018239] [ 5754]   500  5754    52457     1707   0       0             0 php
[562075.020361] [ 5755]   500  5755    53672     3008   0       0             0 php
[562075.022769] [ 5758]   500  5758    53928     3334   1       0             0 php
[562075.024717] [ 5759]   500  5759    53800     3153   0       0             0 php
[562075.026820] [ 5760]   500  5760    52457     1715   1       0             0 php
[562075.029743] [ 5761]   500  5761    53672     2999   1       0             0 php
[562075.031868] [ 5767]   500  5767    52430     1709   0       0             0 php
[562075.033922] [ 5768]   500  5768    53672     3015   1       0             0 php
[562075.035960] [ 5772]   500  5772    52457     1708   1       0             0 php
[562075.038017] [ 5774]   500  5774    53928     3307   0       0             0 php
[562075.040342] [ 5777]   500  5777    52430     1709   0       0             0 php
[562075.042585] [ 5780]   500  5780    52922     2200   1       0             0 php
[562075.044900] [ 5781]   500  5781    29874      419   0       0             0 php
[562075.046949] [ 5782]   500  5782    52430     1710   0       0             0 php
[562075.049163] [ 5787]   500  5787    53736     3111   0       0             0 php
[562075.051348] [ 5788]   500  5788    52428     1681   0       0             0 php
[562075.053411] [ 5789]   500  5789    16226       45   1       0             0 php
[562075.055455] [ 5790]   500  5790    53992     3361   0       0             0 php
[562075.057385] [ 5796]   500  5796    53800     3145   0       0             0 php
[562075.059354] [ 5802]   500  5802    51933     1164   0       0             0 php
[562075.061403] [ 5803]   500  5803    53736     3088   1       0             0 php
[562075.063319] [ 5828]   500  5828    52457     1712   1       0             0 php
[562075.065328] [ 5830]   500  5830    53992     3362   1       0             0 php
[562075.067304] [ 5833]   500  5833    52457     1711   0       0             0 php
[562075.069272] [ 5834]   500  5834    52457     1710   0       0             0 php
[562075.071342] [ 5838]   500  5838    25576       65   0       0             0 php
[562075.073811] [ 5839]    99  5839   121104      248   1       0             0 httpd
[562075.075876] [ 5848]    99  5848   121104      279   1       0             0 httpd
[562075.078224] [ 5849]   500  5849    29230      282   1       0             0 php
[562075.080532] [ 5850]   500  5850    29118      217   1       0             0 php
[562075.082861] [ 5851]   500  5851    53352     2643   0       0             0 php
[562075.084933] [ 5852]    99  5852   121104      281   1       0             0 httpd
[562075.087091] [ 5854]   500  5854    52430     1708   0       0             0 php
[562075.089180] [ 5856]   500  5856    51703      882   0       0             0 php
[562075.091154] [ 5863]    99  5863   121104      286   1       0             0 httpd
[562075.093112] [ 5970]   500  5970    28797      186   1       0             0 php
[562075.095140] [ 5986]     0  5986     5651       60   0       0             0 upcp-running
[562075.097396] [ 5988]   500  5988    29118      223   0       0             0 php
[562075.099621] [ 5991]    99  5991   104720      241   0       0             0 httpd
[562075.101929] [ 5993]   500  5993     3116       43   0       0             0 suphp
[562075.104055] [ 5994]   500  5994    29329      408   0       0             0 php
[562075.106361] [ 6026]   500  6026    29118      223   0       0             0 php
[562075.108660] [ 6027]   500  6027    29118      231   1       0             0 php
[562075.111327] [ 6028]   500  6028     7365       23   1       0             0 php
[562075.113563] [ 6032]    99  6032   121104      279   1       0             0 httpd
[562075.115682] Out of memory: Kill process 9408 (mysqld) score 44 or sacrifice child
[562075.117786] Killed process 9408, UID 402, (mysqld) total-vm:2257588kB, anon-rss:171860kB, file-rss:544kB

PHP seems to be eating all the swap, and kicking mysql out of memory. Any suggestions on what optimizations I should do?

 

[cPanelMichael]

Hello,

How much memory is available on the system? You can check this with a command such as:

free -m

Thank you.

 

[svgr]

Hello,

How much memory is available on the system? You can check this with a command such as:

free -m

Thank you.

free -m output:

             total       used       free     shared    buffers     cached

Mem:          3830       1445       2385          0         23        438

-/+ buffers/cache:        983       2847

Swap:         4095        227       3868

 

[cPanelMichael]

Hello,

Have you noticed any changes since adjusting your MySQL configuration after running the tuner? Have you tried running the tuner again after making the adjustments to your /etc/my.cnf file after the last tuner run?

Thank you.

 

[svgr]

Hello,

Have you noticed any changes since adjusting your MySQL configuration after running the tuner? Have you tried running the tuner again after making the adjustments to your /etc/my.cnf file after the last tuner run?

Thank you.

I took the mysqltuner suggestions about 2-3 times and faced repeated failure, i.e. the out of memory error would occur twice everyday when making modifications based on the tuner. I read some articles and increased table_open_cache to 4000 from 450 and reduced the key_buffer_size to 384M from 624M. I'm no server admin so I don't know if this was the right move or not.

According to the out of memory log, too many php processes are spawned. On examination during a crash, it appears that they were WordPress' index.php processes, reaching CPU usage of 100% twice everyday. I reduced the amount of memory allotted to php, and adjusted the mysql config as mentioned above. This has reduced the frequency of the out of memory error, but not quite solved the problem. I would imagine the problem requires the correct php configuration, WordPress optimisation, and the correct mysql configuration for my website. I'm working on these three with whatever knowledge I can find. Would appreciate any inputs on the configuration front. Thanks!

 

[cPanelMichael]

Hello,

You may also want to consider using a third-party application such as Cloud Linux to manage and control the amount of memory/CPU available to each account if you plan on hosting additional accounts in the future. Another consideration is increasing the amount of memory or CPU power on the system.

Thank you.

 

[twhiting9275]

Though cloudlinux is great option, I would personally go for moving to Litespeed in this specific scenario.
The reason for this is the php processes. These will be limited by Litespeed , far better than CL can, honestly. cPanel (by default) doesn't even bother to limit them.