Out of memory

Everton · Oct 15, 2015

hello, I'm getting this notification by email.
How to solve?

----------------------------------------------------------------------------------------------------
Out of memory: A process was terminated because the system is low on memory.

In order to avoid a system crash due to low memory, the kernel terminated process ID “623315”.

cPanelMichael · Oct 15, 2015

Hello

Could you post the full output of the email in CODE tags, but with actual domain names, IP addresses, and any identifying information removed?

Thank you.

Everton · Oct 16, 2015

ok, is as follows:

Code:

-------------------------------------------------------------------------------------------------------------------------
Out of memory: A process was terminated because the system is low on memory.

In order to avoid a system crash due to low memory, the kernel terminated process ID “713103”.

[B]Server[/B] cloud.example.com
[B]Primary IP Address[/B] 104.200.20.149
[B]Event Time[/B] Friday, October 16, 2015 8:10:01 AM UTC
[B]Process ID[/B] 713103
[B]Process Total Virtual Memory[/B] 166676kB
[B]Process Anonymous Resident Set Size[/B] 52332kB
[B]Process File Resident Set Size[/B] 6604kB
[B]Status[/B] Out of Memory 
[B]Memory Information[/B]
Used 1.69 GB
Available 2.01 GB
Installed 3.7 GB
[B]Load Information[/B] 0.07 0.14 0.13
[B]Uptime[/B] 6 days, 5 hours, 17 minutes, and 3 seconds
[B]IOStat Information[/B] avg-cpu:  %user  %nice %system %iowait  %steal  %idle           3.89    0.60    0.58    0.07    0.00  94.86 Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn sdb              1.54        9.16        12.08    4923424    6491220 sda              24.34      1058.29      336.38  568747229  180776912
[B]Top Processes
PID[/B] [B]Owner[/B] [B]CPU %[/B] [B]Memory %[/B] [B]Command[/B]
817 mysql 0.46 19.04 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib64/mysql/plugin --log-error=/var/lib/mysql/cloud.compactaweb.com.err --open-files-limit=10000 --pid-file=/var/lib/mysql/cloud.compactaweb.com.pid
559472 root 0.03 8.00 /usr/local/cpanel/3rdparty/bin/clamd
360879 compact 9.00 3.31 php -q /home/compact/public_html/central/crons/cron.php
559167 root 0.02 1.45 /usr/local/cpanel/3rdparty/perl/514/bin/spamd --daemonize --max-spare=1 --max-children=3 --allowed-ips=127.0.0.1 --pidfile=/var/run/spamd.pid -i127.0.0.1
658710 root 0.02 1.45 spamd child
For addtional details, please see the attached dmesg log dump.



[B] Preview of “oom_dmesg.txt” [/B]
[537233.224560] [710168] 99 710168 23293 1546 46 83 0 httpd
[537233.226256] [710231] 99 710231 23293 1534 46 88 0 httpd
[537233.227836] [710257] 99 710257 23289 1527 46 83 0 httpd
[537233.229391] [713099] 1017 713099 41670 14733 84 0 0 php
[537233.231255] [713100] 1017 713100 41666 14726 81 1 0 php
[537233.233086] [713101] 1017 713101 41669 14733 83 0 0 php
[537233.234958] [713102] 1017 713102 41666 14677 82 0 0 php
[537233.236801] [713103] 1017 713103 41669 14734 83 0 0 php
[537233.238654] Memory cgroup out of memory: Kill process 713103 (php) score 75 or sacrifice child
[537233.240647] Killed process 713103 (php) total-vm:166676kB, anon-rss:52332kB, file-rss:6604kB
Configure Monitor Settings:
[URL]https://cloud.example.com:2087/scripts/srvmng#service-chkservd[/URL]

Configure chkservd:
[URL]https://cloud.example.com:2087/scripts2/tweaksettings?find=chkservd[/URL]

Disable HTML notifications:
[URL]https://cloud.example.com:2087/scripts2/tweaksettings?find=chkservd_plaintext_notify[/URL]

This notice was generated “Friday, October 16, 2015 8:13:11 AM UTC”.

“Out of Memory” notifications are currently configured to have an importance of “High”. You can change the importance or disable this type of notification in WHM’s Contact Manager at: [URL]https://cloud.example.com:2087/scripts2/editcontact?event=chkservd::OOM[/URL]



Copyright© 2015 cPanel, Inc.




oom_dmesg.txt

[537233.151869] php invoked oom-killer: gfp_mask=0xd0, order=0, oom_score_adj=0
[537233.153676] php cpuset=1017 mems_allowed=0
[537233.154884] CPU: 2 PID: 713100 Comm: php ve: 0 Tainted: PF O-------------- 3.10.0-233.1.2.lve1.3.33.4.el7.x86_64 #1 ovz.4.1
[537233.157696] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.8.2-0-g33fbe13 by qemu-project.org 04/01/2014
[537233.160190] 0000000000000000 00000000547ab0d0 ffff880082d1d800 ffffffff815d3536
[537233.162122] ffff880082d1d890 ffffffff815cf3f4 0000000000000000 00000000000200da
[537233.164238] 0000000000000001 ffff880082d1d860 ffffffff81154a96 ffff880036396230
[537233.166343] Call Trace:
[537233.167203] [<ffffffff815d3536>] dump_stack+0x19/0x1b
[537233.168463] [<ffffffff815cf3f4>] dump_header+0x8e/0x214
[537233.169701] [<ffffffff81154a96>] ? find_lock_task_mm+0x56/0xc0
[537233.171017] [<ffffffff81154f2e>] oom_kill_process+0x24e/0x3b0
[537233.172365] [<ffffffff81154a96>] ? find_lock_task_mm+0x56/0xc0
[537233.173666] [<ffffffff8106b670>] ? has_capability_noaudit+0x20/0x30
[537233.175190] [<ffffffff811ba5ce>] __mem_cgroup_try_charge+0xb9e/0xbe0
[537233.176671] [<ffffffff811bae80>] ? mem_cgroup_charge_common+0xc0/0xc0
[537233.178078] [<ffffffff811bae19>] mem_cgroup_charge_common+0x59/0xc0
[537233.179465] [<ffffffff811bbfba>] mem_cgroup_cache_charge+0x8a/0xb0
[537233.180879] [<ffffffff811518e2>] __add_to_page_cache_locked+0x52/0x270
[537233.182329] [<ffffffff81151b57>] add_to_page_cache_lru+0x37/0xb0
[537233.183674] [<ffffffff81203085>] mpage_readpages+0xb5/0x160
[537233.184995] [<ffffffffa00eb0f0>] ? _ext4_get_block+0x1b0/0x1b0 [ext4]
[537233.186368] [<ffffffffa00eb0f0>] ? _ext4_get_block+0x1b0/0x1b0 [ext4]
[537233.187795] [<ffffffffa01d2691>] ? iolimit_virtinfo+0x381/0x410 [iolimits]
[537233.189374] [<ffffffffa00e93fa>] ext4_readpages+0x3a/0x50 [ext4]
[537233.190740] [<ffffffff8115da2d>] __do_page_cache_readahead+0x1dd/0x260
[537233.192182] [<ffffffff8115e181>] ra_submit+0x21/0x30
[537233.193389] [<ffffffff811539fe>] filemap_fault+0x2fe/0x490
[537233.194725] [<ffffffff8117a8c8>] __do_fault+0x88/0x570
[537233.195991] [<ffffffff8117fb85>] handle_mm_fault+0x3c5/0x7b0
[537233.196917] [<ffffffff815d8132>] ? down_write+0x12/0x30
[537233.197785] [<ffffffff81176fab>] ? anon_vma_interval_tree_insert+0x8b/0x90
[537233.198861] [<ffffffff815dedbe>] __do_page_fault+0x15e/0x530
[537233.199756] [<ffffffff81184352>] ? vma_merge+0x232/0x340
[537233.200633] [<ffffffff811833d9>] ? __vm_enough_memory+0x29/0x170
[537233.201931] [<ffffffff8118545b>] ? do_brk+0x14b/0x390
[537233.203108] [<ffffffff815df1aa>] do_page_fault+0x1a/0x70
[537233.204326] [<ffffffff815de869>] do_async_page_fault+0x29/0xe0
[537233.205614] [<ffffffff815db438>] async_page_fault+0x28/0x30
[537233.206799] Task in /1017 killed as a result of limit of /1017
[537233.208521] memory: usage 262144kB, limit 262144kB, failcnt 258495
[537233.209908] memory+swap: usage 262160kB, limit 9007199254740991kB, failcnt 0
[537233.211369] kmem: usage 0kB, limit 9007199254740991kB, failcnt 0
[537233.212708] Memory cgroup stats for /1017: cache:56KB rss:262088KB rss_huge:0KB mapped_file:0KB swap:16KB inactive_anon:88672KB active_anon:173288KB inactive_file:52KB active_file:0KB unevictable:0KB
[537233.220134] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
[537233.221630] [710166] 99 710166 23289 1542 46 83 0 httpd
[537233.223084] [710167] 99 710167 23293 1532 46 83 0 httpd
[537233.224560] [710168] 99 710168 23293 1546 46 83 0 httpd
[537233.226256] [710231] 99 710231 23293 1534 46 88 0 httpd
[537233.227836] [710257] 99 710257 23289 1527 46 83 0 httpd
[537233.229391] [713099] 1017 713099 41670 14733 84 0 0 php
[537233.231255] [713100] 1017 713100 41666 14726 81 1 0 php
[537233.233086] [713101] 1017 713101 41669 14733 83 0 0 php
[537233.234958] [713102] 1017 713102 41666 14677 82 0 0 php
[537233.236801] [713103] 1017 713103 41669 14734 83 0 0 php
[537233.238654] Memory cgroup out of memory: Kill process 713103 (php) score 75 or sacrifice child
[537233.240647] Killed process 713103 (php) total-vm:166676kB, anon-rss:52332kB, file-rss:6604kB

cPanelMichael · Oct 19, 2015

That output indicates a PHP process was the culprit of the memory usage. The notification indicates the process was automatically killed to prevent that process from crashing the system due to a lack of memory. Was this a one-time notification or does it continue to occur?

Thank you.

Jeffrey Anthony Ansen · Jan 16, 2016

cPanelMichael said: ↑

That output indicates a PHP process was the culprit of the memory usage. The notification indicates the process was automatically killed to prevent that process from crashing the system due to a lack of memory. Was this a one-time notification or does it continue to occur?

Thank you.
Click to expand...

Hi Michael, sorry to jump in but I have the same situation like Everton. It continue to occur

So what might happened? Am I getting a brute force attack or my actually memory was too low (it's only 1GB)?

Can I copy paste full output here?

cPanelMichael · Jan 18, 2016

Could you elaborate on which type of processes are killed? How frequent do you receive this notification?

Thank you.

mageshm · Feb 18, 2016

@ Jeffrey Anthony Ansen,
Are you facing out of memory error very often or rarely ? It seem's PHP process are eating entire server RAM from below account as per your email output.

360879 compact 9.00 3.31 php -q /home/compact/public_html/central/crons/cron.php

I advise you to check or closely watch the particular account activity. I can see cron.php file is culprit, so check whether they configured cron every 5 mins or anything is abnormal on cron jobs.

Hope this will help you to identify your issue.

Forums

The Community Forums

Interact with an entire community of cPanel & WHM users!

Out of memory

Everton Registered

cPanelMichael Forums Analyst
Staff Member

Everton Registered

cPanelMichael Forums Analyst
Staff Member

Jeffrey Anthony Ansen Member

cPanelMichael Forums Analyst
Staff Member

mageshm Well-Known Member

Duplicated database causes memory error when connecting.

MySQL processes using high memory

Hourly CPU and Memory usage record

High Physical memory usage even at no traffic

Server Slow after update memory through php.ini

Share This Page

Useful Searches

The Community Forums

Interact with an entire community of cPanel & WHM users!

Out of memory

Everton Registered

cPanelMichael Forums Analyst Staff Member

Everton Registered

cPanelMichael Forums Analyst Staff Member

Jeffrey Anthony Ansen Member

cPanelMichael Forums Analyst Staff Member

mageshm Well-Known Member

Duplicated database causes memory error when connecting.

MySQL processes using high memory

Hourly CPU and Memory usage record

High Physical memory usage even at no traffic

Server Slow after update memory through php.ini

Share This Page

cPanelMichael Forums Analyst
Staff Member

cPanelMichael Forums Analyst
Staff Member

cPanelMichael Forums Analyst
Staff Member