The Community Forums

Interact with an entire community of cPanel & WHM users!
  1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

weird out of memory cgroups issue

Discussion in 'Workarounds and Optimization' started by sahostking, Feb 5, 2016.

  1. sahostking

    sahostking Well-Known Member

    Joined:
    May 15, 2012
    Messages:
    299
    Likes Received:
    1
    Trophy Points:
    18
    Location:
    Cape Town, South Africa
    cPanel Access Level:
    Root Administrator
    Mysql seems to randomly go down on 2 servers we have with 10Gb. We can increase memory or any resources if required but don't think that will help. We already a few GB a week ago. Same thing

    I see this in logs the whole day? free memory on server is 7GB which is weird. But for no apprent reason mysql just dies, then shortly thereafter apache.

    Wonder if its related to this?
    Code:
    Feb  5 20:00:02 lin02 pam_lve[5494]: (CRON) CAN'T FORK (child_process): Cannot allocate memory
    Feb  5 20:00:02 lin02 pam_lve[5448]: (CRON) CAN'T FORK (child_process): Cannot allocate memory
    Feb  5 20:00:02 lin02 pam_lve[5501]: (CRON) CAN'T FORK (child_process): Cannot allocate memory
    Feb  5 20:00:02 lin02 pam_lve[5438]: (CRON) CAN'T FORK (child_process): Cannot allocate memory
    Feb  5 20:00:02 lin02 pam_lve[5495]: (CRON) CAN'T FORK (child_process): Cannot allocate memory
    Feb  5 20:00:02 lin02 pam_lve[5457]: (CRON) CAN'T FORK (child_process): Cannot allocate memory
    Feb  5 20:00:02 lin02 pam_lve[5447]: (CRON) CAN'T FORK (child_process): Cannot allocate memory
    Feb  5 20:00:02 lin02 pam_lve[5432]: (CRON) CAN'T FORK (child_process): Cannot allocate memory
    Feb  5 20:00:02 lin02 pam_lve[5475]: (CRON) CAN'T FORK (child_process): Cannot allocate memory
    Feb  5 20:00:02 lin02 pam_lve[5431]: (CRON) CAN'T FORK (child_process): Cannot allocate memory
    Feb  5 20:00:02 lin02 pam_lve[5471]: (CRON) CAN'T FORK (child_process): Cannot allocate memory
    Feb  5 20:00:02 lin02 pam_lve[5395]: (CRON) CAN'T FORK (child_process): Cannot allocate memory
    Feb  5 20:12:19 lin02 kernel: memory: usage 1048576kB, limit 1048576kB, failcnt 4118
    Feb  5 20:12:19 lin02 kernel: memory+swap: usage 1048576kB, limit 1048576kB, failcnt 0
    Feb  5 20:12:19 lin02 kernel: Memory cgroup out of memory: Kill process 84642 (php) score 78 or sacrifice child
    Feb  5 20:12:21 lin02 kernel: memory: usage 1048576kB, limit 1048576kB, failcnt 4882
    Feb  5 20:12:21 lin02 kernel: memory+swap: usage 1048576kB, limit 1048576kB, failcnt 0
    Feb  5 20:12:21 lin02 kernel: Memory cgroup out of memory: Kill process 84415 (php) score 80 or sacrifice child
    Feb  5 20:12:22 lin02 kernel: memory: usage 1048576kB, limit 1048576kB, failcnt 5503
    Feb  5 20:12:22 lin02 kernel: memory+swap: usage 1048576kB, limit 1048576kB, failcnt 0
    Feb  5 20:12:22 lin02 kernel: Memory cgroup out of memory: Kill process 84412 (php) score 71 or sacrifice child
    Feb  5 20:12:23 lin02 kernel: memory: usage 1048576kB, limit 1048576kB, failcnt 6148
    Feb  5 20:12:23 lin02 kernel: memory+swap: usage 1048576kB, limit 1048576kB, failcnt 0
    Feb  5 20:12:23 lin02 kernel: Memory cgroup out of memory: Kill process 84417 (php) score 80 or sacrifice child
    Feb  5 20:12:24 lin02 kernel: memory: usage 1048576kB, limit 1048576kB, failcnt 6535
    Feb  5 20:12:24 lin02 kernel: memory+swap: usage 1048576kB, limit 1048576kB, failcnt 0
    Feb  5 20:12:24 lin02 kernel: Memory cgroup out of memory: Kill process 84468 (php) score 70 or sacrifice child
    
    
     
    #1 sahostking, Feb 5, 2016
    Last edited by a moderator: Feb 6, 2016
  2. sahostking

    sahostking Well-Known Member

    Joined:
    May 15, 2012
    Messages:
    299
    Likes Received:
    1
    Trophy Points:
    18
    Location:
    Cape Town, South Africa
    cPanel Access Level:
    Root Administrator
    Hi

    I tried to disable cgroups as per error logs, in adding cgroup_disable=memory into grub.conf line it works great and server stays on without omkiller or reboot but Cloudlinux LVE does not want to start because if it. I'm thinking its a requirement somehow.
    If I take cgroups_disable out then it works again fine ad lve starts up.

    Feb 6 14:26:11 lin02 kernel: init_memory_mapping: [mem 0x00000000-0x000fffff]
    Feb 6 14:26:11 lin02 kernel: init_memory_mapping: [mem 0x43fe00000-0x43fffffff]
    Feb 6 14:26:11 lin02 kernel: init_memory_mapping: [mem 0x43c000000-0x43fdfffff]
    Feb 6 14:26:11 lin02 kernel: init_memory_mapping: [mem 0x400000000-0x43bffffff]
    Feb 6 14:26:11 lin02 kernel: init_memory_mapping: [mem 0x00100000-0xbffdffff]
    Feb 6 14:26:11 lin02 kernel: init_memory_mapping: [mem 0x100000000-0x3ffffffff]
    Feb 6 14:26:11 lin02 kernel: Reserving 161MB of memory at 688MB for crashkernel (System RAM: 16383MB)
    Feb 6 14:26:11 lin02 kernel: Early memory node ranges
    Feb 6 14:26:11 lin02 kernel: PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff]
    Feb 6 14:26:11 lin02 kernel: PM: Registered nosave memory: [mem 0x000a0000-0x000effff]
    Feb 6 14:26:11 lin02 kernel: PM: Registered nosave memory: [mem 0x000f0000-0x000fffff]
    Feb 6 14:26:11 lin02 kernel: PM: Registered nosave memory: [mem 0xbffe0000-0xbfffffff]
    Feb 6 14:26:11 lin02 kernel: PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff]
    Feb 6 14:26:11 lin02 kernel: PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff]
    Feb 6 14:26:11 lin02 kernel: PM: Registered nosave memory: [mem 0xff000000-0xfffbffff]
    Feb 6 14:26:11 lin02 kernel: PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff]
    Feb 6 14:26:11 lin02 kernel: please try 'cgroup_disable=memory' option if you don't want memory cgroups
    Feb 6 14:26:11 lin02 kernel: Initializing cgroup subsys memory
    Feb 6 14:26:11 lin02 kernel: Freeing initrd memory: 16756k freed
    Feb 6 14:26:11 lin02 kernel: Non-volatile memory driver v1.3
    Feb 6 14:26:11 lin02 kernel: crash memory driver: version 1.1
    Feb 6 14:26:11 lin02 kernel: Freeing unused kernel memory: 1636k freed
    Feb 6 14:26:11 lin02 kernel: [TTM] Zone kernel: Available graphics memory: 8131870 kiB
    Feb 6 14:26:11 lin02 kernel: [TTM] Zone dma32: Available graphics memory: 2097152 kiB
    Feb 6 14:30:02 lin02 pam_lve[5739]: (CRON) CAN'T FORK (child_process): Cannot allocate memory

    Feb 6 14:30:04 lin02 crond: /usr/sbin/sendmail: Cannot allocate memory
    Feb 6 14:30:04 lin02 crond: /usr/sbin/sendmail: Cannot allocate memory
    Feb 6 14:30:04 lin02 crond: /usr/sbin/sendmail: Cannot allocate memory
    Feb 6 14:30:04 lin02 crond: /usr/sbin/sendmail: Cannot allocate memory
    Feb 6 14:30:04 lin02 crond: /usr/sbin/sendmail: Cannot allocate memory
    Feb 6 14:30:04 lin02 crond: /usr/sbin/sendmail: Cannot allocate memory
    Feb 6 15:00:02 lin02 pam_lve[18335]: (CRON) CAN'T FORK (child_process): Cannot allocate memory
    Feb 6 15:00:02 lin02 pam_lve[18333]: (CRON) CAN'T FORK (child_process): Cannot allocate memory
    Feb 6 15:00:02 lin02 pam_lve[18312]: (CRON) CAN'T FORK (child_process): Cannot allocate memory
    Feb 6 15:00:02 lin02 pam_lve[18310]: (CRON) CAN'T FORK (child_process): Cannot allocate memory

    Memory is not an issue as at all times 4GB is available for the 10Gb. I've even increased it now to 16GB from 10GB lets see
     
  3. cPanelMichael

    cPanelMichael Forums Analyst
    Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    30,678
    Likes Received:
    654
    Trophy Points:
    113
    cPanel Access Level:
    Root Administrator
    Hello :)

    Are you utilizing the MySQL Governor application with Cloud Linux? Also, this thread is a few years old but it may include information that's helpful to you:

    Application Resource Limits In Cloudlinux

    Thank you.
     
Loading...

Share This Page