Please whitelist cPanel in your adblocker so that you’re able to see our version release promotions, thanks!

The Community Forums

Interact with an entire community of cPanel & WHM users!
  1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Out of memory cpbackup_transp

Discussion in 'Data Protection' started by jimhermann, Sep 12, 2017.

Tags:
  1. jimhermann

    jimhermann Well-Known Member

    Joined:
    Jan 20, 2008
    Messages:
    51
    Likes Received:
    1
    Trophy Points:
    58
    cPanel Community,

    How do I solve this "Out of Memory" problem? I just started using AWS S3 for backups.

    Sep 12 06:41:55 host3 kernel: [310591.852931] httpd invoked oom-killer: gfp_mask=0x24280ca(GFP_HIGHUSER_MOVABLE|__GFP_ZERO), nodemask=0, order=0, oom_score_adj=0
    Sep 12 06:41:55 host3 kernel: [310591.859756] httpd cpuset=/ mems_allowed=0
    Sep 12 06:41:55 host3 kernel: [310591.862377] CPU: 1 PID: 628 Comm: httpd Tainted: G E 4.9.27-14.31.amzn1.x86_64 #1
    Sep 12 06:41:55 host3 kernel: [310591.865290] Hardware name: Xen HVM domU, BIOS 4.2.amazon 02/16/2017
    Sep 12 06:41:55 host3 kernel: [310591.865290] ffffc90001d63b50 ffffffff812f9aff ffffc90001d63cf0 ffff880105a28000
    Sep 12 06:41:55 host3 kernel: [310591.865290] ffffc90001d63be0 ffffffff811f480b 0000000000000000 0000000000000000
    Sep 12 06:41:55 host3 kernel: [310591.865290] ffffc90001d63c08 ffffc90001d63b88 ffffffff811237ea ffffc90001d63bf0
    Sep 12 06:41:55 host3 kernel: [310591.865290] Call Trace:
    Sep 12 06:41:55 host3 kernel: [310591.865290] [<ffffffff812f9aff>] dump_stack+0x63/0x84
    Sep 12 06:41:55 host3 kernel: [310591.865290] [<ffffffff811f480b>] dump_header+0x82/0x212
    Sep 12 06:41:55 host3 kernel: [310591.865290] [<ffffffff811237ea>] ? __delayacct_freepages_end+0x2a/0x30
    Sep 12 06:41:55 host3 kernel: [310591.865290] [<ffffffff8118e52a>] ? do_try_to_free_pages+0x2da/0x340
    Sep 12 06:41:55 host3 kernel: [310591.865290] [<ffffffff8117a6bc>] oom_kill_process+0x21c/0x3f0
    Sep 12 06:41:55 host3 kernel: [310591.865290] [<ffffffff8117ab58>] out_of_memory+0x108/0x4b0
    Sep 12 06:41:55 host3 kernel: [310591.865290] [<ffffffff8117f4d0>] __alloc_pages_slowpath+0x9a0/0xb90
    Sep 12 06:41:55 host3 kernel: [310591.865290] [<ffffffff8117f8a3>] __alloc_pages_nodemask+0x1e3/0x250
    Sep 12 06:41:55 host3 kernel: [310591.865290] [<ffffffff811cf8d5>] alloc_pages_vma+0xa5/0x220
    Sep 12 06:41:55 host3 kernel: [310591.865290] [<ffffffff811ac4d3>] handle_mm_fault+0x11b3/0x13a0
    Sep 12 06:41:55 host3 kernel: [310591.865290] [<ffffffff81060fe5>] __do_page_fault+0x225/0x4a0
    Sep 12 06:41:55 host3 kernel: [310591.865290] [<ffffffff81061282>] do_page_fault+0x22/0x30
    Sep 12 06:41:55 host3 kernel: [310591.865290] [<ffffffff815349f8>] page_fault+0x28/0x30
    Sep 12 06:41:55 host3 kernel: [310591.927529] Mem-Info:
    Sep 12 06:41:55 host3 kernel: [310591.928790] active_anon:957468 inactive_anon:87 isolated_anon:0
    Sep 12 06:41:55 host3 kernel: [310591.928790] active_file:386 inactive_file:385 isolated_file:0
    Sep 12 06:41:55 host3 kernel: [310591.928790] unevictable:0 dirty:1 writeback:1 unstable:0
    Sep 12 06:41:55 host3 kernel: [310591.928790] slab_reclaimable:5315 slab_unreclaimable:5612
    Sep 12 06:41:55 host3 kernel: [310591.928790] mapped:407 shmem:109 pagetables:6378 bounce:0
    Sep 12 06:41:55 host3 kernel: [310591.928790] free:20960 free_pcp:14 free_cma:0
    Sep 12 06:41:55 host3 kernel: [310591.947497] Node 0 active_anon:3829872kB inactive_anon:348kB active_file:1544kB inactive_file:1540kB unevictable:0kB isolated(anon):0kB isola
    ted(file):0kB mapped:2228kB dirty:4kB writeback:4kB shmem:436kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB unstable:0kB pages_scanned:4678 all_unrecla
    imable? yes
    Sep 12 06:41:55 host3 kernel: [310591.962946] Node 0 DMA free:15904kB min:264kB low:328kB high:392kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevic
    table:0kB writepending:0kB present:15988kB managed:15904kB mlocked:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB loca
    l_pcp:0kB free_cma:0kB
    Sep 12 06:41:55 host3 kernel: [310591.979579] lowmem_reserve[]: 0 3730 3920 3920
    Sep 12 06:41:55 host3 kernel: [310591.982286] Node 0 DMA32 free:64712kB min:64064kB low:80080kB high:96096kB active_anon:3679980kB inactive_anon:260kB active_file:1232kB inact
    ive_file:1216kB unevictable:0kB writepending:8kB present:3915776kB managed:3838136kB mlocked:0kB slab_reclaimable:12384kB slab_unreclaimable:11316kB kernel_stack:2368kB pageta
    bles:21776kB bounce:0kB free_pcp:552kB local_pcp:0kB free_cma:0kB
    Sep 12 06:41:55 host3 kernel: [310592.001978] lowmem_reserve[]: 0 0 189 189
    Sep 12 06:41:55 host3 kernel: [310592.004358] Node 0 Normal free:3224kB min:3252kB low:4064kB high:4876kB active_anon:149864kB inactive_anon:88kB active_file:76kB inactive_fil
    e:0kB unevictable:0kB writepending:0kB present:262144kB managed:193976kB mlocked:0kB slab_reclaimable:8876kB slab_unreclaimable:11132kB kernel_stack:1744kB pagetables:3736kB b
    ounce:0kB free_pcp:56kB local_pcp:56kB free_cma:0kB
    Sep 12 06:41:55 host3 kernel: [310592.018456] lowmem_reserve[]: 0 0 0 0
    Sep 12 06:41:55 host3 kernel: [310592.020373] Node 0 DMA: 0*4kB 0*8kB 0*16kB 1*32kB (U) 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 159
    04kB
    Sep 12 06:41:55 host3 kernel: [310592.026825] Node 0 DMA32: 383*4kB (UME) 260*8kB (UME) 645*16kB (UE) 370*32kB (UME) 243*64kB (UME) 119*128kB (UME) 24*256kB (UME) 0*512kB 0*10
    24kB 1*2048kB (H) 0*4096kB = 64748kB
    Sep 12 06:41:55 host3 kernel: [310592.034403] Node 0 Normal: 138*4kB (MEH) 96*8kB (UMEH) 3*16kB (H) 2*32kB (H) 2*64kB (H) 1*128kB (H) 2*256kB (H) 0*512kB 1*1024kB (H) 0*2048kB
    0*4096kB = 3224kB
    Sep 12 06:41:55 host3 kernel: [310592.041505] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
    Sep 12 06:41:55 host3 kernel: [310592.044907] 758 total pagecache pages
    Sep 12 06:41:55 host3 kernel: [310592.046403] 0 pages in swap cache
    Sep 12 06:41:55 host3 kernel: [310592.047768] Swap cache stats: add 0, delete 0, find 0/0
    Sep 12 06:41:55 host3 kernel: [310592.049858] Free swap = 0kB
    Sep 12 06:41:55 host3 kernel: [310592.051456] Total swap = 0kB
    Sep 12 06:41:55 host3 kernel: [310592.053046] 1048477 pages RAM
    Sep 12 06:41:55 host3 kernel: [310592.054745] 0 pages HighMem/MovableOnly
    Sep 12 06:41:55 host3 kernel: [310592.056517] 36473 pages reserved
    *****
    Sep 12 06:41:55 host3 kernel: [310592.486206] Out of memory: Kill process 22990 (cpbackup_transp) score 618 or sacrifice child
    Sep 12 06:41:55 host3 kernel: [310592.489653] Killed process 22990 (cpbackup_transp) total-vm:2863200kB, anon-rss:2576220kB, file-rss:64kB, shmem-rss:0kB
     
  2. cPanelMichael

    cPanelMichael Forums Analyst
    Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    38,165
    Likes Received:
    1,371
    Trophy Points:
    363
    cPanel Access Level:
    Root Administrator
    Hello,

    Do you notice the server is reaching it's memory limit when this happens? EX:

    Code:
    free -m
    Thank you.
     
  3. jimhermann

    jimhermann Well-Known Member

    Joined:
    Jan 20, 2008
    Messages:
    51
    Likes Received:
    1
    Trophy Points:
    58
    No. I was not connected at the time. CPanel Monitoring sent me this report:

    Out of memory: The process “cpbackup_transp” was terminated because the system is low on memory.
    In order to avoid a system crash due to low memory, the kernel terminated the process named “cpbackup_transp” with the PID “22990”.

    Server host3.uuserver.net
    Primary IP Address 172.31.13.215
    Process Name cpbackup_transp
    Event Time 2017-09-12 at 11:41:55 UTC
    PID 22990
    Process UID 0
    Process Username root
    Process Total Virtual Memory 2863200kB
    Process Anonymous Resident Set Size 2576220kB
    Process File Resident Set Size 64kB
    Process OOM Score 618
    Status Out of Memory ⚠
    Memory Information
    Used 1.2 GB
    Available 2.66 GB
    Installed 3.86 GB
    Load Information 0.32 0.20 0.36
    Uptime 3 days, 15 hours, 9 minutes, and 41 seconds
    IOStat Information avg-cpu: %user %nice %system %iowait %steal %idle 7.78 0.25 2.16 1.38 0.07 88.36 Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn xvda 27.70 1551.02 982.35 486679163 308244096 loop0 4.71 13.39 41.25 4202810 12944796 xvdf 6.38 846.72 472.89 265685327 148384808
    Top Processes
    PID
    Owner CPU % Memory % Command
    3720 mysql 1.96 11.98 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib64/mysql/plugin --user=mysql --log-error=host3.uuserver.net.err --open-files-limit=50000 --pid-file=/var/lib/mysql/host3.uuserver.net.pid
    3354 cpanelsolr 0.07 6.81 /usr/lib/jvm/jre-1.8.0/bin/java -server -Xms512m -Xmx512m -XX:NewRatio=3 -XX:SurvivorRatio=4 -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=8 -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:ConcGCThreads=4 -XX:ParallelGCThreads=4 -XX:+CMSScavengeBeforeRemark -XX:PretenureSizeThreshold=64m -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=50 -XX:CMSMaxAbortablePrecleanTime=6000 -XX:+CMSParallelRemarkEnabled -XX:+ParallelRefProcEnabled -XX:-OmitStackTraceInFastThrow -verbose:gc -XX:+PrintHeapAtGC -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime -Xloggc:/home/cpanelsolr/server/logs/solr_gc.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=9 -XX:GCLogFileSize=20M -Dsolr.log.dir=/home/cpanelsolr/server/logs -Djetty.port=8984 -DSTOP.PORT=7984 -DSTOP.KEY=solrrocks -Dhost=127.0.0.1 -Duser.timezone=UTC -Djetty.home=/home/cpanelsolr/server -Dsolr.solr.home=/home/cpanelsolr/server/solr -Dsolr.install.dir=/home/cpanelsolr -Xss256k -Dsolr.autoSoftCommit.maxTime=3000 -Dsolr.log.muteconsole -XX:OnOutOfMemoryError=/home/cpanelsolr/bin/oom_solr.sh 8984 /home/cpanelsolr/server/logs -jar start.jar --module=http
    588 root 0.01 0.77 cpsrvd (SSL) - waiting f --llu=1505133779 --listen=10,11,12,3,4,5,6,7,8,9 --accepted=13
    4007 mailman 0.01 0.71 /usr/bin/python /usr/local/cpanel/3rdparty/mailman/bin/qrunner --runner=IncomingRunner:0:1 -s
    4009 mailman 0.01 0.71 /usr/bin/python /usr/local/cpanel/3rdparty/mailman/bin/qrunner --runner=OutgoingRunner:0:1 -s


    For addtional details, see the attached dmesg log dump.



    Preview of “oom_dmesg.txt”
    [310592.457520] [ 624] 99 624 53670 3076 61 3 0 0 httpd
    [310592.461040] [ 625] 99 625 53670 3076 61 3 0 0 httpd
    [310592.464518] [ 626] 99 626 53670 3076 61 3 0 0 httpd
    [310592.468109] [ 627] 99 627 53670 3070 61 3 0 0 httpd
    [310592.471737] [ 628] 99 628 53670 3071 61 3 0 0 httpd
    [310592.475485] [ 629] 99 629 53670 3071 61 3 0 0 httpd
    [310592.479051] [ 630] 99 630 53670 3076 61 3 0 0 httpd
    [310592.482637] [ 631] 99 631 53670 3070 61 3 0 0 httpd
    [310592.486206] Out of memory: Kill process 22990 (cpbackup_transp) score 618 or sacrifice child
    [310592.489653] Killed process 22990 (cpbackup_transp) total-vm:2863200kB, anon-rss:2576220kB, file-rss:64kB, shmem-rss:0kB


    The system generated this notice on 2017-09-12 at 12:35:04 UTC.
     
    #3 jimhermann, Sep 12, 2017
    Last edited by a moderator: Sep 12, 2017
  4. cPanelMichael

    cPanelMichael Forums Analyst
    Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    38,165
    Likes Received:
    1,371
    Trophy Points:
    363
    cPanel Access Level:
    Root Administrator
    Hello,

    Could you open a support ticket using the link in my signature so we can take a closer look?

    Thank you.
     
  5. jimhermann

    jimhermann Well-Known Member

    Joined:
    Jan 20, 2008
    Messages:
    51
    Likes Received:
    1
    Trophy Points:
    58
    cPanelMichael,

    I don't see your signature or a link.

    Jim
     
  6. jimhermann

    jimhermann Well-Known Member

    Joined:
    Jan 20, 2008
    Messages:
    51
    Likes Received:
    1
    Trophy Points:
    58
    cPanelMichael,

    I see your signature and link when using IE on my PC. It did not appear when using Safari on my iPhone.

    What is the Support Fee if I don't need phone support?

    AWS Technical Support recommended increasing the EC2 Instance from t2.medium (4 GB RAM) to t2.large (8 GB RAM), so I did.

    Also, I increased the Maximum cPanel Process Memory from 768 MB to 2048 MB.

    Thanks,

    Jim
     
  7. jimhermann

    jimhermann Well-Known Member

    Joined:
    Jan 20, 2008
    Messages:
    51
    Likes Received:
    1
    Trophy Points:
    58
    Your Support Request ID is: 8863045
     
Loading...

Share This Page