Please whitelist cPanel in your adblocker so that you’re able to see our version release promotions, thanks!

The Community Forums

Interact with an entire community of cPanel & WHM users!

SOLVED Out of memory cpbackup_transp

Discussion in 'Data Protection' started by jimhermann, Sep 12, 2017.

Tags:
  1. jimhermann

    jimhermann Well-Known Member

    Joined:
    Jan 20, 2008
    Messages:
    62
    Likes Received:
    2
    Trophy Points:
    58
    cPanel Community,

    How do I solve this "Out of Memory" problem? I just started using AWS S3 for backups.
    Code:
    Sep 12 06:41:55 host3 kernel: [310591.852931] httpd invoked oom-killer: gfp_mask=0x24280ca(GFP_HIGHUSER_MOVABLE|__GFP_ZERO), nodemask=0, order=0, oom_score_adj=0
    Sep 12 06:41:55 host3 kernel: [310591.859756] httpd cpuset=/ mems_allowed=0
    Sep 12 06:41:55 host3 kernel: [310591.862377] CPU: 1 PID: 628 Comm: httpd Tainted: G            E   4.9.27-14.31.amzn1.x86_64 #1
    Sep 12 06:41:55 host3 kernel: [310591.865290] Hardware name: Xen HVM domU, BIOS 4.2.amazon 02/16/2017
    Sep 12 06:41:55 host3 kernel: [310591.865290]  ffffc90001d63b50 ffffffff812f9aff ffffc90001d63cf0 ffff880105a28000
    Sep 12 06:41:55 host3 kernel: [310591.865290]  ffffc90001d63be0 ffffffff811f480b 0000000000000000 0000000000000000
    Sep 12 06:41:55 host3 kernel: [310591.865290]  ffffc90001d63c08 ffffc90001d63b88 ffffffff811237ea ffffc90001d63bf0
    Sep 12 06:41:55 host3 kernel: [310591.865290] Call Trace:
    Sep 12 06:41:55 host3 kernel: [310591.865290]  [<ffffffff812f9aff>] dump_stack+0x63/0x84
    Sep 12 06:41:55 host3 kernel: [310591.865290]  [<ffffffff811f480b>] dump_header+0x82/0x212
    Sep 12 06:41:55 host3 kernel: [310591.865290]  [<ffffffff811237ea>] ? __delayacct_freepages_end+0x2a/0x30
    Sep 12 06:41:55 host3 kernel: [310591.865290]  [<ffffffff8118e52a>] ? do_try_to_free_pages+0x2da/0x340
    Sep 12 06:41:55 host3 kernel: [310591.865290]  [<ffffffff8117a6bc>] oom_kill_process+0x21c/0x3f0
    Sep 12 06:41:55 host3 kernel: [310591.865290]  [<ffffffff8117ab58>] out_of_memory+0x108/0x4b0
    Sep 12 06:41:55 host3 kernel: [310591.865290]  [<ffffffff8117f4d0>] __alloc_pages_slowpath+0x9a0/0xb90
    Sep 12 06:41:55 host3 kernel: [310591.865290]  [<ffffffff8117f8a3>] __alloc_pages_nodemask+0x1e3/0x250
    Sep 12 06:41:55 host3 kernel: [310591.865290]  [<ffffffff811cf8d5>] alloc_pages_vma+0xa5/0x220
    Sep 12 06:41:55 host3 kernel: [310591.865290]  [<ffffffff811ac4d3>] handle_mm_fault+0x11b3/0x13a0
    Sep 12 06:41:55 host3 kernel: [310591.865290]  [<ffffffff81060fe5>] __do_page_fault+0x225/0x4a0
    Sep 12 06:41:55 host3 kernel: [310591.865290]  [<ffffffff81061282>] do_page_fault+0x22/0x30
    Sep 12 06:41:55 host3 kernel: [310591.865290]  [<ffffffff815349f8>] page_fault+0x28/0x30
    Sep 12 06:41:55 host3 kernel: [310591.927529] Mem-Info:
    Sep 12 06:41:55 host3 kernel: [310591.928790] active_anon:957468 inactive_anon:87 isolated_anon:0
    Sep 12 06:41:55 host3 kernel: [310591.928790]  active_file:386 inactive_file:385 isolated_file:0
    Sep 12 06:41:55 host3 kernel: [310591.928790]  unevictable:0 dirty:1 writeback:1 unstable:0
    Sep 12 06:41:55 host3 kernel: [310591.928790]  slab_reclaimable:5315 slab_unreclaimable:5612
    Sep 12 06:41:55 host3 kernel: [310591.928790]  mapped:407 shmem:109 pagetables:6378 bounce:0
    Sep 12 06:41:55 host3 kernel: [310591.928790]  free:20960 free_pcp:14 free_cma:0
    Sep 12 06:41:55 host3 kernel: [310591.947497] Node 0 active_anon:3829872kB inactive_anon:348kB active_file:1544kB inactive_file:1540kB unevictable:0kB isolated(anon):0kB isola
    ted(file):0kB mapped:2228kB dirty:4kB writeback:4kB shmem:436kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB unstable:0kB pages_scanned:4678 all_unrecla
    imable? yes
    Sep 12 06:41:55 host3 kernel: [310591.962946] Node 0 DMA free:15904kB min:264kB low:328kB high:392kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevic
    table:0kB writepending:0kB present:15988kB managed:15904kB mlocked:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB loca
    l_pcp:0kB free_cma:0kB
    Sep 12 06:41:55 host3 kernel: [310591.979579] lowmem_reserve[]: 0 3730 3920 3920
    Sep 12 06:41:55 host3 kernel: [310591.982286] Node 0 DMA32 free:64712kB min:64064kB low:80080kB high:96096kB active_anon:3679980kB inactive_anon:260kB active_file:1232kB inact
    ive_file:1216kB unevictable:0kB writepending:8kB present:3915776kB managed:3838136kB mlocked:0kB slab_reclaimable:12384kB slab_unreclaimable:11316kB kernel_stack:2368kB pageta
    bles:21776kB bounce:0kB free_pcp:552kB local_pcp:0kB free_cma:0kB
    Sep 12 06:41:55 host3 kernel: [310592.001978] lowmem_reserve[]: 0 0 189 189
    Sep 12 06:41:55 host3 kernel: [310592.004358] Node 0 Normal free:3224kB min:3252kB low:4064kB high:4876kB active_anon:149864kB inactive_anon:88kB active_file:76kB inactive_fil
    e:0kB unevictable:0kB writepending:0kB present:262144kB managed:193976kB mlocked:0kB slab_reclaimable:8876kB slab_unreclaimable:11132kB kernel_stack:1744kB pagetables:3736kB b
    ounce:0kB free_pcp:56kB local_pcp:56kB free_cma:0kB
    Sep 12 06:41:55 host3 kernel: [310592.018456] lowmem_reserve[]: 0 0 0 0
    Sep 12 06:41:55 host3 kernel: [310592.020373] Node 0 DMA: 0*4kB 0*8kB 0*16kB 1*32kB (U) 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 159
    04kB
    Sep 12 06:41:55 host3 kernel: [310592.026825] Node 0 DMA32: 383*4kB (UME) 260*8kB (UME) 645*16kB (UE) 370*32kB (UME) 243*64kB (UME) 119*128kB (UME) 24*256kB (UME) 0*512kB 0*10
    24kB 1*2048kB (H) 0*4096kB = 64748kB
    Sep 12 06:41:55 host3 kernel: [310592.034403] Node 0 Normal: 138*4kB (MEH) 96*8kB (UMEH) 3*16kB (H) 2*32kB (H) 2*64kB (H) 1*128kB (H) 2*256kB (H) 0*512kB 1*1024kB (H) 0*2048kB
     0*4096kB = 3224kB
    Sep 12 06:41:55 host3 kernel: [310592.041505] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
    Sep 12 06:41:55 host3 kernel: [310592.044907] 758 total pagecache pages
    Sep 12 06:41:55 host3 kernel: [310592.046403] 0 pages in swap cache
    Sep 12 06:41:55 host3 kernel: [310592.047768] Swap cache stats: add 0, delete 0, find 0/0
    Sep 12 06:41:55 host3 kernel: [310592.049858] Free swap  = 0kB
    Sep 12 06:41:55 host3 kernel: [310592.051456] Total swap = 0kB
    Sep 12 06:41:55 host3 kernel: [310592.053046] 1048477 pages RAM
    Sep 12 06:41:55 host3 kernel: [310592.054745] 0 pages HighMem/MovableOnly
    Sep 12 06:41:55 host3 kernel: [310592.056517] 36473 pages reserved
    *****
    Sep 12 06:41:55 host3 kernel: [310592.486206] Out of memory: Kill process 22990 (cpbackup_transp) score 618 or sacrifice child
    Sep 12 06:41:55 host3 kernel: [310592.489653] Killed process 22990 (cpbackup_transp) total-vm:2863200kB, anon-rss:2576220kB, file-rss:64kB, shmem-rss:0kB
    
     
    #1 jimhermann, Sep 12, 2017
    Last edited by a moderator: Oct 4, 2017
  2. cPanelMichael

    cPanelMichael Technical Support Community Manager
    Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    44,802
    Likes Received:
    1,895
    Trophy Points:
    363
    cPanel Access Level:
    Root Administrator
    Twitter:
    Hello,

    Do you notice the server is reaching it's memory limit when this happens? EX:

    Code:
    free -m
    Thank you.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  3. jimhermann

    jimhermann Well-Known Member

    Joined:
    Jan 20, 2008
    Messages:
    62
    Likes Received:
    2
    Trophy Points:
    58
    No. I was not connected at the time. CPanel Monitoring sent me this report:

    Out of memory: The process “cpbackup_transp” was terminated because the system is low on memory.
    In order to avoid a system crash due to low memory, the kernel terminated the process named “cpbackup_transp” with the PID “22990”.

    Server host3.uuserver.net
    Primary IP Address 172.31.13.215
    Process Name cpbackup_transp
    Event Time 2017-09-12 at 11:41:55 UTC
    PID 22990
    Process UID 0
    Process Username root
    Process Total Virtual Memory 2863200kB
    Process Anonymous Resident Set Size 2576220kB
    Process File Resident Set Size 64kB
    Process OOM Score 618
    Status Out of Memory ⚠
    Memory Information
    Used 1.2 GB
    Available 2.66 GB
    Installed 3.86 GB
    Load Information 0.32 0.20 0.36
    Uptime 3 days, 15 hours, 9 minutes, and 41 seconds
    IOStat Information avg-cpu: %user %nice %system %iowait %steal %idle 7.78 0.25 2.16 1.38 0.07 88.36 Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn xvda 27.70 1551.02 982.35 486679163 308244096 loop0 4.71 13.39 41.25 4202810 12944796 xvdf 6.38 846.72 472.89 265685327 148384808
    Top Processes
    PID
    Owner CPU % Memory % Command
    3720 mysql 1.96 11.98 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib64/mysql/plugin --user=mysql --log-error=host3.uuserver.net.err --open-files-limit=50000 --pid-file=/var/lib/mysql/host3.uuserver.net.pid
    3354 cpanelsolr 0.07 6.81 /usr/lib/jvm/jre-1.8.0/bin/java -server -Xms512m -Xmx512m -XX:NewRatio=3 -XX:SurvivorRatio=4 -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=8 -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:ConcGCThreads=4 -XX:ParallelGCThreads=4 -XX:+CMSScavengeBeforeRemark -XX:PretenureSizeThreshold=64m -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=50 -XX:CMSMaxAbortablePrecleanTime=6000 -XX:+CMSParallelRemarkEnabled -XX:+ParallelRefProcEnabled -XX:-OmitStackTraceInFastThrow -verbose:gc -XX:+PrintHeapAtGC -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime -Xloggc:/home/cpanelsolr/server/logs/solr_gc.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=9 -XX:GCLogFileSize=20M -Dsolr.log.dir=/home/cpanelsolr/server/logs -Djetty.port=8984 -DSTOP.PORT=7984 -DSTOP.KEY=solrrocks -Dhost=127.0.0.1 -Duser.timezone=UTC -Djetty.home=/home/cpanelsolr/server -Dsolr.solr.home=/home/cpanelsolr/server/solr -Dsolr.install.dir=/home/cpanelsolr -Xss256k -Dsolr.autoSoftCommit.maxTime=3000 -Dsolr.log.muteconsole -XX:OnOutOfMemoryError=/home/cpanelsolr/bin/oom_solr.sh 8984 /home/cpanelsolr/server/logs -jar start.jar --module=http
    588 root 0.01 0.77 cpsrvd (SSL) - waiting f --llu=1505133779 --listen=10,11,12,3,4,5,6,7,8,9 --accepted=13
    4007 mailman 0.01 0.71 /usr/bin/python /usr/local/cpanel/3rdparty/mailman/bin/qrunner --runner=IncomingRunner:0:1 -s
    4009 mailman 0.01 0.71 /usr/bin/python /usr/local/cpanel/3rdparty/mailman/bin/qrunner --runner=OutgoingRunner:0:1 -s


    For addtional details, see the attached dmesg log dump.



    Preview of “oom_dmesg.txt”
    [310592.457520] [ 624] 99 624 53670 3076 61 3 0 0 httpd
    [310592.461040] [ 625] 99 625 53670 3076 61 3 0 0 httpd
    [310592.464518] [ 626] 99 626 53670 3076 61 3 0 0 httpd
    [310592.468109] [ 627] 99 627 53670 3070 61 3 0 0 httpd
    [310592.471737] [ 628] 99 628 53670 3071 61 3 0 0 httpd
    [310592.475485] [ 629] 99 629 53670 3071 61 3 0 0 httpd
    [310592.479051] [ 630] 99 630 53670 3076 61 3 0 0 httpd
    [310592.482637] [ 631] 99 631 53670 3070 61 3 0 0 httpd
    [310592.486206] Out of memory: Kill process 22990 (cpbackup_transp) score 618 or sacrifice child
    [310592.489653] Killed process 22990 (cpbackup_transp) total-vm:2863200kB, anon-rss:2576220kB, file-rss:64kB, shmem-rss:0kB


    The system generated this notice on 2017-09-12 at 12:35:04 UTC.
     
    #3 jimhermann, Sep 12, 2017
    Last edited by a moderator: Sep 12, 2017
  4. cPanelMichael

    cPanelMichael Technical Support Community Manager
    Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    44,802
    Likes Received:
    1,895
    Trophy Points:
    363
    cPanel Access Level:
    Root Administrator
    Twitter:
    Hello,

    Could you open a support ticket using the link in my signature so we can take a closer look?

    Thank you.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  5. jimhermann

    jimhermann Well-Known Member

    Joined:
    Jan 20, 2008
    Messages:
    62
    Likes Received:
    2
    Trophy Points:
    58
    cPanelMichael,

    I don't see your signature or a link.

    Jim
     
  6. jimhermann

    jimhermann Well-Known Member

    Joined:
    Jan 20, 2008
    Messages:
    62
    Likes Received:
    2
    Trophy Points:
    58
    cPanelMichael,

    I see your signature and link when using IE on my PC. It did not appear when using Safari on my iPhone.

    What is the Support Fee if I don't need phone support?

    AWS Technical Support recommended increasing the EC2 Instance from t2.medium (4 GB RAM) to t2.large (8 GB RAM), so I did.

    Also, I increased the Maximum cPanel Process Memory from 768 MB to 2048 MB.

    Thanks,

    Jim
     
  7. jimhermann

    jimhermann Well-Known Member

    Joined:
    Jan 20, 2008
    Messages:
    62
    Likes Received:
    2
    Trophy Points:
    58
    Your Support Request ID is: 8863045
     
  8. jbourque

    jbourque Member

    Joined:
    Jul 18, 2008
    Messages:
    6
    Likes Received:
    0
    Trophy Points:
    51
    I have been having this problem for the past few weeks every time my backups are being transported my server becomes unstable. I have been using Amazon WS for over a year and this has not been an issue, I'm now having to hard power my server in order to gain control.

    Has anyone found a solution to this?

    Joe
     
  9. jimhermann

    jimhermann Well-Known Member

    Joined:
    Jan 20, 2008
    Messages:
    62
    Likes Received:
    2
    Trophy Points:
    58
    I had to increase my RAM to 16GB because the cpbackup_transp process uses 14 GB at times. Also, I increased my SWAP size to 8GB.

    I don't use compression. cPanel blamed the problem on large (20-30 GB) web sites AND on other processes, like MySQL.

    The transfer still takes 8 hours for 250 GB backup.

    Jim
     
  10. jimhermann

    jimhermann Well-Known Member

    Joined:
    Jan 20, 2008
    Messages:
    62
    Likes Received:
    2
    Trophy Points:
    58
    Folks,

    cPanel Support recommended that I stop using AWS S3 storage (not an option) until the developers fix the problem with cPanel case ID CPANEL-160066.

    They said that there is a memory problem with the cpbackup_transp program. It keeping on using more and more memory as it runs and it takes hours and hours to complete.

    Today, it's been running for 9 hours for a 300 GB backup and is using 14 GB RAM (RSS).

    USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
    root 16893 24.1 86.1 20000468 14157680 ? RN 02:47 131:07 cpbackup_transporter - Processing next task

    'nuff said,

    Jim
     
  11. cPanelMichael

    cPanelMichael Technical Support Community Manager
    Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    44,802
    Likes Received:
    1,895
    Trophy Points:
    363
    cPanel Access Level:
    Root Administrator
    Twitter:
    Hello,

    I'll monitor internal case CPANEL-16066 and update this thread with more information as it becomes available.

    Thank you.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  12. mooseontheloose

    Joined:
    Jul 25, 2003
    Messages:
    22
    Likes Received:
    2
    Trophy Points:
    228
    cPanel Access Level:
    DataCenter Provider
    We've been using S3 for a couple years now without issue until recently. Now have all kind of memory issues.

    Code:
    root     11843 13.5 53.1 15742860 13062672 ?   RN   04:34 100:44 cpbackup_transporter - Processing next task
    You can guess when the issue started happening:

    memory-year.png
     
  13. jimhermann

    jimhermann Well-Known Member

    Joined:
    Jan 20, 2008
    Messages:
    62
    Likes Received:
    2
    Trophy Points:
    58
    Would it be possible to use the AWS Python program called aws?

    It can transfer data at 20-30 MiB/s. It only takes about 4 hours to transfer 330 GB of backup data.

    Thanks,

    Jim
     
  14. cPanelMichael

    cPanelMichael Technical Support Community Manager
    Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    44,802
    Likes Received:
    1,895
    Trophy Points:
    363
    cPanel Access Level:
    Root Administrator
    Twitter:
    Hello,

    There's no update to report on internal case CPANEL-16066 at this time. I'll continue to monitor the case and update this thread with new information as it becomes available.

    Changes to how the Amazon transport works would require a feature request. You can open a feature request using the following URL:

    Submit A Feature Request

    Thank you.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  15. mooseontheloose

    Joined:
    Jul 25, 2003
    Messages:
    22
    Likes Received:
    2
    Trophy Points:
    228
    cPanel Access Level:
    DataCenter Provider
    Any updates on this?

    We had a server lock up due to the memory leak, forcing us to disable S3 backups.
     
  16. cPanelMichael

    cPanelMichael Technical Support Community Manager
    Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    44,802
    Likes Received:
    1,895
    Trophy Points:
    363
    cPanel Access Level:
    Root Administrator
    Twitter:
    There are no new updates to report at this time. I'll update this thread as soon as an update is available.

    Thank you.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  17. Kelli

    Kelli cPanel Product Owner
    Staff Member

    Joined:
    Nov 30, 2016
    Messages:
    17
    Likes Received:
    4
    Trophy Points:
    78
    Location:
    Houston
    cPanel Access Level:
    Root Administrator
    Hi everyone,

    We're currently investigating this issue, looking for a root cause so that we know how to fix it. CPANEL-16066
     
  18. cPanelMichael

    cPanelMichael Technical Support Community Manager
    Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    44,802
    Likes Received:
    1,895
    Trophy Points:
    363
    cPanel Access Level:
    Root Administrator
    Twitter:
    Hello,

    To update, the source of the memory leak was identified and the resolution is now undergoing testing before it's released. I'll provide another update once the resolution is published.

    Thank you.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  19. cPanelMichael

    cPanelMichael Technical Support Community Manager
    Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    44,802
    Likes Received:
    1,895
    Trophy Points:
    363
    cPanel Access Level:
    Root Administrator
    Twitter:
    Hello,

    This is fixed in cPanel version 68.0.21 as part of the following case:

    Fixed case CPANEL-16066: Update cpanel-perl-524-IO-Socket-SSL to 2.047-3.cp1162.

    The full cPanel version 68 change log is available at:

    68 Change Log - Change Logs - cPanel Documentation

    Thank you.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
Loading...

Share This Page

  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice