SOLVED Out of memory cpbackup_transp

[jimhermann]

cPanel Community,

How do I solve this "Out of Memory" problem? I just started using AWS S3 for backups.

Sep 12 06:41:55 host3 kernel: [310591.852931] httpd invoked oom-killer: gfp_mask=0x24280ca(GFP_HIGHUSER_MOVABLE|__GFP_ZERO), nodemask=0, order=0, oom_score_adj=0
Sep 12 06:41:55 host3 kernel: [310591.859756] httpd cpuset=/ mems_allowed=0
Sep 12 06:41:55 host3 kernel: [310591.862377] CPU: 1 PID: 628 Comm: httpd Tainted: G            E   4.9.27-14.31.amzn1.x86_64 #1
Sep 12 06:41:55 host3 kernel: [310591.865290] Hardware name: Xen HVM domU, BIOS 4.2.amazon 02/16/2017
Sep 12 06:41:55 host3 kernel: [310591.865290]  ffffc90001d63b50 ffffffff812f9aff ffffc90001d63cf0 ffff880105a28000
Sep 12 06:41:55 host3 kernel: [310591.865290]  ffffc90001d63be0 ffffffff811f480b 0000000000000000 0000000000000000
Sep 12 06:41:55 host3 kernel: [310591.865290]  ffffc90001d63c08 ffffc90001d63b88 ffffffff811237ea ffffc90001d63bf0
Sep 12 06:41:55 host3 kernel: [310591.865290] Call Trace:
Sep 12 06:41:55 host3 kernel: [310591.865290]  [<ffffffff812f9aff>] dump_stack+0x63/0x84
Sep 12 06:41:55 host3 kernel: [310591.865290]  [<ffffffff811f480b>] dump_header+0x82/0x212
Sep 12 06:41:55 host3 kernel: [310591.865290]  [<ffffffff811237ea>] ? __delayacct_freepages_end+0x2a/0x30
Sep 12 06:41:55 host3 kernel: [310591.865290]  [<ffffffff8118e52a>] ? do_try_to_free_pages+0x2da/0x340
Sep 12 06:41:55 host3 kernel: [310591.865290]  [<ffffffff8117a6bc>] oom_kill_process+0x21c/0x3f0
Sep 12 06:41:55 host3 kernel: [310591.865290]  [<ffffffff8117ab58>] out_of_memory+0x108/0x4b0
Sep 12 06:41:55 host3 kernel: [310591.865290]  [<ffffffff8117f4d0>] __alloc_pages_slowpath+0x9a0/0xb90
Sep 12 06:41:55 host3 kernel: [310591.865290]  [<ffffffff8117f8a3>] __alloc_pages_nodemask+0x1e3/0x250
Sep 12 06:41:55 host3 kernel: [310591.865290]  [<ffffffff811cf8d5>] alloc_pages_vma+0xa5/0x220
Sep 12 06:41:55 host3 kernel: [310591.865290]  [<ffffffff811ac4d3>] handle_mm_fault+0x11b3/0x13a0
Sep 12 06:41:55 host3 kernel: [310591.865290]  [<ffffffff81060fe5>] __do_page_fault+0x225/0x4a0
Sep 12 06:41:55 host3 kernel: [310591.865290]  [<ffffffff81061282>] do_page_fault+0x22/0x30
Sep 12 06:41:55 host3 kernel: [310591.865290]  [<ffffffff815349f8>] page_fault+0x28/0x30
Sep 12 06:41:55 host3 kernel: [310591.927529] Mem-Info:
Sep 12 06:41:55 host3 kernel: [310591.928790] active_anon:957468 inactive_anon:87 isolated_anon:0
Sep 12 06:41:55 host3 kernel: [310591.928790]  active_file:386 inactive_file:385 isolated_file:0
Sep 12 06:41:55 host3 kernel: [310591.928790]  unevictable:0 dirty:1 writeback:1 unstable:0
Sep 12 06:41:55 host3 kernel: [310591.928790]  slab_reclaimable:5315 slab_unreclaimable:5612
Sep 12 06:41:55 host3 kernel: [310591.928790]  mapped:407 shmem:109 pagetables:6378 bounce:0
Sep 12 06:41:55 host3 kernel: [310591.928790]  free:20960 free_pcp:14 free_cma:0
Sep 12 06:41:55 host3 kernel: [310591.947497] Node 0 active_anon:3829872kB inactive_anon:348kB active_file:1544kB inactive_file:1540kB unevictable:0kB isolated(anon):0kB isola
ted(file):0kB mapped:2228kB dirty:4kB writeback:4kB shmem:436kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB unstable:0kB pages_scanned:4678 all_unrecla
imable? yes
Sep 12 06:41:55 host3 kernel: [310591.962946] Node 0 DMA free:15904kB min:264kB low:328kB high:392kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevic
table:0kB writepending:0kB present:15988kB managed:15904kB mlocked:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB loca
l_pcp:0kB free_cma:0kB
Sep 12 06:41:55 host3 kernel: [310591.979579] lowmem_reserve[]: 0 3730 3920 3920
Sep 12 06:41:55 host3 kernel: [310591.982286] Node 0 DMA32 free:64712kB min:64064kB low:80080kB high:96096kB active_anon:3679980kB inactive_anon:260kB active_file:1232kB inact
ive_file:1216kB unevictable:0kB writepending:8kB present:3915776kB managed:3838136kB mlocked:0kB slab_reclaimable:12384kB slab_unreclaimable:11316kB kernel_stack:2368kB pageta
bles:21776kB bounce:0kB free_pcp:552kB local_pcp:0kB free_cma:0kB
Sep 12 06:41:55 host3 kernel: [310592.001978] lowmem_reserve[]: 0 0 189 189
Sep 12 06:41:55 host3 kernel: [310592.004358] Node 0 Normal free:3224kB min:3252kB low:4064kB high:4876kB active_anon:149864kB inactive_anon:88kB active_file:76kB inactive_fil
e:0kB unevictable:0kB writepending:0kB present:262144kB managed:193976kB mlocked:0kB slab_reclaimable:8876kB slab_unreclaimable:11132kB kernel_stack:1744kB pagetables:3736kB b
ounce:0kB free_pcp:56kB local_pcp:56kB free_cma:0kB
Sep 12 06:41:55 host3 kernel: [310592.018456] lowmem_reserve[]: 0 0 0 0
Sep 12 06:41:55 host3 kernel: [310592.020373] Node 0 DMA: 0*4kB 0*8kB 0*16kB 1*32kB (U) 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 159
04kB
Sep 12 06:41:55 host3 kernel: [310592.026825] Node 0 DMA32: 383*4kB (UME) 260*8kB (UME) 645*16kB (UE) 370*32kB (UME) 243*64kB (UME) 119*128kB (UME) 24*256kB (UME) 0*512kB 0*10
24kB 1*2048kB (H) 0*4096kB = 64748kB
Sep 12 06:41:55 host3 kernel: [310592.034403] Node 0 Normal: 138*4kB (MEH) 96*8kB (UMEH) 3*16kB (H) 2*32kB (H) 2*64kB (H) 1*128kB (H) 2*256kB (H) 0*512kB 1*1024kB (H) 0*2048kB
 0*4096kB = 3224kB
Sep 12 06:41:55 host3 kernel: [310592.041505] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Sep 12 06:41:55 host3 kernel: [310592.044907] 758 total pagecache pages
Sep 12 06:41:55 host3 kernel: [310592.046403] 0 pages in swap cache
Sep 12 06:41:55 host3 kernel: [310592.047768] Swap cache stats: add 0, delete 0, find 0/0
Sep 12 06:41:55 host3 kernel: [310592.049858] Free swap  = 0kB
Sep 12 06:41:55 host3 kernel: [310592.051456] Total swap = 0kB
Sep 12 06:41:55 host3 kernel: [310592.053046] 1048477 pages RAM
Sep 12 06:41:55 host3 kernel: [310592.054745] 0 pages HighMem/MovableOnly
Sep 12 06:41:55 host3 kernel: [310592.056517] 36473 pages reserved
*****
Sep 12 06:41:55 host3 kernel: [310592.486206] Out of memory: Kill process 22990 (cpbackup_transp) score 618 or sacrifice child
Sep 12 06:41:55 host3 kernel: [310592.489653] Killed process 22990 (cpbackup_transp) total-vm:2863200kB, anon-rss:2576220kB, file-rss:64kB, shmem-rss:0kB

 

[cPanelMichael]

Hello,

Do you notice the server is reaching it's memory limit when this happens? EX:

free -m

Thank you.

 

[jimhermann]

Hello,

Do you notice the server is reaching it's memory limit when this happens? EX:

free -m

Thank you.

No. I was not connected at the time. CPanel Monitoring sent me this report:

Out of memory: The process “cpbackup_transp” was terminated because the system is low on memory.
In order to avoid a system crash due to low memory, the kernel terminated the process named “cpbackup_transp” with the PID “22990”.

Server host3.uuserver.net
Primary IP Address 172.31.13.215
Process Name cpbackup_transp
Event Time 2017-09-12 at 11:41:55 UTC
PID 22990
Process UID 0
Process Username root
Process Total Virtual Memory 2863200kB
Process Anonymous Resident Set Size 2576220kB
Process File Resident Set Size 64kB
Process OOM Score 618
Status Out of Memory
Memory Information
Used 1.2 GB
Available 2.66 GB
Installed 3.86 GB
Load Information 0.32 0.20 0.36
Uptime 3 days, 15 hours, 9 minutes, and 41 seconds
IOStat Information avg-cpu: %user %nice %system %iowait %steal %idle 7.78 0.25 2.16 1.38 0.07 88.36 Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn xvda 27.70 1551.02 982.35 486679163 308244096 loop0 4.71 13.39 41.25 4202810 12944796 xvdf 6.38 846.72 472.89 265685327 148384808
Top Processes
PID Owner CPU % Memory % Command
3720 mysql 1.96 11.98 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib64/mysql/plugin --user=mysql --log-error=host3.uuserver.net.err --open-files-limit=50000 --pid-file=/var/lib/mysql/host3.uuserver.net.pid
3354 cpanelsolr 0.07 6.81 /usr/lib/jvm/jre-1.8.0/bin/java -server -Xms512m -Xmx512m -XX:NewRatio=3 -XX:SurvivorRatio=4 -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=8 -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:ConcGCThreads=4 -XX:ParallelGCThreads=4 -XX:+CMSScavengeBeforeRemark -XX:PretenureSizeThreshold=64m -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=50 -XX:CMSMaxAbortablePrecleanTime=6000 -XX:+CMSParallelRemarkEnabled -XX:+ParallelRefProcEnabled -XX:-OmitStackTraceInFastThrow -verbose:gc -XX:+PrintHeapAtGC -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime -Xloggc:/home/cpanelsolr/server/logs/solr_gc.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=9 -XX:GCLogFileSize=20M -Dsolr.log.dir=/home/cpanelsolr/server/logs -Djetty.port=8984 -DSTOP.PORT=7984 -DSTOP.KEY=solrrocks -Dhost=127.0.0.1 -Duser.timezone=UTC -Djetty.home=/home/cpanelsolr/server -Dsolr.solr.home=/home/cpanelsolr/server/solr -Dsolr.install.dir=/home/cpanelsolr -Xss256k -Dsolr.autoSoftCommit.maxTime=3000 -Dsolr.log.muteconsole -XX:OnOutOfMemoryError=/home/cpanelsolr/bin/oom_solr.sh 8984 /home/cpanelsolr/server/logs -jar start.jar --module=http
588 root 0.01 0.77 cpsrvd (SSL) - waiting f --llu=1505133779 --listen=10,11,12,3,4,5,6,7,8,9 --accepted=13
4007 mailman 0.01 0.71 /usr/bin/python /usr/local/cpanel/3rdparty/mailman/bin/qrunner --runner=IncomingRunner:0:1 -s
4009 mailman 0.01 0.71 /usr/bin/python /usr/local/cpanel/3rdparty/mailman/bin/qrunner --runner=OutgoingRunner:0:1 -s


For addtional details, see the attached dmesg log dump.



Preview of “oom_dmesg.txt”
[310592.457520] [ 624] 99 624 53670 3076 61 3 0 0 httpd
[310592.461040] [ 625] 99 625 53670 3076 61 3 0 0 httpd
[310592.464518] [ 626] 99 626 53670 3076 61 3 0 0 httpd
[310592.468109] [ 627] 99 627 53670 3070 61 3 0 0 httpd
[310592.471737] [ 628] 99 628 53670 3071 61 3 0 0 httpd
[310592.475485] [ 629] 99 629 53670 3071 61 3 0 0 httpd
[310592.479051] [ 630] 99 630 53670 3076 61 3 0 0 httpd
[310592.482637] [ 631] 99 631 53670 3070 61 3 0 0 httpd
[310592.486206] Out of memory: Kill process 22990 (cpbackup_transp) score 618 or sacrifice child
[310592.489653] Killed process 22990 (cpbackup_transp) total-vm:2863200kB, anon-rss:2576220kB, file-rss:64kB, shmem-rss:0kB


The system generated this notice on 2017-09-12 at 12:35:04 UTC.

 

[cPanelMichael]

Hello,

Could you open a support ticket using the link in my signature so we can take a closer look?

Thank you.

 

[jimhermann]

cPanelMichael,

I don't see your signature or a link.

Jim

 

[jimhermann]

cPanelMichael,

I see your signature and link when using IE on my PC. It did not appear when using Safari on my iPhone.

What is the Support Fee if I don't need phone support?

AWS Technical Support recommended increasing the EC2 Instance from t2.medium (4 GB RAM) to t2.large (8 GB RAM), so I did.

Also, I increased the Maximum cPanel Process Memory from 768 MB to 2048 MB.

Thanks,

Jim

 

[jimhermann]

Your Support Request ID is: 8863045

 

[jbourque]

I have been having this problem for the past few weeks every time my backups are being transported my server becomes unstable. I have been using Amazon WS for over a year and this has not been an issue, I'm now having to hard power my server in order to gain control.

Has anyone found a solution to this?

Joe

 

[jimhermann]

I had to increase my RAM to 16GB because the cpbackup_transp process uses 14 GB at times. Also, I increased my SWAP size to 8GB.

I don't use compression. cPanel blamed the problem on large (20-30 GB) web sites AND on other processes, like MySQL.

The transfer still takes 8 hours for 250 GB backup.

Jim

 

[jimhermann]

Folks,

cPanel Support recommended that I stop using AWS S3 storage (not an option) until the developers fix the problem with cPanel case ID CPANEL-160066.

They said that there is a memory problem with the cpbackup_transp program. It keeping on using more and more memory as it runs and it takes hours and hours to complete.

Today, it's been running for 9 hours for a 300 GB backup and is using 14 GB RAM (RSS).

USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 16893 24.1 86.1 20000468 14157680 ? RN 02:47 131:07 cpbackup_transporter - Processing next task

'nuff said,

Jim

 

[cPanelMichael]

Hello,

I'll monitor internal case CPANEL-16066 and update this thread with more information as it becomes available.

Thank you.

 

[mooseontheloose]

We've been using S3 for a couple years now without issue until recently. Now have all kind of memory issues.

root     11843 13.5 53.1 15742860 13062672 ?   RN   04:34 100:44 cpbackup_transporter - Processing next task

You can guess when the issue started happening:

 

[jimhermann]

Hello,

I'll monitor internal case CPANEL-16066 and update this thread with more information as it becomes available.

Thank you.

Would it be possible to use the AWS Python program called aws?

It can transfer data at 20-30 MiB/s. It only takes about 4 hours to transfer 330 GB of backup data.

Thanks,

Jim

 

[cPanelMichael]

Hello,

There's no update to report on internal case CPANEL-16066 at this time. I'll continue to monitor the case and update this thread with new information as it becomes available.

Would it be possible to use the AWS Python program called aws?

Changes to how the Amazon transport works would require a feature request. You can open a feature request using the following URL:

Submit A Feature Request

Thank you.

 

[mooseontheloose]

Any updates on this?

We had a server lock up due to the memory leak, forcing us to disable S3 backups.

 

[cPanelMichael]

Any updates on this?

There are no new updates to report at this time. I'll update this thread as soon as an update is available.

Thank you.

 

[Kelli]

Hi everyone,

We're currently investigating this issue, looking for a root cause so that we know how to fix it. CPANEL-16066

 

[cPanelMichael]

Hello,

To update, the source of the memory leak was identified and the resolution is now undergoing testing before it's released. I'll provide another update once the resolution is published.

Thank you.

 

[cPanelMichael]

Hello,

This is fixed in cPanel version 68.0.21 as part of the following case:

Fixed case CPANEL-16066: Update cpanel-perl-524-IO-Socket-SSL to 2.047-3.cp1162.

The full cPanel version 68 change log is available at:

68 Change Log - Change Logs - cPanel Documentation

Thank you.