Slow upload to Amazon S3

[davorg]

Hello

For last 4 or 5 days my backup are uploading very slow to Amazon S3. I have tried to cancel cpbbackup_transport service and then rerun backup again, but it is same. Backup for one account, which is 30 GB is not finished in 18 hours. I'm using S3 in Ireland, my server with cPanel and CloudLinux is located in Frankfurt. I have create new bucket for S3 in Frankfurt and upload is a little bit faster but it has started with backup today at 1 AM and it is still not finished, file which is 4,9 GB takes 4 hours.

Log for this file:

[2018-10-05 04:17:37 +0200] info [cpbackup_transporter] Uploading account backup /home3/backup/2018-10-05/accounts/nextboar15.tar.gz to /backup/2018-10-05/accounts/nextboar15.tar.gz (from 2018-10-05/accounts/nextboar15.tar.gz) 

[2018-10-05 04:17:37 +0200] info [cpbackup_transporter] Attempting to upload /home3/backup/2018-10-05/accounts/nextboar15.tar.gz to /backup/2018-10-05/accounts/nextboar15.tar.gz for destination:  S3

[2018-10-05 04:17:37 +0200] info [cpbackup_transporter] Upload attempt #1 starting for /home3/backup/2018-10-05/accounts/nextboar15.tar.gz to /backup/2018-10-05/accounts/nextboar15.tar.gz for destination:  S3

[2018-10-05 08:25:57 +0200] info [cpbackup_transporter] Successful transfer of /home3/backup/2018-10-05/accounts/nextboar15.tar.gz to /backup/2018-10-05/accounts/nextboar15.tar.gz for destination S3

[2018-10-05 08:25:57 +0200] info [cpbackup_transporter] The backup has been successfully uploaded at least once, now we will delete the local copy since keeplocal (0) is disabled.

I have also turn on acceleration for S3 on Amazon, but this also nothing helped. My server is VPS with 16 GB RAM and 8 CPU. LOAD of server is 0,6 to 0,8. CPanel version is latest and CloudLinux 7 is latest with all patches.

Any Idea or advice?

Regards, Davor

 

[davorg]

I have found this ... after 2 hours today I have this in log:

[2018-10-08 13:55:34 +0200] warn [cpbackup_transporter] Upload attempt failed: Amazon::S3: Amazon responded with 403 Forbidden

 at /usr/local/cpanel/3rdparty/perl/526/lib64/perl5/cpanel_lib/Amazon/S3/Bucket.pm line 136.

 at /usr/local/cpanel/Cpanel/LoggerAdapter.pm line 27.

Cpanel::LoggerAdapter::warn(Cpanel::LoggerAdapter=HASH(0x218fc70), "Upload attempt failed: Amazon::S3: Amazon responded with 403 "...) called at /usr/local/cpanel/Cpanel/Backup/Queue.pm line 609

Cpanel::Backup::Queue::transport_backup::attempt_to_upload_file(Cpanel::Backup::Queue::transport_backup=HASH(0x1ff9190), Cpanel::Transport::Files::AmazonS3=HASH(0x3b2dda0), "/home3/backup/2018-10-07/accounts/citystud.tar.gz", "/backup/2018-10-07/accounts/citystud.tar.gz", Cpanel::LoggerAdapter=HASH(0x218fc70)) called at /usr/local/cpanel/Cpanel/Backup/Queue.pm line 301

Cpanel::Backup::Queue::transport_backup::process_task(Cpanel::Backup::Queue::transport_backup=HASH(0x1ff9190), cPanel::TaskQueue::Task=HASH(0x40b3c80), Cpanel::LoggerAdapter=HASH(0x218fc70)) called at /usr/local/cpanel/3rdparty/perl/526/lib64/perl5/cpanel_lib/cPanel/TaskQueue.pm line 624

eval {...} called at /usr/local/cpanel/3rdparty/perl/526/lib64/perl5/cpanel_lib/cPanel/TaskQueue.pm line 627

cPanel::TaskQueue::__ANON__() called at /usr/local/cpanel/3rdparty/perl/526/lib64/perl5/cpanel_lib/cPanel/StateFile.pm line 223

eval {...} called at /usr/local/cpanel/3rdparty/perl/526/lib64/perl5/cpanel_lib/cPanel/StateFile.pm line 223

cPanel::StateFile::Guard::call_unlocked(cPanel::StateFile::Guard=HASH(0x41dd060), CODE(0x2c58bf8)) called at /usr/local/cpanel/3rdparty/perl/526/lib64/perl5/cpanel_lib/cPanel/TaskQueue.pm line 632

cPanel::TaskQueue::process_next_task(cPanel::TaskQueue=HASH(0x2c2e0a8)) called at /usr/local/cpanel/bin/cpbackup_transporter line 167

eval {...} called at /usr/local/cpanel/bin/cpbackup_transporter line 165

 

[ronaldst]

Please advice that I am no expert.

My first thoughts are that this might be due to network congestion or routing problems.

If there are routing or network congestion problems from your server to S3 the obvious result would be poor performance.

Have you run any tests related to the server/isp speeds and routing to S3?


Another possible reason is that the connection drops/fails during the transfer process, and restarts without continuing where it left off.

I would make a "dummy backup account" (with a reasonable size) and transfer it to S3 and see how that goes.

Do a 'tail -f' on the backup logfile while running the backup and look for errors or signs of failure.

 

[cPanelLauren]

I agree with @ronaldst this sounds a lot like networking related issues. You might want to also check with your server provider to see if they can test their ability to connect to Amazon S3's network.

Thanks!