Please whitelist cPanel in your adblocker so that you’re able to see our version release promotions, thanks!

The Community Forums

Interact with an entire community of cPanel & WHM users!

Solution for large files being sent to Dropbox?

Discussion in 'Data Protection' started by martin MHC, Jan 24, 2019.

  1. martin MHC

    martin MHC Well-Known Member

    Joined:
    Sep 14, 2016
    Messages:
    145
    Likes Received:
    23
    Trophy Points:
    18
    Location:
    UK
    cPanel Access Level:
    Root Administrator
    I have set up a backup system to my Dropbox -- external to cPanel and the local server environment. I have used the script described here: Custom Backup Transport: Dropbox | cPanel Blog as well as various Forum posts to set up the Dropbox backup.

    The script and the WHM backup interface appear to work correctly for most accounts, but there are persistent accounts that fail on _most_ but not all backups.

    I believe the reason may be something like the time taken to transport the data to the external Dropbox fails due to the size of the transfers (each failed account is well over 1Gb+ due to email accounts).

    Backups are compressed - backups are maximised timeout (50,000 seconds) - smaller backups are transported ok.

    Example result error message is:

    Unable to send “/backup/2019-01-22/accounts/accountname.tar.gz” to destination “Dropbox”

    So;

    It seems to be a timeout issue for the 50,000 seconds being needed to transport 1Gb+ (so on a minimum sized 1Gb backup file the backup needs to be transported at 1Mbps as a MINIMUM, so for a 3-4Gb file (of which we have a few accounts); thy need to be transported consistently at 3-4Mbps which from one server to another across the open net seems to be unlikely to transport :-(

    Can cPanel provide any assistance to making this work? IS this likely to be a Dropbox issue or a Panel issue?
     
  2. martin MHC

    martin MHC Well-Known Member

    Joined:
    Sep 14, 2016
    Messages:
    145
    Likes Received:
    23
    Trophy Points:
    18
    Location:
    UK
    cPanel Access Level:
    Root Administrator
    To clarify further:

    1) Can we increase the 50k seconds hardlimit timeout value and see if this helps move larger files across to the backup destination?

    2) If not, can the 50k timeout limit be set on a per-file basis rather than a per-transaction basis, if not already? (I am unclear on this)

    3) Can the transport mechanism be able to report why the backup failed, for instance; if the timeout was reached?


    Thanks!
     
  3. martin MHC

    martin MHC Well-Known Member

    Joined:
    Sep 14, 2016
    Messages:
    145
    Likes Received:
    23
    Trophy Points:
    18
    Location:
    UK
    cPanel Access Level:
    Root Administrator
    Yes there is a support ticket now open. I will update here as news comes through.
     
    #3 martin MHC, Jan 24, 2019
    Last edited by a moderator: Jan 24, 2019
  4. cPanelMichael

    cPanelMichael Technical Support Community Manager Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    47,335
    Likes Received:
    2,162
    Trophy Points:
    363
    cPanel Access Level:
    Root Administrator
    Twitter:
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  5. martin MHC

    martin MHC Well-Known Member

    Joined:
    Sep 14, 2016
    Messages:
    145
    Likes Received:
    23
    Trophy Points:
    18
    Location:
    UK
    cPanel Access Level:
    Root Administrator
    Hello @cPanelMichael - sure, as referenced just a couple of posts above - are you able to give any answers to points a,b,c -if these are possible currently and if so how to achieve them, or if not currently possible any chance they'd be candidates for future versions?

    Searching the online feature requests system I can't see anything for extending the 50k time limit or asking for more verbose error reporting in email (if possible), should I put these as future feature requests, or is there somethig I'm unaware of as to these topics?

    Cheers
     
  6. cPWilliamL

    cPWilliamL cP Technical Analyst II Staff Member

    Joined:
    May 15, 2017
    Messages:
    258
    Likes Received:
    30
    Trophy Points:
    103
    Location:
    America
    cPanel Access Level:
    Root Administrator
    Hi @martin MHC

    This should now be corrected in the 'development' branch. Please update the Github issues if you have any problems.

    Thanks!
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
    martin MHC likes this.
  7. quanin

    quanin Well-Known Member

    Joined:
    Aug 18, 2011
    Messages:
    114
    Likes Received:
    3
    Trophy Points:
    68
    cPanel Access Level:
    Root Administrator
    @cPWilliamL, have you corrected for the issue of large files filling up /tmp in the development branch as well? I understand the original version of this script can't feed files over 1 GB due to API limitations. I had opened a support ticket previously without coming to the forums, and was told this was being worked on.

    Also, please tell me you're advocating for this to be added to CPanel by default. Please? I find Dropbox a lot less finicky than Google Drive, which I only signed up for because CPanel backups.
     
  8. cPanelMichael

    cPanelMichael Technical Support Community Manager Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    47,335
    Likes Received:
    2,162
    Trophy Points:
    363
    cPanel Access Level:
    Root Administrator
    Twitter:
    Hello @quanin,

    It looks like the issue was addressed, but I did want to answer your earlier questions:

    1. A timeout value between 300 and 50,000 seconds is required and entering a value higher than 50,000 seconds is not supported.

    2. The timeout is enforced on each upload attempt (e.g. each attempt to upload account.tar.gz to the remote destination). This is not a configurable option.

    3. Transport failures/errors are logged to the /usr/local/cpanel/logs/cpbackup_transporter.log file. Is there a specific event/error that isn't currently logged?

    I don't have anything new to report on the status of native support for DropBox integrated into the product as a backup destination, but I do see a fair amount of votes on the official feature request page:

    Add remote backup destination Dropbox

    I suggest adding voting and adding feedback directly to the feature request page if you haven't already done so. I've voted for it myself, and will push others to vote for it whenever I see questions/feedback about the use of DropBox as a backup destination. This is the best approach to getting native support added to the product.

    Thank you.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  9. martin MHC

    martin MHC Well-Known Member

    Joined:
    Sep 14, 2016
    Messages:
    145
    Likes Received:
    23
    Trophy Points:
    18
    Location:
    UK
    cPanel Access Level:
    Root Administrator
    Hello @cPanelMichael
    Just as a final note - William has put a lot of effort into this issue for me/us (and others) and the development branch on the GitHub is now the main branch/master.

    CpanelInc/backup-transport-dropbox

    This script now successfully and correctly resolves my issues raised in this thread and causes complete file transfers to my Dropbox destination.

    Many thanks to all for this :)
     
    cPanelMichael likes this.
Loading...

Share This Page

  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice