The Community Forums

Interact with an entire community of cPanel & WHM users!
  1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Backups hanging

Discussion in 'Data Protection' started by theallan, May 30, 2015.

  1. theallan

    theallan Active Member

    Joined:
    Nov 17, 2005
    Messages:
    32
    Likes Received:
    1
    Trophy Points:
    8
    I've just realised that the automated back I've set up in WHM (11.48.4 build 4, CENTOS 6.6 x86_64) hasn't been running for the last month (yikes).

    This appears to be caused by a backup process that had hung and therefore (presumably) no other backup process would run.

    I've just manually tried running the backup script (/usr/local/cpanel/bin/backup) and it runs, and then hangs while creating the archive for an account:

    [2015-05-30 18:38:39 +0100] Copying cpuser file.......[2015-05-30 18:38:39 +0100] Done
    [2015-05-30 18:38:39 +0100] Copying crontab file.......[2015-05-30 18:38:39 +0100] Done
    [2015-05-30 18:38:39 +0100] Copying quota info.......[2015-05-30 18:38:39 +0100] Done
    [2015-05-30 18:38:39 +0100] Storing Subdomains....
    [2015-05-30 18:38:39 +0100] Done
    [2015-05-30 18:38:39 +0100] Storing Parked Domains....
    [2015-05-30 18:38:39 +0100] Done
    [2015-05-30 18:38:39 +0100] Storing Addon Domains....
    [2015-05-30 18:38:39 +0100] Done
    [2015-05-30 18:38:39 +0100] Storing SSL domain......[2015-05-30 18:38:39 +0100] Done
    [2015-05-30 18:38:39 +0100] Copying password.......[2015-05-30 18:38:39 +0100] Done
    [2015-05-30 18:38:39 +0100] Copying shell.......[2015-05-30 18:38:39 +0100] Done
    [2015-05-30 18:38:39 +0100] Creating Archive ....................................................................................
    ................................................................................​

    A few more lines of dots and then nothing. strace on the backup process unfortunately shows very little of use:

    Process 15233 attached - interrupt to quit
    wait4(16698, ​

    So I guess the most important question is, how can I complete a backup?!

    Secondary question - should I not have received a warning that backups were not running due to an existing backup process?

    Allan
     
  2. theallan

    theallan Active Member

    Joined:
    Nov 17, 2005
    Messages:
    32
    Likes Received:
    1
    Trophy Points:
    8
    A little update on this - I've disabled the backups for the account that the hang appears to occur on and the back now completes for all other accounts. But of course I have one account that it now not being backed up due to this issue.

    Does anyone know what might be causing this and how to resolve it?

    Allan
     
  3. cPanelMichael

    cPanelMichael Forums Analyst
    Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    30,854
    Likes Received:
    676
    Trophy Points:
    113
    cPanel Access Level:
    Root Administrator
    Hello,

    How large is the account that fails to backup? Have you tried manually generating a backup for this account via "/scripts/pkgacct"?

    Thank you.
     
  4. theallan

    theallan Active Member

    Joined:
    Nov 17, 2005
    Messages:
    32
    Likes Received:
    1
    Trophy Points:
    8
    Thanks for your reply. The account causing the problem is 767MB is size. Running `/scripts/pkgacct` for that account also hangs while "Creating Archive".
     
  5. cPanelMichael

    cPanelMichael Forums Analyst
    Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    30,854
    Likes Received:
    676
    Trophy Points:
    113
    cPanel Access Level:
    Root Administrator
    Do you have any process resource monitors installed that could be preventing the backup from completing? How's your resource usage during the backup? Are you reaching any memory limits, or VPS resource limits (if it's a VPS)?

    Thank you.
     
  6. theallan

    theallan Active Member

    Joined:
    Nov 17, 2005
    Messages:
    32
    Likes Received:
    1
    Trophy Points:
    8
    From `top` at the point when the hang occurs:

    top - 14:08:14 up 382 days, 4:35, 3 users, load average: 1.74, 1.15, 0.66
    Tasks: 379 total, 2 running, 377 sleeping, 0 stopped, 0 zombie
    Cpu(s): 5.2%us, 2.3%sy, 0.0%ni, 87.4%id, 5.0%wa, 0.0%hi, 0.1%si, 0.0%st
    Mem: 32836812k total, 27873704k used, 4963108k free, 711472k buffers
    Swap: 16490488k total, 685112k used, 15805376k free, 21474548k cached​

    During the "Creating Archive" step up until that point gzip is using 100% CPU. Then it hangs and gzip disappears from the top list.

    The backup process is still running:

    # ps aux | grep pkg
    root 4248 2.7 0.1 265344 56704 pts/1 S+ 14:05 0:07 pkgacct - myAccount - av: 3
    myAccount 4572 1.4 0.1 265344 54288 pts/1 S+ 14:05 0:03 pkgacct - myAccount - av: 3
    root 5145 0.0 0.0 103244 892 pts/2 S+ 14:09 0:00 grep pkg​

    gzip also appears to still be running

    # ps aux | grep gzip
    root 4571 39.2 0.0 4300 804 pts/1 S+ 14:05 2:03 /usr/bin/gzip -6
    root 5250 0.0 0.0 103244 892 pts/2 S+ 14:11 0:00 grep gzip​

    And there is loads of disk space available (no disk has above 50% usage).

    Allan
     
  7. cPanelMichael

    cPanelMichael Forums Analyst
    Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    30,854
    Likes Received:
    676
    Trophy Points:
    113
    cPanel Access Level:
    Root Administrator
    Feel free to open a support ticket using the link in my signature so we can take a closer look. You can post the ticket number here so we can update this thread with the outcome.

    Thank you.
     
  8. theallan

    theallan Active Member

    Joined:
    Nov 17, 2005
    Messages:
    32
    Likes Received:
    1
    Trophy Points:
    8
  9. theallan

    theallan Active Member

    Joined:
    Nov 17, 2005
    Messages:
    32
    Likes Received:
    1
    Trophy Points:
    8
    For anyone else who comes across this, the issue was a FIFO file that had been created by `screen`. The gzip process would just hang on that.

    A fix is apparently in place and coming in a future release according to the ticket. Until then, removing the FIFO or adding it to the backup ignore files will do the job.

    Allan
     
  10. cPanelMichael

    cPanelMichael Forums Analyst
    Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    30,854
    Likes Received:
    676
    Trophy Points:
    113
    cPanel Access Level:
    Root Administrator
    Hello,

    Internal case number 185949 is open to address an issue where pkgacct hangs on pipe files. There's currently no time frame available for a resolution, but you can monitor our change log for the case number. The temporary workaround at this time is to exclude the path to the file in the /etc/cpbackup-exclude.conf per the instructions at:

    How to exclude files from backup

    Thank you.
     
  11. theallan

    theallan Active Member

    Joined:
    Nov 17, 2005
    Messages:
    32
    Likes Received:
    1
    Trophy Points:
    8
    Ah - thanks you for the clarification. I misunderstood the reply in the ticket in that case.

    Out of interest, this issue caused my backs to not run for about a month - should there be anything that monitors that and would notify me (other than me realising that the backups were not running - which I clean didn't!).

    Allan
     
  12. cPanelMichael

    cPanelMichael Forums Analyst
    Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    30,854
    Likes Received:
    676
    Trophy Points:
    113
    cPanel Access Level:
    Root Administrator
    You should already receive a notification if the backup fails. It looks like the backup process actually hung in this case due to the internal case that's open, and thus no notification was sent.

    Thank you.
     
  13. brianoz

    brianoz Well-Known Member

    Joined:
    Mar 13, 2004
    Messages:
    1,146
    Likes Received:
    6
    Trophy Points:
    38
    Location:
    Melbourne, Australia
    cPanel Access Level:
    Root Administrator
    This is obviously a frightening bug as it means a FIFO created in one account can stop backups across an *entire server*. I'm glad a fix is going through the process.

    The FIFO we had one one of our servers was called tmp/spamhack.fifo so another helpful trick might be to run this once to exclude all files with .fifo on the end. Limited, but it helps a little.

    Code:
    echo "*.fifo" >> /etc/cpbackup-exclude.conf
     
  14. cPanelMichael

    cPanelMichael Forums Analyst
    Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    30,854
    Likes Received:
    676
    Trophy Points:
    113
    cPanel Access Level:
    Root Administrator
Loading...

Share This Page