The Community Forums

Interact with an entire community of cPanel & WHM users!
  1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Any SCP heroes here?

Discussion in 'General Discussion' started by Data 1, Feb 16, 2010.

  1. Data 1

    Data 1 Well-Known Member

    Joined:
    May 25, 2008
    Messages:
    113
    Likes Received:
    0
    Trophy Points:
    16
    Location:
    Columbus Ohio
    cPanel Access Level:
    DataCenter Provider
    I'm transferring about 250G of files from a non cpanel server to a cpanel server.
    I'm doing a SCP server to server transfer, going well.

    My question is... can you open multiple sessions and transfer different folders during an open session?

    If you can... do you gain anything?

    Thanks-

    Jim
     
  2. Eric

    Eric Administrator
    Staff Member

    Joined:
    Nov 25, 2007
    Messages:
    746
    Likes Received:
    11
    Trophy Points:
    18
    Location:
    Texas
    cPanel Access Level:
    Root Administrator
    Howdy,

    Sure you can have more than one transfer going, but if you have large files the data transfer rates will be split between them.

    When you have small files scp can take some time starting files and checking them after it's done with them. So you might gain some by doing a multi transfer there. But the way around this would be to tar/compress the files into a single file and unzip on the other side.

    Also multi-session helps when you're doing bi-directional transfers. Say you're downloading and uploading at the same time. Two connections is the way to go.
     
  3. Data 1

    Data 1 Well-Known Member

    Joined:
    May 25, 2008
    Messages:
    113
    Likes Received:
    0
    Trophy Points:
    16
    Location:
    Columbus Ohio
    cPanel Access Level:
    DataCenter Provider
    Thank you cPanelEricE for the response.

    You really think I should try and compress 250G of info? I'll try it with a folder I know is about 9G and see what I net.

    Thank you again, I have never started 2 SCP sessions in my life and didn't know if I was going to kill the existing one or not.

    Jim
     
  4. Data 1

    Data 1 Well-Known Member

    Joined:
    May 25, 2008
    Messages:
    113
    Likes Received:
    0
    Trophy Points:
    16
    Location:
    Columbus Ohio
    cPanel Access Level:
    DataCenter Provider
    Wow I have 5 sessions started and no noticeable drop in transfer speed on any of them.

    Thanks for the tips- I'm going to keep opening them till I'm out of folders!

    Jim
     
  5. Eric

    Eric Administrator
    Staff Member

    Joined:
    Nov 25, 2007
    Messages:
    746
    Likes Received:
    11
    Trophy Points:
    18
    Location:
    Texas
    cPanel Access Level:
    Root Administrator
    Howdy,

    I'd try using rar or split and make several moderate sized files that you can move over while keeping an eye on the md5sum hash to make sure the data is move across accurately.
     
  6. Data 1

    Data 1 Well-Known Member

    Joined:
    May 25, 2008
    Messages:
    113
    Likes Received:
    0
    Trophy Points:
    16
    Location:
    Columbus Ohio
    cPanel Access Level:
    DataCenter Provider
    This ended up being much deeper than it looked, still working on it. If I don't die of old age before I finish the project I'll post the quite interesting resulting saga.

    Tens of thousands of error log files. So large I couldn't view them. After 3 days SCP running 4 windows I gave up and started asking why.

    Thanks-
     
  7. Data 1

    Data 1 Well-Known Member

    Joined:
    May 25, 2008
    Messages:
    113
    Likes Received:
    0
    Trophy Points:
    16
    Location:
    Columbus Ohio
    cPanel Access Level:
    DataCenter Provider
    If anyone doesn't have a life and is following this...

    The files ended up being not needed. It was from a misfiring phpBB3 install that put 4 cached sql queries in the folder EVERY TIME ANYONE VISITED. So you can imagine the exponential gain in files. I never did determine exactly how many files were actually there but it was so many that "rm" did not work. Returned an error.

    I ended up using "find . -name "sql_*" -exec rm -rf {} \;" and after about a half day it did delete them all.
     
  8. Lyttek

    Lyttek Well-Known Member

    Joined:
    Jan 2, 2004
    Messages:
    770
    Likes Received:
    3
    Trophy Points:
    18

    Great.... another post confirming that I don't have a life :p
     
Loading...

Share This Page