AnsweredAssumed Answered

What is the fastest way to move data from one volume to another?

Question asked by stefanpapp on Jul 27, 2018
Latest reply on Jul 30, 2018 by MichaelSegel

I want to move data from one volume to another. The folders and file sizes vary. Files can be up to 100 GB, but we can have also a lot of small files. If there is data in the destination volume at that particular folder, it can be overwritten.

So far, I tried (Code has been simplified for demonstration purposes)

   (1)for root, directories, files in os.walk(src):         for file in files:             mv -v <src> <dest>     (2)hadoop distcp -overwrite -m100 <src> <dest>

Less than 10 GB, the mv option is faster. At 10 GB both options take approx 2 minutes transfer time.

Is there a faster way?

distcp

 mapr-fs .

 

Outcomes