I want to move data from one volume to another. The folders and file sizes vary. Files can be up to 100 GB, but we can have also a lot of small files. If there is data in the destination volume at that particular folder, it can be overwritten.
So far, I tried (Code has been simplified for demonstration purposes)
(1)for root, directories, files in os.walk(src): for file in files: mv -v <src> <dest> (2)hadoop distcp -overwrite -m100 <src> <dest>
Less than 10 GB, the mv option is faster. At 10 GB both options take approx 2 minutes transfer time.
Is there a faster way?