How to implement copyFromLocal functionality in Java Application

Document created by snayeem Employee on Feb 27, 2016Last modified by snayeem Employee on Feb 27, 2016
Version 2Show Document
  • View in full screen mode

Author: Sanjamala Nayeem

Original Publication Date: April 25, 2015

 

If there is a requirement to implement copyFromLocal functionality in customer's application, the below sample will help them to understand.  It is a sample program to copy file from local system to Mapr file system.This example need to run on one of the node, this program is same implementation as command hadoop fs –copyFromLocal.

 

Basic elements of Copy from localHadoop has generic configuration object which we will use to specify to write to MaprFileSystem.import org.apache.hadoop.conf.Configuration;

  • Now create a conf object
Configuration conf = new Configuration();
  • Specify to write to MaprFileSystem in configuration object, this specifically set the configuration to take destination location as Mapr file system. This is optional if you are setting environment variable MAPR_HOME.
conf.set("fs.maprfs.impl", "com.mapr.fs.MapRFileSystem");

 

  • Get the filesystem access to the destination path
Path dstPath = new Path(dst); 
FileSystem dstFs = dstPath.getFileSystem(conf);

 

  • Then call method copyFromLocal on filesystem object. First parameter Boolean value indicate whether to delete the src after copying is complete, Second parameter boolean value indicating whether to overwrite an existing file in destination true indicate to overwrite, third parameter is source on local system, and fourth is the destination path in mapr file system.
dstFs.copyFromLocalFile(false, true, new Path(src), new Path(dst));

Sample Standalone program.

 

import org.apache.hadoop.conf.Configuration;

import org.apache.hadoop.fs.FileSystem;

import org.apache.hadoop.fs.Path;

 

public class TestLocalCopy {

  public static void main(String[] args) {

  try {

  if(args.length <3)

  printUsage(); 

  Configuration conf = new Configuration();

  conf.set("fs.maprfs.impl", "com.mapr.fs.MapRFileSystem");

  System.out.println("config done");

  String src = args[0];

  System.out.println("src is "+src);

  String dst = args[1];

  System.out.println("dst is "+dst);

  Path dstPath = new Path(dst);

  FileSystem dstFs = dstPath.getFileSystem(conf);

  System.out.println("copy from local is initiated for src "+src+" to dest "+dst);

  dstFs.copyFromLocalFile(false, true, new Path(src), new Path(dst));

  System.out.println("copy done");

  }

  catch (Exception ex){

  ex.printStackTrace();

  }

  }

  public static void printUsage()

  {

  System.out.println("TestLocalCopy <local File> <MFS path>");

  System.out.println("Example ");

  System.out.println("TestLocalCopy /root/FileTocopy /user/root");

  }

}

Compiling and executing the code

  • The following example commands are given based on MapR 3.0.2 version. Please modify the jar file name (dependencies) based on your MapR Version.
  • The code above would need the following libraries to compile successfully

 

maprfs-1.0.3-mapr-3.0.2.jar
hadoop-0.20.2-dev-core.jar

 

  • So copy the above to any convenient location and compile the program as follows:

 

javac -cp .:"lib/*" TestLocalCopy.java
[root@ip-10-147-237-177 3.1securityon]# ll src/
total 8
-rw-r--r--. 1 rootroot3979 Apr8 15:34 TestLocalCopy.class
-rw-r--r--. 1 root     root     3061 Apr  8 15:08 TestLocalCopy.java

 

  • Now execute this and it will generate the following output on the console :

java -Dmapr.home.dir=/opt/mapr -Djava.library.path=/opt/mapr/hadoop/hadoop-0.20.2/lib/native/Linux-amd64-64 -classpath /opt/mapr/hadoop/hadoop-0.20.2/lib/zookeeper-3.3.6.jar:/opt/mapr/hadoop/hadoop-0.20.2/lib/maprfs-1.0.3-mapr-3.0.2.jar:/opt/mapr/hadoop/hadoop-0.20.2/lib/commons-logging-1.0.4.jar:/opt/mapr/hadoop/hadoop-0.20.2/lib/commons-logging-api-1.0.4.jar:/opt/mapr/hadoop/hadoop-0.20.2/lib/log4j-1.2.15.jar:/opt/mapr/hadoop/hadoop-0.20.2/lib/hadoop-0.20.2-dev-core.jar:/root/testloclcopy.jar TestLocalCopy /root/<localFile> /user/root/

Attachments

    Outcomes