Accessing MapR FileSystem without installing MapR Client

Document created by snayeem Employee on Feb 27, 2016
Version 1Show Document
  • View in full screen mode

Author: Sanjamala Nayeem

Original Publication Date: May 10, 2015

 

Env:

  • MapR v3.1.1+ latest EBF patch (mapr-patch-3.1.1.26113.GA-31296 onwards)
  • MapR v4.0.2

Goal:

To access MapR file system from application without installing MapR client for un-secure cluster.

Solutions:

To achieve this MapR has provided following configuration parameter to set cluster details at run time which allows to connect to that cluster.

conf.set("dfs.nameservices", clusterName); 
conf.set("dfs.ha.namenodes." + clusterName, "cldb1,cldb2,cldb3");
conf.set("dfs.namenode.rpc-address." + clusterName + ".cldb1",cldb1node + ":7222");
  • In the above snippet, we need to provide cluster name against dfs.nameservices.
  • If we have HA implemented for CLDB, need to specify how many CLDB's  we have. In this example, we are having three cldbs, configured as cldb1,cldb2.cldb3 and against dfs.namenodes.<clustername>
  • Hostname of CLDB or IP address of CLDB with port  7222 need to be set against dfs.namenode.rpc-address.<clustername>.<cldbno>

 

import org.apache.hadoop.conf.Configuration;

import org.apache.hadoop.fs.FileSystem;

import org.apache.hadoop.fs.Path;

 

public class SampleFsAccess {

 

public static void main(String[] args) {

Configuration conf = new Configuration();

String clusterName = "gauth";

String cldb1node = "10.10.70.1";

String cldb2node = "10.10.70.2";

String cldb3node = "10.10.70.3";

 

conf.setBoolean("fs.mapr.impl.clustername.unique", false);

 

conf.set("dfs.nameservices", clusterName);

conf.set("dfs.ha.namenodes." + clusterName, "cldb1,cldb2,cldb3");

conf.set("dfs.namenode.rpc-address." + clusterName + ".cldb1",cldb1node + ":7222");

conf.set("dfs.namenode.rpc-address." + clusterName + ".cldb2",cldb2node + ":7222");

conf.set("dfs.namenode.rpc-address." + clusterName + ".cldb3",cldb3node + ":7222");

System.out.println("Check for FS");

try {

FileSystem maprFs = FileSystem.get(conf);

Path p = new Path("maprfs:/mapr/gauth/test"

+ System.currentTimeMillis());

System.out.println("Path: " + p);

maprFs.create(p);

} catch (Exception ex) {

ex.printStackTrace();

}

 

}

 

}

  • In the above code, we are trying to create a file named "filetest<current-timestamp>".
1 person found this helpful

Attachments

    Outcomes