AnsweredAssumed Answered

MapR maven dependency conflicts with existing Hadoop dependency

Question asked by niranjan on Sep 19, 2016
Latest reply on Oct 30, 2017 by snayeem

Hi, 


I'm working on a project which has a dependency on Apache Hadoop related jars, in this project I also have a module which reads from or writes to HDFS. 

I tried testing this module against MapR-FS and it works fine (I've installed the client as required). But in order to make this work I did the following 

I had to remove Apache related Hadoop jars from my classpath, having both MapR client dependency /opt/mapr/lib and Apache Hadoop jars in classpath resulted in the following exception  

 

         

 java.io.IOException: No FileSystem for scheme: maprfs.

 

I could understand there is a jar conflict but is there a way or some configuration that I could specify in my module or client so that it could force my application to load correct dependency? 

 

For instance would something like this help?

 

Configuration conf = new Configuration();
conf.set("fs.defaultFS",url);
conf.set("fs.hdfs.impl","com.mapr.fs.MapRFileSystem");
FileSystem fs = FileSystem.get( conf);

The above didn't work actually but looking for a suggestion something like above.

 

 Can anyone help me to find out the why there is a conflict or explain me what is happening here? Is this a general class loading issue? 

 

Sanjamala Nayeem Ted Dunning MC Srivas Nabeel Moidu

Outcomes