AnsweredAssumed Answered

MapR maven dependency conflicts with existing Hadoop dependency

Question asked by niranjan on Feb 19, 2015
Latest reply on Sep 19, 2016 by niranjan

Hi, 
Sanjamala NayeemMadalina Papacica
I'm working on a project which has a dependency on Apache Hadoop related jars, in this project I also have a module which reads from or writes to HDFS. 

I tried testing this module against MapR-FS and it works fine (I've installed the client as required). But in order to make this work I did the following 

I had to remove Apache related Hadoop jars from my classpath, having both MapR client dependency /opt/mapr/lib and Apache Hadoop jars in classpath resulted in the following exception  

          java.io.IOException: No FileSystem for scheme: maprfs.

 

I could understand there is a jar conflict but is there a way or some configuration that I could specify in my module or client so that it could force my application to load correct dependency? 

 

For instance would something like this help?

 

Configuration conf = new Configuration();
conf.set("fs.defaultFS",url);
conf.set("fs.hdfs.impl","com.mapr.fs.MapRFileSystem");
FileSystem fs = FileSystem.get( conf);

 

The above didn't work actually but maybe something like above.

 

 Can anyone help to find out the why there is a conflict? Is this a general class loading issue? I thought I won't run into any class loading issue as both MapR and Apache Hadoop jars would be binary compatible. 

Outcomes