AnsweredAssumed Answered

Submitting jobs to MapR without full client installation

Question asked by dominic on May 10, 2013
Latest reply on May 11, 2013 by Ted Dunning
We are working on a Spring Batch/Spring Hadoop project that will submit jars to MapR.

The project works in that it submits jars containing a Tool class successfully to the MapR JobTracker we have installed on a VM, but now we are trying to configure our project so developers who need to run integration tests locally with our VM do not have to install the MapR client.  Same situation for CI servers.

It seems clear that code executing in hadoop-core-1.0.3-mapr.2.1.2.1.jar or maprfs-1.0.3-mapr-2.1.2.1.jar is scanning for a MapR client on my development machine.  I have no environment variables that point to /opt/mapr, but it is clearly finding it.  When I removed /opt/mapr my app fails to run.

In my Hadoop Configuration object I am only setting "mapred.job.tracker" and "fs.default.name".  Do I need to pickup all the other xml files in opt/mapr/hadoop/hadoop-0.20.2/conf as well?

My questions:
1) What exactly does it need to find in the MapR client installation?  If it's just configuration files I can add those to my project.

2) Are the native libraries in /opt/mapr/... picked up at runtime?  My project runs fine without -Djava.library.path=/opt/mapr/..., but I am wondering if they are discovered at in the /opt/mapr/... at run-time as well.

Thanks,
~Dominic

Outcomes