Configure SpringXD on MapR 4.x

Document created by wade on Feb 27, 2016
Version 1Show Document
  • View in full screen mode

Author: Jitendra Yadav

Original Publication Date: February 20, 2015


Below are the steps to configure SpringXD on MapR 4.x.


1. Download SpringXD tar from “” and unzip it.


2.  Switch to SpringXD directory.

cd spring-xd-1.0.3.RELEASE


3. Add below JARs under spring-xd-1.0.3.RELEASE/xd/lib/hadoop24/server and spring-xd-1.0.3.RELEASE/shell/lib

(You may create a symbolic link of these jars)


4. Inside file spring-xd-1.0.3.RELEASE/xd/config/servers.yml, change MapR zookeeper info and MapRFS Uri


# Zookeeper properties

# namespace is the path under the root where XD's top level nodes will be created

# client connect string: host1:port1,host2:port2,...,hostN:portN


namespace: xd




# Hadoop properties



fsUri: maprfs:///


5. Export below Linux Env variable in shell before starting XD server

export JAVA_OPTS=" -Djava.library.path=/opt/mapr/lib/ -Dzookeeper.sasl.client=false"

6. Start XD server with below option

./xd/bin/xd-singlenode  --hadoopDistro hadoop24 &

7. Once XD server started, start XD shell from a different Linux window and create a streaming job.

./bin/xd-shell --hadoopDistro hadoop24  
xd:>hadoop config fs --namenode maprfs:///
xd:>hadoop fs ls /
xd:>stream create --name ticktockhdfs --definition "time | hdfs"
xd:>stream deploy ticktockhdfs
xd:>stream destroy ticktockhdfs

8. Check the output file under below directory, the file should contain proper date time formate as created during spring stream job.

hadoop fs -ls /xd/ticktockhdfs/ hadoop fs -cat  /xd/ticktockhdfs/ticktockhdfs<>.txt

Common Issues

1. Symtom:

Caused by:
ion: No FileSystem for scheme: maprfs


Make sure we have proper MapR specific jar files inside xd server and shell lib, please refer step 3. Also note that these jars version may differ according to your current MapR release.


2. Symptom:

Caused by: failure to login: No LoginModules configured for hadoop_simple


Since MapR has its customize Java authentication module, therefore we need to add MapR Auth conf in SpringXD jvm option.

Please refer step 5.

1 person found this helpful