Configure SpringXD on MapR 4.x

Document created by wade on Feb 27, 2016
Version 1Show Document
  • View in full screen mode

Author: Jitendra Yadav

Original Publication Date: February 20, 2015

 

Below are the steps to configure SpringXD on MapR 4.x.

 

1. Download SpringXD tar from “spring-xd-1.0.3.RELEASE-dist.zip” and unzip it.

 

2.  Switch to SpringXD directory.

cd spring-xd-1.0.3.RELEASE

 

3. Add below JARs under spring-xd-1.0.3.RELEASE/xd/lib/hadoop24/server and spring-xd-1.0.3.RELEASE/shell/lib

(You may create a symbolic link of these jars)

/opt/mapr/lib/hadoop-auth-2.4.1.jar 
/opt/mapr/lib/hadoop-0.20.2-dev-core.jar
/opt/mapr/lib/hadoop-common-2.4.1.jar
/opt/mapr/lib/maprfs-4.0.1-mapr.jar
/opt/mapr/lib/json-20080701.jar

4. Inside file spring-xd-1.0.3.RELEASE/xd/config/servers.yml, change MapR zookeeper info and MapRFS Uri

 

# Zookeeper properties

# namespace is the path under the root where XD's top level nodes will be created

# client connect string: host1:port1,host2:port2,...,hostN:portN

zk:

namespace: xd

client:  

connect: 10.10.70.92:5181 

 

# Hadoop properties

spring:  

hadoop:   

fsUri: maprfs:///

 

5. Export below Linux Env variable in shell before starting XD server

export JAVA_OPTS=" -Djava.library.path=/opt/mapr/lib/ -Djava.security.auth.login.config=/opt/mapr/conf/mapr.login.conf -Dzookeeper.sasl.client=false"

6. Start XD server with below option

./xd/bin/xd-singlenode  --hadoopDistro hadoop24 &

7. Once XD server started, start XD shell from a different Linux window and create a streaming job.

./bin/xd-shell --hadoopDistro hadoop24  
xd:>hadoop config fs --namenode maprfs:///
xd:>hadoop fs ls /
xd:>stream create --name ticktockhdfs --definition "time | hdfs"
xd:>stream deploy ticktockhdfs
xd:>stream destroy ticktockhdfs

8. Check the output file under below directory, the file should contain proper date time formate as created during spring stream job.

hadoop fs -ls /xd/ticktockhdfs/ hadoop fs -cat  /xd/ticktockhdfs/ticktockhdfs<>.txt

Common Issues

1. Symtom:

Caused by: java.io.IOExcept
ion: No FileSystem for scheme: maprfs

Solution:

Make sure we have proper MapR specific jar files inside xd server and shell lib, please refer step 3. Also note that these jars version may differ according to your current MapR release.

 

2. Symptom:

Caused by: java.io.IOException: failure to login: No LoginModules configured for hadoop_simple

Solution:

Since MapR has its customize Java authentication module, therefore we need to add MapR Auth conf in SpringXD jvm option.

Please refer step 5.

1 person found this helpful

Attachments

    Outcomes