AnsweredAssumed Answered

How to configure Spring Batch tasklet for launching Spark jobs in YARN

Question asked by Velumani on Jun 2, 2016
Latest reply on Jun 3, 2016 by Velumani

Hi,

     Currently I am trying to launch spark jobs using Spring Batch tasklet. I am following the spring doc 10. Apache Spark integration  and trying to execute the sample project springone-2015/batch-spark at master · trisberg/springone-2015 · GitHub  .

     I am running the code in my eclipse and i have Mapr-client installed in windows.

     In that code, the hadoop configuration uses fsUri: hdfs://localhost:8020, since there will be no Name Node in Mapr. I with fsUri: maprfs://localhost:7222 as well as fsUri: maprfs:///

 

 

I am getting No FileSystem for scheme: maprfs Exception

 

[2016-06-02 17:26:24.698] boot - 13344  WARN [main] --- AnnotationConfigApplicationContext: Exception encountered during context initialization - cancelling refresh attempt

org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'hadoopFsShell' defined in class path resource [org/springframework/data/hadoop/boot/HadoopFsShellAutoConfiguration.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.data.hadoop.fs.FsShell]: Factory method 'fsShell' threw exception; nested exception is org.springframework.data.hadoop.HadoopException: Cannot create shell No FileSystem for scheme: maprfs

  at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:599)

  at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1123)

  at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1018)

  at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:510)

  at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:482)

  at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:305)

  at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)

  at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:301)

  at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:196)

  at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:772)

  at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:834)

  at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:537)

  at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:667)

  at org.springframework.boot.SpringApplication.doRun(SpringApplication.java:342)

  at org.springframework.boot.SpringApplication.run(SpringApplication.java:273)

  at org.springframework.boot.SpringApplication.run(SpringApplication.java:980)

  at org.springframework.boot.SpringApplication.run(SpringApplication.java:969)

  at com.springdeveloper.demo.SparkYarnApplication.main(SparkYarnApplication.java:23)

Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.data.hadoop.fs.FsShell]: Factory method 'fsShell' threw exception; nested exception is org.springframework.data.hadoop.HadoopException: Cannot create shell No FileSystem for scheme: maprfs

  at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:189)

  at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:588)

  ... 17 more

Caused by: org.springframework.data.hadoop.HadoopException: Cannot create shell No FileSystem for scheme: maprfs

  at org.springframework.data.hadoop.fs.FsShell.<init>(FsShell.java:100)

  at org.springframework.data.hadoop.fs.FsShell.<init>(FsShell.java:84)

  at org.springframework.data.hadoop.boot.HadoopFsShellAutoConfiguration.fsShell(HadoopFsShellAutoConfiguration.java:47)

  at org.springframework.data.hadoop.boot.HadoopFsShellAutoConfiguration$$EnhancerBySpringCGLIB$$453f28ce.CGLIB$fsShell$0(<generated>)

  at org.springframework.data.hadoop.boot.HadoopFsShellAutoConfiguration$$EnhancerBySpringCGLIB$$453f28ce$$FastClassBySpringCGLIB$$1ddddbfc.invoke(<generated>)

  at org.springframework.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)

  at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:318)

  at org.springframework.data.hadoop.boot.HadoopFsShellAutoConfiguration$$EnhancerBySpringCGLIB$$453f28ce.fsShell(<generated>)

  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

  at java.lang.reflect.Method.invoke(Method.java:498)

  at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:162)

  ... 18 more

Caused by: java.io.IOException: No FileSystem for scheme: maprfs

  at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2644)

  at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2651)

  at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:92)

  at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2687)

  at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2669)

  at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:371)

  at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:170)

  at org.springframework.data.hadoop.fs.FsShell.<init>(FsShell.java:96)

  ... 30 more

 

I saw the document in mapr community about configuring springXD on MapR 4.x (Configure SpringXD on MapR 4.x ) and suggestions about the above exception but I don't know how to fix my issue as I don't where to copy the mapr dependencies

 

Outcomes