AnsweredAssumed Answered

How do you start the MiniYarnCluster in MapR 5.1 for testing Spark Applications in Yarn

Question asked by jleach on May 4, 2016
Latest reply on Jun 3, 2016 by maprcommunity


When I start the MiniYarnCluster it complains with this stack trace...  The goal here is to unit test Spark Applications in Yarn in maven based, java only projects.  Help...

 

 

15:39:46,392 (main) INFO  [c.m.h.y.r.RMVolumeManager] - Checking for ResourceManager volume. If volume not present command will create and mount it. Command invoked is : /opt/mapr/server/createJTVolume.sh MAPR_CLIENT /var/mapr/cluster/yarn/rm /var/mapr/cluster/yarn/rm yarn

15:39:46,500 (main) INFO  [c.m.h.y.r.RMVolumeManager] - Retrying check for ResourceManager volume ...

15:39:46,607 (main) INFO  [c.m.h.y.r.RMVolumeManager] - Retrying check for ResourceManager volume ...

15:39:46,612 (main) FATAL [o.a.h.y.s.a.ConfigurableAuxServices] - Failed to initialize RMVolumeManager

org.apache.hadoop.service.ServiceStateException: java.io.IOException: Cannot run program "/opt/mapr/server/createJTVolume.sh": error=2, No such file or directory

  at org.apache.hadoop.service.ServiceStateException.convert(ServiceStateException.java:59)

  at org.apache.hadoop.service.AbstractService.init(AbstractService.java:172)

  at org.apache.hadoop.yarn.server.api.ConfigurableAuxServices.serviceInit(ConfigurableAuxServices.java:94)

  at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)

  at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)

  at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:268)

  at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)

  at org.apache.hadoop.yarn.server.MiniYARNCluster.initResourceManager(MiniYARNCluster.java:293)

  at org.apache.hadoop.yarn.server.MiniYARNCluster.access$400(MiniYARNCluster.java:97)

  at org.apache.hadoop.yarn.server.MiniYARNCluster$ResourceManagerWrapper.serviceInit(MiniYARNCluster.java:444)

  at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)

  at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)

  at org.apache.hadoop.yarn.server.MiniYARNCluster.serviceInit(MiniYARNCluster.java:269)

  at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)

  at org.apache.hadoop.yarn.server.TestMiniYARNClusterForHA.setup(TestMiniYARNClusterForHA.java:30)

  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

  at java.lang.reflect.Method.invoke(Method.java:606)

  at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)

  at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)

  at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)

  at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)

  at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)

  at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)

  at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)

  at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)

  at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)

  at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)

  at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)

  at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)

  at org.junit.runners.ParentRunner.run(ParentRunner.java:363)

  at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:344)

  at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:269)

  at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:240)

  at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:184)

  at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:286)

  at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:240)

  at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:121)

Caused by: java.io.IOException: Cannot run program "/opt/mapr/server/createJTVolume.sh": error=2, No such file or directory

  at java.lang.ProcessBuilder.start(ProcessBuilder.java:1047)

  at org.apache.hadoop.util.Shell.runCommand(Shell.java:486)

  at org.apache.hadoop.util.Shell.run(Shell.java:456)

  at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:722)

  at com.mapr.hadoop.yarn.resourcemanager.RMVolumeManager.createVolume(RMVolumeManager.java:95)

  at com.mapr.hadoop.yarn.resourcemanager.RMVolumeManager.serviceInit(RMVolumeManager.java:46)

  at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)

  ... 37 more

Caused by: java.io.IOException: error=2, No such file or directory

  at java.lang.UNIXProcess.forkAndExec(Native Method)

  at java.lang.UNIXProcess.<init>(UNIXProcess.java:184)

  at java.lang.ProcessImpl.start(ProcessImpl.java:130)

  at java.lang.ProcessBuilder.start(ProcessBuilder.java:1028)

  ... 43 more

Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 2.151 sec <<< FAILURE! - in org.apache.hadoop.yarn.server.TestMiniYARNClusterForHA

testClusterWorks(org.apache.hadoop.yarn.server.TestMiniYARNClusterForHA)  Time elapsed: 2.151 sec  <<< ERROR!

org.apache.hadoop.service.ServiceStateException: java.io.IOException: Cannot run program "/opt/mapr/server/createJTVolume.sh": error=2, No such file or directory

  at org.apache.hadoop.yarn.server.TestMiniYARNClusterForHA.setup(TestMiniYARNClusterForHA.java:30)

Caused by: java.io.IOException: Cannot run program "/opt/mapr/server/createJTVolume.sh": error=2, No such file or directory

  at org.apache.hadoop.yarn.server.TestMiniYARNClusterForHA.setup(TestMiniYARNClusterForHA.java:30)

Caused by: java.io.IOException: error=2, No such file or directory

  at org.apache.hadoop.yarn.server.TestMiniYARNClusterForHA.setup(TestMiniYARNClusterForHA.java:30)

 

 

Results :

 

Tests in error:

  TestMiniYARNClusterForHA.setup:30 » ServiceState java.io.IOException: Cannot r...

Outcomes