AnsweredAssumed Answered

attempted duplicate class definition for name: "com/mapr/fs/jni/MapRConstants"

Question asked by nboger on Feb 14, 2018
Latest reply on Feb 16, 2018 by maprcommunity

Hi,

 

I'm trying to connect to MapR cluster using Scala.

MapR version 5.2.2 Spark 2.1.0

Here is a snippet of my pom.xml file

 

<dependency>
   <groupId>org.apache.spark</groupId>
   <artifactId>spark-core_2.11</artifactId>
   <version>2.1.0</version>
   <exclusions>
      <exclusion>
         <groupId>org.apache.hadoop</groupId>
         <artifactId>hadoop-client</artifactId>
      </exclusion>
   </exclusions>
</dependency>
<dependency>
   <groupId>org.apache.hadoop</groupId>
   <artifactId>hadoop-core</artifactId>
   <version>1.0.3-mapr-5.2.2</version>
</dependency>

 

Any idea why I'm getting this error? Have anyone knows how to solve it?

 

java.lang.reflect.InvocationTargetException
at sun.reflect.GeneratedMethodAccessor1.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at com.mapr.fs.ShimLoader.injectNativeLoader(ShimLoader.java:290)
at com.mapr.fs.ShimLoader.load(ShimLoader.java:223)
at com.mapr.fs.MapRFileSystem.<clinit>(MapRFileSystem.java:107)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at org.apache.hadoop.conf.Configuration.getClassByNameOrNull(Configuration.java:2159)
at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2124)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2218)
at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2678)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2691)
at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:98)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2773)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2755)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:411)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:176)
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:505)
at org.apache.spark.sql.hive.client.HiveClientImpl.<init>(HiveClientImpl.scala:188)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
at org.apache.spark.sql.hive.client.IsolatedClientLoader.createClient(IsolatedClientLoader.scala:264)
at org.apache.spark.sql.hive.HiveUtils$.newClientForMetadata(HiveUtils.scala:358)
at org.apache.spark.sql.hive.HiveUtils$.newClientForMetadata(HiveUtils.scala:262)
at org.apache.spark.sql.hive.HiveExternalCatalog.<init>(HiveExternalCatalog.scala:66)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
at org.apache.spark.sql.internal.SharedState$.org$apache$spark$sql$internal$SharedState$$reflect(SharedState.scala:166)
at org.apache.spark.sql.internal.SharedState.<init>(SharedState.scala:86)
at org.apache.spark.sql.SparkSession$$anonfun$sharedState$1.apply(SparkSession.scala:101)
at org.apache.spark.sql.SparkSession$$anonfun$sharedState$1.apply(SparkSession.scala:101)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.sql.SparkSession.sharedState$lzycompute(SparkSession.scala:101)
at org.apache.spark.sql.SparkSession.sharedState(SparkSession.scala:100)
at org.apache.spark.sql.internal.SessionState.<init>(SessionState.scala:157)
at org.apache.spark.sql.hive.HiveSessionState.<init>(HiveSessionState.scala:32)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
at org.apache.spark.sql.SparkSession$.org$apache$spark$sql$SparkSession$$reflect(SparkSession.scala:978)
at org.apache.spark.sql.SparkSession.sessionState$lzycompute(SparkSession.scala:110)
at org.apache.spark.sql.SparkSession.sessionState(SparkSession.scala:109)
at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:592)
at org.apache.spark.sql.SQLContext.sql(SQLContext.scala:699)
at com.delver.spark.JobRunner.TestJob.run(TestJob.scala:21)
at com.delver.spark.JobRunner.TestJob.run(TestJob.scala:12)
at com.delver.spark.JobRunner.SparkJobBase.start(SparkJobBase.scala:30)
at com.delver.spark.JobRunner.SparkDriver$.com$delver$spark$JobRunner$SparkDriver$$startJob(SparkDriver.scala:84)
at com.delver.spark.JobRunner.SparkDriver$$anonfun$main$2.apply(SparkDriver.scala:68)
at com.delver.spark.JobRunner.SparkDriver$$anonfun$main$2.apply(SparkDriver.scala:51)
at scala.Option.map(Option.scala:146)
at com.delver.spark.JobRunner.SparkDriver$.main(SparkDriver.scala:51)
at com.delver.spark.JobRunner.SparkDriver.main(SparkDriver.scala)
Caused by: java.lang.LinkageError: loader (instance of org/apache/spark/sql/hive/client/IsolatedClientLoader$$anon$1): attempted duplicate class definition for name: "com/mapr/fs/jni/MapRConstants"
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:760)
... 59 more
java.lang.RuntimeException: Failure loading MapRClient.
at com.mapr.fs.ShimLoader.injectNativeLoader(ShimLoader.java:305)
at com.mapr.fs.ShimLoader.load(ShimLoader.java:223)
at com.mapr.fs.MapRFileSystem.<clinit>(MapRFileSystem.java:107)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at org.apache.hadoop.conf.Configuration.getClassByNameOrNull(Configuration.java:2159)
at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2124)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2218)
at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2678)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2691)
at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:98)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2773)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2755)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:411)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:176)
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:505)
at org.apache.spark.sql.hive.client.HiveClientImpl.<init>(HiveClientImpl.scala:188)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
at org.apache.spark.sql.hive.client.IsolatedClientLoader.createClient(IsolatedClientLoader.scala:264)
at org.apache.spark.sql.hive.HiveUtils$.newClientForMetadata(HiveUtils.scala:358)
at org.apache.spark.sql.hive.HiveUtils$.newClientForMetadata(HiveUtils.scala:262)
at org.apache.spark.sql.hive.HiveExternalCatalog.<init>(HiveExternalCatalog.scala:66)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
at org.apache.spark.sql.internal.SharedState$.org$apache$spark$sql$internal$SharedState$$reflect(SharedState.scala:166)
at org.apache.spark.sql.internal.SharedState.<init>(SharedState.scala:86)
at org.apache.spark.sql.SparkSession$$anonfun$sharedState$1.apply(SparkSession.scala:101)
at org.apache.spark.sql.SparkSession$$anonfun$sharedState$1.apply(SparkSession.scala:101)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.sql.SparkSession.sharedState$lzycompute(SparkSession.scala:101)
at org.apache.spark.sql.SparkSession.sharedState(SparkSession.scala:100)
at org.apache.spark.sql.internal.SessionState.<init>(SessionState.scala:157)
at org.apache.spark.sql.hive.HiveSessionState.<init>(HiveSessionState.scala:32)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
at org.apache.spark.sql.SparkSession$.org$apache$spark$sql$SparkSession$$reflect(SparkSession.scala:978)
at org.apache.spark.sql.SparkSession.sessionState$lzycompute(SparkSession.scala:110)
at org.apache.spark.sql.SparkSession.sessionState(SparkSession.scala:109)
at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:592)
at org.apache.spark.sql.SQLContext.sql(SQLContext.scala:699)
at com.delver.spark.JobRunner.TestJob.run(TestJob.scala:21)
at com.delver.spark.JobRunner.TestJob.run(TestJob.scala:12)
at com.delver.spark.JobRunner.SparkJobBase.start(SparkJobBase.scala:30)
at com.delver.spark.JobRunner.SparkDriver$.com$delver$spark$JobRunner$SparkDriver$$startJob(SparkDriver.scala:84)
at com.delver.spark.JobRunner.SparkDriver$$anonfun$main$2.apply(SparkDriver.scala:68)
at com.delver.spark.JobRunner.SparkDriver$$anonfun$main$2.apply(SparkDriver.scala:51)
at scala.Option.map(Option.scala:146)
at com.delver.spark.JobRunner.SparkDriver$.main(SparkDriver.scala:51)
at com.delver.spark.JobRunner.SparkDriver.main(SparkDriver.scala)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.GeneratedMethodAccessor1.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at com.mapr.fs.ShimLoader.injectNativeLoader(ShimLoader.java:290)
... 55 more
Caused by: java.lang.LinkageError: loader (instance of org/apache/spark/sql/hive/client/IsolatedClientLoader$$anon$1): attempted duplicate class definition for name: "com/mapr/fs/jni/MapRConstants"
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:760)
... 59 more
Exception in thread "main" java.lang.IllegalArgumentException: Error while instantiating 'org.apache.spark.sql.hive.HiveSessionState':
at org.apache.spark.sql.SparkSession$.org$apache$spark$sql$SparkSession$$reflect(SparkSession.scala:981)
at org.apache.spark.sql.SparkSession.sessionState$lzycompute(SparkSession.scala:110)
at org.apache.spark.sql.SparkSession.sessionState(SparkSession.scala:109)
at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:592)
at org.apache.spark.sql.SQLContext.sql(SQLContext.scala:699)
at com.delver.spark.JobRunner.TestJob.run(TestJob.scala:21)
at com.delver.spark.JobRunner.TestJob.run(TestJob.scala:12)
at com.delver.spark.JobRunner.SparkJobBase.start(SparkJobBase.scala:30)
at com.delver.spark.JobRunner.SparkDriver$.com$delver$spark$JobRunner$SparkDriver$$startJob(SparkDriver.scala:84)
at com.delver.spark.JobRunner.SparkDriver$$anonfun$main$2.apply(SparkDriver.scala:68)
at com.delver.spark.JobRunner.SparkDriver$$anonfun$main$2.apply(SparkDriver.scala:51)
at scala.Option.map(Option.scala:146)
at com.delver.spark.JobRunner.SparkDriver$.main(SparkDriver.scala:51)
at com.delver.spark.JobRunner.SparkDriver.main(SparkDriver.scala)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
at org.apache.spark.sql.SparkSession$.org$apache$spark$sql$SparkSession$$reflect(SparkSession.scala:978)
... 13 more
Caused by: java.lang.IllegalArgumentException: Error while instantiating 'org.apache.spark.sql.hive.HiveExternalCatalog':
at org.apache.spark.sql.internal.SharedState$.org$apache$spark$sql$internal$SharedState$$reflect(SharedState.scala:169)
at org.apache.spark.sql.internal.SharedState.<init>(SharedState.scala:86)
at org.apache.spark.sql.SparkSession$$anonfun$sharedState$1.apply(SparkSession.scala:101)
at org.apache.spark.sql.SparkSession$$anonfun$sharedState$1.apply(SparkSession.scala:101)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.sql.SparkSession.sharedState$lzycompute(SparkSession.scala:101)
at org.apache.spark.sql.SparkSession.sharedState(SparkSession.scala:100)
at org.apache.spark.sql.internal.SessionState.<init>(SessionState.scala:157)
at org.apache.spark.sql.hive.HiveSessionState.<init>(HiveSessionState.scala:32)
... 18 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
at org.apache.spark.sql.internal.SharedState$.org$apache$spark$sql$internal$SharedState$$reflect(SharedState.scala:166)
... 26 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
at org.apache.spark.sql.hive.client.IsolatedClientLoader.createClient(IsolatedClientLoader.scala:264)
at org.apache.spark.sql.hive.HiveUtils$.newClientForMetadata(HiveUtils.scala:358)
at org.apache.spark.sql.hive.HiveUtils$.newClientForMetadata(HiveUtils.scala:262)
at org.apache.spark.sql.hive.HiveExternalCatalog.<init>(HiveExternalCatalog.scala:66)
... 31 more
Caused by: java.lang.ExceptionInInitializerError
at com.mapr.fs.ShimLoader.load(ShimLoader.java:243)
at com.mapr.fs.MapRFileSystem.<clinit>(MapRFileSystem.java:107)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at org.apache.hadoop.conf.Configuration.getClassByNameOrNull(Configuration.java:2159)
at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2124)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2218)
at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2678)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2691)
at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:98)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2773)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2755)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:411)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:176)
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:505)
at org.apache.spark.sql.hive.client.HiveClientImpl.<init>(HiveClientImpl.scala:188)
... 39 more
Caused by: java.lang.RuntimeException: Failure loading MapRClient.
at com.mapr.fs.ShimLoader.injectNativeLoader(ShimLoader.java:305)
at com.mapr.fs.ShimLoader.load(ShimLoader.java:223)
... 54 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.GeneratedMethodAccessor1.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at com.mapr.fs.ShimLoader.injectNativeLoader(ShimLoader.java:290)
... 55 more
Caused by: java.lang.LinkageError: loader (instance of org/apache/spark/sql/hive/client/IsolatedClientLoader$$anon$1): attempted duplicate class definition for name: "com/mapr/fs/jni/MapRConstants"
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:760)
... 59 more

Outcomes