AnsweredAssumed Answered

Kafka + MapR problems

Question asked by fredrik_emilsson on May 30, 2012
Latest reply on Jun 4, 2012 by fredrik_emilsson
Hello,

  When I tried to use Kafka hadoop-consumer to move data into the MapR HDFS I got the following error:

12/05/31 11:09:12 INFO producer.SyncProducer: Disconnecting from localhost:9092
Exception in thread "main" java.io.IOException: No FileSystem for scheme: maprfs
        at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1375)
        at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66)
        at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1390)
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:196)
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:95)
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:180)
        at org.apache.hadoop.fs.Path.getFileSystem(Path.java:175)
        at kafka.etl.impl.DataGenerator.generateOffsets(DataGenerator.java:116)
        at kafka.etl.impl.DataGenerator.run(DataGenerator.java:108)
        at kafka.etl.impl.DataGenerator.main(DataGenerator.java:139)

I guess I have to add some more libs into the classpath but which? The current classpath is as follows:

/usr/lib/jvm/java-6-sun/bin/java -cp dist::./../../core/target/scala_2.8.0/kafka-0.7.0.jar:./../../contrib/hadoop-consumer/lib_managed/scala_2.8.0/compile/ant-1.6.5.jar:./../../contrib/hadoop-consumer/lib_managed/scala_2.8.0/compile/commons-cli-1.2.jar:./../../contrib/hadoop-consumer/lib_managed/scala_2.8.0/compile/commons-codec-1.3.jar:./../../contrib/hadoop-consumer/lib_managed/scala_2.8.0/compile/commons-el-1.0.jar:./../../contrib/hadoop-consumer/lib_managed/scala_2.8.0/compile/commons-httpclient-3.0.1.jar:./../../contrib/hadoop-consumer/lib_managed/scala_2.8.0/compile/commons-logging-1.1.1.jar:./../../contrib/hadoop-consumer/lib_managed/scala_2.8.0/compile/commons-net-1.4.1.jar:./../../contrib/hadoop-consumer/lib_managed/scala_2.8.0/compile/core-3.1.1.jar:./../../contrib/hadoop-consumer/lib_managed/scala_2.8.0/compile/hadoop-core-0.20.2.jar:./../../contrib/hadoop-consumer/lib_managed/scala_2.8.0/compile/hsqldb-1.8.0.10.jar:./../../contrib/hadoop-consumer/lib_managed/scala_2.8.0/compile/jasper-compiler-5.5.12.jar:./../../contrib/hadoop-consumer/lib_managed/scala_2.8.0/compile/jasper-runtime-5.5.12.jar:./../../contrib/hadoop-consumer/lib_managed/scala_2.8.0/compile/jets3t-0.7.1.jar:./../../contrib/hadoop-consumer/lib_managed/scala_2.8.0/compile/jetty-6.1.14.jar:./../../contrib/hadoop-consumer/lib_managed/scala_2.8.0/compile/jetty-util-6.1.14.jar:./../../contrib/hadoop-consumer/lib_managed/scala_2.8.0/compile/joda-time-1.6.jar:./../../contrib/hadoop-consumer/lib_managed/scala_2.8.0/compile/jopt-simple-3.2.jar:./../../contrib/hadoop-consumer/lib_managed/scala_2.8.0/compile/jsp-2.1-6.1.14.jar:./../../contrib/hadoop-consumer/lib_managed/scala_2.8.0/compile/jsp-api-2.1-6.1.14.jar:./../../contrib/hadoop-consumer/lib_managed/scala_2.8.0/compile/kfs-0.3.jar:./../../contrib/hadoop-consumer/lib_managed/scala_2.8.0/compile/log4j-1.2.15.jar:./../../contrib/hadoop-consumer/lib_managed/scala_2.8.0/compile/oro-2.0.8.jar:./../../contrib/hadoop-consumer/lib_managed/scala_2.8.0/compile/servlet-api-2.5-6.1.14.jar:./../../contrib/hadoop-consumer/lib_managed/scala_2.8.0/compile/xmlenc-0.52.jar:./../../contrib/hadoop-consumer/target/scala_2.8.0/hadoop-consumer_2.8.0-0.7.0.jar:./../../contrib/hadoop-consumer/lib/maprfs-0.1.jar:./../../contrib/hadoop-consumer/lib/maprfs-test-0.1.jar:./../../contrib/hadoop-consumer/lib/pig-0.8.0-core.jar:./../../contrib/hadoop-consumer/lib/piggybank.jar:./../../project/boot/scala-2.8.0/lib/scala-library.jar:/opt/mapr/hadoop/hadoop-0.20.2/conf kafka.etl.impl.DataGenerator test/test.properties

 Regards,
    Fredrik Emilsson
  

Outcomes