AnsweredAssumed Answered

Missing org.apache.hadoop.fs.s3a.S3AFileSystem - MapR 5.0

Question asked by peterjenks on Jul 30, 2015
Latest reply on Jan 14, 2016 by peterjenks
Branched to a new discussion
I'm trying to get s3 integration working within the file system. I have added provided values for __fs.s3a.awsAccessKeyId__ and __fs.s3a.awsSecretAccessKey__ in __/opt/mapr/hadoop/hadoop-2.7.0/etc/hadoop/core-site.xml__.

When I now try __hadoop fs -ls "s3a://mybucket/"__ I get the following error.

java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.s3a.S3AFileSystem not found
        at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2208)
        at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2674)
        at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2687)
        at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:94)
        at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2723)
        at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2705)
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:407)
        at org.apache.hadoop.fs.Path.getFileSystem(Path.java:309)
        at org.apache.hadoop.fs.shell.PathData.expandAsGlob(PathData.java:325)
        at org.apache.hadoop.fs.shell.Command.expandArgument(Command.java:235)
        at org.apache.hadoop.fs.shell.Command.expandArguments(Command.java:218)
        at org.apache.hadoop.fs.shell.Command.processRawArguments(Command.java:201)
        at org.apache.hadoop.fs.shell.Command.run(Command.java:165)
        at org.apache.hadoop.fs.FsShell.run(FsShell.java:287)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
        at org.apache.hadoop.fs.FsShell.main(FsShell.java:340)
Caused by: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.s3a.S3AFileSystem not found
        at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2114)
        at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2206)
        ... 16 more

I suspect I am missing a key library from my CLASSPATH but am not sure which one.

Does anyone know how I can resolve this please?

Outcomes