AnsweredAssumed Answered

LZ4 not found

Question asked by brianjohnson on Apr 15, 2014
I'm getting the following error while running importtsv

<pre>
14/04/15 19:17:35 INFO mapred.JobClient: Task Id : attempt_201403240207_4368_r_000000_2, Status : FAILED on node ip-10-145-138-231.ec2.internal
java.lang.RuntimeException: java.lang.ClassNotFoundException: org.apache.hadoop.io.compress.Lz4Codec
at org.apache.hadoop.hbase.io.hfile.Compression$Algorithm$5.buildCodec(Compression.java:237)
at org.apache.hadoop.hbase.io.hfile.Compression$Algorithm$5.getCodec(Compression.java:222)
at org.apache.hadoop.hbase.io.hfile.Compression$Algorithm.getCompressor(Compression.java:303)
at org.apache.hadoop.hbase.io.hfile.HFileBlock$Writer.<init>(HFileBlock.java:745)
at org.apache.hadoop.hbase.io.hfile.HFileWriterV2.finishInit(HFileWriterV2.java:135)
at org.apache.hadoop.hbase.io.hfile.HFileWriterV2.<init>(HFileWriterV2.java:126)
at org.apache.hadoop.hbase.io.hfile.HFileWriterV2$WriterFactoryV2.createWriter(HFileWriterV2.java:105)
at org.apache.hadoop.hbase.io.hfile.HFile$WriterFactory.create(HFile.java:400)
at org.apache.hadoop.hbase.regionserver.StoreFile$Writer.<init>(StoreFile.java:1078)
at org.apache.hadoop.hbase.regionserver.StoreFile$Writer.<init>(StoreFile.java:1022)
at org.apache.hadoop.hbase.regionserver.StoreFile$WriterBuilder.build(StoreFile.java:897)
at org.apache.hadoop.hbase.mapreduce.HFileOutputFormat$1.getNewWriter(HFileOutputFormat.java:209)
at org.apache.hadoop.hbase.mapreduce.HFileOutputFormat$1.write(HFileOutputFormat.java:163)
at org.apache.hadoop.hbase.mapreduce.HFileOutputFormat$1.write(HFileOutputFormat.java:124)
at org.apache.hadoop.mapred.ReduceTask$NewTrackingRecordWriter.write(ReduceTask.java:544)
at org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)
at org.apache.hadoop.hbase.mapreduce.PutSortReducer.reduce(PutSortReducer.java:72)
at org.apache.hadoop.hbase.mapreduce.PutSortReducer.reduce(PutSortReducer.java:40)
at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:176)
at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:602)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:444)
at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127)
at org.apache.hadoop.mapred.Child.main(Child.java:264)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.io.compress.Lz4Codec
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at org.apache.hadoop.hbase.io.hfile.Compression$Algorithm$5.buildCodec(Compression.java:232)
... 25 more

</pre>
If I run with compression off importtsv works and I get the following error with completebulkload. I think the issue is the out put from importtsv importtsv.bulk.output which only contains

DATABLK*


T_PUT_TS
hfile.LASTKEY
<pre>
Exception in thread "main" java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.hbase.mapreduce.Driver.main(Driver.java:51)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.main(RunJar.java:197)
Caused by: java.lang.NullPointerException
at org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.doBulkLoad(LoadIncrementalHFiles.java:197)
at org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.run(LoadIncrementalHFiles.java:778)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
at org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.main(LoadIncrementalHFiles.java:783)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68)
at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139)
... 10 more
</pre>
I've done quite a bit of debugging on the NullPointerException and it appears that table.getConnection() is returning null. The strange thing is that it even returns NULL if I create a non-null connection and pass it into the constructor. If I call isTableAvailable on that connection I get the following

<pre>
14/04/16 09:20:31 INFO zookeeper.RecoverableZooKeeper: The identifier of this process is 6241@maprdemo.local
14/04/16 09:20:31 ERROR zookeeper.ZooKeeperNodeTracker: Check the value configured in 'zookeeper.znode.parent'. There could be a mismatch with the one configured in the master.
Exception in thread "main" java.lang.IllegalArgumentException: Check the value configured in 'zookeeper.znode.parent'. There could be a mismatch with the one configured in the master.
at org.apache.hadoop.hbase.zookeeper.RootRegionTracker.waitRootRegionLocation(RootRegionTracker.java:81)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:987)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:1100)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:998)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:959)
</pre>

Outcomes