AnsweredAssumed Answered

Mapr HBASE LZ4

Question asked by rhinomike on Mar 19, 2015
Hi there,

Can someone confirm if the ecosystem package version of HBase supports LZ4?

Looking at  the config dump I get:

    <name>io.compression.codecs</name>
    <value>org.apache.hadoop.io.compress.DefaultCodec,org.apache.hadoop.io.compress.GzipCodec,org.apache.hadoop.io.compress.BZip2Codec,org.apache.hadoop.io.compress.DeflateCodec,org.apache.hadoop.io.compress.SnappyCodec
    </value>

However, using

    $  hbase org.apache.hadoop.hbase.util.CompressionTest maprfs:///testo/ lz4
    2015-03-19 17:52:09,294 INFO  [main] Configuration.deprecation: hadoop.native.lib is deprecated. Instead, use io.native.lib.available
    2015-03-19 17:52:09,975 INFO  [main] util.ChecksumType: Checksum using org.apache.hadoop.util.PureJavaCrc32
    2015-03-19 17:52:09,976 INFO  [main] util.ChecksumType: Checksum can use org.apache.hadoop.util.PureJavaCrc32C
    2015-03-19 17:52:10,173 INFO  [main] compress.CodecPool: Got brand-new compressor [.lz4]
    2015-03-19 17:52:10,178 INFO  [main] compress.CodecPool: Got brand-new compressor [.lz4]
    2015-03-19 17:52:10,183 ERROR [main] hbase.KeyValue: Unexpected getShortMidpointKey result, fakeKey:testkey, firstKeyInBlock:testkey
    2015-03-19 17:52:10,280 INFO  [main] compress.CodecPool: Got brand-new decompressor [.lz4]
    SUCCESS

    $  hbase org.apache.hadoop.hbase.util.CompressionTest maprfs:///testo/ snappy
    2015-03-19 17:53:29,580 INFO  [main] Configuration.deprecation: hadoop.native.lib is deprecated. Instead, use io.native.lib.available
    2015-03-19 17:53:30,274 INFO  [main] util.ChecksumType: Checksum using org.apache.hadoop.util.PureJavaCrc32
    2015-03-19 17:53:30,276 INFO  [main] util.ChecksumType: Checksum can use org.apache.hadoop.util.PureJavaCrc32C
    2015-03-19 17:53:30,479 INFO  [main] compress.CodecPool: Got brand-new compressor [.snappy]
    2015-03-19 17:53:30,484 INFO  [main] compress.CodecPool: Got brand-new compressor [.snappy]
    2015-03-19 17:53:30,491 ERROR [main] hbase.KeyValue: Unexpected getShortMidpointKey result, fakeKey:testkey, firstKeyInBlock:testkey
    2015-03-19 17:53:30,597 INFO  [main] compress.CodecPool: Got brand-new decompressor [.snappy]
    SUCCESS

    hbase(main):001:0> create 'testo2', { NAME => 'tt2', COMPRESSION => 'LZ4' }
    0 row(s) in 1.3940 seconds
    
    => Hbase::Table - testo2
    hbase(main):002:0> describe 'testo2'
    DESCRIPTION                                                                                            ENABLED
     'testo2', {NAME => 'tt2', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'ROW', REPLICATION_SCOPE => ' true
     0', VERSIONS => '1', COMPRESSION => 'LZ4', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS
     => 'false', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}
    1 row(s) in 0.0670 seconds



Given the output of both compressions is the same, I would assume LZ4 is supported, however I couldn't find any mention of it on

http://doc.mapr.com/display/MapR/HBase#HBase-compression

Is that just a documentation issue or is the LZ4 on HBase not supported? (Or maybe I got the test

Cheers

Outcomes