AnsweredAssumed Answered

Stripe Compactions in HBase 0.98

Question asked by tc_dev on Oct 19, 2014
Latest reply on Feb 6, 2015 by tc_dev

I am interested in deploying a new "experimental" feature of HBase 0.98, which is a different HStore engine that supports sub-dividing regions into multiple HFiles - therefore supporting more efficient management and compaction of large regions. I am following directions from HBase Book (on MapR v4.0.1):

 

http://hbase.apache.org/book/regions.arch.html#ops.stripe

 

I successfully converted an existing table into a Stripe format by issuing the command as directed:

 

   

alter 'mytable', CONFIGURATION => {"hbase.hstore.engine.class" => "org.apache.hadoop.hbase.regionserver.StripeStoreEngine", "hbase.hstore.blockingStoreFiles" => 200, "hbase.store.stripe.sizeToSplit" => 2000000000, "hbase.store.stripe.compaction.flushToL0" => true }

 

However I am seeing some suspicious behavior. When triggering a major compaction from the command line (major_compact 'mytable') or WCS, I do not see the storefiles being actually compacted / rewritten / change timestamp, unlike how this happens with the default storage engine.

 

I wonder if StripeStoreEngine has been tested by others on MapR and if anyone can share their experience with it...

 

EDIT: I have now confirmed that StripeStoreEngine also loses data. After writing to a table and trying to manually flush the memstore from HBase shell I've confirmed the loss of data with these exceptions written to the regionserver logs -

 

    2014-10-21 01:30:57,882 WARN  [PriorityRpcServer.handler=1,queue=0,port=60020] regionserver.HStore: Failed validating store file null, retrying num=9
    java.lang.NullPointerException
        at org.apache.hadoop.hbase.regionserver.HStore.flushCache(HStore.java:802)
        at org.apache.hadoop.hbase.regionserver.HStore$StoreFlusherImpl.flushCache(HStore.java:1949)
        at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1777)
        at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1659)
        at org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:1572)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.flushRegion(HRegionServer.java:3826)
        at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:20168)
        at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2028)
        at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:98)
        at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114)
        at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94)
        at java.lang.Thread.run(Thread.java:744)
    2014-10-21 01:30:57,883 FATAL [PriorityRpcServer.handler=1,queue=0,port=60020] regionserver.HRegionServer: ABORTING region server myhostname.net,60020,1413795908231: Replay of HLog required. Forcing server shutdown
    org.apache.hadoop.hbase.DroppedSnapshotException: region: rkey_request,,1413608633505.f8225cfd995babc935f4c4daaaa19440.
        at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1802)
        at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1659)
        at org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:1572)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.flushRegion(HRegionServer.java:3826)
        at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:20168)
        at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2028)
        at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:98)
        at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114)
        at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94)
        at java.lang.Thread.run(Thread.java:744)
    Caused by: java.io.IOException: java.lang.NullPointerException
        at org.apache.hadoop.hbase.regionserver.HStore.flushCache(HStore.java:812)
        at org.apache.hadoop.hbase.regionserver.HStore$StoreFlusherImpl.flushCache(HStore.java:1949)
        at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1777)
        ... 9 more
    Caused by: java.lang.NullPointerException
        at org.apache.hadoop.hbase.regionserver.HStore.flushCache(HStore.java:802)
        ... 11 more
    2014-10-21 01:30:57,884 FATAL [PriorityRpcServer.handler=1,queue=0,port=60020] regionserver.HRegionServer: RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint, xxx]

Outcomes