AnsweredAssumed Answered

Sqoop MaprDB Integration

Question asked by arnab1980 on Oct 24, 2017
Latest reply on Oct 31, 2017 by cathy

I am not sure if this works with maprdb but it does work with hbase.From a client node which has both mapr client and hbase client installed i tried to run sqoop that would pull data out of rdbms and will dynamically create a new MaprDB table and then load data into it.

 

The sqoop map job does get kick off and even creates a new maprdb table but it fails to proceed giving a strange error.To me it appears that there are two classes , one specific to maprDB and one hbase and seems to be a kind of confusion between them.The app team wants to load data into maprdb table and wanted to skip the process or creating Hfiles and directly get this done via sqoop.

 

Sqoop syntax

 

sqoop import --connect jdbc:oracle:thin:@djpslp01.nomura.com:8235/djpslp01 --username ****--password
****--table test IM --columns "test_ID,test_USER" --hbase-create-table --hbase-table
/a/b/c/d/project/details/tabname --hbase-row-key test_ID --column-family USERS -m 1

 

The error we see is this is as below and all the four map jobs fail with same error

 

Error: tried to access field com.mapr.fs.hbase.HTableImpl.maprTable from class com.mapr.fs.hbase.HTableImpl11

 

2017-10-24 22:04:31,281 FATAL [main] org.apache.hadoop.mapred.YarnChild: Error running child : java.lang.IllegalAccessError: tried to access field com.mapr.fs.hbase.HTableImpl.maprTable from class com.mapr.fs.hbase.HTableImpl11      at com.mapr.fs.hbase.HTableImpl11.close(HTableImpl11.java:73)      at org.apache.hadoop.hbase.client.HRegionLocator.close(HRegionLocator.java:104)      at org.apache.hadoop.hbase.client.HTable.close(HTable.java:1821)      at org.apache.sqoop.hbase.HBasePutProcessor.close(HBasePutProcessor.java:169)      at org.apache.sqoop.mapreduce.DelegatingOutputFormat$DelegatingRecordWriter.close(DelegatingOutputFormat.java:119)      at org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.close(MapTask.java:679)      at org.apache.hadoop.mapred.MapTask.closeQuietly(MapTask.java:2031)      at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:806)      at org.apache.hadoop.mapred.MapTask.run(MapTask.java:346)      at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)      at java.security.AccessController.doPrivileged(Native Method)      at javax.security.auth.Subject.doAs(Subject.java:422)      at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1595)      at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)

 

I can see the table created in the shell.

 

The jar mapr-hbase-5.2.1-mapr.jar is in the sqoop lib which has both these classes.

Outcomes