Failed to load Hive table in Pig using HCatalog

Document created by Hao Zhu Employee on Feb 17, 2016
Version 1Show Document
  • View in full screen mode

Author: Hao Zhu

Original Publication Date: November 25, 2014

 

Symptom:

In Pig, loading a Hive table using HCatLoader fails with the below error message:

grunt> a = load 'passwords' using org.apache.hcatalog.pig.HCatLoader();

2014-11-25 11:19:35,474 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1070: Could not resolve org.apache.hcatalog.pig.HCatLoader using imports: [, java.lang., org.apache.pig.builtin., org.apache.pig.impl.builtin.]

Details at logfile: /root/pig_1416943163875.log

Root Cause:

The option "-useHCatalog" is not specified when starting the pig shell.

Solution:

One of the following options can be used to resolve the above error.  The root cause is that the HCatalog classes are not made available to Pig at run time.  Accordingly the Pig classpath needs to be updated to include the HCatalog jars provided by MapR's Hive package.  This can be done either by manually setting HADOOP_CLASSPATH before loading the Pig shell or by using the '-useHCatalog' command-line option when running Pig.If you are using mapr-pig-0.12 with hive-0.13, you may see the below errors when using the '-useHCatalog' option:

 

[root@n1a conf]# pig -useHCatalog

ls: cannot access /opt/mapr/hive/hive-0.13/bin/..//lib/slf4j-api-*.jar: No such file or directory

ls: cannot access /opt/mapr/hive/hive-0.13/hcatalog/bin/..//share/hcatalog/hcatalog-core-*.jar: No such file or directory

ls: cannot access /opt/mapr/hive/hive-0.13/hcatalog/bin/..//share/hcatalog/hcatalog-*.jar: No such file or directory

ls: cannot access /opt/mapr/hive/hive-0.13/hcatalog/bin/..//share/hcatalog/hcatalog-pig-adapter-*.jar: No such file or directory

This is because the hive-0.12's hcatalog jar file names are like hcatalog-xxx.jar, while hive-0.13's hcatalog jar file names are like hive-hcatalog-xxx.jar. In this case, you have to use option 1 below to set the correct HADOOP_CLASSPATH or you can upgrade pig to mapr-pig-0.13 which does not have this issue.

 

1. Import needed jars in HADOOP_CLASSPATH.Using hive-0.13 as an example:

export HADOOP_CLASSPATH=/opt/mapr/hive/hive-0.13/hcatalog/share/hcatalog/hive-hcatalog-core-0.13.0-mapr-1409.jar:/opt/mapr/hive/hive-0.13/hcatalog/share/hcatalog/hive-hcatalog-streaming-0.13.0-mapr-1409.jar:/opt/mapr/hive/hive-0.13/hcatalog/share/hcatalog/hive-hcatalog-pig-adapter-0.13.0-mapr-1409.jar:/opt/mapr/hive/hive-0.13/hcatalog/share/hcatalog/hive-hcatalog-server-extensions-0.13.0-mapr-1409.jar

2. Start pig using "-useHCatalog" option.

pig -useHCatalog

After using either option above attempt to load the Hive table in Pig again using HCatLoader and it should work. Ex:

 

grunt> a = load 'passwords' using org.apache.hcatalog.pig.HCatLoader();

SLF4J: Class path contains multiple SLF4J bindings.

SLF4J: Found binding in [jar:file:/opt/mapr/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: Found binding in [jar:file:/opt/mapr/hbase/hbase-0.94.21/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.

2014-11-25 11:25:43,929 [main] INFO hive.metastore - Trying to connect to metastore with URI thrift://xx.xx.xx.xx:9083

2014-11-25 11:25:43,965 [main] INFO hive.metastore - Connected to metastore.

If you continue to see issues loading Hive tables in Pig with HCatalog, capture the error and the Pig classpath and contact MapR support or post a comment below.

Attachments

    Outcomes