How to run pig in local mode?

Document created by prasoon Employee on Feb 8, 2016
Version 1Show Document
  • View in full screen mode

Author: Prasoon Keloth

 

Original Publication Date: February 2, 2015

Issue:

Running pig shell provided in the mapr-pig package in local mode (pig -x local) fails with the error below :

 

Caused by: org.apache.pig.backend.executionengine.ExecException: ERROR 0: java.lang.IllegalArgumentException: Buffer size <= 0 
at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.launchPig(HExecutionEngine.java:283)
at org.apache.pig.PigServer.launchPlan(PigServer.java:1367)
at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1352)
at org.apache.pig.PigServer.storeEx(PigServer.java:1011)
... 14 more
Caused by: java.lang.IllegalArgumentException: Buffer size <= 0

Root Cause:

 

The behaviour of the io.file.buffer.size parameter needs to be configured for local filesystem with a different value than the default for Mapr-FS. See the discussion here for some explanation. http://answers.mapr.com/questions/9338/setting-io-file-buffer-size.html

 

Resolution:

 

We need to add below options in /opt/mapr/pig/pig-0.13/conf/pig.properies file to run pig in local mode.

 

fs.file.impl=org.apache.hadoop.fs.LocalFileSystem io.file.buffer.size=4096

 

 

Attachments

    Outcomes