How to run pig in local mode?

Document created by prasoon Employee on Feb 8, 2016
Version 1Show Document
  • View in full screen mode

Author: Prasoon Keloth


Original Publication Date: February 2, 2015


Running pig shell provided in the mapr-pig package in local mode (pig -x local) fails with the error below :


Caused by: org.apache.pig.backend.executionengine.ExecException: ERROR 0: java.lang.IllegalArgumentException: Buffer size <= 0 
at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.launchPig(
at org.apache.pig.PigServer.launchPlan(
at org.apache.pig.PigServer.executeCompiledLogicalPlan(
at org.apache.pig.PigServer.storeEx(
... 14 more
Caused by: java.lang.IllegalArgumentException: Buffer size <= 0

Root Cause:


The behaviour of the io.file.buffer.size parameter needs to be configured for local filesystem with a different value than the default for Mapr-FS. See the discussion here for some explanation.




We need to add below options in /opt/mapr/pig/pig-0.13/conf/pig.properies file to run pig in local mode.


fs.file.impl=org.apache.hadoop.fs.LocalFileSystem io.file.buffer.size=4096