AnsweredAssumed Answered

PIG execution is hung and status shows pending

Question asked by raj_hadoop on Feb 17, 2013
Latest reply on Feb 18, 2013 by raj_hadoop
Hi,

I am trying to execute basic pig commands and the process is pending/stuck (below log). Could anyone let me know what is missing here?

Something to do with Snappy native library since it is not loaded?
<pre>

grunt> records = LOAD '/user/ciber/poc/pig_test.txt' AS (year:chararray, temperature:int, quality:int);                                                       

grunt> filtered_records = FILTER records BY temperature != 9999 AND (quality == 0 OR quality == 1 OR quality == 4 OR quality == 5 OR quality == 9);

grunt> grouped_records = GROUP filtered_records BY year;                                                                                                      

grunt> max_temp = FOREACH grouped_records GENERATE group, MAX(filtered_records.temperature);                                                                   

grunt> DUMP max_temp;                                                                                                                                         



2013-02-15 19:25:30,491 [main] INFO  org.apache.pig.tools.pigstats.ScriptState - Pig features used in the script: GROUP_BY,FILTER

2013-02-15 19:25:30,786 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - File concatenation threshold:    100 optimistic? false

2013-02-15 19:25:30,807 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.CombinerOptimizer - Choosing to move algebraic foreach to combiner

2013-02-15 19:25:30,857 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size before optimization: 1

2013-02-15 19:25:30,858 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size after optimization: 1

2013-02-15 19:25:30,980 [main] INFO  org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added to the job

2013-02-15 19:25:31,016 [main] INFO 
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3

2013-02-15 19:25:31,024 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - creating jar file Job2663466069648886656.jar

2013-02-15 19:25:37,255 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - jar file Job2663466069648886656.jar created

2013-02-15 19:25:37,286 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Setting up single store job

2013-02-15 19:25:37,351 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - BytesPerReducer=1000000000 maxReducers=999 totalInputFileSize=91

2013-02-15 19:25:37,352 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Neither PARALLEL nor default parallelism is set for this job. Setting number of reducers to 1

2013-02-15 19:25:37,437 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 1 map-reduce job(s) waiting for submission.

2013-02-15 19:25:37,938 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 0% complete

2013-02-15 19:25:38,045 [Thread-8] INFO  org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths to process : 1

2013-02-15 19:25:38,046 [Thread-8] INFO  org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 1

2013-02-15 19:25:38,064 [Thread-8] INFO  org.apache.hadoop.util.NativeCodeLoader - Loaded the native-hadoop library

2013-02-15 19:25:38,065 [Thread-8] WARN  org.apache.hadoop.io.compress.snappy.LoadSnappy - Snappy native library not loaded

2013-02-15 19:25:38,069 [Thread-8] INFO  org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths (combined) to process : 1

2013-02-15 19:25:39,112 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - HadoopJobId: job_201302151243_0004

2013-02-15 19:25:39,112 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - More information at: Job URL


As per the above url, More information at: job url, it shows below information. It is pending for long.


User: ciber

Job Name: Job2663466069648886656.jar

Job File: hdfs:--> link

Submit Host: cbr-srv-hp0725

Submit Host Address: <namenode IP>

Job-ACLs: All users are allowed

Job Setup: Pending

Job Cleanup: Pending


Regards,
Raj

Outcomes