AnsweredAssumed Answered

No space left on device at

Question asked by lewis on Jan 26, 2014
Latest reply on Jan 28, 2014 by lewis
I had a question about my pig script that an error occured when pig did group operation. I am using M7 v3.1.0 and V0.11.2 of Pig.
My M7 have 8 task nodes to do mapreduce task.
the following is my pig script sample:

bcn_logs_find = FILTER bcn_logs_dl BY  (resCode =='200' and not url matches '.*\\/mcs\\/mcs_ping\\.txt&.*' and not url matches '.*&hppid=42&.*');
bcn_logs_find_grp = GROUP bcn_logs_find BY (resCode,LOWER(url));
 bcn_logs_find_cnt = foreach bcn_logs_find_grp {
       latestrecord = order bcn_logs_find by date desc;
       findurl = SUM(bcn_logs_find.thisLineClicks);
       byteurl = SUM(bcn_logs_find.byteCode);
       latesturl = limit latestrecord.url 1;
       latestdate = limit 1;
       generate FLATTEN(group) AS (resCode,lurl),findurl as findurl,byteurl as byteurl,FLATTEN((IsEmpty(latesturl)?null:latesturl)) AS url,FLATTEN((IsEmpty(latestdate)?null:latestdate)) AS date;

Here are some steps I used the pig scripts:

 1. I changed the pig.cachedbag.memusage to 0, then I ran the pig script.
 2. My pig script read two log files which contains 158,869,700 records.
 3. If my pig script read 158,869,700 records that was working fine. after that, the pig script read log files which increased 10 times that contains about 1,588,697,000 records, and then the pig task was failed. The following is the failed message:

 Failed Jobs:
JobId   Alias   Feature Message Outputs
job_201312130533_1841   1-1402,1-1403,bcn_logs_find_cnt,bcn_logs_find_grp,latestdate,latesturl  GROUP_BY        Message: Job failed! Error - NA

Successfully read 783733000 records from: "/Elements/0332/logs/Metric_access_log-20130604{,.gz}"
Successfully read 804964000 records from: "/Elements/0486g/logs/Metric_access_log-20130604{,.gz}"
Successfully read 1 records from: "/Elements/test/fields.log"

Successfully stored 0 records in: "/jobs/output/bcn/robot"
Successfully stored 0 records in: "/jobs/output/bcn/red"
Successfully stored 176 records (18592 bytes) in: "/jobs/output/bcn/err"
Successfully stored 1 records in: "/jobs/output/bcn/fields"
Successfully stored 2 records in: "/jobs/output/bcn/box"
Successfully stored 141628 records in: "/jobs/output/bcn/hb"

Total records written : 141807
Total bytes written : 18592
Spillable Memory Manager spill count : 0
Total bags proactively spilled: 1058
Total records proactively spilled: 64060225

Job DAG:
job_201312130533_1837   ->      job_201312130533_1841,job_201312130533_1838,job_201312130533_1840,job_201312130533_1839,
job_201312130533_1841   ->      null,
null    ->      null,
null    ->      null,null,
job_201312130533_1838   ->      job_201312130533_1843,job_201312130533_1845,
job_201312130533_1843   ->      job_201312130533_1848,job_201312130533_1847,
job_201312130533_1848   ->      job_201312130533_1855,
job_201312130533_1855   ->      null,
job_201312130533_1840   ->      job_201312130533_1846,
job_201312130533_1846   ->      job_201312130533_1849,job_201312130533_1852,job_201312130533_1851,
job_201312130533_1849   ->      null,null,
job_201312130533_1845   ->      job_201312130533_1850,job_201312130533_1847,
job_201312130533_1850   ->      job_201312130533_1856,
job_201312130533_1856   ->      job_201312130533_1858,
job_201312130533_1858   ->      job_201312130533_1859,
job_201312130533_1859   ->      null,null,
job_201312130533_1839   ->      job_201312130533_1844,job_201312130533_1842,
job_201312130533_1844   ->      null,null,
job_201312130533_1842   ->      job_201312130533_1847,
job_201312130533_1847   ->      job_201312130533_1857,
job_201312130533_1852   ->      job_201312130533_1854,job_201312130533_1853,
job_201312130533_1851   ->      job_201312130533_1854,
job_201312130533_1853   ->      null,


 1. I checked the log from Mapr System Console:

java.lang.RuntimeException: No space left on device at at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POPackage.getNext( at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Reduce.processOnePackageOutput( at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Reduce.reduce( at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Reduce.reduce( at at org.apache.hadoop.mapred.ReduceTask.runNewReducer( at at org.apache.hadoop.mapred.Child$ at Method) at at at org.apache.hadoop.mapred.Child.main( Caused by: No space left on device at Method) at at at at at at at at at at at at at ... 12 more
 2. I found my tmp directory whose size is 15G that usage almost full when pig was running the above task.
 3. according to the log, I thought the issuse is that there is not enought space in my tmp directory. but I am new to use Pig, So I am not sure for that, Could you give me some good suggestions?
 4. or How could I optimize my pig script? I need your help, thank you so much.
 Ps. my english is very poor, so I hope you can understand my questions that I described above.