AnsweredAssumed Answered

No space left on device at:org.apache.pig.data.InternalCachedBag.add(InternalCachedBag.java:106) at

Question asked by lewis on Jan 26, 2014
Latest reply on Jan 28, 2014 by lewis
I had a question about my pig script that an error occured when pig did group operation. I am using M7 v3.1.0 and V0.11.2 of Pig.
My M7 have 8 task nodes to do mapreduce task.
the following is my pig script sample:

<code>
bcn_logs_find = FILTER bcn_logs_dl BY  (resCode =='200' and not url matches '.*\\/mcs\\/mcs_ping\\.txt&.*' and not url matches '.*&hppid=42&.*');
bcn_logs_find_grp = GROUP bcn_logs_find BY (resCode,LOWER(url));
 bcn_logs_find_cnt = foreach bcn_logs_find_grp {
       latestrecord = order bcn_logs_find by date desc;
       findurl = SUM(bcn_logs_find.thisLineClicks);
       byteurl = SUM(bcn_logs_find.byteCode);
       latesturl = limit latestrecord.url 1;
       latestdate = limit latestrecord.date 1;
       generate FLATTEN(group) AS (resCode,lurl),findurl as findurl,byteurl as byteurl,FLATTEN((IsEmpty(latesturl)?null:latesturl)) AS url,FLATTEN((IsEmpty(latestdate)?null:latestdate)) AS date;
 };
</code>

Here are some steps I used the pig scripts:

 1. I changed the pig.cachedbag.memusage to 0, then I ran the pig script.
 2. My pig script read two log files which contains 158,869,700 records.
 3. If my pig script read 158,869,700 records that was working fine. after that, the pig script read log files which increased 10 times that contains about 1,588,697,000 records, and then the pig task was failed. The following is the failed message:
<pre>

 Failed Jobs:
JobId   Alias   Feature Message Outputs
job_201312130533_1841   1-1402,1-1403,bcn_logs_find_cnt,bcn_logs_find_grp,latestdate,latesturl  GROUP_BY        Message: Job failed! Error - NA

Input(s):
Successfully read 783733000 records from: "/Elements/0332/logs/Metric_access_log-20130604{,.gz}"
Successfully read 804964000 records from: "/Elements/0486g/logs/Metric_access_log-20130604{,.gz}"
Successfully read 1 records from: "/Elements/test/fields.log"

Output(s):
Successfully stored 0 records in: "/jobs/output/bcn/robot"
Successfully stored 0 records in: "/jobs/output/bcn/red"
Successfully stored 176 records (18592 bytes) in: "/jobs/output/bcn/err"
Successfully stored 1 records in: "/jobs/output/bcn/fields"
Successfully stored 2 records in: "/jobs/output/bcn/box"
Successfully stored 141628 records in: "/jobs/output/bcn/hb"

Counters:
Total records written : 141807
Total bytes written : 18592
Spillable Memory Manager spill count : 0
Total bags proactively spilled: 1058
Total records proactively spilled: 64060225

Job DAG:
job_201312130533_1837   ->      job_201312130533_1841,job_201312130533_1838,job_201312130533_1840,job_201312130533_1839,
job_201312130533_1841   ->      null,
null    ->      null,
null    ->      null,null,
job_201312130533_1838   ->      job_201312130533_1843,job_201312130533_1845,
job_201312130533_1843   ->      job_201312130533_1848,job_201312130533_1847,
job_201312130533_1848   ->      job_201312130533_1855,
job_201312130533_1855   ->      null,
job_201312130533_1840   ->      job_201312130533_1846,
job_201312130533_1846   ->      job_201312130533_1849,job_201312130533_1852,job_201312130533_1851,
job_201312130533_1849   ->      null,null,
job_201312130533_1845   ->      job_201312130533_1850,job_201312130533_1847,
job_201312130533_1850   ->      job_201312130533_1856,
job_201312130533_1856   ->      job_201312130533_1858,
job_201312130533_1858   ->      job_201312130533_1859,
job_201312130533_1859   ->      null,null,
job_201312130533_1839   ->      job_201312130533_1844,job_201312130533_1842,
job_201312130533_1844   ->      null,null,
null
job_201312130533_1836
job_201312130533_1842   ->      job_201312130533_1847,
job_201312130533_1847   ->      job_201312130533_1857,
job_201312130533_1857
job_201312130533_1852   ->      job_201312130533_1854,job_201312130533_1853,
job_201312130533_1851   ->      job_201312130533_1854,
job_201312130533_1854
job_201312130533_1853   ->      null,
null

</pre>

 1. I checked the log from Mapr System Console:
<code>

java.lang.RuntimeException: java.io.IOException: No space left on device at org.apache.pig.data.InternalCachedBag.add(InternalCachedBag.java:106) at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POPackage.getNext(POPackage.java:276) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Reduce.processOnePackageOutput(PigGenericMapReduce.java:422) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Reduce.reduce(PigGenericMapReduce.java:413) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Reduce.reduce(PigGenericMapReduce.java:257) at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:176) at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:602) at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:444) at org.apache.hadoop.mapred.Child$4.run(Child.java:270) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127) at org.apache.hadoop.mapred.Child.main(Child.java:264) Caused by: java.io.IOException: No space left on device at java.io.FileOutputStream.writeBytes(Native Method) at java.io.FileOutputStream.write(FileOutputStream.java:282) at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65) at java.io.BufferedOutputStream.write(BufferedOutputStream.java:109) at java.io.DataOutputStream.write(DataOutputStream.java:90) at java.io.DataOutputStream.writeUTF(DataOutputStream.java:384) at java.io.DataOutputStream.writeUTF(DataOutputStream.java:306) at org.apache.pig.data.utils.SedesHelper.writeChararray(SedesHelper.java:66) at org.apache.pig.data.BinInterSedes.writeDatum(BinInterSedes.java:543) at org.apache.pig.data.BinInterSedes.writeDatum(BinInterSedes.java:435) at org.apache.pig.data.utils.SedesHelper.writeGenericTuple(SedesHelper.java:135) at org.apache.pig.data.BinInterSedes.writeTuple(BinInterSedes.java:613) at org.apache.pig.data.BinInterSedes.writeDatum(BinInterSedes.java:443) at org.apache.pig.data.InternalCachedBag.add(InternalCachedBag.java:97) ... 12 more
</pre>
 2. I found my tmp directory whose size is 15G that usage almost full when pig was running the above task.
 3. according to the log, I thought the issuse is that there is not enought space in my tmp directory. but I am new to use Pig, So I am not sure for that, Could you give me some good suggestions?
 4. or How could I optimize my pig script? I need your help, thank you so much.
 Ps. my english is very poor, so I hope you can understand my questions that I described above.

Outcomes