AnsweredAssumed Answered

DistibutedCache - Returning bad page to cache page

Question asked by fistan684 on Jan 11, 2012
Latest reply on Jan 11, 2012 by tomer
We're recently moving over to MapR and we're trying to migrate a streaming job over to the new distribution. So far things were going smoothly but today we ran into an issue where we are seeing some strange errors coming up. To give you a little background, this particular streaming job uses a Perl Mapper and Java Reducer. To make the various perl modules available we create a Jar of the needed modules and place it into distributed cache. This has worked fine in past distributions, but for some reason we're getting the following errors showing up in the log:
<pre>
ERROR com.mapr.fs.Inode: 2049.271687.1958124 /mapreduce/perl-lib.jar Returning bad page to cache page: (2049.271687.1958124 0, id: 9, state Invalid) 2012-01-11 18:32:17,905 WARN org.apache.hadoop.mapred.TaskTracker: Exception while localization java.io.IOException: Error -5 at com.mapr.fs.Inode.throwIfFailed(Inode.java:158) at com.mapr.fs.Inode.getDataIntoCache(Inode.java:568) at com.mapr.fs.MapRFsInStream.dropAndRefillCurrentPage(MapRFsInStream.java:98) at com.mapr.fs.MapRFsInStream.read(MapRFsInStream.java:133) at java.io.DataInputStream.read(DataInputStream.java:83) at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:55) at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:87) at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:224) at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:157) at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1335) at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1316) at org.apache.hadoop.filecache.TrackerDistributedCacheManager.downloadCacheObject(TrackerDistributedCacheManager.java:539) at org.apache.hadoop.filecache.TrackerDistributedCacheManager.localizePublicCacheObject(TrackerDistributedCacheManager.java:593) at org.apache.hadoop.filecache.TrackerDistributedCacheManager.getLocalCache(TrackerDistributedCacheManager.java:201) at org.apache.hadoop.filecache.TaskDistributedCacheManager.setupCache(TaskDistributedCacheManager.java:183) at org.apache.hadoop.mapred.TaskTracker$4.run(TaskTracker.java:1560) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1109) at org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1551) at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1433) at org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskTracker.java:3718) at org.apache.hadoop.mapred.TaskTracker$TaskLauncher.run(TaskTracker.java:3641)
</pre>
If anyone has any suggestions on where to begin to track this issue down it would be greatly appreciated. The job obviously fails straight away before even beginning with this particular error. Thanks.

Outcomes