AnsweredAssumed Answered

com.mapr.fs.Inode: Write failed for file ...

Question asked by mark_kidwell on Sep 6, 2012
Latest reply on Sep 25, 2012 by mark_kidwell
Hi everyone,

On a completely new MapR cluster set up with MapR 2.0 M3 edition, some (50+%) of every MR job's tasks are getting errors like the one below. The errors occur on every fileserver in the cluster, and in every type of job we run (test jobs like DFSIO, Hive query jobs, etc.) In the best case, it causes jobs to run longer due to task retries, in the worst case jobs simply fail. Has anyone else run into anything like this?

Please see the details below, and thanks in advance.


Mark Kidwell

Sr Manager Database Engineering,

Cluster info:

4 node Ubuntu 10.04, 2.6.38-13-server kernel
MapR M3

Example of task error:

    2012-09-05 18:38:28,817 ERROR com.mapr.fs.Inode: Write failed for file: /var/mapr/local/<hostname>/mapred/taskTracker/output/job_201209051646_0025/attempt_201209051646_0025_m_000001_0/output.00000, error: Invalid argument

Corresponding fileserver errors from /opt/mapr/logs/mfs.log:

    2012-09-05 18:38:28,8151 ERROR x.x.0.0:0 ContainerAssignProc failed with status 22
    2012-09-05 18:38:28,8152 ERROR x.x.0.0:0 Assign() for volume 251957651 clientIP failed 22
    2012-09-05 18:38:28,8152 ERROR x.x.0.0:0 ContainerAssign failed 22
    2012-09-05 18:38:28,8152 ERROR x.x.0.0:0 AllocateFidMap: prim fid 2174.75.114490 chunkIndex 0 Shard create failed (22) for primary Fid 2174.75.114490