AnsweredAssumed Answered

com.mapr.fs.Inode: Write failed for file ...

Question asked by mark_kidwell on Sep 6, 2012
Latest reply on Sep 25, 2012 by mark_kidwell
Hi everyone,

On a completely new MapR cluster set up with MapR 2.0 M3 edition, some (50+%) of every MR job's tasks are getting errors like the one below. The errors occur on every fileserver in the cluster, and in every type of job we run (test jobs like DFSIO, Hive query jobs, etc.) In the best case, it causes jobs to run longer due to task retries, in the worst case jobs simply fail. Has anyone else run into anything like this?

Please see the details below, and thanks in advance.

Regards,

Mark Kidwell

Sr Manager Database Engineering, Gracenote.com

Cluster info:

4 node Ubuntu 10.04, 2.6.38-13-server kernel
MapR M3 2.0.0.15153GA-1

Example of task error:

    2012-09-05 18:38:28,817 ERROR com.mapr.fs.Inode: Write failed for file: /var/mapr/local/<hostname>/mapred/taskTracker/output/job_201209051646_0025/attempt_201209051646_0025_m_000001_0/output.00000, error: Invalid argument

Corresponding fileserver errors from /opt/mapr/logs/mfs.log:

    2012-09-05 18:38:28,8151 ERROR  volclient.cc:134 x.x.0.0:0 ContainerAssignProc failed with status 22
    2012-09-05 18:38:28,8152 ERROR  pcldb.cc:118 x.x.0.0:0 Assign() for volume 251957651 clientIP 10.11.130.154:0 failed 22
    2012-09-05 18:38:28,8152 ERROR  shard.cc:74 x.x.0.0:0 ContainerAssign failed 22
    2012-09-05 18:38:28,8152 ERROR  allocatefid.cc:378 x.x.0.0:0 AllocateFidMap: prim fid 2174.75.114490 chunkIndex 0 Shard create failed (22) for primary Fid 2174.75.114490

Outcomes