AnsweredAssumed Answered

org.apache.hadoop.fs.PathId.getFidInfo

Question asked by stormcrow on Jul 8, 2013
Latest reply on Jul 9, 2013 by gera
Reduce tasks for most cluster jobs all failing with the following error:

    Diagnostic Messages for this Task:
    Error: java.lang.NumberFormatException: For input string: "2244811652"
            at java.lang.NumberFormatException.forInputString(NumberFormatException.java:48)
            at java.lang.Integer.parseInt(Integer.java:461)
            at java.lang.Integer.valueOf(Integer.java:554)
            at org.apache.hadoop.fs.PathId.getFidInfo(PathId.java:56)
            at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputLocation.<init>(ReduceTask.java:940)
            at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.getMapCompletionEvents(ReduceTask.java:2933)
            at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.run(ReduceTask.java:2838)
    
    
    FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask


It looks to us like our "fileID" of wherever the Reduce task is trying
to send output have exceeded the size of a 32 bit signed integer
(2.1B)....note it's trying to parse an int from "2244811652" (2.2B).


Have we reached the maximum number of files on a volume somewhere or
something?

What exactly is PathId.getFidInfo() trying to read?

Outcomes