AnsweredAssumed Answered

HDFS rename API failing with no information

Question asked by chriscurtin on May 7, 2012
Latest reply on May 9, 2012 by nabeel
Hi,

I'm moving a Cascading application to MapR and I've encountered an issue with the 'rename' method in the FileSystem class. Basically, it is failing with no information about why.

The logic uses a Cascading TemplateTap to build 1 or more 'bins' of files in HDFS that I then go back and using 'rename' move to a final home within another reducer.

For example the TemplateTap creates directories /offline/aat/100_100, /offline/aat/100_200 etc. A second Map/Reduce job moves /offline/aat/100_100 to /offlined/1_1232/2012_04/100_100_aat (so moves it to a directory based on the month it was created)

All I get back is a 'false' from the call to rename and no details nor exception.

Sample code:

    private int moveFile(Path a_directory, String a_destinationDir, String a_newName) {
        int numMoved = 0;
        try {
            Configuration config = new Configuration();
            FileSystem hdfs = FileSystem.get(config);

            FileStatus files[] = hdfs.listStatus(a_directory);
            if (files.length != 0) {
                for (int cnt = 0; cnt < files.length; cnt++) {
                    if (files[cnt].isDir()) continue;
                    hdfs.mkdirs(new Path(a_destinationDir));
                    m_logger.info("Source File: " + files[cnt].getPath().toString() );
                    if (!hdfs.rename(files[cnt].getPath(), new Path(a_destinationDir + a_newName))) {
                        m_logger.error("Error Moving Files to final HDFS Directory [" + a_destinationDir + a_newName + "]");
                    } else {
                        numMoved++;
                    }
                }
            } else {
                m_logger.info("No Files to move to HDFS? : " + a_directory.getName());
            }
        }
        catch (java.io.IOException ioe) {
            m_logger.error("Unable to move file to final HDFS Directory. [" + a_destinationDir + a_newName + "]", ioe);
            numMoved = 0;
        }
        return numMoved;
    }

Output in the reducer:

2012-05-08 10:27:49,046 INFO com.silverpop.clouds.tracking.offline.MoveFiles: Source File: maprfs://10.121.41.42:7222/offline/working/1/aat/1162872_123115393/part-00016
2012-05-08 10:27:49,048 ERROR com.silverpop.clouds.tracking.offline.MoveFiles: Error Moving Files to final HDFS Directory [/offlined/1_1511/2012_04/1162872_123115393_243474_242392_1162870_20120501_aat]
2012-05-08 10:27:49,230 INFO com.silverpop.clouds.tracking.offline.MoveFiles: Source File: maprfs://10.121.41.42:7222/offline/working/1/aat/1238888_124340092/part-00016
2012-05-08 10:27:49,231 ERROR com.silverpop.clouds.tracking.offline.MoveFiles: Error Moving Files to final HDFS Directory [/offlined/1_1325/2012_04/1238888_124340092_506424_218782_1237409_20120501_aat]

Any thoughts on how to figure out why the rename is failing?

Outcomes