AnsweredAssumed Answered

Queries fail after upgrading to YARN

Question asked by kokonihau on Jun 11, 2015
Latest reply on Jun 11, 2015 by kokonihau
After upgrading to YARN (4.1 MRv2 / hive 0.12) fails to query external tables due to what seem's to be a problem accessing data on MFS.

for example, creating an external table :  

    CREATE EXTERNAL TABLE DB.t3(a string) LOCATION '/user/DB/filename.txt';
running `select * from DB.t3;` -- success 
running `select count(*) from DB.t3;` -- fails with the following error :

    15/06/11 16:10:05 ERROR exec.Task: Job Submission failed with exception 'java.io.IOException(cannot find dir = maprfs:///user/DB/filename.txt in pathToPartitionInfo: [maprfs:/user/DB/filename.txt])'
    java.io.IOException: cannot find dir = maprfs:///user/DB/filename.txt in pathToPartitionInfo: [maprfs:/user/DB/filename.txt]
            at org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getPartitionDescFromPathRecursively(HiveFileFormatUtils.java:344)
            at org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getPartitionDescFromPathRecursively(HiveFileFormatUtils.java:306)
            at org.apache.hadoop.hive.ql.io.CombineHiveInputFormat$CombineHiveInputSplit.<init>(CombineHiveInputFormat.java:105)
            at org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getSplits(CombineHiveInputFormat.java:420)
            at org.apache.hadoop.mapreduce.JobSubmitter.writeOldSplits(JobSubmitter.java:520)
            at org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:512)
            at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:394)
            at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1285)
            at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1282)
            at java.security.AccessController.doPrivileged(Native Method)
            at javax.security.auth.Subject.doAs(Subject.java:422)
            at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1566)
            at org.apache.hadoop.mapreduce.Job.submit(Job.java:1282)
            at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:562)
            at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:557)
            at java.security.AccessController.doPrivileged(Native Method)
            at javax.security.auth.Subject.doAs(Subject.java:422)
            at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1566)
            at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:557)
            at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:548)
            at org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:425)
            at org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute(MapRedTask.java:144)
            at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:150)
            at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:65)
            at org.apache.hadoop.hive.ql.exec.TaskRunner.run(TaskRunner.java:55)

Trying to create the table with the following location : `'maprfs:/user/DB/filename.txt'`
**fails** exactly the same.

But, creating the table with this location : `'maprfs://my.cluster.com/user/DB/filename.txt'`
**works** well!

Also, this only happens to locations that are pointing exactly to a file and not a directory.
So, putting `filename.txt` in a directory `dir` and setting the location to : `'/user/DB/dir'`
also works.

you can see that when it fails, it tries to access the file using `maprfs:///` 
and when it succeeds in the logs its looks like : `maprfs:/` 
for example : `INFO exec.Utilities: Adding input file maprfs:/user/DB/dir`

Also, all the queries are throwing an exception in the end of the execution saying :

    ERROR fs.MapRFileSystem: Failed to delete path maprfs:/user/hadoop/tmp/hive/hive_2015-06-11_16-13-19_358_548055336019337510-1/-mr-10000, error: No such file or directory (2)
The queries do succeed but its still weird.

This works perfectly well on another 3.0.1 MRv1 Cluster.


Outcomes