AnsweredAssumed Answered

Hive Can't Rename Across Volumes on Insert

Question asked by mandoskippy on Nov 1, 2012
Latest reply on Nov 1, 2012 by mandoskippy
I am trying to run a small insert, and I am getting the error below.   Now it says it can't rename across partitions which I get, but I am having a hard time figuring out what the source and destination on the move is.  /user/hive is the volume, and in that there is the warehouse directory and the scratch directory (i.e. both scratch and the final data are in the same volume) This is a different volume from the root, however, I need to know what is causing the issue. Very frustrating at this point.

12/11/01 11:14:07 INFO metadata.Hive: New loading path = maprfs:/user/hive/scratch/hive-darkness/hive_2012-11-01_11-13-45_109_7241101075339812186/-ext-10000/day=2012-10-31/source=source1 with partSpec {day=2012-10-31, source=source1}
12/11/01 11:14:07 INFO metastore.HiveMetaStore: 0: get_table : db=%Database% tbl=%tablename%
12/11/01 11:14:07 INFO hive.log: DDL: struct {%tablename%}{%Columns%}
12/11/01 11:14:07 DEBUG columnar.ColumnarSerDe: ColumnarSerDe initialized with: %LOTS OF COLUMNS%
12/11/01 11:14:07 INFO metastore.HiveMetaStore: 0: get_partition_with_auth : db=intel_flow tbl=%tablename%[2012-10-31,source2]
Failed with exception copyFiles: error while moving files!!!
12/11/01 11:14:07 ERROR exec.Task: Failed with exception copyFiles: error while moving files!!!
org.apache.hadoop.hive.ql.metadata.HiveException: copyFiles: error while moving files!!!
at org.apache.hadoop.hive.ql.metadata.Hive.copyFiles(Hive.java:1971)
at org.apache.hadoop.hive.ql.metadata.Hive.loadPartition(Hive.java:1152)
at org.apache.hadoop.hive.ql.metadata.Hive.loadDynamicPartitions(Hive.java:1231)
at org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:259)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:134)
at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:57)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1326)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1118)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:951)
at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:258)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:215)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:406)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:341)
at org.apache.hadoop.hive.cli.CliDriver.processReader(CliDriver.java:439)
at org.apache.hadoop.hive.cli.CliDriver.processFile(CliDriver.java:449)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:647)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:557)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:197)
Caused by: java.io.IOException: Cannot rename across volumes
at com.mapr.fs.MapRFileSystem.rename(MapRFileSystem.java:462)
at org.apache.hadoop.hive.ql.metadata.Hive.copyFiles(Hive.java:1965)
... 21 more

FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.MoveTask
12/11/01 11:14:07 ERROR ql.Driver: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.MoveTask
12/11/01 11:14:07 DEBUG fs.FileSystem: Removing filesystem for maprfs:///
12/11/01 11:14:07 DEBUG fs.FileSystem: Removing filesystem for file:///
12/11/01 11:14:07 DEBUG fs.FileSystem: Removing filesystem for file:///

Outcomes