AnsweredAssumed Answered

error with sqoop incremental import with merge-key

Question asked by ronaldm on Apr 28, 2016
Latest reply on Jul 25, 2017 by maprcommunity

Running a sqoop job:

sqoop job --create testjob -- import --connect jdbc:mysql://localhost/testdb --username username --password password --table member_info -m 2 --check-column updated_ts --incremental lastmodified --target-dir /tmp/test --fields-terminated-by ',' --merge-key id

 

During the merge phase (I believe), encountered the following problem:

16/04/14 18:33:40 INFO mapreduce.Job: Task Id : attempt_1460535083590_0043_m_000002_0, Status : FAILED

Error: java.io.IOException: File maprfs:///user/mapr/_sqoop/be8ef6242d5d40458eaf3a8afad4b660_member_info/part-m-00000 is not under new path maprfs:/user/mapr/_sqoop/be8ef6242d5d40458eaf3a8afad4b660_member_info or old path maprfs:///tmp/test

  at org.apache.sqoop.mapreduce.MergeMapperBase.setup(MergeMapperBase.java:64)

  at org.apache.sqoop.mapreduce.MergeTextMapper.setup(MergeTextMapper.java:47)

  at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143)

  at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:796)

  at org.apache.hadoop.mapred.MapTask.run(MapTask.java:346)

  at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)

  at java.security.AccessController.doPrivileged(Native Method)

  at javax.security.auth.Subject.doAs(Subject.java:422)

  at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1595)

  at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)

 

 

Sqoop merge uses a couple parameters (--new-data, --onto) that seem related to the above error.

However, sqoop import does not provide any parameters related to this.

 

Any ideas where I am doing this wrong?

Outcomes