AnsweredAssumed Answered

Does Sqoop 2 support impersonation?

Question asked by dodoman on Jun 9, 2016
Latest reply on Sep 16, 2016 by dodoman

Hi there,

I changed my cluster setup to impersonation, which works with nearly every component.

But Sqoop isn't working anymore, since I changed the cluster configuration of hadoop, hive and hue.

Now i changed the configuration of sqoop2 to impersonation by enabling these lines in the sqoop.properties

 

org.apache.sqoop.security.authentication.enable.doAs=true
org.apache.sqoop.security.authentication.proxyuser.mapr.users=*
org.apache.sqoop.security.authentication.proxyuser.mapr.groups=*
org.apache.sqoop.security.authentication.proxyuser.mapr.hosts=*

 

While mapr is my impersonation user.

 

Even if I enable the impersonation mode, I am still running into the following Exceptions:

 

Jun 09, 2016 12:54:08 PM org.apache.catalina.core.StandardWrapperValve invoke
SEVERE: Servlet.service() for servlet v1.JobServlet threw exception
java.lang.NoSuchMethodError: <init>
        at com.mapr.fs.jni.MapRClient.getattr(Native Method)
        at com.mapr.fs.MapRClientImpl.getFileStatus(MapRClientImpl.java:372)
        at com.mapr.fs.MapRFileSystem.getMapRFileStatus(MapRFileSystem.java:1397)
        at com.mapr.fs.MapRFileSystem.getFileStatus(MapRFileSystem.java:1024)
        at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1460)
        at org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:116)
        at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:147)
        at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1290)
        at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1287)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1595)
        at org.apache.hadoop.mapreduce.Job.submit(Job.java:1287)
        at org.apache.sqoop.submission.mapreduce.MapreduceSubmissionEngine.submitToCluster(MapreduceSubmissionEngine.java:274)
        at org.apache.sqoop.submission.mapreduce.MapreduceSubmissionEngine.submit(MapreduceSubmissionEngine.java:255)
        at org.apache.sqoop.driver.JobManager.start(JobManager.java:288)
        at org.apache.sqoop.handler.JobRequestHandler.startJob(JobRequestHandler.java:380)
        at org.apache.sqoop.handler.JobRequestHandler.handleEvent(JobRequestHandler.java:116)
        at org.apache.sqoop.server.v1.JobServlet.handlePutRequest(JobServlet.java:96)
        at org.apache.sqoop.server.SqoopProtocolServlet.doPut(SqoopProtocolServlet.java:79)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:646)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:723)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:604)
        at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationFilter.doFilter(DelegationTokenAuthenticationFilter.java:277)
        at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:567)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
        at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:861)
        at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:606)
        at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
        at java.lang.Thread.run(Thread.java:745)

 

I am also wondering why starting the job in hue sets user to null, while the commandline sets the commandline user. But maybe this has nothing in common with the exception.

 

2016-06-09 12:50:11,015 INFO  default: user=null        ip=10.238.70.84 op=update       obj=job objId=9
2016-06-09 12:50:11,131 INFO  default: user=null        ip=10.238.70.84 op=get  obj=jobs
        objId=all
2016-06-09 12:50:11,278 INFO  default: user=null        ip=10.238.70.84 op=get  obj=jobs
        objId=all
2016-06-09 12:50:11,954 INFO  default: user=null        ip=10.238.70.84 op=get  obj=job objId=9
2016-06-09 12:50:11,978 INFO  default: user=null        ip=10.238.70.84 op=submit       obj=job objId=9
2016-06-09 12:53:26,274 INFO  default: user=mapr        ip=10.238.70.84 op=submit       obj=job objId=9
2016-06-09 12:53:47,196 INFO  default: user=mapr        ip=10.238.70.84 op=submit       obj=job objId=9
2016-06-09 12:54:04,176 INFO  default: user=mapr        ip=10.238.70.84 op=submit       obj=job objId=9

 

Do you have any Idea how to fix this?

 

Here the versions I use:

MapR 5.1

Sqoop 1.99.6

Hue 3.9.0

Outcomes