AnsweredAssumed Answered

Does anyone verified the webhdfs success with enabled kerberos?

Question asked by zzx on Dec 27, 2017
Latest reply on Jan 10, 2018 by maprcommunity

Does anyone verified the webhdfs success with enabled kerberos on mapr_ecosystem_version 4.0.0?

 

I installed the mapr 6.0.0 and mapr_ecosystem_version  6.0.0 with enabled the kerberos, and ensure the basic hadoop command is ok, but when I test the webhdfs, it always failed. 

 

[mapr@mo-39520c3ff hadoop]$ hadoop fs -ls webhdfs://mo-39520c3ff.mo.sap.corp:14000/
17/12/20 07:44:06 WARN web.WebHdfsFileSystem: Unexpected end of file from server
ls: Unexpected end of file from server

Error in spark-shell is the same as in shell:
java.net.SocketException: Unexpected end of file from server
at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:851)
at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:678)
at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:848)
at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:678)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1587)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)
at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)
at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.validateResponse(WebHdfsFileSystem.java:341)
at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$200(WebHdfsFileSystem.java:90)
at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.runWithRetry(WebHdfsFileSystem.java:620)
at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.access$100(WebHdfsFileSystem.java:470)
at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner$1.run(WebHdfsFileSystem.java:499)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1631)
at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.run(WebHdfsFileSystem.java:495)
at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getHdfsFileStatus(WebHdfsFileSystem.java:850)
at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getFileStatus(WebHdfsFileSystem.java:865)
at org.apache.hadoop.fs.Globber.getFileStatus(Globber.java:57)
at org.apache.hadoop.fs.Globber.glob(Globber.java:253)
at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:1714)
at org.apache.hadoop.mapred.FileInputFormat.singleThreadedListStatus(FileInputFormat.java:259)
at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:229)
at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:317)
at org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:206)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:252)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:250)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.rdd.RDD.partitions(RDD.scala:250)
at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:35)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:252)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:250)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.rdd.RDD.partitions(RDD.scala:250)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:1958)
at org.apache.spark.rdd.RDD.count(RDD.scala:1158)
... 48 elided


But the basic function is OK and the livy test all are ok except for webhdfs.


[mapr@mo-39520c3ff server]$ hadoop fs -ls maprfs://mo-7b87031da.mo.sap.corp:7222/
Found 6 items
drwxr-xr-x - mapr mapr 1 2017-12-19 07:33 maprfs://mo-7b87031da.mo.sap.corp:7222/apps
drwxr-xr-x - mapr mapr 0 2017-12-19 07:34 maprfs://mo-7b87031da.mo.sap.corp:7222/opt
-rwxrwxrwx 3 mapr mapr 32 2017-12-20 02:27 maprfs://mo-7b87031da.mo.sap.corp:7222/people.txt
drwxrwxrwx - mapr mapr 0 2017-12-19 07:32 maprfs://mo-7b87031da.mo.sap.corp:7222/tmp
drwxrwxrwx - mapr mapr 1 2017-12-19 07:34 maprfs://mo-7b87031da.mo.sap.corp:7222/user
drwxr-xr-x - mapr mapr 1 2017-12-19 07:33 maprfs://mo-7b87031da.mo.sap.corp:7222/var

 

The configurations for httpfs:
Step 1: Set up a Kerberos Principal and keytab File
https://maprdocs.mapr.com/home/HttpFS/c-setup-kerberos-principal-keytab-file.html?hl=kerberos%2Cauthentication%2Chttpfs
Step 2: Verify Credentials in the keytab File
https://maprdocs.mapr.com/home/HttpFS/c-verify-creds-keytab-file.html
Step 3: Modify the httpfs-site.xml File
https://maprdocs.mapr.com/home/HttpFS/modify-httpfs-site_xml-file.html
Configuring SPNEGO on MapR
https://maprdocs.mapr.com/home/SecurityGuide/ConfiguringSPNEGOonMapR.html

Outcomes