AnsweredAssumed Answered

Cannot configure SPARK with HUE

Question asked by thomasd on Mar 24, 2016
Latest reply on Apr 19, 2016 by maprcommunity

Hi guys,

 

I am faced to a main issue for a couple of days. I am currently trying to install Hue. The installation has been successful for Hive but is failing for spark. I am following mapR instructions on mapR website for the installation.

You will find below some information about my configuration :

  • Hue 3.9.0
  • Spark 1.5.2

 

Hue.ini :

 

[spark]

# Host address of the Livy Server.

livy_server_host=localhost

 

# Port of the Livy Server.

livy_server_port=8998

 

# Configure livy to start with 'process', 'thread', or 'yarn' workers.

  livy_server_session_kind=yarn

 

Hue.sh :

 

export HUE_HOME=/opt/mapr/hue/hue-3.9.0

export HADOOP_CONF_DIR=/opt/mapr/hadoop/hadoop-2.7.0/etc/Hadoop

 

The log in Hue are the following one :

 

[23/Mar/2016 09:19:01 -0700] conf ERROR    failed to get spark status

Traceback (most recent call last):

  File "/opt/mapr/hue/hue-3.9.0/apps/spark/src/spark/conf.py", line 100, in get_spark_status

get_api(user).get_status()

  File "/opt/mapr/hue/hue-3.9.0/apps/spark/src/spark/job_server_api.py", line 84, in get_status

return self._root.get('sessions')

  File "/opt/mapr/hue/hue-3.9.0/desktop/core/src/desktop/lib/rest/resource.py", line 97, in get

return self.invoke("GET", relpath, params, headers=headers, allow_redirects=True)

  File "/opt/mapr/hue/hue-3.9.0/desktop/core/src/desktop/lib/rest/resource.py", line 78, in invoke

urlencode=self._urlencode)

  File "/opt/mapr/hue/hue-3.9.0/desktop/core/src/desktop/lib/rest/http_client.py", line 161, in execute

raise self._exc_class(ex)

RestException: ('Connection aborted.', BadStatusLine('',))

[23/Mar/2016 09:19:01 -0700] connectionpool INFO     Starting new HTTP connection (1): localhost

 

There is also something suspicious in the logs of hue-mapr-livy_server-localhost:

 

07:59:53.082 [main] WARN  c.c.h.j.u.c.AbstractLifeCycle - FAILED ServerConnector@fe{HTTP/1.1}{localhost:8998}: java.net.BindException: Address already in use
java.net.BindException: 998

 at sun.nio.ch.Net.bind0(Native Method) ~[na:1.7.0_71]
 at sun.nio.ch.Net.bind(Net.java:444) ~[na:1.7.0_71]
 at sun.nio.ch.Net.bind(Net.java:436) ~[na:1.7.0_71]
 at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214) ~[na:1.7.0_71]
 at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74) ~[na:1.7.0_71]
 at com.cloudera.hue.jetty.server.ServerConnector.open(ServerConnector.java:321) ~[livy-assembly.jar:na]
 at com.cloudera.hue.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:80) ~[livy-assembly.jar:na]
 at com.cloudera.hue.jetty.server.ServerConnector.doStart(ServerConnector.java:236) ~[livy-assembly.jar:na]
 at com.cloudera.hue.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) [livy-assembly.jar:na]
 at com.cloudera.hue.jetty.server.Server.doStart(Server.java:366) [livy-assembly.jar:na]
 at com.cloudera.hue.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) [livy-assembly.jar:na]
 at com.cloudera.hue.livy.WebServer.start(WebServer.scala:64) [livy-assembly.jar:na]
 at com.cloudera.hue.livy.server.Main$.main(Main.scala:54) [livy-assembly.jar:na]
 at com.cloudera.hue.livy.server.Main.main(Main.scala) [livy-assembly.jar:na]
07:59:53.082 [main] WARN  c.c.h.j.u.c.AbstractLifeCycle - FAILED com.cloudera.hue.jetty.server.Server@68407cdd: java.net.BindException: Address already in use
java.net.BindException: Address already in use

 at sun.nio.ch.Net.bind0(Native Method) ~[na:1.7.0_71]
 at sun.nio.ch.Net.bind(Net.java:444) ~[na:1.7.0_71]
 at sun.nio.ch.Net.bind(Net.java:436) ~[na:1.7.0_71]
 at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214) ~[na:1.7.0_71]
 at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74) ~[na:1.7.0_71]
 at com.cloudera.hue.jetty.server.ServerConnector.open(ServerConnector.java:321) ~[livy-assembly.jar:na]
 at com.cloudera.hue.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:80) ~[livy-assembly.jar:na]
 at com.cloudera.hue.jetty.server.ServerConnector.doStart(ServerConnector.java:236) ~[livy-assembly.jar:na]
 at com.cloudera.hue.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) ~[livy-assembly.jar:na]
 at com.cloudera.hue.jetty.server.Server.doStart(Server.java:366) ~[livy-assembly.jar:na]
 at com.cloudera.hue.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) ~[livy-assembly.jar:na]
 at com.cloudera.hue.livy.WebServer.start(WebServer.scala:64) [livy-assembly.jar:na]
 at com.cloudera.hue.livy.server.Main$.main(Main.scala:54) [livy-assembly.jar:na]
 at com.cloudera.hue.livy.server.Main.main(Main.scala) [livy-assembly.jar:na]
Exception in thread "main" java.net.BindException: Address already in use
 at sun.nio.ch.Net.bind0(Native Method)
 at sun.nio.ch.Net.bind(Net.java:444)
 at sun.nio.ch.Net.bind(Net.java:436)
 at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214)
 at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
 at com.cloudera.hue.jetty.server.ServerConnector.open(ServerConnector.java:321)
 at com.cloudera.hue.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:80)
 at com.cloudera.hue.jetty.server.ServerConnector.doStart(ServerConnector.java:236)
 at com.cloudera.hue.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
 at com.cloudera.hue.jetty.server.Server.doStart(Server.java:366)
 at com.cloudera.hue.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
 at com.cloudera.hue.livy.WebServer.start(WebServer.scala:64)
 at com.cloudera.hue.livy.server.Main$.main(Main.scala:54)
      at com.cloudera.hue.livy.server.Main.main(Main.scala)

 

The error message once I am opening Hue UI is

Spark

The app won't work without a running Livy Spark Server

 

Thanks for your help, I am really stuck on this point.

 

Thomas

Outcomes