AnsweredAssumed Answered

Zookeeper does not start properly

Question asked by mahdi on Jul 29, 2013
Latest reply on Aug 26, 2015 by 00gavin
Hi All,

I'm trying to set up a 3-node cluster with the following configuration:
<code>
node1: cldb, zookeeper, fileserver, tasktracker
node2: zookeeper, webserver, fileserver, tasktracker
node3: zookeeper, fileserver, jobtracker, tasktracker
</code>

I set up keyless ssh between the nodes and I made sure to have SELinux and iptables disabled.  I start zookeeper on the nodes as the first step to bring up the cluster, but when I check the status of zookeeper, this is what I get:
<code>
    [root@mr1 mapr]# service mapr-zookeeper qstatus
    JMX enabled by default
    Using config: /opt/mapr/zookeeper/zookeeper-3.3.6/conf/zoo.cfg
    Error contacting service. It is probably not running.
<code>
And this is my zookeeper.log:

<pre>
[root@mr1 mapr]# more /opt/mapr/zookeeper/zookeeper-3.3.6/logs/zookeeper.log
2013-07-29 12:14:00,976 - INFO  [main:QuorumPeerConfig@90] - Reading configuration from: /opt/mapr/zookeeper/zookeeper-3.3.6/conf/zoo.cfg
2013-07-29 12:14:00,982 - INFO  [main:QuorumPeerConfig@310] - Defaulting to majority quorums
2013-07-29 12:14:00,995 - INFO  [main:QuorumPeerMain@119] - Starting quorum peer
2013-07-29 12:14:01,016 - INFO  [main:NIOServerCnxn$Factory@143] - binding to port 0.0.0.0/0.0.0.0:5181
2013-07-29 12:14:01,029 - INFO  [main:QuorumPeer@819] - tickTime set to 2000
2013-07-29 12:14:01,029 - INFO  [main:QuorumPeer@830] - minSessionTimeout set to -1
2013-07-29 12:14:01,029 - INFO  [main:QuorumPeer@841] - maxSessionTimeout set to -1
2013-07-29 12:14:01,030 - INFO  [main:QuorumPeer@856] - initLimit set to 20
2013-07-29 12:14:01,043 - INFO  [Thread-1:QuorumCnxManager$Listener@473] - My election bind port: 3888
2013-07-29 12:14:01,053 - INFO  [QuorumPeer:/0:0:0:0:0:0:0:0:5181:QuorumPeer@621] - LOOKING
2013-07-29 12:14:01,055 - INFO  [QuorumPeer:/0:0:0:0:0:0:0:0:5181:FastLeaderElection@663] - New election. My id =  0, Proposed zxid = 0
2013-07-29 12:14:01,059 - INFO  [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 0 (n.leader), 0 (n.zxid), 1 (n.round), LOOKING (n.state), 0
(n.sid), LOOKING (my state)
2013-07-29 12:14:01,062 - WARN  [WorkerSender Thread:QuorumCnxManager@384] - Cannot open channel to 1 at election address mr2/10.10.10.6:3888
java.net.NoRouteToHostException: No route to host
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:708)
at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:115)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:371)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:340)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:360)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:333)
at java.lang.Thread.run(Thread.java:724)
2013-07-29 12:14:01,064 - WARN  [WorkerSender Thread:QuorumCnxManager@384] - Cannot open channel to 2 at election address mr3/10.10.10.4:3888
java.net.NoRouteToHostException: No route to host
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:708)
at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:115)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:371)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:340)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:360)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:333)
at java.lang.Thread.run(Thread.java:724)
2013-07-29 12:14:01,262 - WARN  [QuorumPeer:/0:0:0:0:0:0:0:0:5181:QuorumCnxManager@384] - Cannot open channel to 2 at election address mr3/10.10.10.4:3888
java.net.NoRouteToHostException: No route to host
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:708)
at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:115)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:371)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:404)
at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:688)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:622)
2013-07-29 12:14:01,264 - WARN  [QuorumPeer:/0:0:0:0:0:0:0:0:5181:QuorumCnxManager@384] - Cannot open channel to 1 at election address mr2/10.10.10.6:3888
java.net.NoRouteToHostException: No route to host
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:708)
at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:115)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:371)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:404)
at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:688)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:622)
2013-07-29 12:14:01,265 - INFO  [QuorumPeer:/0:0:0:0:0:0:0:0:5181:FastLeaderElection@697] - Notification time out: 400
2013-07-29 12:14:01,667 - WARN  [QuorumPeer:/0:0:0:0:0:0:0:0:5181:QuorumCnxManager@384] - Cannot open channel to 2 at election address mr3/10.10.10.4:3888
java.net.NoRouteToHostException: No route to host
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:708)
at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:115)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:371)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:404)
at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:688)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:622)
2013-07-29 12:14:01,669 - WARN  [QuorumPeer:/0:0:0:0:0:0:0:0:5181:QuorumCnxManager@384] - Cannot open channel to 1 at election address mr2/10.10.10.6:3888
java.net.NoRouteToHostException: No route to host
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:708)
at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:115)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:371)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:404)
at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:688)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:622)
2013-07-29 12:14:01,670 - INFO  [QuorumPeer:/0:0:0:0:0:0:0:0:5181:FastLeaderElection@697] - Notification time out: 800
2013-07-29 12:14:02,472 - WARN  [QuorumPeer:/0:0:0:0:0:0:0:0:5181:QuorumCnxManager@384] - Cannot open channel to 2 at election address mr3/10.10.10.4:3888
java.net.NoRouteToHostException: No route to host
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:708)
at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:115)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:371)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:404)
at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:688)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:622)
2013-07-29 12:14:02,474 - WARN  [QuorumPeer:/0:0:0:0:0:0:0:0:5181:QuorumCnxManager@384] - Cannot open channel to 1 at election address mr2/10.10.10.6:3888
java.net.NoRouteToHostException: No route to host
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:708)
at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:115)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:371)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:404)
at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:688)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:622)
2013-07-29 12:14:02,475 - INFO  [QuorumPeer:/0:0:0:0:0:0:0:0:5181:FastLeaderElection@697] - Notification time out: 1600
2013-07-29 12:14:04,077 - WARN  [QuorumPeer:/0:0:0:0:0:0:0:0:5181:QuorumCnxManager@384] - Cannot open channel to 2 at election address mr3/10.10.10.4:3888
java.net.NoRouteToHostException: No route to host
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:708)
at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:115)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:371)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:404)
at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:688)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:622)
2013-07-29 12:14:04,079 - WARN  [QuorumPeer:/0:0:0:0:0:0:0:0:5181:QuorumCnxManager@384] - Cannot open channel to 1 at election address mr2/10.10.10.6:3888
java.net.NoRouteToHostException: No route to host
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:708)
at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:115)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:371)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:404)
at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:688)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:622)
2013-07-29 12:14:04,080 - INFO  [QuorumPeer:/0:0:0:0:0:0:0:0:5181:FastLeaderElection@697] - Notification time out: 3200
2013-07-29 12:14:07,282 - WARN  [QuorumPeer:/0:0:0:0:0:0:0:0:5181:QuorumCnxManager@384] - Cannot open channel to 2 at election address mr3/10.10.10.4:3888
java.net.NoRouteToHostException: No route to host
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:708)
at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:115)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:371)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:404)
at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:688)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:622)
2013-07-29 12:14:07,284 - WARN  [QuorumPeer:/0:0:0:0:0:0:0:0:5181:QuorumCnxManager@384] - Cannot open channel to 1 at election address mr2/10.10.10.6:3888
java.net.NoRouteToHostException: No route to host
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:708)
at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:115)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:371)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:404)
at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:688)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:622)
2013-07-29 12:14:07,285 - INFO  [QuorumPeer:/0:0:0:0:0:0:0:0:5181:FastLeaderElection@697] - Notification time out: 6400
2013-07-29 12:14:13,686 - WARN  [QuorumPeer:/0:0:0:0:0:0:0:0:5181:QuorumCnxManager@384] - Cannot open channel to 2 at election address mr3/10.10.10.4:3888
java.net.NoRouteToHostException: No route to host
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:708)
at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:115)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:371)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:404)
at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:688)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:622)
2013-07-29 12:14:13,688 - WARN  [QuorumPeer:/0:0:0:0:0:0:0:0:5181:QuorumCnxManager@384] - Cannot open channel to 1 at election address mr2/10.10.10.6:3888
java.net.NoRouteToHostException: No route to host
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:708)
at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:115)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:371)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:404)
at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:688)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:622)
2013-07-29 12:14:13,689 - INFO  [QuorumPeer:/0:0:0:0:0:0:0:0:5181:FastLeaderElection@697] - Notification time out: 12800
2013-07-29 12:14:26,491 - WARN  [QuorumPeer:/0:0:0:0:0:0:0:0:5181:QuorumCnxManager@384] - Cannot open channel to 2 at election address mr3/10.10.10.4:3888
java.net.NoRouteToHostException: No route to host
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:708)
at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:115)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:371)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:404)
at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:688)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:622)
2013-07-29 12:14:26,492 - WARN  [QuorumPeer:/0:0:0:0:0:0:0:0:5181:QuorumCnxManager@384] - Cannot open channel to 1 at election address mr2/10.10.10.6:3888
java.net.NoRouteToHostException: No route to host
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:708)
at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:115)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:371)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:404)
at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:688)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:622)
2013-07-29 12:14:26,493 - INFO  [QuorumPeer:/0:0:0:0:0:0:0:0:5181:FastLeaderElection@697] - Notification time out: 25600
2013-07-29 12:14:46,138 - INFO  [main:FourLetterWordMain@43] - connecting to localhost 5181
2013-07-29 12:14:46,154 - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:5181:NIOServerCnxn$Factory@251] - Accepted socket connection from /127.0.0.1:42852
2013-07-29 12:14:46,157 - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:5181:NIOServerCnxn@1237] - Processing srvr command from /127.0.0.1:42852
2013-07-29 12:14:46,160 - INFO  [Thread-2:NIOServerCnxn@1435] - Closed socket connection for client /127.0.0.1:42852 (no session established for client)
2013-07-29 12:14:52,095 - WARN  [QuorumPeer:/0:0:0:0:0:0:0:0:5181:QuorumCnxManager@384] - Cannot open channel to 2 at election address mr3/10.10.10.4:3888
java.net.NoRouteToHostException: No route to host
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:708)
at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:115)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:371)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:404)
at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:688)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:622)
2013-07-29 12:14:52,096 - WARN  [QuorumPeer:/0:0:0:0:0:0:0:0:5181:QuorumCnxManager@384] - Cannot open channel to 1 at election address mr2/10.10.10.6:3888
java.net.NoRouteToHostException: No route to host
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:708)
at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:115)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:371)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:404)
at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:688)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:622)
2013-07-29 12:14:52,096 - INFO  [QuorumPeer:/0:0:0:0:0:0:0:0:5181:FastLeaderElection@697] - Notification time out: 51200
2013-07-29 12:15:43,298 - WARN  [QuorumPeer:/0:0:0:0:0:0:0:0:5181:QuorumCnxManager@384] - Cannot open channel to 2 at election address mr3/10.10.10.4:3888
java.net.NoRouteToHostException: No route to host
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:708)
at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:115)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:371)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:404)
at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:688)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:622)
2013-07-29 12:15:43,300 - WARN  [QuorumPeer:/0:0:0:0:0:0:0:0:5181:QuorumCnxManager@384] - Cannot open channel to 1 at election address mr2/10.10.10.6:3888
java.net.NoRouteToHostException: No route to host
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:708)
at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:115)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:371)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:404)
at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:688)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:622)
2013-07-29 12:15:43,300 - INFO  [QuorumPeer:/0:0:0:0:0:0:0:0:5181:FastLeaderElection@697] - Notification time out: 60000
2013-07-29 12:16:43,302 - WARN  [QuorumPeer:/0:0:0:0:0:0:0:0:5181:QuorumCnxManager@384] - Cannot open channel to 2 at election address mr3/10.10.10.4:3888
java.net.NoRouteToHostException: No route to host
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:708)
at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:115)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:371)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:404)
at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:688)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:622)
2013-07-29 12:16:43,304 - WARN  [QuorumPeer:/0:0:0:0:0:0:0:0:5181:QuorumCnxManager@384] - Cannot open channel to 1 at election address mr2/10.10.10.6:3888
java.net.NoRouteToHostException: No route to host
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:708)
at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:115)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:371)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:404)
at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:688)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:622)
2013-07-29 12:16:43,305 - INFO  [QuorumPeer:/0:0:0:0:0:0:0:0:5181:FastLeaderElection@697] - Notification time out: 60000
2013-07-29 12:17:43,306 - WARN  [QuorumPeer:/0:0:0:0:0:0:0:0:5181:QuorumCnxManager@384] - Cannot open channel to 2 at election address mr3/10.10.10.4:3888
java.net.NoRouteToHostException: No route to host
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:708)
at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:115)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:371)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:404)
at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:688)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:622)
2013-07-29 12:17:43,308 - WARN  [QuorumPeer:/0:0:0:0:0:0:0:0:5181:QuorumCnxManager@384] - Cannot open channel to 1 at election address mr2/10.10.10.6:3888
java.net.NoRouteToHostException: No route to host
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:708)
at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:115)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:371)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:404)
at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:688)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:622)
2013-07-29 12:17:43,309 - INFO  [QuorumPeer:/0:0:0:0:0:0:0:0:5181:FastLeaderElection@697] - Notification time out: 60000
2013-07-29 12:18:43,311 - WARN  [QuorumPeer:/0:0:0:0:0:0:0:0:5181:QuorumCnxManager@384] - Cannot open channel to 2 at election address mr3/10.10.10.4:3888
java.net.NoRouteToHostException: No route to host
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:708)
at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:115)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:371)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:404)
at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:688)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:622)
2013-07-29 12:18:43,313 - WARN  [QuorumPeer:/0:0:0:0:0:0:0:0:5181:QuorumCnxManager@384] - Cannot open channel to 1 at election address mr2/10.10.10.6:3888
java.net.NoRouteToHostException: No route to host
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:708)
at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:115)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:371)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:404)
at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:688)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:622)
2013-07-29 12:18:43,313 - INFO  [QuorumPeer:/0:0:0:0:0:0:0:0:5181:FastLeaderElection@697] - Notification time out: 60000
2013-07-29 12:19:36,081 - INFO  [main:FourLetterWordMain@43] - connecting to localhost 5181
2013-07-29 12:19:36,092 - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:5181:NIOServerCnxn$Factory@251] - Accepted socket connection from /127.0.0.1:43187
2013-07-29 12:19:36,092 - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:5181:NIOServerCnxn@1237] - Processing srvr command from /127.0.0.1:43187
2013-07-29 12:19:36,094 - INFO  [Thread-3:NIOServerCnxn@1435] - Closed socket connection for client /127.0.0.1:43187 (no session established for client)
2013-07-29 12:19:43,315 - WARN  [QuorumPeer:/0:0:0:0:0:0:0:0:5181:QuorumCnxManager@384] - Cannot open channel to 2 at election address mr3/10.10.10.4:3888
java.net.NoRouteToHostException: No route to host
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:708)
at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:115)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:371)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:404)
at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:688)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:622)
2013-07-29 12:19:43,317 - WARN  [QuorumPeer:/0:0:0:0:0:0:0:0:5181:QuorumCnxManager@384] - Cannot open channel to 1 at election address mr2/10.10.10.6:3888
java.net.NoRouteToHostException: No route to host
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:708)
at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:115)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:371)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:404)
at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:688)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:622)
2013-07-29 12:19:43,317 - INFO  [QuorumPeer:/0:0:0:0:0:0:0:0:5181:FastLeaderElection@697] - Notification time out: 60000


</pre>

Any ideas what's going wrong?

Thanks,
Mahdi

Outcomes