AnsweredAssumed Answered

CLDB shutdown when add newing second node to cluster

Question asked by bencetamas on Jan 30, 2015
Latest reply on Sep 14, 2015 by sumit_gupta
Hello,

I'm trying to set a 2-node demo cluster in docker containers. For the first node I've installed resourcemanager, nodemanager, cldb, fileserver, zookeeper, etc. When I start the warden on it, everything is fine. But when I start warden on the other node (with nodemanager and fileserver), CLDB shuts down.

Looking at **cldb.log** at the first node, I suspect, that something is wrong with the topology, because node 1 (mapr1.example.com) and node2 (mapr2.example.com) both want to register to- /default-rack/mapr1.example.com. Here is a part of **cldb.log:**

<pre><code>  

    2015-01-28 13:24:15,145 WARN Topology [RPC-1]: FileSever on mapr1.exam
    ple.com reported an invalid topology . Ignoring reported topology
    2015-01-28 13:24:15,204 INFO CLDBServer [RPC-1]: FSRegister: Registere
    d FileServer: 172.17.0.5- at topology /default-rack/mapr1.example.com
    2015-01-28 13:24:15,232 INFO log [main]: Started SelectChannelConnecto
    r@0.0.0.0:7221
    2015-01-28 13:24:15,948 INFO CLDBServer [Lookup-1]: Rejecting RPC 2345
    .4 from 172.17.0.5:36859 with status 3 as CLDB is waiting for local kv
    store to become master.
    2015-01-28 13:24:16,791 INFO CLDBServer [RPC-3]: FSRegister: Request 
    FSID: 818516103053193757 FSNetworkLocation:  FSHost:Port: 172.17.0.6-
    FSHostName: mapr2.example.com StoragePools f9cfaa95199c44620054c925c10
    c3897- Capacity: 0 Available: 0 Used: 0 Role: 0 isDCA: false Received
    registration request
    2015-01-28 13:24:16,792 INFO CLDBServer [RPC-3]: FSRegister: Re-regist
    ered FileServer 172.17.0.6- at topology /default-rack/mapr1.example.co
    m
    2015-01-28 13:24:17,050 INFO ZooKeeperClient [RPC-4]: Storing KvStoreC
    ontainerInfo to ZooKeeper  Container ID:1 Master:172.17.0.6-3-BM(81851
    6103053193757) Servers:  172.17.0.6-3-BM(818516103053193757) Inactive:
      Unused:  Epoch:3 SizeMB:0
    2015-01-28 13:25:19,704 INFO CLDBServer [Lookup-3]: Rejecting RPC 2345
    .4 from 172.17.0.6:1111 with status 3 as CLDB is waiting for local kvs
    tore to become master.
    2015-01-28 13:25:23,966 INFO Topology [FCR-1]: New storage pool 700ded
    8af70c55ba0054c9230f01f67e has been added to FileServer 81851610305319
    3757 with IPaddress 172.17.0.6
    2015-01-28 13:25:24,163 INFO ZooKeeperClient [RPC-2]: Storing KvStoreC
    ontainerInfo to ZooKeeper  Container ID:1 Master:172.17.0.6-3(81851610
    3053193757) Servers:  172.17.0.6-3(818516103053193757) Inactive:  Unus
    ed:  Epoch:3 SizeMB:0
    2015-01-28 13:25:24,309 INFO CLDBServer [RPC-2]: CLDB Mode: MASTER_REG
    ISTER_READY CLDB HA Check
    2015-01-28 13:25:24,316 INFO LicenseManager [RPC-2]: using default lic
    ense validator
    2015-01-28 13:26:20,387 INFO CLDBServer [Lookup-8]: Rejecting RPC 2345
    .4 from 172.17.0.5:1111 with status 3 as CLDB is waiting for local kvs
    tore to become master.


</code></pre>

When setting topology info in conf/cldb.conf or conf/mfs.conf, nothings happen, the nodes register to the same /default-rack/mapr1.example.com.
 (I've set net.topology.table.file.name=/tmp/topo.txt in cldb.conf and the corresponding text file, and mfs.network.location=/test-rack-mfs/test-mfs-2/test3/ in mfs.conf)

What I'm doing wrong? How can I se topology informations? The problem is likely with the topology or with something else?

Or maybe there is a licensing issue? I didn't installed any license on the cluster.

Thank you for answers!
Bence Tamas

Outcomes