How can we manage configurations or make configurations consistent across cluster in Mapr for deployments ?
Here are some instructions for setting up Central Configuration. This allows you to create customized configuration files for services, and it will then copy the custom configuration files to all the nodes in the cluster. So you can make changes in once place, that will be propagated to the other nodes.
Sorry I should have been more specific, is there any possibility of changing configurations from MCS and deploy cluster wide ? Just like other hadoop distributions do ?
When you say "configurations," are you talking about configuration files (such as yarn-site.xml or warden.conf), or something else?
I was referring to all the primary configurations for ecosystem components like hive-site.xml, oozie-site.xml etc., and core components warden.conf as well.
1) Once new server or service is added , we want to deploy all the existing configurations with right permissions(close to Deploy configurations in cloudera) through MCS.
2) we want to change the configurations in MCS and deploy to all nodes in cluster via MCS .
Does that make sense ?
Makes perfect sense. We do not currently have a way to do that through the MCS - the Centralized Configuration is what you want. It is fairly easy to set up, even though it's not GUI-driven.
Thanks for the quick response, Is there any scope in near future to do so?
Not in the near future. Beyond that, I can't say...
I'm really sorry for Deborah, but we don't recommend using Central Configuration at all these days with our customers where I am located. We tend to think about CC as a legacy technology.
We find it's far easier to use a tool like Cluster Shell to quickly and efficiently manage the configurations of your cluster. From what I can follow in the thread, this tool will be a perfect fit for your use case, except for the fact it's a command line tool, not an MCS tool.
As part of the pre-install routine, we always install Cluster Shell so that we can copy files around the cluster and run commands on multiple nodes from a single node. it's super easy and painless.
To manage configuration, just make any change you want, then "clush -ac /opt/mapr/conf/some_file" and that file is now copied across the cluster to all nodes.
Run configure.sh? clush -a /opt/mapr/bin/configure.sh -R
restart warden on all nodes? clush -a service mapr-warden restart
and so on. it's super easy.
Another important point, the Cluster Validation tools, which we strongly recommend as part of the pre and post installation process in every production cluster, also requires cluster shell.
Mathieu Dumoulin and Deborah Littlefield for your responses. But with cluster shell there are certain drawbacks like
1) How can we ensure all the configurations are copied ?
2) If we have different configurations for different nodes , how can we ensure that right configuration is copied ?
Currently above drawbacks are addressed via cluster shell groups . But i think there is a room for improvement there. I think use of GUI for deploying configurations might be a better solution.
Starting with your comment: is there room for improvement? I agree with you 100%
Please add your comments to the Product Ideas section (https://community.mapr.com/community/answers/pages/ideas ). It's a great way to tell MapR you think this is something we should improve.
1) by using cluster shell's copy feature: clush -ac /path/to/my/file OR clush -g worker -c /path/to/my/file
this will copy to all nodes in the first case, or only to nodes with the worker label (defined in cluster shells configuration by you).
2) by using labels to define group of nodes in cluster shell's configuration. then you can use the -g <label> to copy only to the relevant nodes.
in my work, I use the following groups:
cldb: node[01-03]zk: node[01-03]web: node[04-05]rm: node[04-05]hs: node04mfs: node[01-10]nfs: node[01-10]nm: node[01-10]all: node[01-10]
web = MCS server nodes, rm resource manager, hs history server, nm node manager, etc.
Thanks Mathieu for the response, we use similar approach 1,2 currently in our Mapr Environments. I will provide my comments in Product Ideas section (https://community.mapr.com/community/answers/pages/ideas ).
Great discussion and feedback everyone! Will close this discussion unless bgajjela have more questions.
Please go ahead and close this discussion. Thanks every one for your valuable feedback
Retrieving data ...