In addition to all the Hadoop shell commands, MapR provides a fully complementary toolset that builds on Hadoop to give you a lot more power and insight into the MapR File System. These tools are incredibly useful for administrators who operate the Hadoop cluster, as well as for developers trying to debug Hadoop applications.
Maprcli has a large number of options organized into subcommands. By typing maprcli without any options you will get a usage summary showing all the subcommands. Use that to remind yourself of the subcommands and options.
The MapR CLI commands uses the following conventions:
- [Square brackets] indicate an optional parameter
- <Angle brackets> indicate a value to enter
These tool is incredibly useful for administrators who operate the Hadoop cluster, as well as for developers trying to debug Hadoop applications.
The objective of this blog is to get you started with the Administering operations using command line interface for MapR Streams.
This section assumes that you performed the following:
- The Cluster is up and running, see MapR CLI 101 Tutorial on how tobring up the cluster
If you have a basic understanding of MapR Streams's components and the typical flow of messages from producers to consumers, you can give MapR Streams a try. Read MapR Streams and Applications.
On a MapR VM 5.2, log in a “mapr” user to a terminal and follow these steps
Security in Streams
The topics that are in a stream are all protected by a single set of security permissions.
You can set these permissions with access-control expressions when you create or edit a stream. For general information about access-control expressions. Different types of permissions are adminperm, copyperm, consumeperm, produceperm, topicperm.
Check MapR documentation for more information on Security
Create a Stream
Run this command if you plan to run the producer and consumer with the same user ID that you are using to create the stream:
maprcli stream create -path /<path to and name of the stream>
maprcli stream create -path /streams/mystreams
/streams is a volume (mount) and mystreams is the name of the stream
if no permissions are specified while creating the stream, produce/consume/topic permissions defaulting to creator. To change, execute 'maprcli stream edit –path’
Edit a Stream
Edit a stream if you plan to run the producer and consumer with user IDs that are different from the user ID that you are using to create the stream:
maprcli stream create -path /<path to and name of the stream> -consumeperm u:<user ID> -produceperm u:<user ID>
Grant producer and consumer permissions to user01
maprcli stream create -path /streams/mystreams -consumeperm u:user01 -produceperm u:user01
To list the information of stream, run the following command
maprcli stream info -path /streams/mystream
Output as follows:
produceperm defaultpartitions topicperm consumeperm autocreate logicalsize path copyperm compression ttl numtopics physicalsize adminperm
u:mapr 1 u:mapr u:mapr true 49152 /streams/mystreams u:mapr lz4 604800 1 98304 u:mapr
Create a topic
Run this command to create the topic:
When you create or edit a stream, you can specify the default number of partitions to create for each topic that is subsequently created in the stream. By default only 1 partition will be created.
maprcli stream topic create -path <path and name of the stream> -topic <name of the topic>
maprcli stream topic create -path /streams/mystreams -topic topic1
For further reading: Creating Topics
Edit a topic
Edit a topic to specify number of partitions to 3
maprcli stream topic edit -path /streams/mystreams -topic topic1 -partitions 3
For more information partitions: Setting Partitions for Topics
To list the info for a topic
maprcli stream topic info -path /streams/mystreams -topic topic1
Output as follows:
mintimestamp partitionid mintimestampacrossconsumers logicalsize minoffsetacrossconsumers maxtimestamp maxoffset physicalsize
1970-01-01T12:00:00.000+0000 0 1970-01-01T12:00:00.000+0000 0 0 1970-01-01T12:00:00.000+0000 0 0
1970-01-01T12:00:00.000+0000 1 1970-01-01T12:00:00.000+0000 0 0 1970-01-01T12:00:00.000+0000 0 0
1970-01-01T12:00:00.000+0000 2 1970-01-01T12:00:00.000+0000 0 0 1970-01-01T12:00:00.000+0000 0 0
List all the available topics in a Stream
To List all the available topics in a given stream, run the following command
maprcli stream topic list -path /streams/mystreams
Output as follows:
topic partitions logicalsize consumers maxlag physicalsize
topic2 1 0 0 0 0
topic1 3 0 0 0 0
Delete a topic
To delete a topic, run the following command
maprcli stream topic delete -path /streams/mystreams -topic topic1
Delete a Stream
To delete a stream, run the following command
maprcli stream delete -path /streams/mystreams
Search for maprcli