vgutta

MapR Streams CLI Tutorial

Blog Post created by vgutta on Feb 15, 2017

Intro 

 

In addition to all the Hadoop shell commands, MapR provides a fully complementary toolset that builds on Hadoop to give you a lot more power and insight into the MapR File System. These tools are incredibly useful for administrators who operate the Hadoop cluster, as well as for developers trying to debug Hadoop applications.

 

Maprcli has a large number of options organized into subcommands. By typing maprcli without any options you will get a usage summary showing all the subcommands. Use that to remind yourself of the subcommands and options.

The MapR CLI commands uses the following conventions:

  • [Square brackets] indicate an optional parameter
  • <Angle brackets> indicate a value to enter

These tool is incredibly useful for administrators who operate the Hadoop cluster, as well as for developers trying to debug Hadoop applications.

MapR Streams

The objective of this blog is to get you started with the Administering operations using command line interface for MapR Streams.

This section assumes that you performed the following:

  1. The Cluster is up and running, see MapR CLI 101 Tutorial on how tobring up the cluster

If you have a basic understanding of MapR Streams's components and the typical flow of messages from producers to consumers, you can give MapR Streams a try. Read MapR Streams and Applications.

On a MapR VM 5.2, log in a “mapr” user to a terminal and follow these steps

Security in Streams

The topics that are in a stream are all protected by a single set of security permissions.

You can set these permissions with access-control expressions when you create or edit a stream. For general information about access-control expressions. Different types of permissions are adminperm, copyperm, consumeperm, produceperm, topicperm.

Check MapR documentation for more information on Security 

 

Create a Stream

Run this command if you plan to run the producer and consumer with the same user ID that you are using to create the stream:

maprcli stream create -path /<path to and name of the stream>

maprcli stream create -path /streams/mystreams

/streams is a volume (mount) and mystreams is the name of the stream

if no permissions are specified while creating the stream, produce/consume/topic permissions defaulting to creator. To change, execute 'maprcli stream edit –path’

Edit a Stream

Edit a stream if you plan to run the producer and consumer with user IDs that are different from the user ID that you are using to create the stream:

maprcli stream create -path /<path to and name of the stream> -consumeperm u:<user ID> -produceperm u:<user ID>

Grant producer and consumer permissions to user01

maprcli stream create -path /streams/mystreams -consumeperm u:user01 -produceperm u:user01
Similarly groups also can be granted permissions.

Stream Info

To list the information of stream, run the following command

maprcli stream info -path /streams/mystream

Output as follows:

produceperm  defaultpartitions  topicperm  consumeperm  autocreate  logicalsize  path                copyperm  compression  ttl     numtopics  physicalsize  adminperm 

u:mapr       1                  u:mapr     u:mapr       true        49152        /streams/mystreams  u:mapr    lz4          604800  1          98304         u:mapr    

Create a topic

Run this command to create the topic:

When you create or edit a stream, you can specify the default number of partitions to create for each topic that is subsequently created in the stream. By default only 1 partition will be created.

maprcli stream topic create -path <path and name of the stream> -topic <name of the topic>

maprcli stream topic create -path /streams/mystreams -topic topic1

For further reading: Creating Topics

Edit a topic

Edit a topic to specify number of partitions to 3

maprcli stream topic edit -path /streams/mystreams -topic topic1 -partitions 3

For more information partitions: Setting Partitions for Topics 

Topic Info

To list the info for a topic

maprcli stream topic info  -path /streams/mystreams -topic topic1

Output as follows:

mintimestamp                  partitionid  mintimestampacrossconsumers   logicalsize  minoffsetacrossconsumers  maxtimestamp                  maxoffset  physicalsize 

1970-01-01T12:00:00.000+0000  0            1970-01-01T12:00:00.000+0000  0            0                         1970-01-01T12:00:00.000+0000  0          0            

1970-01-01T12:00:00.000+0000  1            1970-01-01T12:00:00.000+0000  0            0                         1970-01-01T12:00:00.000+0000  0          0            

1970-01-01T12:00:00.000+0000  2            1970-01-01T12:00:00.000+0000  0            0                         1970-01-01T12:00:00.000+0000  0          0 

List all the available topics in a Stream

To List all the available topics in a given stream, run the following command

maprcli stream topic list -path /streams/mystreams

Output as follows:

topic   partitions  logicalsize  consumers  maxlag  physicalsize 

topic2  1           0            0          0       0            

topic1  3           0            0          0       0          

Delete a topic   

To delete a topic, run the following command

maprcli stream topic delete  -path /streams/mystreams -topic topic1   

Delete a Stream

To delete a stream, run the following command

 

maprcli stream delete  -path /streams/mystreams

 

Related Resources

Visit MapR-Streams 

Search for maprcli

Getting Started with the MapR Command Line

Outcomes