rupal

How to install StreamSets on MapR 6.0 Sandbox

Blog Post created by rupal on Apr 19, 2018

Pre-requisites

  1. Install MapR 6.0 Sandbox: https://mapr.com/products/mapr-sandbox-hadoop/download/
  2. Ensure you have enough space on the Sandbox to install StreamSets Data Collector and StreamSets Data Collector Edge. Keep at least 5GB of space available. To check how much space is available and/or to add more space, follow this guide: https://community.mapr.com/docs/DOC-1608-how-to-extend-the-mapr-sandbox-vms-storage-space

 

Install StreamSets Data Collector

  1. SSH into the Sandbox and login as root

 

$ ssh mapr@localhost -p 2222

Password:

Last login: Wed Jan 31 21:30:50 2018

Welcome to your Mapr Demo virtual machine.

[mapr@maprdemo ~]$ su -

Password:

Last login: Wed Jan 31 21:30:54 PST 2018 on pts/0

[root@maprdemo ~]#

 

  1. Download the RPM and extract the binaries

Get the latest version to install from https://archives.streamsets.com/index.html

Note: We’ll be using StreamSets Data Collector version 3.0.3.0.

 

 

URL: http://archives.streamsets.com/datacollector/3.0.3.0/rpm/el7/streamsets-datacollector-3.0.3.0-el7-all-rpms.tar

 

[root@maprdemo ~]# wget http://archives.streamsets.com/datacollector/3.0.3.0/rpm/el7/streamsets-datacollector-3.0.3.0-el7-all-rpms.tar

 

Note: If the download link does not work, use the fully qualified download link: https://s3-us-west-2.amazonaws.com/archives.streamsets.com/datacollector/3.0.3.0/rpm/el7/streamsets-datacollector-3.0.3.0-el7-all-rpms.tar

 

--2018-02-01 05:37:42--  http://archives.streamsets.com/datacollector/3.0.3.0/rpm/el7/streamsets-datacollector-3.0.3.0-el7-all-rpms.tar

Resolving archives.streamsets.com (archives.streamsets.com)... 151.101.48.69

Connecting to archives.streamsets.com (archives.streamsets.com)|151.101.48.69|:80... connected.

HTTP request sent, awaiting response... 200 OK

Length: 3914629120 (3.6G) [application/x-tar]

Saving to: ‘streamsets-datacollector-3.0.3.0-el7-all-rpms.tar’

 

[root@maprdemo ~]# tar -xf streamsets-datacollector-3.0.3.0-el7-all-rpms.tar

[root@maprdemo ~]# ls

anaconda-ks.cfg  config.sandbox original-ks.cfg  streamsets-datacollector-3.0.3.0-el7-all-rpms  streamsets-datacollector-3.0.3.0-el7-all-rpms.tar

[root@maprdemo ~]#

 

  1. Remove unrequired stage libraries

StreamSets installs each package as a stage library. You can choose to do a full install with all the stage libraries or selectively install only what’s required. The full install will take ~3.5GB of space. We do not need to do a full install because half the stage libraries will not be required for MapR. Remove these unwanted stage libraries as follows:

 

[root@maprdemo ~]# cd streamsets-datacollector-3.0.3.0-el7-all-rpms/

[root@maprdemo streamsets-datacollector-3.0.3.0-el7-all-rpms]# rm -rf streamsets-datacollector-cdh* && rm -rf streamsets-datacollector-hdp* && rm -rf streamsets-datacollector-apache-kudu* && rm -rf streamsets-datacollector-mapr_5*

 

  1. Install

[root@maprdemo streamsets-datacollector-3.0.3.0-el7-all-rpms]# pwd

/root/streamsets-datacollector-3.0.3.0-el7-all-rpms

[root@maprdemo streamsets-datacollector-3.0.3.0-el7-all-rpms]# yum localinstall streamsets*.rpm

Loaded plugins: fastestmirror, langpacks

Examining streamsets-datacollector-3.0.3.0-1.noarch.rpm: streamsets-datacollector-3.0.3.0-1.noarch

Marking streamsets-datacollector-3.0.3.0-1.noarch.rpm to be installed

Examining streamsets-datacollector-apache-kafka_0_10-lib-3.0.3.0-1.noarch.rpm: streamsets-datacollector-apache-kafka_0_10-lib-3.0.3.0-1.noarch

Marking streamsets-datacollector-apache-kafka_0_10-lib-3.0.3.0-1.noarch.rpm to be installed

Examining streamsets-datacollector-apache-kafka_0_11-lib-3.0.3.0-1.noarch.rpm: streamsets-datacollector-apache-kafka_0_11-lib-3.0.3.0-1.noarch

Marking streamsets-datacollector-apache-kafka_0_11-lib-3.0.3.0-1.noarch.rpm to be installed

Examining streamsets-datacollector-apache-kafka_0_9-lib-3.0.3.0-1.noarch.rpm: streamsets-datacollector-apache-kafka_0_9-lib-3.0.3.0-1.noarch

Marking streamsets-datacollector-apache-kafka_0_9-lib-3.0.3.0-1.noarch.rpm to be installed

Examining streamsets-datacollector-apache-kafka_1_0-lib-3.0.3.0-1.noarch.rpm: streamsets-datacollector-apache-kafka_1_0-lib-3.0.3.0-1.noarch

Marking streamsets-datacollector-apache-kafka_1_0-lib-3.0.3.0-1.noarch.rpm to be installed

Examining streamsets-datacollector-apache-solr_6_1_0-lib-3.0.3.0-1.noarch.rpm: streamsets-datacollector-apache-solr_6_1_0-lib-3.0.3.0-1.noarch

Marking streamsets-datacollector-apache-solr_6_1_0-lib-3.0.3.0-1.noarch.rpm to be installed

Examining streamsets-datacollector-aws-lib-3.0.3.0-1.noarch.rpm: streamsets-datacollector-aws-lib-3.0.3.0-1.noarch

Marking streamsets-datacollector-aws-lib-3.0.3.0-1.noarch.rpm to be installed

Examining streamsets-datacollector-azure-lib-3.0.3.0-1.noarch.rpm: streamsets-datacollector-azure-lib-3.0.3.0-1.noarch

Marking streamsets-datacollector-azure-lib-3.0.3.0-1.noarch.rpm to be installed

Examining streamsets-datacollector-basic-lib-3.0.3.0-1.noarch.rpm: streamsets-datacollector-basic-lib-3.0.3.0-1.noarch

Marking streamsets-datacollector-basic-lib-3.0.3.0-1.noarch.rpm to be installed

Examining streamsets-datacollector-bigtable-lib-3.0.3.0-1.noarch.rpm: streamsets-datacollector-bigtable-lib-3.0.3.0-1.noarch

Marking streamsets-datacollector-bigtable-lib-3.0.3.0-1.noarch.rpm to be installed

Examining streamsets-datacollector-cassandra_3-lib-3.0.3.0-1.noarch.rpm: streamsets-datacollector-cassandra_3-lib-3.0.3.0-1.noarch

Marking streamsets-datacollector-cassandra_3-lib-3.0.3.0-1.noarch.rpm to be installed

Examining streamsets-datacollector-cyberark-credentialstore-lib-3.0.3.0-1.noarch.rpm: streamsets-datacollector-cyberark-credentialstore-lib-3.0.3.0-1.noarch

Marking streamsets-datacollector-cyberark-credentialstore-lib-3.0.3.0-1.noarch.rpm to be installed

Examining streamsets-datacollector-dev-lib-3.0.3.0-1.noarch.rpm: streamsets-datacollector-dev-lib-3.0.3.0-1.noarch

Marking streamsets-datacollector-dev-lib-3.0.3.0-1.noarch.rpm to be installed

Examining streamsets-datacollector-elasticsearch_5-lib-3.0.3.0-1.noarch.rpm: streamsets-datacollector-elasticsearch_5-lib-3.0.3.0-1.noarch

Marking streamsets-datacollector-elasticsearch_5-lib-3.0.3.0-1.noarch.rpm to be installed

Examining streamsets-datacollector-google-cloud-lib-3.0.3.0-1.noarch.rpm: streamsets-datacollector-google-cloud-lib-3.0.3.0-1.noarch

Marking streamsets-datacollector-google-cloud-lib-3.0.3.0-1.noarch.rpm to be installed

Examining streamsets-datacollector-groovy_2_4-lib-3.0.3.0-1.noarch.rpm: streamsets-datacollector-groovy_2_4-lib-3.0.3.0-1.noarch

Marking streamsets-datacollector-groovy_2_4-lib-3.0.3.0-1.noarch.rpm to be installed

Examining streamsets-datacollector-influxdb_0_9-lib-3.0.3.0-1.noarch.rpm: streamsets-datacollector-influxdb_0_9-lib-3.0.3.0-1.noarch

Marking streamsets-datacollector-influxdb_0_9-lib-3.0.3.0-1.noarch.rpm to be installed

Examining streamsets-datacollector-jdbc-lib-3.0.3.0-1.noarch.rpm: streamsets-datacollector-jdbc-lib-3.0.3.0-1.noarch

Marking streamsets-datacollector-jdbc-lib-3.0.3.0-1.noarch.rpm to be installed

Examining streamsets-datacollector-jks-credentialstore-lib-3.0.3.0-1.noarch.rpm: streamsets-datacollector-jks-credentialstore-lib-3.0.3.0-1.noarch

Marking streamsets-datacollector-jks-credentialstore-lib-3.0.3.0-1.noarch.rpm to be installed

Examining streamsets-datacollector-jms-lib-3.0.3.0-1.noarch.rpm: streamsets-datacollector-jms-lib-3.0.3.0-1.noarch

Marking streamsets-datacollector-jms-lib-3.0.3.0-1.noarch.rpm to be installed

Examining streamsets-datacollector-jython_2_7-lib-3.0.3.0-1.noarch.rpm: streamsets-datacollector-jython_2_7-lib-3.0.3.0-1.noarch

Marking streamsets-datacollector-jython_2_7-lib-3.0.3.0-1.noarch.rpm to be installed

Examining streamsets-datacollector-kinetica_6_0-lib-3.0.3.0-1.noarch.rpm: streamsets-datacollector-kinetica_6_0-lib-3.0.3.0-1.noarch

Marking streamsets-datacollector-kinetica_6_0-lib-3.0.3.0-1.noarch.rpm to be installed

Examining streamsets-datacollector-mapr_6_0-lib-3.0.3.0-1.noarch.rpm: streamsets-datacollector-mapr_6_0-lib-3.0.3.0-1.noarch

Marking streamsets-datacollector-mapr_6_0-lib-3.0.3.0-1.noarch.rpm to be installed

Examining streamsets-datacollector-mapr_6_0-mep4-lib-3.0.3.0-1.noarch.rpm: streamsets-datacollector-mapr_6_0-mep4-lib-3.0.3.0-1.noarch

Marking streamsets-datacollector-mapr_6_0-mep4-lib-3.0.3.0-1.noarch.rpm to be installed

Examining streamsets-datacollector-mapr_spark_2_1_mep_3_0-lib-3.0.3.0-1.noarch.rpm: streamsets-datacollector-mapr_spark_2_1_mep_3_0-lib-3.0.3.0-1.noarch

Marking streamsets-datacollector-mapr_spark_2_1_mep_3_0-lib-3.0.3.0-1.noarch.rpm to be installed

Examining streamsets-datacollector-mongodb_3-lib-3.0.3.0-1.noarch.rpm: streamsets-datacollector-mongodb_3-lib-3.0.3.0-1.noarch

Marking streamsets-datacollector-mongodb_3-lib-3.0.3.0-1.noarch.rpm to be installed

Examining streamsets-datacollector-mysql-binlog-lib-3.0.3.0-1.noarch.rpm: streamsets-datacollector-mysql-binlog-lib-3.0.3.0-1.noarch

Marking streamsets-datacollector-mysql-binlog-lib-3.0.3.0-1.noarch.rpm to be installed

Examining streamsets-datacollector-omniture-lib-3.0.3.0-1.noarch.rpm: streamsets-datacollector-omniture-lib-3.0.3.0-1.noarch

Marking streamsets-datacollector-omniture-lib-3.0.3.0-1.noarch.rpm to be installed

Examining streamsets-datacollector-rabbitmq-lib-3.0.3.0-1.noarch.rpm: streamsets-datacollector-rabbitmq-lib-3.0.3.0-1.noarch

Marking streamsets-datacollector-rabbitmq-lib-3.0.3.0-1.noarch.rpm to be installed

Examining streamsets-datacollector-redis-lib-3.0.3.0-1.noarch.rpm: streamsets-datacollector-redis-lib-3.0.3.0-1.noarch

Marking streamsets-datacollector-redis-lib-3.0.3.0-1.noarch.rpm to be installed

Examining streamsets-datacollector-salesforce-lib-3.0.3.0-1.noarch.rpm: streamsets-datacollector-salesforce-lib-3.0.3.0-1.noarch

Marking streamsets-datacollector-salesforce-lib-3.0.3.0-1.noarch.rpm to be installed

Examining streamsets-datacollector-stats-lib-3.0.3.0-1.noarch.rpm: streamsets-datacollector-stats-lib-3.0.3.0-1.noarch

Marking streamsets-datacollector-stats-lib-3.0.3.0-1.noarch.rpm to be installed

Examining streamsets-datacollector-vault-credentialstore-lib-3.0.3.0-1.noarch.rpm: streamsets-datacollector-vault-credentialstore-lib-3.0.3.0-1.noarch

Marking streamsets-datacollector-vault-credentialstore-lib-3.0.3.0-1.noarch.rpm to be installed

Examining streamsets-datacollector-windows-lib-3.0.3.0-1.noarch.rpm: streamsets-datacollector-windows-lib-3.0.3.0-1.noarch

Marking streamsets-datacollector-windows-lib-3.0.3.0-1.noarch.rpm to be installed

Resolving Dependencies

--> Running transaction check

---> Package streamsets-datacollector.noarch 0:3.0.3.0-1 will be installed

---> Package streamsets-datacollector-apache-kafka_0_10-lib.noarch 0:3.0.3.0-1 will be installed

---> Package streamsets-datacollector-apache-kafka_0_11-lib.noarch 0:3.0.3.0-1 will be installed

---> Package streamsets-datacollector-apache-kafka_0_9-lib.noarch 0:3.0.3.0-1 will be installed

---> Package streamsets-datacollector-apache-kafka_1_0-lib.noarch 0:3.0.3.0-1 will be installed

---> Package streamsets-datacollector-apache-solr_6_1_0-lib.noarch 0:3.0.3.0-1 will be installed

---> Package streamsets-datacollector-aws-lib.noarch 0:3.0.3.0-1 will be installed

---> Package streamsets-datacollector-azure-lib.noarch 0:3.0.3.0-1 will be installed

---> Package streamsets-datacollector-basic-lib.noarch 0:3.0.3.0-1 will be installed

---> Package streamsets-datacollector-bigtable-lib.noarch 0:3.0.3.0-1 will be installed

---> Package streamsets-datacollector-cassandra_3-lib.noarch 0:3.0.3.0-1 will be installed

---> Package streamsets-datacollector-cyberark-credentialstore-lib.noarch 0:3.0.3.0-1 will be installed

---> Package streamsets-datacollector-dev-lib.noarch 0:3.0.3.0-1 will be installed

---> Package streamsets-datacollector-elasticsearch_5-lib.noarch 0:3.0.3.0-1 will be installed

---> Package streamsets-datacollector-google-cloud-lib.noarch 0:3.0.3.0-1 will be installed

---> Package streamsets-datacollector-groovy_2_4-lib.noarch 0:3.0.3.0-1 will be installed

---> Package streamsets-datacollector-influxdb_0_9-lib.noarch 0:3.0.3.0-1 will be installed

---> Package streamsets-datacollector-jdbc-lib.noarch 0:3.0.3.0-1 will be installed

---> Package streamsets-datacollector-jks-credentialstore-lib.noarch 0:3.0.3.0-1 will be installed

---> Package streamsets-datacollector-jms-lib.noarch 0:3.0.3.0-1 will be installed

---> Package streamsets-datacollector-jython_2_7-lib.noarch 0:3.0.3.0-1 will be installed

---> Package streamsets-datacollector-kinetica_6_0-lib.noarch 0:3.0.3.0-1 will be installed

---> Package streamsets-datacollector-mapr_6_0-lib.noarch 0:3.0.3.0-1 will be installed

---> Package streamsets-datacollector-mapr_6_0-mep4-lib.noarch 0:3.0.3.0-1 will be installed

---> Package streamsets-datacollector-mapr_spark_2_1_mep_3_0-lib.noarch 0:3.0.3.0-1 will be installed

---> Package streamsets-datacollector-mongodb_3-lib.noarch 0:3.0.3.0-1 will be installed

---> Package streamsets-datacollector-mysql-binlog-lib.noarch 0:3.0.3.0-1 will be installed

---> Package streamsets-datacollector-omniture-lib.noarch 0:3.0.3.0-1 will be installed

---> Package streamsets-datacollector-rabbitmq-lib.noarch 0:3.0.3.0-1 will be installed

---> Package streamsets-datacollector-redis-lib.noarch 0:3.0.3.0-1 will be installed

---> Package streamsets-datacollector-salesforce-lib.noarch 0:3.0.3.0-1 will be installed

---> Package streamsets-datacollector-stats-lib.noarch 0:3.0.3.0-1 will be installed

---> Package streamsets-datacollector-vault-credentialstore-lib.noarch 0:3.0.3.0-1 will be installed

---> Package streamsets-datacollector-windows-lib.noarch 0:3.0.3.0-1 will be installed

--> Finished Dependency Resolution

MapR_Core                                                                                                                                                                            | 1.4 kB 00:00:00

MapR_Core/primary                                                                                                                                                                    | 4.7 kB 00:00:00

MapR_Ecosystem                                                                                                                                                                       | 1.4 kB 00:00:00

MapR_Ecosystem/primary                                                                                                                                                               | 14 kB 00:00:00

base/7/x86_64                                                                                                                                                                        | 3.6 kB 00:00:00

base/7/x86_64/group_gz                                                                                                                                                               | 156 kB 00:00:00

base/7/x86_64/primary_db                                                                                                                                                             | 5.7 MB 00:00:02

epel/x86_64/metalink                                                                                                                                                                 | 13 kB 00:00:00

epel/x86_64                                                                                                                                                                          | 4.7 kB 00:00:00

epel/x86_64/group_gz                                                                                                                                                                 | 266 kB 00:00:00

epel/x86_64/updateinfo                                                                                                                                                               | 880 kB 00:00:00

epel/x86_64/primary_db                                                                                                                                                               | 6.2 MB 00:00:01

extras/7/x86_64                                                                                                                                                                      | 3.4 kB 00:00:00

extras/7/x86_64/primary_db                                                                                                                                                           | 166 kB 00:00:00

updates/7/x86_64                                                                                                                                                                     | 3.4 kB 00:00:00

updates/7/x86_64/primary_db                                                                                                                                                          | 6.0 MB 00:00:01

 

Dependencies Resolved

 

============================================================================================================================================================================================================

Package                                                            Arch Version Repository                                                               Size

============================================================================================================================================================================================================

Installing:

streamsets-datacollector                                           noarch 3.0.3.0-1 /streamsets-datacollector-3.0.3.0-1.noarch                                           162 M

streamsets-datacollector-apache-kafka_0_10-lib                     noarch 3.0.3.0-1 /streamsets-datacollector-apache-kafka_0_10-lib-3.0.3.0-1.noarch                      38 M

streamsets-datacollector-apache-kafka_0_11-lib                     noarch 3.0.3.0-1 /streamsets-datacollector-apache-kafka_0_11-lib-3.0.3.0-1.noarch                      40 M

streamsets-datacollector-apache-kafka_0_9-lib                      noarch 3.0.3.0-1 /streamsets-datacollector-apache-kafka_0_9-lib-3.0.3.0-1.noarch                       38 M

streamsets-datacollector-apache-kafka_1_0-lib                      noarch 3.0.3.0-1 /streamsets-datacollector-apache-kafka_1_0-lib-3.0.3.0-1.noarch                       40 M

streamsets-datacollector-apache-solr_6_1_0-lib                     noarch 3.0.3.0-1 /streamsets-datacollector-apache-solr_6_1_0-lib-3.0.3.0-1.noarch                      17 M

streamsets-datacollector-aws-lib                                   noarch 3.0.3.0-1 /streamsets-datacollector-aws-lib-3.0.3.0-1.noarch                                    46 M

streamsets-datacollector-azure-lib                                 noarch 3.0.3.0-1 /streamsets-datacollector-azure-lib-3.0.3.0-1.noarch                                  18 M

streamsets-datacollector-basic-lib                                 noarch 3.0.3.0-1 /streamsets-datacollector-basic-lib-3.0.3.0-1.noarch                                  36 M

streamsets-datacollector-bigtable-lib                              noarch 3.0.3.0-1 /streamsets-datacollector-bigtable-lib-3.0.3.0-1.noarch                               55 M

streamsets-datacollector-cassandra_3-lib                           noarch 3.0.3.0-1 /streamsets-datacollector-cassandra_3-lib-3.0.3.0-1.noarch                            17 M

streamsets-datacollector-cyberark-credentialstore-lib              noarch 3.0.3.0-1 /streamsets-datacollector-cyberark-credentialstore-lib-3.0.3.0-1.noarch              5.2 M

streamsets-datacollector-dev-lib                                   noarch 3.0.3.0-1 /streamsets-datacollector-dev-lib-3.0.3.0-1.noarch                                    14 M

streamsets-datacollector-elasticsearch_5-lib                       noarch 3.0.3.0-1 /streamsets-datacollector-elasticsearch_5-lib-3.0.3.0-1.noarch                        18 M

streamsets-datacollector-google-cloud-lib                          noarch 3.0.3.0-1 /streamsets-datacollector-google-cloud-lib-3.0.3.0-1.noarch                           28 M

streamsets-datacollector-groovy_2_4-lib                            noarch 3.0.3.0-1 /streamsets-datacollector-groovy_2_4-lib-3.0.3.0-1.noarch                             19 M

streamsets-datacollector-influxdb_0_9-lib                          noarch 3.0.3.0-1 /streamsets-datacollector-influxdb_0_9-lib-3.0.3.0-1.noarch                           14 M

streamsets-datacollector-jdbc-lib                                  noarch 3.0.3.0-1 /streamsets-datacollector-jdbc-lib-3.0.3.0-1.noarch                                   27 M

streamsets-datacollector-jks-credentialstore-lib                   noarch 3.0.3.0-1 /streamsets-datacollector-jks-credentialstore-lib-3.0.3.0-1.noarch                   2.6 M

streamsets-datacollector-jms-lib                                   noarch 3.0.3.0-1 /streamsets-datacollector-jms-lib-3.0.3.0-1.noarch                                    17 M

streamsets-datacollector-jython_2_7-lib                            noarch 3.0.3.0-1 /streamsets-datacollector-jython_2_7-lib-3.0.3.0-1.noarch                             53 M

streamsets-datacollector-kinetica_6_0-lib                          noarch 3.0.3.0-1 /streamsets-datacollector-kinetica_6_0-lib-3.0.3.0-1.noarch                           32 M

streamsets-datacollector-mapr_6_0-lib                              noarch 3.0.3.0-1 /streamsets-datacollector-mapr_6_0-lib-3.0.3.0-1.noarch                               43 M

streamsets-datacollector-mapr_6_0-mep4-lib                         noarch 3.0.3.0-1 /streamsets-datacollector-mapr_6_0-mep4-lib-3.0.3.0-1.noarch                          94 M

streamsets-datacollector-mapr_spark_2_1_mep_3_0-lib                noarch 3.0.3.0-1 /streamsets-datacollector-mapr_spark_2_1_mep_3_0-lib-3.0.3.0-1.noarch                152 M

streamsets-datacollector-mongodb_3-lib                             noarch 3.0.3.0-1 /streamsets-datacollector-mongodb_3-lib-3.0.3.0-1.noarch                              16 M

streamsets-datacollector-mysql-binlog-lib                          noarch 3.0.3.0-1 /streamsets-datacollector-mysql-binlog-lib-3.0.3.0-1.noarch                           16 M

streamsets-datacollector-omniture-lib                              noarch 3.0.3.0-1 /streamsets-datacollector-omniture-lib-3.0.3.0-1.noarch                               15 M

streamsets-datacollector-rabbitmq-lib                              noarch 3.0.3.0-1 /streamsets-datacollector-rabbitmq-lib-3.0.3.0-1.noarch                               16 M

streamsets-datacollector-redis-lib                                 noarch 3.0.3.0-1 /streamsets-datacollector-redis-lib-3.0.3.0-1.noarch                                  14 M

streamsets-datacollector-salesforce-lib                            noarch 3.0.3.0-1 /streamsets-datacollector-salesforce-lib-3.0.3.0-1.noarch                             20 M

streamsets-datacollector-stats-lib                                 noarch 3.0.3.0-1 /streamsets-datacollector-stats-lib-3.0.3.0-1.noarch                                  32 M

streamsets-datacollector-vault-credentialstore-lib                 noarch 3.0.3.0-1 /streamsets-datacollector-vault-credentialstore-lib-3.0.3.0-1.noarch                 3.8 M

streamsets-datacollector-windows-lib                               noarch 3.0.3.0-1 /streamsets-datacollector-windows-lib-3.0.3.0-1.noarch                                14 M

 

Transaction Summary

============================================================================================================================================================================================================

Install  34 Packages

 

Total size: 1.1 G

Installed size: 1.1 G

Is this ok [y/d/N]: y

Downloading packages:

Running transaction check

Running transaction test

Transaction test succeeded

Running transaction

 Installing : streamsets-datacollector-3.0.3.0-1.noarch                                                                                                                                               1/34

 Installing : streamsets-datacollector-salesforce-lib-3.0.3.0-1.noarch                                                                                                                                2/34

 Installing : streamsets-datacollector-groovy_2_4-lib-3.0.3.0-1.noarch                                                                                                                                3/34

 Installing : streamsets-datacollector-cyberark-credentialstore-lib-3.0.3.0-1.noarch                                                                                                                  4/34

 Installing : streamsets-datacollector-aws-lib-3.0.3.0-1.noarch                                                                                                                                       5/34

 Installing : streamsets-datacollector-cassandra_3-lib-3.0.3.0-1.noarch                                                                                                                               6/34

 Installing : streamsets-datacollector-rabbitmq-lib-3.0.3.0-1.noarch                                                                                                                                  7/34

 Installing : streamsets-datacollector-mapr_spark_2_1_mep_3_0-lib-3.0.3.0-1.noarch                                                                                                                    8/34

 Installing : streamsets-datacollector-jdbc-lib-3.0.3.0-1.noarch                                                                                                                                      9/34

 Installing : streamsets-datacollector-apache-kafka_1_0-lib-3.0.3.0-1.noarch                                                                                                                         10/34

 Installing : streamsets-datacollector-dev-lib-3.0.3.0-1.noarch                                                                                                                                      11/34

 Installing : streamsets-datacollector-omniture-lib-3.0.3.0-1.noarch                                                                                                                                 12/34

 Installing : streamsets-datacollector-mongodb_3-lib-3.0.3.0-1.noarch                                                                                                                                13/34

 Installing : streamsets-datacollector-redis-lib-3.0.3.0-1.noarch                                                                                                                                    14/34

 Installing : streamsets-datacollector-windows-lib-3.0.3.0-1.noarch                                                                                                                                  15/34

 Installing : streamsets-datacollector-jks-credentialstore-lib-3.0.3.0-1.noarch                                                                                                                      16/34

 Installing : streamsets-datacollector-jython_2_7-lib-3.0.3.0-1.noarch                                                                                                                               17/34

 Installing : streamsets-datacollector-kinetica_6_0-lib-3.0.3.0-1.noarch                                                                                                                             18/34

 Installing : streamsets-datacollector-jms-lib-3.0.3.0-1.noarch                                                                                                                                      19/34

 Installing : streamsets-datacollector-stats-lib-3.0.3.0-1.noarch                                                                                                                                    20/34

 Installing : streamsets-datacollector-elasticsearch_5-lib-3.0.3.0-1.noarch                                                                                                                          21/34

 Installing : streamsets-datacollector-apache-solr_6_1_0-lib-3.0.3.0-1.noarch                                                                                                                        22/34

 Installing : streamsets-datacollector-apache-kafka_0_11-lib-3.0.3.0-1.noarch                                                                                                                        23/34

 Installing : streamsets-datacollector-mapr_6_0-lib-3.0.3.0-1.noarch                                                                                                                                 24/34

 Installing : streamsets-datacollector-azure-lib-3.0.3.0-1.noarch                                                                                                                                    25/34

 Installing : streamsets-datacollector-mysql-binlog-lib-3.0.3.0-1.noarch                                                                                                                             26/34

 Installing : streamsets-datacollector-vault-credentialstore-lib-3.0.3.0-1.noarch                                                                                                                    27/34

 Installing : streamsets-datacollector-apache-kafka_0_10-lib-3.0.3.0-1.noarch                                                                                                                        28/34

 Installing : streamsets-datacollector-basic-lib-3.0.3.0-1.noarch                                                                                                                                    29/34

 Installing : streamsets-datacollector-influxdb_0_9-lib-3.0.3.0-1.noarch                                                                                                                             30/34

 Installing : streamsets-datacollector-apache-kafka_0_9-lib-3.0.3.0-1.noarch                                                                                                                         31/34

 Installing : streamsets-datacollector-mapr_6_0-mep4-lib-3.0.3.0-1.noarch                                                                                                                            32/34

 Installing : streamsets-datacollector-bigtable-lib-3.0.3.0-1.noarch                                                                                                                                 33/34

 Installing : streamsets-datacollector-google-cloud-lib-3.0.3.0-1.noarch                                                                                                                             34/34

 Verifying  : streamsets-datacollector-salesforce-lib-3.0.3.0-1.noarch                                                                                                                                1/34

 Verifying  : streamsets-datacollector-groovy_2_4-lib-3.0.3.0-1.noarch                                                                                                                                2/34

 Verifying  : streamsets-datacollector-cyberark-credentialstore-lib-3.0.3.0-1.noarch                                                                                                                  3/34

 Verifying  : streamsets-datacollector-aws-lib-3.0.3.0-1.noarch                                                                                                                                       4/34

 Verifying  : streamsets-datacollector-cassandra_3-lib-3.0.3.0-1.noarch                                                                                                                               5/34

 Verifying  : streamsets-datacollector-rabbitmq-lib-3.0.3.0-1.noarch                                                                                                                                  6/34

 Verifying  : streamsets-datacollector-mapr_spark_2_1_mep_3_0-lib-3.0.3.0-1.noarch                                                                                                                    7/34

 Verifying  : streamsets-datacollector-jdbc-lib-3.0.3.0-1.noarch                                                                                                                                      8/34

 Verifying  : streamsets-datacollector-apache-kafka_1_0-lib-3.0.3.0-1.noarch                                                                                                                          9/34

 Verifying  : streamsets-datacollector-dev-lib-3.0.3.0-1.noarch                                                                                                                                      10/34

 Verifying  : streamsets-datacollector-omniture-lib-3.0.3.0-1.noarch                                                                                                                                 11/34

 Verifying  : streamsets-datacollector-mongodb_3-lib-3.0.3.0-1.noarch                                                                                                                                12/34

 Verifying  : streamsets-datacollector-redis-lib-3.0.3.0-1.noarch                                                                                                                                    13/34

 Verifying  : streamsets-datacollector-windows-lib-3.0.3.0-1.noarch                                                                                                                                  14/34

 Verifying  : streamsets-datacollector-jks-credentialstore-lib-3.0.3.0-1.noarch                                                                                                                      15/34

 Verifying  : streamsets-datacollector-jython_2_7-lib-3.0.3.0-1.noarch                                                                                                                               16/34

 Verifying  : streamsets-datacollector-kinetica_6_0-lib-3.0.3.0-1.noarch                                                                                                                             17/34

 Verifying  : streamsets-datacollector-jms-lib-3.0.3.0-1.noarch                                                                                                                                      18/34

 Verifying  : streamsets-datacollector-stats-lib-3.0.3.0-1.noarch                                                                                                                                    19/34

 Verifying  : streamsets-datacollector-elasticsearch_5-lib-3.0.3.0-1.noarch                                                                                                                          20/34

 Verifying  : streamsets-datacollector-apache-solr_6_1_0-lib-3.0.3.0-1.noarch                                                                                                                        21/34

 Verifying  : streamsets-datacollector-apache-kafka_0_11-lib-3.0.3.0-1.noarch                                                                                                                        22/34

 Verifying  : streamsets-datacollector-mapr_6_0-lib-3.0.3.0-1.noarch                                                                                                                                 23/34

 Verifying  : streamsets-datacollector-azure-lib-3.0.3.0-1.noarch                                                                                                                                    24/34

 Verifying  : streamsets-datacollector-mysql-binlog-lib-3.0.3.0-1.noarch                                                                                                                             25/34

 Verifying  : streamsets-datacollector-vault-credentialstore-lib-3.0.3.0-1.noarch                                                                                                                    26/34

 Verifying  : streamsets-datacollector-apache-kafka_0_10-lib-3.0.3.0-1.noarch                                                                                                                        27/34

 Verifying  : streamsets-datacollector-basic-lib-3.0.3.0-1.noarch                                                                                                                                    28/34

 Verifying  : streamsets-datacollector-influxdb_0_9-lib-3.0.3.0-1.noarch                                                                                                                             29/34

 Verifying  : streamsets-datacollector-apache-kafka_0_9-lib-3.0.3.0-1.noarch                                                                                                                         30/34

 Verifying  : streamsets-datacollector-mapr_6_0-mep4-lib-3.0.3.0-1.noarch                                                                                                                            31/34

 Verifying  : streamsets-datacollector-bigtable-lib-3.0.3.0-1.noarch                                                                                                                                 32/34

 Verifying  : streamsets-datacollector-google-cloud-lib-3.0.3.0-1.noarch                                                                                                                             33/34

 Verifying  : streamsets-datacollector-3.0.3.0-1.noarch                                                                                                                                              34/34

 

Installed:

 streamsets-datacollector.noarch 0:3.0.3.0-1                                                         streamsets-datacollector-apache-kafka_0_10-lib.noarch 0:3.0.3.0-1

 streamsets-datacollector-apache-kafka_0_11-lib.noarch 0:3.0.3.0-1                                   streamsets-datacollector-apache-kafka_0_9-lib.noarch 0:3.0.3.0-1

 streamsets-datacollector-apache-kafka_1_0-lib.noarch 0:3.0.3.0-1                                    streamsets-datacollector-apache-solr_6_1_0-lib.noarch 0:3.0.3.0-1

 streamsets-datacollector-aws-lib.noarch 0:3.0.3.0-1                                                 streamsets-datacollector-azure-lib.noarch 0:3.0.3.0-1

 streamsets-datacollector-basic-lib.noarch 0:3.0.3.0-1                                               streamsets-datacollector-bigtable-lib.noarch 0:3.0.3.0-1

 streamsets-datacollector-cassandra_3-lib.noarch 0:3.0.3.0-1                                         streamsets-datacollector-cyberark-credentialstore-lib.noarch 0:3.0.3.0-1

 streamsets-datacollector-dev-lib.noarch 0:3.0.3.0-1                                                 streamsets-datacollector-elasticsearch_5-lib.noarch 0:3.0.3.0-1

 streamsets-datacollector-google-cloud-lib.noarch 0:3.0.3.0-1                                        streamsets-datacollector-groovy_2_4-lib.noarch 0:3.0.3.0-1

 streamsets-datacollector-influxdb_0_9-lib.noarch 0:3.0.3.0-1                                        streamsets-datacollector-jdbc-lib.noarch 0:3.0.3.0-1

 streamsets-datacollector-jks-credentialstore-lib.noarch 0:3.0.3.0-1                                 streamsets-datacollector-jms-lib.noarch 0:3.0.3.0-1

 streamsets-datacollector-jython_2_7-lib.noarch 0:3.0.3.0-1                                          streamsets-datacollector-kinetica_6_0-lib.noarch 0:3.0.3.0-1

 streamsets-datacollector-mapr_6_0-lib.noarch 0:3.0.3.0-1                                            streamsets-datacollector-mapr_6_0-mep4-lib.noarch 0:3.0.3.0-1

 streamsets-datacollector-mapr_spark_2_1_mep_3_0-lib.noarch 0:3.0.3.0-1                              streamsets-datacollector-mongodb_3-lib.noarch 0:3.0.3.0-1

 streamsets-datacollector-mysql-binlog-lib.noarch 0:3.0.3.0-1                                        streamsets-datacollector-omniture-lib.noarch 0:3.0.3.0-1

 streamsets-datacollector-rabbitmq-lib.noarch 0:3.0.3.0-1                                            streamsets-datacollector-redis-lib.noarch 0:3.0.3.0-1

 streamsets-datacollector-salesforce-lib.noarch 0:3.0.3.0-1                                          streamsets-datacollector-stats-lib.noarch 0:3.0.3.0-1

 streamsets-datacollector-vault-credentialstore-lib.noarch 0:3.0.3.0-1                               streamsets-datacollector-windows-lib.noarch 0:3.0.3.0-1

 

Complete!

[root@maprdemo streamsets-datacollector-3.0.3.0-el7-all-rpms]#

 

  1. Setup connectivity to MapR

The command modifies configuration files, creates the required symbolic links, and installs the appropriate MapR stage libraries.

[root@maprdemo streamsets-datacollector-3.0.3.0-el7-all-rpms]# cd /opt/streamsets-datacollector/

[root@maprdemo streamsets-datacollector]# ls

api-lib  bin cli-lib  container-lib libexec  libs-common-lib root-lib  sdc-static-web streamsets-libs  user-libs

[root@maprdemo streamsets-datacollector]# export SDC_HOME=/opt/streamsets-datacollector

[root@maprdemo streamsets-datacollector]# export SDC_CONF=/etc/sdc

[root@maprdemo streamsets-datacollector]# export MAPR_MEP_VERSION=4

[root@maprdemo streamsets-datacollector]# $SDC_HOME/bin/streamsets setup-mapr

...

+ printf 'Done\n'

Done

+ echo Succeeded

Succeeded

 

  1. Start the service

[root@maprdemo streamsets-datacollector-3.0.3.0-el7-all-rpms]# systemctl start sdc

 

  1. Check Service Status

[root@maprdemo streamsets-datacollector-3.0.3.0-el7-all-rpms]# systemctl status sdc

  • sdc.service - StreamSets Data Collector (SDC)

  Loaded: loaded (/usr/lib/systemd/system/sdc.service; static; vendor preset: disabled)

  Active: active (running) since Thu 2018-02-01 06:19:20 PST; 26s ago

Main PID: 31899 (_sdc)

  CGroup: /system.slice/sdc.service

          ├─31899 /bin/bash /opt/streamsets-datacollector/libexec/_sdc -verbose

          └─31939 /usr/bin/java -classpath /opt/streamsets-datacollector/libexec/bootstrap-libs/main/streamsets-datacollector-bootstrap-3.0.3.0.jar:/opt/streamsets-datacollector/root-lib/* -Djava.secu...

 

Feb 01 06:19:20 maprdemo.local streamsets[31899]: API_CLASSPATH                  : /opt/streamsets-datacollector/api-lib/*.jar

Feb 01 06:19:20 maprdemo.local streamsets[31899]: CONTAINER_CLASSPATH            : /etc/sdc:/opt/streamsets-datacollector/container-lib/*.jar

Feb 01 06:19:20 maprdemo.local streamsets[31899]: LIBS_COMMON_LIB_DIR            : /opt/streamsets-datacollector/libs-common-lib/

Feb 01 06:19:20 maprdemo.local streamsets[31899]: STREAMSETS_LIBRARIES_DIR       : /opt/streamsets-datacollector/streamsets-libs

Feb 01 06:19:20 maprdemo.local streamsets[31899]: STREAMSETS_LIBRARIES_EXTRA_DIR : /opt/streamsets-datacollector/streamsets-libs-extras/

Feb 01 06:19:20 maprdemo.local streamsets[31899]: USER_LIBRARIES_DIR             : /opt/streamsets-datacollector/user-libs/

Feb 01 06:19:20 maprdemo.local streamsets[31899]: JAVA OPTS                      : -Djava.security.manager -Djava.security.policy=file:///etc/sdc/sdc-security.policy -Xmx1024m -Xms1024m -s...amsets-dataco

Feb 01 06:19:20 maprdemo.local streamsets[31899]: MAIN CLASS                     : com.streamsets.datacollector.main.DataCollectorMain

Feb 01 06:19:21 maprdemo.local streamsets[31899]: Logging initialized @945ms to org.eclipse.jetty.util.log.Slf4jLog

Feb 01 06:19:34 maprdemo.local streamsets[31899]: Running on URI : 'http://maprdemo:18630'

Hint: Some lines were ellipsized, use -l to show in full.

 

  1. Enable port forwarding

To access the UI for StreamSets Data Collector, enable port 18630 to be accessible.

Here’s how you’ll do that if you use Virtual Box.

 

Select Settings for the Sandbox and then click on Network settings

 

 

Add an entry for host port 18630

 

Select OK.

 

  1. Log into SDC & verify MapR stages are visible

Log into the SDC UI with the following url: http://localhost:18630

Default login is: admin/admin

Verify that you see MapR stages in the UI by first creating a pipeline.

 

Create a new pipeline

 

If all goes well, you should be able to see all the MapR stages as shown above.

Outcomes