AWS Pre-defined Examples

Document created by wochanda Employee on Aug 23, 2017Last modified by nsen on Oct 10, 2017
Version 5Show Document
  • View in full screen mode

Introduction

Because each organization's design, standards, and security policies are different, MapR provides several deployment examples in the form of AWS CloudFormation Templates (CFTs).  Users are encouraged to use these CFT examples as-is, or customize them to better fit a particular situation.

 

Below is the most current list of pre-defined templates released by MapR.  Within the list below, some templates are listed as 'deployment templates', meaning they actually deploy a MapR cluster, while others are listed as 'helper' templates, which create a portion of the cloud environment but don't perform a full MapR deployment. Helper templates can help split tasks between multiple people in an organization in cases where AWS permissions are split between multiple users. These are hosted in github for you to download and customize.

 

TypeTemplateDescriptionPrivileges Required
Deploymentaws_cf_maprcluster_with_role.ymlCreates a MapR cluster along with all the privileges and roles required by the MapR Installer. Use this template if you are privileged to create roles and policies in AWS.

iam:GetRole
iam:PassRole
iam:PutRolePolicy
iam:CreateRole
iam:AddRoleToInstanceProfile
iam:CreateInstanceProfile

iam:autoscaling:*

iam:cloudformation:*

iam:ec2:*

iam:PassRole

iam:s3:*

Deploymentaws_cf_maprcluster.ymlCreates a MapR cluster. This template requires the IAM role to be provided as an input. If you do not have the permission to create roles and privileges, ask your administrator to create the role for them. Theaws_cf_maprcluster_role.yml template can be used for this purpose.

am:autoscaling:*

iam:cloudformation:*

iam:ec2:*

iam:PassRole

iam:s3:*

Deploymentaws_cf_maprcluster_ami.ymlCreates a MapR cluster using an arbitrary AMI running a MapR-supported OS.  This is ideal for situations where a company has organizational restrictions about which AMIs are allowed to be deployed. For best results, the root disk (where /opt/mapr is located) should have a minimum of 128 GB, and swap space should be at least 10% of memory or less than 2 GB.

iam:autoscaling:*

iam:cloudformation:*

iam:ec2:*

iam:PassRole

iam:s3:*

Helperaws_cf_vpc_openvpn.yml

Deploys a VPC capable of supporting a MapR cluster.  It can also optionally setup a OpenVPN Access Server. The OpenVPN Access Server supports 2 free client connections for testing purposes. You can buy licenses for it from the OpenVPN site

Once created, you can pass the subnet id into any of the deployment templates. 

iam:cloudformation:*

iam:ec2:*

Helper

aws_cf_maprcluster_role.yml

Creates a role with the privileges necessary for the MapR Installer to create, modify, and release AWS instances.  Once created, you can pass the role name into the aws_cf_maprcluster.yml template.

iam:cloudformation:*

iam:GetRole
iam:PassRole
iam:PutRolePolicy
iam:CreateRole
iam:AddRoleToInstanceProfile

Helper

aws_cf_maprcluster_nodes.yml

Creates launch configuration and auto scaling group. If you are doing a custom deployment, you can customize this to your needs. 

iam:autoscaling:*

iam:cloudformation:*

Launching Via CloudFormation Example

Create a Stack in AWS CloudFormation

Either click one of the links above, or:

  1. Browse to AWS CloudFormation
  2. Click 'Create New Stack'
  3. Click 'Specify an Amazon S3 template URL'
  4. Enter URL for CloudFormation Template

e.g. https://s3.us-east-2.amazonaws.com/mapr-installer/templates/aws/aws_cf_maprcluster_with_role.yml 

 

Configure Your Stack

(Selections here will differ between examples.)

  1. Stack name - Unique name for this deployment within your AWS account.  You may need this later to modify or delete your cluster.
  2. AWS Infrastructure Configuration
    1. keyName - AWS keypair you'll be using to SSH into your deployed cluster. 
      1. If you don't have one, create one from AWS > Services > EC2 > Key Pairs.  Hold on to your private key.
  3. MapR Cluster Configuration
    1. clusterName - Use a unique, DNS-esque identifier for your cluster.
    2. clusterAdminPassword - This will be used along with the 'mapr' user to log into your cluster management console.
    3. MEP - This is the version of the 'MapR Ecosystem Pack' that will be deployed.
    4. provisioningTemplate - This specifies the specific Apache open-source components that will be deployed on your cluster.
      1. Converged-Platform - Deploys Apache Hadoop, Hive, Spark, and Drill along with the core MapR Data Services - MapR-XD, MapR-DB, and MapR-ES.
      2. Data-Lake - Deploys Apache Hadoop, Hive, Spark, and Drill along with MapR-XD.
      3. Operational-Analytics - Deploys Apache Drill and MapR-DB.
      4. Real-Time-Analytics - Deploys Apache Spark with MapR-ES.
      5. Custom-Configuration - Allows you to fully customize the components to install at a later stage (see last step).
  4. Node Configuration - Specify the instance type and storage configuration.  Some tips.
    1. i3 instances are optimized for local storage.  If these instance type is selected we will ignore your disk choices and instead use the disks that come with the instance.
    2. d2 instances are also optimized for local storage. If these instance type is selected and you choose to use useInstanceStore, we will ignore your disk choices and instead use the disks that come with the instance.
    3. All other instances will use only EBS storage, so you should specify type/amount.
  5. Network, Access, Security
    1. useExistingSubnet - Specify subnet id that you either created manually or using the aws_cf_vpc_openvpn.yml template.
    2. securityGroups  - Specify the security groups that the subnet you specified above requires to allow access the subnet
    3. installOpenVPN - If your VPC/subnet are not accessible to the wide Internet, you can use OpenVPN to connect your computer to the MapR subnet to access it.

 

Create Your Stack

Once you've configured your stack, click 'Next' at the next couple of pages.

When asked, check the box next to 'I acknowledge that AWS CloudFormation might create IAM resources with custom names' and click 'Create'.

 

Once you click 'create', you can view progress on the next page.  Make sure you filter for 'In Progress'.

 

A 3 node cluster takes 15-20 minutes to create. Larger cluster may take longer to create. 

 

(Optional) Customize Your Cluster

If for provisioningTemplate you selected 'Custom-Configuration' you can now fully customize your MapR cluster before installation.

 

  1. Find your stack in the CloudFormation Stack list.  Make sure it is in status of 'CREATE_COMPLETE'.
  2. Select the your stack, and in the bottom panel dropdown select 'Outputs'.
  3. Find your 'MaprInstallerUrl' and try to browse to it.
    1. If you can't access the URL, you may need to VPN into it.  If you selected OpenVPN=true when configuring your stack, you should have other outputs like 'OpenVPNUrl' to help you VPN into your cluster network.
  4. Sign In to your cluster using 'mapr' user and the password you configured.
  5. Click 'Next' and configure your cluster as you would for a non-cloud installation.
  6. Once complete, you'll see a page of URL endpoints, including a link to MCS for managing your cluster.

 

Access Your Cluster

  1. Find your stack in the CloudFormation Stack list.  Make sure it is in status of 'CREATE_COMPLETE'.
  2. Select the your stack, and in the bottom panel dropdown select 'Outputs'.
  3. Find your 'MaprInstallerUrl' and try to browse to it.
    1. If you can't access the URL, you may need to VPN into it.  If you selected OpenVPN=true when configuring your stack, you should have other outputs like 'OpenVPNUrl' to help you VPN into your cluster network.
  4. Sign In to your cluster using 'mapr' user and the password you configured.
  5. You will see a page containing a list of URL endpoints, including a link to MCS for managing your cluster.

Attachments

    Outcomes