Manual AWS Deployment

Document created by wochanda Employee on Aug 23, 2017Last modified by nsen on Aug 28, 2017
Version 13Show Document
  • View in full screen mode

Manually Deploying MapR in AWS

This document assumes you are familiar with the AWS console and can navigate and create resources there.

 

The high level tasks are:

 

 

Create AWS Key-pair

See http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-key-pairs.html on creating key-pairs

 

 

Create Policies and Roles

If there are restriction in creating roles and policies, you can add the AWS API credentials on the installer node. See section on “Setup the Installer”

 

There is cloud-formation template that can create these for you: Launch Stack

 

Otherwise, follow these steps:

  1. Create Policy

       The policy statement would be:  

         {

            "Version": "2012-10-17",

            "Statement": [

                {

                    "Effect": "Allow",

                    "Action": [

                        "autoscaling:Describe*",

                        "autoscaling:CreateLaunchConfiguration",

                        "autoscaling:DeleteLaunchConfiguration",

                        "autoscaling:SuspendProcesses",

                        "autoscaling:UpdateAutoScalingGroup",

                        "cloudformation:DescribeStack*",

                        "cloudformation:GetTemplate",

                        "cloudformation:UpdateStack",

                        "ec2:CreateKeyPair",

                        "ec2:DeleteKeyPair",           

                        "ec2:ImportKeyPair",

                        "ec2:DescribeKeyPairs",

                        "ec2:DescribeInstances",

                        "ec2:DescribeInstanceAttribute",

                        "ec2:ModifyInstanceAttribute",

                        "ec2:DescribeSubnets",

                        "ec2:RunInstances",

                        "ec2:StartInstances",

                        "ec2:StopInstances",

                        "ec2:CreateVolume",

                        "ec2:AttachVolume",

                        "ec2:DescribeVolumes"

                 ],

                 "Resource": "*"

             }

         ]

      }

 

    2. Create Role with the policy created above.

 

Note: You as a user must have the IAM:PassRole permission

 

 

 

 

Create VPC, subnets, Internet Gateway, NAT Gateway, Security Groups

You can use an existing VPC if that satisfies the following requirements:

  • Instances can be created in a subnet that can talk to the Internet (to download MapR packages)
  • Instances can talk to each other within the VPC

 

If not use can use the following steps to create the VPC, subnets, Internet Gateway, NAT Gateway, Security Groups.

 

There is cloud-formation template that can create these for you: Launch Stack

 

Otherwise, follow these steps:

  1. Create VPC
  2. Create two subnets
    1. These will be configured such that one can be accessed from the Internet (public-subnet), another can talk to the Internet but cannot be accessed from outside (private-subnet)
  3. Create Internet Gateway
  4. Attach Internet Gateway to the VPC
  5. Create Elastic IP
  6. Create NAT Gateway
  7. Update default Route Table for VPC to allow Internet access via Internet Gateway
  8. Create a new Route
    1. Update the route to allow the private-subnet to talk to the Internet via NAT Gateway
    2. Associate private subnet to the new route
  9. Create Security groups
    1. SSH Access
      1. Ports: 22
    2. Web Access
      1. Ports: 9443, 8443, 8042, 8047, 8088, 8888, 18080, 19888



Create VPC



Create subnet. You are creating two subnets.


 

Create Internet Gateway

 

Attach Internet Gateway to VPC

 

Create NAT Gateway

 

 

Update default route table for VPC to route all trafic to Internet Gateway

 

 

Create a new route table for private subnet

 

 

Update the route table to route all traffic to the NAT gateway

 

Associate route table with the private subnet

 

Create Security Groups

Add Inbound security rules



Create a Launch Configuration

From AWS console, create launch configuration:

  • Choose AMI

  • Choose machine type

  • Provide launch configuration name

    • Choose either the 2nd or 3rd option for IP Address Type under Advanced options

  • Add storage

    • Increase the root disk size to minimum of 128G

    • Add as many disks, along with their size, for data disks on the nodes

  • Select security groups

    • The default security group for the VPC that will allow the instances to talk to each other

    • The security group created to allow web access to the MapR services

  • Select the AWS Keypair to be used

 

You can go on to creating an auto scaling group with the launch configuration created from the final screen.

 

Create an Auto-scale Group

From AWS console, create auto scaling group:

  • Set auto-scale group properties

    • Give it a name

    • Select the VPC and the private-subnet

    • Choose 0 instances to start with (and let the MapR installer manage it)

  • Select to keep the group at initial size

  • After auto-scale group is created, edit the group to Suspend the following 4 services

    • Terminate, ReplaceUnhealthy, AZRebalance, HealthCheck

 

Suspend auto-scale management processes:

 

Create an EC2 instance for the installer

From AWS console, create an EC2 instance using an AMI with supported OS:

  1. Assign it to the VPC created and public-subnet. Assign a public IP address.

  2. Give role to be able to execute AWS commands without credentials

    1. Alternatively, create aws credentials files on the installer (see next section)

  3. Min disk size: 128G

  4. Assign the following security groups:

    1. Default security group to allow the node to talk to all nodes in the cluster

    2. The security group created to allow web access to the MapR services

    3. The security group created to allow SSH access to the nodes

 

 

Setup MapR installer

SSH to mapr installer and as root run:

 

Note: If the installer has not been setup with an IAM Role to allow it to execute AWS commands, follow the section "Create AWS credentials file" first.

Note: replace <clustername> with the name of you wish to give your cluster

  1. wget http://package.mapr.com/releases/installer/mapr-setup.sh
  2. chmod +x mapr-setup.sh
  3. ./mapr-setup.sh -y
  4. cd /opt/mapr/installer
  5. source build/installer/bin/activate
  6. cd data
  7. ssh-keygen -b 2048 -t rsa -f <clustername> -q -N '' -C "maprinstaller@maprcluster"
  8. aws ec2 import-key-pair --key-name <clustername> --region us-east-2 --public-key-material "`cat <clustername>.pub`"
  9. chmod 400 <clustername>
  10. touch config.yml
  11. chown mapr:mapr <clustername> <clustername>.pub config.yml
  12. vi config.yml

Copy and paste the content below. Update the values as per your environment.

Environment:

 mapr_core_version: 5.2.2

config:

 ssh_id: centos

 ssh_key_file: /opt/mapr/installer/data/<clustername>

 cluster_name: <clustername>

 mep_version: 3.0.1

 provider:

   id: AWS

   config:

     aws_region: us-east-2

     auto_scaling_group: mapr-test-asg

     count: 3

     key_name: <clustername>

     disk_type: gp2

     disk_size: 100

     disk_count: 3

 hosts: []

  1. sudo -u mapr ../bin/mapr-installer-cli import -f -n --config -t config.yml
  2. passwd mapr

 

Next, launch the installer and complete the rest of the installation via the installer. The installer would be listening on:

https://<mapr-installer-public-hostname>:9443

 

Create AWS credentials file

If you did not create an AWS role to be used by the MapR installer, you must create an AWS credentials file with AWS access key and secret. Information on how to obtain access key and secret can be found at http://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html

 

  1. mkdir -p ~/home/mapr/.aws
  2. vi ~/home/mapr/.aws/credentials

[default]

aws_access_key_id = <your aws access_key>

aws_secret_access_key = <your aws secret key>




Attachments

    Outcomes