Amazon AWS CloudFormation stack with Eclipse- Step by step guide_Part1
To follow this step by step guide, please install Eclipse with AWS toolkit.
This article will guide you through the installation and configuration.
Before using CloudFormation, It’s a good idea to navigate different services through the web console; create a VPC with subnets, and assign them CIDR IP blocks; launch EC2 instances, and pay attention to the different options available in the wizard; create security groups, Routes, and NACL rules.
The web console knowledge gained will help you read CF templates, and appreciate the logical relationships between a Resource and its properties, as well as relationships between different sections.
A CloudFormation template can have up to 8 sections, but only the Resources section is required.
If you use some of the optional sections, you will most likely need to reference the data in those sections using Intrinsic Functions.
For example, if you create a Mappings sections; inside your Resources section, you will use the function Fn::FindInMap to return the value corresponding to the key you declared under Mappings.
Let’s take a look at this closely:
“ImageId” : { “Fn::FindInMap” : [ “ImageMap”, { “Ref” : “AWS::Region” }, “MonitoringAMI” ]},
Here, ImageId is a property of AWS::EC2::Instance Resource, and as the name implies, it defines the AMI that will be used for the EC2 instance
We could have easily assigned an AMI inline without using Intrinsic functions:
“ImageId” : “ami-79fd7eee”,
Using Mappings; however, will make your CF templates more readable and maintainable.
For instance, in the example below, you can add multiple mappings that will cover the regions where you intend to run your stack.
"Mappings" : {
"ImageMap" : {"us-east-1" : { "OpenVpnAMI" : "ami-bc3566ab", "MonitoringAMI" : "ami-b73b63a0","NiFiAMI" : "ami-b73b63a0", "ClouderaAMI" : "ami-20b6c437", "RstatAMI" : "ami-b73b63a0", "VisualAMI" : "ami-b73b63a0" },
"us-east-2" :{ Hop on to your webconsole, and fill in us-east-2 AMI mappings}
}
},
As you know, an AMI number is region specific, so for the same image, the ID will be different in each region. Mapping AMI IDs to a region will aid you in optimizing your template, and not needing to create a separate template for each region.
This pseudo parameter that’s predefined by cloudformation is what passes the region name back to the Mappings using the Ref function: AWS::Region
You can expand upon this ImageMap section by creating a mapping for each of the 14 Amazon AWS regions available. (I excluded the US government region)
Amazon AWS documentation is top notch, so there is no need to replicate what’s already available, and in great details from their website. Here is a link that describes the different template sections, and their use:
http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/template-anatomy.html
The best way to learn is by doing, so let’s get started with an example of creating a stack for a Big data in the Cloud project.
This project will be housed in a VPN with one public subnet and 6 private subnets. Each subnet will run an EC2 instance that preforms a task in this miniaturized big data ecosystem.
I will be unconventional, and I will list how I started and the end result.
In future articles, we can expand more by explaining the stack creation process in detail, updating our CF stack with security groups, RDS database, and more updates to route table or NACL to tighten up security.
If you are a visual person, and you need to see it to believe it, then I would recommend using the web console CF template designer to kick start the building of your stack.
Here is a screenshot from my web console:
After a certain point, where you will need to fill out properties of your resources, that’s where you can save the template to your local drive and open it in Eclipse.
The designer needs to keep track of the dimensions and placement of boxes , objects, and lines in your template, so it adds Metadata containing this extraneous information throughout your template.
It will look like this:
"Metadata": {
"AWS::CloudFormation::Designer": {
"c733e469-afeb-4ccb-b0c1-f6c4125295f8": {
"size": {
"width": 1200,
"height": 1230
},
"position": {
"x": -80,
"y": -130
},
"z": 0,
"embeds": [
"8c3863c1-d144-4baf-8b8b-167eb0c83aae",
"01c6050d-1dc2-40e2-ace6-9c595b881719",
"7d4f0b22-bcdd-4595-a650-b9911f4479ef",
"8cfe599e-6f64-4c67-b720-82a8d3ee91ca",
"d4d7a5a5-7a23-44a8-b8e4-6e4b65d23407",
"b7769298-ed5a-4cea-9507-4b8e1c0709d6",
"74ad0ea8-0d45-4c27-92bc-5d70cac6d2ad",
"55003a29-f97f-4b22-8990-c8c5989a293d",
"4517c146-da09-4c8c-a9bc-ce8613f52f83",
"a0c8a6f3-037c-4957-b48a-415508e57fac",
"375e3d1a-007b-4393-8b05-c5ee7a7a6e15",
"d31fc2ee-1ca6-4ffa-982d-f914481aa62c",
"ed8ca7d4-12f7-42a8-a211-0b6788bef0fd",
"55912648-c1f4-4e93-a12d-1ef206bc28f8",
"71d77869-3266-4421-aaa1-b0efc9b9f19c"
]
},
"8c3863c1-d144-4baf-8b8b-167eb0c83aae": {
},
"size": {
"width": 490,
"height": 120
},
"position": {
"x": 0,
"y": -110
},
"z": 1,
"parent": "c733e469-afeb-4ccb-b0c1-f6c4125295f8",
"embeds": [
"10172244-abe9-49d4-923b-597566a0f720"
]
},
"01c6050d-1dc2-40e2-ace6-9c595b881719": {
"size": {
"width": 490,
"height": 120
},
…
None of that information is going to be used in creating your stack, so I have decided to clean up my template from all this Metadata, and continue building the stack in Eclipse.
I have decided to use the following sections: Format version, Description, Parameters, Mappings, and Resources. Here is the final CF stack, please note that it’s missing the RDS database, and security groups.
{
"AWSTemplateFormatVersion": "2010-09-09",
"Description" : "AWS CloudFormation template for a VPC with one public subnet and six private subnets for running a Big Data ecosystem. The following instances will be deployed in each subnet with different tasks: OpenVpn for authentication, Nagios or Kabana for monitoring, Apache Nifi for ETL, RDS MySql for database storage, Cloudera for BigData, OpenR or Revo for analytics, Qlik or Tabelaux for visualization",
"Parameters" : {
"InstanceType" : {
"Description" : " Lab instances are t1/t2.micro, or t1.small",
"Type" : "String",
"Default" : "t2.micro",
"AllowedValues" : ["t1.micro","t2.micro","t2.small"]
},
"ClouderaInstanceType" : {
"Description" : " Lab instances are t1/t2.micro, or t1.small",
"Type" : "String",
"Default" : "t1.micro",
"AllowedValues" : ["t1.micro","t2.micro","t2.small"]
},
"KeyName" : {
"Description" : "Name of an existing EC2 keyPair to enable SSH access to the instance",
"Type": "AWS::EC2::KeyPair::KeyName",
"ConstraintDescription" : "Must be the name of an existing Key pair"
},
"CFKeyName" : {
"Description" : "Name of an existing EC2 keyPair to enable SSH access to the instance",
"Type": "AWS::EC2::KeyPair::KeyName",
"ConstraintDescription" : "Must be the name of an existing Key pair"
},
"SSHLocation": {
"Description": "The IP address range that can be used to SSH to the EC2 instances",
"Type": "String",
"MinLength": "9",
"MaxLength": "18",
"Default": "0.0.0.0/0",
"ConstraintDescription": "must be a valid IP CIDR range of the form x.x.x.x/x."
}
},
"Mappings" : {
"ImageMap" : {
"us-east-1" : { "OpenVpnAMI" : "ami-bc3566ab", "MonitoringAMI" : "ami-b73b63a0","NiFiAMI" : "ami-b73b63a0", "ClouderaAMI" : "ami-20b6c437", "RstatAMI" : "ami-b73b63a0", "VisualAMI" : "ami-b73b63a0" },
"us-east-2" :{ }
}
}
},
"Resources": {
"VPC": {
"Type": "AWS::EC2::VPC",
"Properties": {
"CidrBlock" : "10.0.0.0/16",
"EnableDnsSupport" : "true",
"EnableDnsHostnames": "true",
"InstanceTenancy": "default"
}
},
"BasicSecurityGroup" : {
"Type" : "AWS::EC2::SecurityGroup",
"Properties" : {
"VpcId" : { "Ref" : "VPC" },
"GroupDescription" : "Enable SSH access",
"SecurityGroupIngress" : [
{"IpProtocol" : "tcp", "FromPort" : "22", "ToPort" : "22", "CidrIp" : { "Ref" : "SSHLocation"}}
]
}
},
"PublicAuthentication": {
"Type": "AWS::EC2::Subnet",
"Properties": {
"VpcId": {
"Ref": "VPC"
},
"CidrBlock" : "10.0.0.0/24",
"AvailabilityZone" : "us-east-1e"
}
},
"PrivateDataLanding": {
"Type": "AWS::EC2::Subnet",
"Properties": {
"VpcId": {
"Ref": "VPC"
},
"CidrBlock" : "10.0.1.0/24",
"AvailabilityZone" : "us-east-1c"
}
},
"PrivateDatabase": {
"Type": "AWS::EC2::Subnet",
"Properties": {
"VpcId": {
"Ref": "VPC"
},
"CidrBlock" : "10.0.2.0/24",
"AvailabilityZone" : "us-east-1d"
}
},
"PrivateDatabase2": {
"Type": "AWS::EC2::Subnet",
"Properties": {
"VpcId": {
"Ref": "VPC"
},
"CidrBlock" : "10.0.3.0/24",
"AvailabilityZone" : "us-east-1a"
}
},
"PrivateDataLake": {
"Type": "AWS::EC2::Subnet",
"Properties": {
"VpcId": {
"Ref": "VPC"
},
"CidrBlock" : "10.0.4.0/24",
"AvailabilityZone" : "us-east-1d"
}
},
"PrivateAnalytics": {
"Type": "AWS::EC2::Subnet",
"Properties": {
"VpcId": {
"Ref": "VPC"
},
"CidrBlock" : "10.0.5.0/24",
"AvailabilityZone" : "us-east-1d"
}
},
"PrivateVisualization": {
"Type": "AWS::EC2::Subnet",
"Properties": {
"VpcId": {
"Ref": "VPC"
},
"CidrBlock" : "10.0.6.0/24",
"AvailabilityZone" : "us-east-1a"
}
},
"PrivateMonitoring": {
"Type": "AWS::EC2::Subnet",
"Properties": {
"VpcId": {
"Ref": "VPC"
},
"CidrBlock" : "10.0.7.0/24",
"AvailabilityZone" : "us-east-1a"
}
},
"InternetGateway" : {
"Type" : "AWS::EC2::InternetGateway",
"Properties" : {
"Tags" : [ {"Key" : "Application", "Value" : { "Ref" : "AWS::StackId"} } ]
}
},
"AttachGateway" : {
"Type" : "AWS::EC2::VPCGatewayAttachment",
"Properties" : {
"VpcId" : { "Ref" : "VPC" },
"InternetGatewayId" : { "Ref" : "InternetGateway" }
}
},
"PublicRouteTable" : {
"Type" : "AWS::EC2::RouteTable",
"Properties" : {
"VpcId" : {"Ref" : "VPC"},
"Tags" : [ {"Key" : "Application", "Value" : { "Ref" : "AWS::StackId"} } ]
}
},
"Route" : {
"Type" : "AWS::EC2::Route",
"DependsOn" : "AttachGateway",
"Properties" : {
"RouteTableId" : { "Ref" : "PublicRouteTable" },
"DestinationCidrBlock" : "0.0.0.0/0",
"GatewayId" : { "Ref" : "InternetGateway" }
}
},
"SubnetRouteTableAssociation" : {
"Type" : "AWS::EC2::SubnetRouteTableAssociation",
"Properties" : {
"SubnetId" : { "Ref" : "PublicAuthentication" },
"RouteTableId" : { "Ref" : "PublicRouteTable" }
}
},
"NetworkAcl" : {
"Type" : "AWS::EC2::NetworkAcl",
"Properties" : {
"VpcId" : {"Ref" : "VPC"},
"Tags" : [ {"Key" : "Application", "Value" : { "Ref" : "AWS::StackId"} } ]
}
},
"InboundHTTPNetworkAclEntry" : {
"Type" : "AWS::EC2::NetworkAclEntry",
"Properties" : {
"NetworkAclId" : {"Ref" : "NetworkAcl"},
"RuleNumber" : "100",
"Protocol" : "6",
"RuleAction" : "allow",
"Egress" : "false",
"CidrBlock" : "0.0.0.0/0",
"PortRange" : {"From" : "80", "To" : "80"}
}
},
"InboundSSHNetworkAclEntry" : {
"Type" : "AWS::EC2::NetworkAclEntry",
"Properties" : {
"NetworkAclId" : {"Ref" : "NetworkAcl"},
"RuleNumber" : "101",
"Protocol" : "6",
"RuleAction" : "allow",
"Egress" : "false",
"CidrBlock" : "0.0.0.0/0",
"PortRange" : {"From" : "22", "To" : "22"}
}
},
"InboundResponsePortsNetworkAclEntry" : {
"Type" : "AWS::EC2::NetworkAclEntry",
"Properties" : {
"NetworkAclId" : {"Ref" : "NetworkAcl"},
"RuleNumber" : "102",
"Protocol" : "6",
"RuleAction" : "allow",
"Egress" : "false",
"CidrBlock" : "0.0.0.0/0",
"PortRange" : {"From" : "1024", "To" : "65535"}
}
},
"OutBoundHTTPNetworkAclEntry" : {
"Type" : "AWS::EC2::NetworkAclEntry",
"Properties" : {
"NetworkAclId" : {"Ref" : "NetworkAcl"},
"RuleNumber" : "100",
"Protocol" : "6",
"RuleAction" : "allow",
"Egress" : "true",
"CidrBlock" : "0.0.0.0/0",
"PortRange" : {"From" : "80", "To" : "80"}
}
},
"OutBoundHTTPSNetworkAclEntry" : {
"Type" : "AWS::EC2::NetworkAclEntry",
"Properties" : {
"NetworkAclId" : {"Ref" : "NetworkAcl"},
"RuleNumber" : "101",
"Protocol" : "6",
"RuleAction" : "allow",
"Egress" : "true",
"CidrBlock" : "0.0.0.0/0",
"PortRange" : {"From" : "443", "To" : "443"}
}
},
"OutBoundResponsePortsNetworkAclEntry" : {
"Type" : "AWS::EC2::NetworkAclEntry",
"Properties" : {
"NetworkAclId" : {"Ref" : "NetworkAcl"},
"RuleNumber" : "102",
"Protocol" : "6",
"RuleAction" : "allow",
"Egress" : "true",
"CidrBlock" : "0.0.0.0/0",
"PortRange" : {"From" : "1024", "To" : "65535"}
}
},
"SubnetNetworkAclAssociation" : {
"Type" : "AWS::EC2::SubnetNetworkAclAssociation",
"Properties" : {
"SubnetId" : { "Ref" : "PublicAuthentication" },
"NetworkAclId" : { "Ref" : "NetworkAcl" }
}
},
"OpenVPNSFTP": {
"Type": "AWS::EC2::Instance",
"Properties": {
"InstanceType" : {
"Ref" : "InstanceType"
},
"ImageId" : { "Fn::FindInMap" : [ "ImageMap", { "Ref" : "AWS::Region" }, "OpenVpnAMI" ]},
"KeyName" : {
"Ref" : "CFKeyName"
},
"NetworkInterfaces": [ {
"AssociatePublicIpAddress": "true",
"DeviceIndex": "0",
"GroupSet" : [ {"Ref" : "BasicSecurityGroup"} ],
"SubnetId": { "Ref" : "PublicAuthentication" }
} ]
}
},
"NagiosOrKabana": {
"Type": "AWS::EC2::Instance",
"Properties": {
"InstanceType" : {
"Ref" : "InstanceType"
},
"ImageId" : { "Fn::FindInMap" : [ "ImageMap", { "Ref" : "AWS::Region" }, "MonitoringAMI" ]},
"KeyName" : {
"Ref" : "KeyName"
},
"NetworkInterfaces": [ {
"AssociatePublicIpAddress": "false",
"DeviceIndex": "0",
"GroupSet" : [ {"Ref" : "BasicSecurityGroup"} ],
"SubnetId": { "Ref" : "PrivateMonitoring" }
} ]
}
},
"ApacheNiFi": {
"Type": "AWS::EC2::Instance",
"Properties": {
"ImageId" : { "Fn::FindInMap" : [ "ImageMap", { "Ref" : "AWS::Region" }, "NiFiAMI" ]},
"InstanceType" : {
"Ref" : "InstanceType"
},
"KeyName" : {
"Ref" : "KeyName"
},
"NetworkInterfaces": [ {
"AssociatePublicIpAddress": "false",
"DeviceIndex": "0",
"GroupSet" : [ {"Ref" : "BasicSecurityGroup"} ],
"SubnetId": { "Ref" : "PrivateDataLanding" }
} ]
}
},
"Cloudera": {
"Type": "AWS::EC2::Instance",
"Properties": {
"ImageId" : { "Fn::FindInMap" : [ "ImageMap", { "Ref" : "AWS::Region" }, "ClouderaAMI" ]},
"InstanceType" : {
"Ref" : "ClouderaInstanceType"
},
"KeyName" : {
"Ref" : "KeyName"
},
"NetworkInterfaces": [ {
"AssociatePublicIpAddress": "false",
"DeviceIndex": "0",
"GroupSet" : [ {"Ref" : "BasicSecurityGroup"} ],
"SubnetId": { "Ref" : "PrivateDataLake" }
} ]
}
},
"OpenOrRevoR": {
"Type": "AWS::EC2::Instance",
"Properties": {
"ImageId" : { "Fn::FindInMap" : [ "ImageMap", { "Ref" : "AWS::Region" }, "RstatAMI" ]},
"InstanceType" : {
"Ref" : "InstanceType"
},
"KeyName" : {
"Ref" : "KeyName"
},
"NetworkInterfaces": [ {
"AssociatePublicIpAddress": "false",
"DeviceIndex": "0",
"GroupSet" : [ {"Ref" : "BasicSecurityGroup"} ],
"SubnetId": { "Ref" : "PrivateAnalytics" }
} ]
}
},
"QlikQlikviewTableau": {
"Type": "AWS::EC2::Instance",
"Properties": {
"ImageId" : { "Fn::FindInMap" : [ "ImageMap", { "Ref" : "AWS::Region" }, "VisualAMI" ]},
"InstanceType" : {
"Ref" : "InstanceType"
},
"KeyName" : {
"Ref" : "KeyName"
},
"NetworkInterfaces": [ {
"AssociatePublicIpAddress": "false",
"DeviceIndex": "0",
"GroupSet" : [ {"Ref" : "BasicSecurityGroup"} ],
"SubnetId": { "Ref" : "PrivateVisualization" }
} ]
}
}
}
}
Go ahead try and run this stack from your eclipse by right clicking on the page, click on “Run on AWS”, then click “Create stack”.
You can then hop on to your console to watch it in action as your VPC, subnets, and EC2 machines are being created.
Some tips before I conclude the article:
- There is no delete stack command in Eclipse, so I deleted mine using AWS CLI with the following command:” C:\Program Files\Amazon\AWSCLI> aws cloudformation delete-stack –stack-name SunTest2 “
- You can also delete the stack from the cloudformation web console. But it’s not recommended that you delete individual stack components manually. It also defeats the purpose of using CF.
- If you need to update anything in your stack, you can do it in Eclipse with the “Update Stack” command.
- Please restrict your Stack names to the following characters: [Az az 0-9], or your stack creation will fail.
- Resource Properties and Parameters are case sensitive, so Default is not the same as default.
That’s it for now, and stay tuned for follow up articles that go on more details about CF stack creation, updating, and troubleshooting.
