Cloud Computing- Data and AI

In the wake of yesterday’s ransomware incident, Information security is in the lime light, and before I connected to the internet today, I clicked restart on the annoying windows updates message, instead of postponing it another 4 hours.

Computer is up to date with all security fixes and anti-virus definition files have been updated. I am ready to go.

I wanted to experiment with a web application, but didn’t want to spend the time configuring LAMP, and a PHP framework, so I opted for using an Amazon AWS EC2 community API pre-loaded with the application stack.

I was lucky to find one, so I went ahead and launched it, and accessed it via the server IP.

I created my RDS DB in order to change the “database.php” config file to point to my own DB.

When I opened the config file to edit it; there I found some brazilian website “.br” as the host name, and mysql root credentials.

I thought, there is no way this is still live, so I tried to connect via command line to the remote host.

I got the prompt back!

I had root on their MySQL database with no effort.

I am not sharing any screenshots on this blog post to protect the server. But this incident is a reminder to make sure you don’t share AMIs until you remove all credentials, user history, etc.

Follow this guide for more info: https://aws.amazon.com/articles/0155828273219400

Another tip is to run the mysql secure installation script.

It’s also a good idea to block remote connections to your SQL server.

I went ahead and contacted AWS security to alert them to this AMI, so they can take care of taking it down, and contacting the site owner.

In the wake of security breaches, one is amazed how these criminals are so successful with their exploits. It comes down to our deficiency in not following basic, annoying security steps one has to take. I say annoying, because they kind of are. Nobody wants to have multiple locks to their house, and then a lock for every room, security guard outside, etc… You get where I am going with this. However, today’s reality where most of our computing is in public networks makes it a priority to follow good security practices. Creating hard to guess passwords, implementing password expiration, multi factor authentication, software patching, updates to both OS and anti-virus definition files should be part of our disciplined approach to cyber security.

The ransomware that ravaged UK NHS computers was caused by running unpatched OS. It’s easy for me to sit here and speculate about the why?!

What’s apparent is that the NHS CTO or CSO (if they have one) decided to take on that risk, and we now know it was a costly decision.

Update: AWS Security replied to my e-mail where they referred to their “Shared responsibility model“, and that they can not take down the EC2 instance. So, until the owner takes action, this EC2 image is still there for people to use.

Excerpt from AWS Security message:

“To be clear, the security concern you have reported cannot be resolved by AWS but must be addressed by the customer, who may not be aware of or be following our recommended security best practices.

We have passed your security concern on to the specific customer for their awareness and potential mitigation.“

I forget my password regularly, and I don’t have e-mail setup on my WordPress site; therefore, clicking on the forget password link doesn’t do me any good. This article will serve as a step by step process I can use in the future in case this happens again, and it can save you time as well if you face the same problem

An AWS bitnami WordPress EC2 machine comes with a packaged installation of LAMP, WordPress, and PhpMyAdmin that’s accessible through localhost.

In order to reset your WordPress password, you will need access to the database either through the command line or through a MySql database management tool like PhpMyAdmin.

This article will guide you through the process of resetting your WordPress password through PhpMyAdmin.

These instructions are for a windows environment using putty SSH client. Same concepts apply to other platforms.

1. Create an SSH tunnel using putty. You will need to have your EC2 private key, and you need to know your domain name/IP. (Image 1 and Image 2)
2. After you have entered your HostName under Session Category, your private key under Connection>SSH>Auth; click on Tunnels to create a secure tunnel by forwarding port 80 on the server to port 8888 on the local host (127.0.0.1 or localhost). In Source Port type port “8888”, and in the destination type “127.0.0.1:80”, the click add. If you happen to be using port 8888 for anything in your local machine, then change the source port to any non-privileged open port. (Image 3)
3. Login using your user name, the default user is “bitnami” if you haven’t changed it. (Image 4)
4. Now that you have a tunnel between your local machine and your server, you can access phpMyAdmin through the URL: http://127.0.0.1:8888/phpmyadmin (Image 5)
5. Next, you will need a DB user and password. Since you are on the server, navigate to “/opt/bitnami/apps/wordpress/htdocs” and open wp-config.php, look for the database user and password your WordPress is using to connect, and use that to login to PhpMyAdmin.
6. Once Logged in, search for the table: “wp_users”, and double click on the name to open it.
7. 
8. Click on edit next to the username you want to edit its password.
9. The password is MD5 hashed, so you can’t recover it there, but you can delete it, and copy and paste the MD5 hash of a new password. Use this website to get the hash: http://www.miraclesalad.com/webtools/md5.php
10. Once you paste your new password (Hint, don’t use “new password”, but your real password), click Go to update the table row for that user.
11. Repeat same process for other users.
12. Test logging in to WordPress, it should work now!

Today is Saturday, and I am out for an early breakfast at Panera, as usual, and I have my laptop with me to work on some AWS stuff.

My plan is to finish setting up installing Laravel PHP framework as part of a tutorial I am following for storing laravel sessions on DynamoDB. Since my Web Server is hosted on an EC2 instance on a private Subnet, I need to first VPN into my VPC to be able to get to it.

So I proceeded to connect through VPN to my OpenVPN EC2 on my public subnet. Connection is established. Now, I need to SSH to my 10.0.3.208 machine, but nothing is happening, and eventually the connection times out!

I checked my security groups, Routes, and NACLs; everything looks good, and I haven’t changed my setup from when I successfully connected the day before.

Is AWS flaky? What’s going on?

I run Wireshark to help me figure out what’s causing the connection timeout. I am trying to connect through port 22 with putty, and port 80.

I am seeing ICMP packets from a 10.128.128.128 device telling me that the destination is unreachable, and that the communication is administratively filtered.

Not Sure what this device is, but let’s do an ipconfig on the local machine:

Ok, so Panera bread, wireless router is rejecting my request for a server that it thinks is in it’s 10.0.0.0/8 network.

My VPC is on 10.0.0.0/16

My private subnet is on 10.0.3.0/24

Ok, so I guess the wireless router will not let this slide by him so my packets can travel through the VPN tunnel to my EC2 machine on my VPC.

I did some google search to see if I can make this work through some Ninja networking tricks. The threads I read unanimously agree that you need to design your VPC with a CIDR block that doesn’t conflict with your local network.

Every now and then, some network guru will tell you about all the hoops he went through to make it somehow work, but why go through the trouble, when you could design your VPC with no conflict in mind? Plus, I needed access to the local network router to try those solutions, and Panera wasn’t about to hand me the credentials to their router.

A Cisco article I read presents a solution through the use of NAT and DNS. A use case for that could be two companies merging with the same private  address space, and what you would do to prevent re-addressing the local network of one of them.

Anyway, I can’t stress enough the importance of choosing the right VPC CIDR.

Avoid the 192.168.x.x since most home network routers use that RFC 1918  IP addresses.

To solve my issue, I need to go home and connect. Surely enough, as soon as I connected to my home network, I was able to VPN, and browse to my private web server and SSH to it:

Traffic through wireshark looks good. Syns are being acknowledged, and traffic is flowing nicely through the VPN adapter:

One caveat is that if you want to see network traffic for your private IP address as the destination IP before it gets encapsulated into VPN encrypted packets, you need to make sure you are listening to the VPN adapter not your wireless adapter, or your network card if you are connected with an Ethernet cable to your router. (It took me some googling to figure that out)

My VPN server is configured with this CIDR: 172.27.224.0/20, which it uses to assign IPs to connected clients.

I have tried to produce some conflict again with my VPC, and I changed my VPN server CIDR to 10.0.0.0/16:

My VPN server couldn’t route traffic correctly, I couldn’t resolve domains to browse the internet, so that’s another place to look at in case you have problems with your VPN.

Software defined networks enables services such Amazon AWS VPC, which gives us greater control over our cloud network topology.

“With great power comes great responsibility”, so choose your CIDR blocks wisely.

Recently, I have gotten involved in a project where, among other things, we needed to create EC2 instances from a web application.
We are using CloudFormation templates to create VPC with public and private subnets, Internet gateway, Security Groups, NACLs, etc.
We have decided to have separate stacks for network infrastructure, servers, ELB and auto-scalers, and we are also considering a separate security stack.
What makes this possible is using the cross-stack reference option.
“To create a cross-stack reference, use the Export output field to flag the value of a resource output for export. Then, use the Fn::ImportValue intrinsic function to import the value”
So simply put, you create a new stack where you import values from a parent stack where those same values are exported in the “Outputs” section.
In this example, I am exporting values from my main template to be consumed later by my EC2 instances template:

"Outputs" : {
"PrivateSubnet" : {
"Description" : "The subnet ID to use for web servers behind an ELB",
"Value" : { "Ref" : "PrivateWebServer1" },
"Export" : { "Name" : {"Fn::Sub": "${AWS::StackName}-SubnetID" }}
},
"BasicSecurityGroup" : {
"Description" : "The security group ID to use for private web servers",
"Value" : { "Fn::GetAtt" : ["BasicSecurityGroup", "GroupId"] },
"Export" : { "Name" : {"Fn::Sub": "${AWS::StackName}-SecurityGroupID" }}
}
}

Notice that I am first referencing the value that need to be exported using the “Ref” function.
Next, I am using the Export parameter to export the SubnetID, and in the next declaration the SecurityGroupID of a Stack Name.
Using the Fn-Sub function, I am substituting the $(AWS::StackName) variable with an input parameter.
This Parameter is defined in our Parameters section of the child template:

“"Parameters": {
"NetworkStackName": {
"Description": "Name of an active CloudFormation stack that contains the networking resources, such as the subnet and security group, that will be used in this stack.",
"Type": "String",
"MinLength" : 1,
"MaxLength" : 255,
"AllowedPattern" : "^[a-zA-Z][-a-zA-Z0-9]*$",
"Default" : "Your-Parent-Template"
}
}

This is the complete child stack that will import values from its parent stack:

{
"AWSTemplateFormatVersion": "2010-09-09",

"Description" : "CF stack to create an EC2 instance, with Parent VPC name as parameter",

"Parameters": {
"NetworkStackName": {
"Description": "Name of an active CloudFormation stack that contains the networking resources, such as the subnet and security group, that will be used in this stack.",
"Type": "String",
"MinLength" : 1,
"MaxLength" : 255,
"AllowedPattern" : "^[a-zA-Z][-a-zA-Z0-9]*$",
"Default" : "MyVPC3"
}
},

"Resources": {

"WebServer": {

"Type": "AWS::EC2::Instance",

"Properties": {

"InstanceType" : "t2.micro",

"ImageId" : "xxxxxxxx",

"KeyName" : "xxxxxxxxx",

"NetworkInterfaces": [ {
"AssociatePublicIpAddress": "false",
"DeviceIndex": "0",
"GroupSet" : [{ "Fn::ImportValue" : {"Fn::Sub": "${NetworkStackName}-SecurityGroupID" } }],

"SubnetId":{ "Fn::ImportValue" : {"Fn::Sub": "${NetworkStackName}-SubnetID" } }
} ]

}
}
}
}

when the child stack is ran, it will create an EC2 machine inside the VPC and subnet created in the parent stack, and assign it the appropriate security group.
After testing that everything works, now it’s time to create an EC2 instance from a web application.
We will use PHP, and the well documented AWS SDK for PHP.
Best way to install the SDK is to use Composer.
Go through the trouble of installing it because it’s worth it. You won’t have to worry about troubleshooting issues with dependencies, because composer takes care of that.
I am using PHP with IIS on a windows server.
Download the exe file, and install composer. Check if it’s installed:

Next, install the PHP SDK by following these instructions, and run the following command from the vendor directory of your PHP project: (If PHP binary is not in your path, then specify the full command path)

php -r "eval('?>'.file_get_contents('https://getcomposer.org/installer'));"

To use the PHP SDK for any AWS service, you will need to first start with creating a client. For more information on how to get started with calling an AWS service API, read this guide:
“The basic usage pattern of the SDK is that you instantiate a Client object for the AWS service you want to interact with“

My Cloudformation service client looks like this: (You can use credentials online, or use a profile in your credentials file. For security, use profiles!)

<?php require 'C:\awscf\vendor\autoload.php'; //Create a client use Aws\CloudFormation\CloudFormationClient; $client = CloudFormationClient::factory(array( 'region' => 'us-east-1',
    'credentials' => [
        'key'    => 'xxxxxxxxxxxxxxxxxxx',
        'secret' => 'xxxxxxxxxxxxxxxxxxxx',
    ],
));

// Create your stack stored on S3 that imports data from your network stack
    $result = $client->createStack(array(
    // StackName is required
    'StackName' => 'Webserver1',
    'TemplateURL' => 'https://s3.amazonaws.com/yourbucket/yourfile.template',
    ));
?>

Very straight forward way of creating a stack from your PHP web application. We have only used 2 elements of the array to keep it simple, but you are welcome to take advantage of the full capabilities of the API call.
For example, you could pass on your parameter of parent stack name to the child CF stack, instead of using the default.

I hope you found this article helpful to get you started quickly with using the AWS cloudformation PHP SDK.

To follow this step by step guide, please install Eclipse with AWS toolkit.

This article will guide you through the installation and configuration.

Before using CloudFormation, It’s a good idea to navigate different services through the web console; create a VPC with subnets, and assign them CIDR IP blocks; launch EC2 instances, and pay attention to the different options available in the wizard; create security groups, Routes, and NACL rules.

The web console knowledge gained will help you read CF templates, and appreciate the logical relationships between a Resource and its properties, as well as relationships between different sections.

A CloudFormation template can have up to 8 sections, but only the Resources section is required.
If you use some of the optional sections, you will most likely need to reference the data in those sections using Intrinsic Functions.
For example, if you create a Mappings sections; inside your Resources section, you will use the function Fn::FindInMap to return the value corresponding to the key you declared under Mappings.

Let’s take a look at this closely:

“ImageId” : { “Fn::FindInMap” : [ “ImageMap”, { “Ref” : “AWS::Region” }, “MonitoringAMI” ]},

Here, ImageId is a property of AWS::EC2::Instance Resource, and as the name implies, it defines the AMI that will be used for the EC2 instance 

We could have easily assigned an AMI inline without using Intrinsic functions:
“ImageId” : “ami-79fd7eee”,

Using Mappings; however, will make your CF templates more readable and maintainable.

For instance, in the example below, you can add multiple mappings that will cover the regions where you intend to run your stack.

 "Mappings" : {
"ImageMap" : {"us-east-1" : { "OpenVpnAMI" : "ami-bc3566ab", "MonitoringAMI" : "ami-b73b63a0","NiFiAMI" : "ami-b73b63a0", "ClouderaAMI" : "ami-20b6c437", "RstatAMI" : "ami-b73b63a0", "VisualAMI" : "ami-b73b63a0" },
"us-east-2" :{ Hop on to your webconsole, and fill in us-east-2 AMI mappings}
 }
},

As you know, an AMI number is region specific, so for the same image, the ID will be different in each region. Mapping AMI IDs to a region will aid you in optimizing your template, and not needing to create a separate template for each region.
This pseudo parameter that’s predefined by cloudformation is what passes the region name back to the Mappings using the Ref function: AWS::Region

You can expand upon this ImageMap section by creating a mapping for each of the 14 Amazon AWS regions available. (I excluded the US government region)

Amazon AWS documentation is top notch, so there is no need to replicate what’s already available, and in great details from their website. Here is a link that describes the different template sections, and their use: 

http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/template-anatomy.html

The best way to learn is by doing, so let’s get started with an example of creating a stack for a Big data in the Cloud project.

This project will be housed in a VPN with one public subnet and 6 private subnets. Each subnet will run an EC2 instance that preforms a task in this miniaturized big data ecosystem.

I will be unconventional, and I will list how I started and the end result.

In future articles, we can expand more by explaining the stack creation process in detail, updating our CF stack with security groups, RDS database, and more updates to route table or NACL to tighten up security.

If you are a visual person, and you need to see it to believe it, then I would recommend using the web console CF template designer to kick start the building of your stack.

Here is a screenshot from my web console:

After a certain point, where you will need to fill out properties of your resources, that’s where you can save the template to your local drive and open it in Eclipse.

The designer needs to keep track of the dimensions and placement of boxes , objects, and lines in your template, so it adds Metadata containing this extraneous information throughout your template.

It will look like this:

"Metadata": {
"AWS::CloudFormation::Designer": {
"c733e469-afeb-4ccb-b0c1-f6c4125295f8": {
"size": {
"width": 1200,
"height": 1230
},
"position": {
"x": -80,
"y": -130
},
"z": 0,
"embeds": [
"8c3863c1-d144-4baf-8b8b-167eb0c83aae",
"01c6050d-1dc2-40e2-ace6-9c595b881719",
"7d4f0b22-bcdd-4595-a650-b9911f4479ef",
"8cfe599e-6f64-4c67-b720-82a8d3ee91ca",
"d4d7a5a5-7a23-44a8-b8e4-6e4b65d23407",
"b7769298-ed5a-4cea-9507-4b8e1c0709d6",
"74ad0ea8-0d45-4c27-92bc-5d70cac6d2ad",
"55003a29-f97f-4b22-8990-c8c5989a293d",
"4517c146-da09-4c8c-a9bc-ce8613f52f83",
"a0c8a6f3-037c-4957-b48a-415508e57fac",
"375e3d1a-007b-4393-8b05-c5ee7a7a6e15",
"d31fc2ee-1ca6-4ffa-982d-f914481aa62c",
"ed8ca7d4-12f7-42a8-a211-0b6788bef0fd",
"55912648-c1f4-4e93-a12d-1ef206bc28f8",
"71d77869-3266-4421-aaa1-b0efc9b9f19c"
]
},
"8c3863c1-d144-4baf-8b8b-167eb0c83aae": {
},
"size": {
"width": 490,
"height": 120
},
"position": {
"x": 0,
"y": -110
},
"z": 1,
"parent": "c733e469-afeb-4ccb-b0c1-f6c4125295f8",
"embeds": [
"10172244-abe9-49d4-923b-597566a0f720"
]
},
"01c6050d-1dc2-40e2-ace6-9c595b881719": {
"size": {
"width": 490,
"height": 120
},

…

None of that information is going to be used in creating your stack, so I have decided to clean up my template from all this Metadata, and continue building the stack in Eclipse.

I have decided to use the following sections: Format version, Description, Parameters, Mappings, and Resources. Here is the final CF stack, please note that it’s missing the RDS database, and security groups.

{
  "AWSTemplateFormatVersion": "2010-09-09",
  
  "Description" : "AWS CloudFormation template for a VPC with one public subnet and six private subnets for running a Big Data ecosystem. The following instances will be deployed in each subnet with different tasks: OpenVpn for authentication, Nagios or Kabana for monitoring, Apache Nifi for ETL, RDS MySql for database storage, Cloudera for BigData, OpenR or Revo for analytics,  Qlik or Tabelaux for visualization", 
 
  "Parameters" : {
    "InstanceType" : {
      "Description" : " Lab instances are t1/t2.micro, or t1.small",
      "Type" : "String",
      "Default" : "t2.micro",
      "AllowedValues" : ["t1.micro","t2.micro","t2.small"]
    },
    
    "ClouderaInstanceType" : {
      "Description" : " Lab instances are t1/t2.micro, or t1.small",
      "Type" : "String",
      "Default" : "t1.micro",
      "AllowedValues" : ["t1.micro","t2.micro","t2.small"]
    },
    
    "KeyName" : {
      "Description" : "Name of an existing EC2 keyPair to enable SSH access to the instance",
      "Type": "AWS::EC2::KeyPair::KeyName",
      "ConstraintDescription" : "Must be the name of an existing Key pair"
       },
       
      "CFKeyName" : {
      "Description" : "Name of an existing EC2 keyPair to enable SSH access to the instance",
      "Type": "AWS::EC2::KeyPair::KeyName",
      "ConstraintDescription" : "Must be the name of an existing Key pair"
       },
      
    "SSHLocation": {
      "Description": "The IP address range that can be used to SSH to the EC2 instances",
      "Type": "String",
      "MinLength": "9",
      "MaxLength": "18",
      "Default": "0.0.0.0/0",
   
      "ConstraintDescription": "must be a valid IP CIDR range of the form x.x.x.x/x."
    }
}, 
   
  "Mappings" : {
    
     "ImageMap" : {
      "us-east-1" : { "OpenVpnAMI" : "ami-bc3566ab", "MonitoringAMI" : "ami-b73b63a0","NiFiAMI" : "ami-b73b63a0", "ClouderaAMI" : "ami-20b6c437", "RstatAMI" : "ami-b73b63a0", "VisualAMI" : "ami-b73b63a0" },
      "us-east-2" :{ }
      }   
    }
 }, 
    
  "Resources": {
 
    "VPC": {
      "Type": "AWS::EC2::VPC",
      "Properties": {
        "CidrBlock" : "10.0.0.0/16",
        "EnableDnsSupport" : "true",
        "EnableDnsHostnames": "true",
        "InstanceTenancy": "default"
          }
          
         },
         
      "BasicSecurityGroup" : {
      "Type" : "AWS::EC2::SecurityGroup",
      "Properties" : {
        "VpcId" : { "Ref" : "VPC" },
        "GroupDescription" : "Enable SSH access",
        "SecurityGroupIngress" : [
          {"IpProtocol" : "tcp", "FromPort" : "22", "ToPort" : "22", "CidrIp" : { "Ref" : "SSHLocation"}}
        ]
      }
    },
      
      "PublicAuthentication": {
      "Type": "AWS::EC2::Subnet",
      "Properties": {
        "VpcId": {
          "Ref": "VPC"
        },
        "CidrBlock" : "10.0.0.0/24",
       "AvailabilityZone" : "us-east-1e"
      }
      
    }, 
  
    "PrivateDataLanding": {
      "Type": "AWS::EC2::Subnet",
      "Properties": {
        "VpcId": {
          "Ref": "VPC"
        },
       "CidrBlock" : "10.0.1.0/24",
       "AvailabilityZone" : "us-east-1c"
      }
     
    },
    "PrivateDatabase": {
      "Type": "AWS::EC2::Subnet",
      "Properties": {
        "VpcId": {
          "Ref": "VPC"
        },
        "CidrBlock" : "10.0.2.0/24",
        "AvailabilityZone" : "us-east-1d"
      }
      
    },
    
    "PrivateDatabase2": {
      "Type": "AWS::EC2::Subnet",
      "Properties": {
        "VpcId": {
          "Ref": "VPC"
        },
        "CidrBlock" : "10.0.3.0/24",
        "AvailabilityZone" : "us-east-1a"
      }
      
    },
    "PrivateDataLake": {
      "Type": "AWS::EC2::Subnet",
      "Properties": {
        "VpcId": {
          "Ref": "VPC"
        },
       "CidrBlock" : "10.0.4.0/24",
       "AvailabilityZone" : "us-east-1d"
      }
    },
    "PrivateAnalytics": {
      "Type": "AWS::EC2::Subnet",
      "Properties": {
        "VpcId": {
          "Ref": "VPC"
        },
         "CidrBlock" : "10.0.5.0/24",
       "AvailabilityZone" : "us-east-1d"
      }
     
    },
    "PrivateVisualization": {
      "Type": "AWS::EC2::Subnet",
      "Properties": {
        "VpcId": {
          "Ref": "VPC"
        },
        "CidrBlock" : "10.0.6.0/24",
       "AvailabilityZone" : "us-east-1a"
      }
     
    },
   
    "PrivateMonitoring": {
      "Type": "AWS::EC2::Subnet",
      "Properties": {
        "VpcId": {
          "Ref": "VPC"
        },
       "CidrBlock" : "10.0.7.0/24",
       "AvailabilityZone" : "us-east-1a"
      }
      
    },
    
      "InternetGateway" : {
      "Type" : "AWS::EC2::InternetGateway",
      "Properties" : {
        "Tags" : [ {"Key" : "Application", "Value" : { "Ref" : "AWS::StackId"} } ]
      }
    },

    "AttachGateway" : {
       "Type" : "AWS::EC2::VPCGatewayAttachment",
       "Properties" : {
         "VpcId" : { "Ref" : "VPC" },
         "InternetGatewayId" : { "Ref" : "InternetGateway" }
       }
    },
    
    "PublicRouteTable" : {
      "Type" : "AWS::EC2::RouteTable",
      "Properties" : {
        "VpcId" : {"Ref" : "VPC"},
        "Tags" : [ {"Key" : "Application", "Value" : { "Ref" : "AWS::StackId"} } ]
      }
    },

    "Route" : {
      "Type" : "AWS::EC2::Route",
      "DependsOn" : "AttachGateway",
      "Properties" : {
        "RouteTableId" : { "Ref" : "PublicRouteTable" },
        "DestinationCidrBlock" : "0.0.0.0/0",
        "GatewayId" : { "Ref" : "InternetGateway" }
      }
    },

    "SubnetRouteTableAssociation" : {
      "Type" : "AWS::EC2::SubnetRouteTableAssociation",
      "Properties" : {
        "SubnetId" : { "Ref" : "PublicAuthentication" },
        "RouteTableId" : { "Ref" : "PublicRouteTable" }
      }
    },

    "NetworkAcl" : {
      "Type" : "AWS::EC2::NetworkAcl",
      "Properties" : {
        "VpcId" : {"Ref" : "VPC"},
        "Tags" : [ {"Key" : "Application", "Value" : { "Ref" : "AWS::StackId"} } ]
      }
    },

    "InboundHTTPNetworkAclEntry" : {
      "Type" : "AWS::EC2::NetworkAclEntry",
      "Properties" : {
        "NetworkAclId" : {"Ref" : "NetworkAcl"},
        "RuleNumber" : "100",
        "Protocol" : "6",
        "RuleAction" : "allow",
        "Egress" : "false",
        "CidrBlock" : "0.0.0.0/0",
        "PortRange" : {"From" : "80", "To" : "80"}
      }
    },

    "InboundSSHNetworkAclEntry" : {
      "Type" : "AWS::EC2::NetworkAclEntry",
      "Properties" : {
        "NetworkAclId" : {"Ref" : "NetworkAcl"},
        "RuleNumber" : "101",
        "Protocol" : "6",
        "RuleAction" : "allow",
        "Egress" : "false",
        "CidrBlock" : "0.0.0.0/0",
        "PortRange" : {"From" : "22", "To" : "22"}
      }
    },

    "InboundResponsePortsNetworkAclEntry" : {
      "Type" : "AWS::EC2::NetworkAclEntry",
      "Properties" : {
        "NetworkAclId" : {"Ref" : "NetworkAcl"},
        "RuleNumber" : "102",
        "Protocol" : "6",
        "RuleAction" : "allow",
        "Egress" : "false",
        "CidrBlock" : "0.0.0.0/0",
        "PortRange" : {"From" : "1024", "To" : "65535"}
      }
    },

    "OutBoundHTTPNetworkAclEntry" : {
      "Type" : "AWS::EC2::NetworkAclEntry",
      "Properties" : {
        "NetworkAclId" : {"Ref" : "NetworkAcl"},
        "RuleNumber" : "100",
        "Protocol" : "6",
        "RuleAction" : "allow",
        "Egress" : "true",
        "CidrBlock" : "0.0.0.0/0",
        "PortRange" : {"From" : "80", "To" : "80"}
      }
    },

    "OutBoundHTTPSNetworkAclEntry" : {
      "Type" : "AWS::EC2::NetworkAclEntry",
      "Properties" : {
        "NetworkAclId" : {"Ref" : "NetworkAcl"},
        "RuleNumber" : "101",
        "Protocol" : "6",
        "RuleAction" : "allow",
        "Egress" : "true",
        "CidrBlock" : "0.0.0.0/0",
        "PortRange" : {"From" : "443", "To" : "443"}
      }
    },

    "OutBoundResponsePortsNetworkAclEntry" : {
      "Type" : "AWS::EC2::NetworkAclEntry",
      "Properties" : {
        "NetworkAclId" : {"Ref" : "NetworkAcl"},
        "RuleNumber" : "102",
        "Protocol" : "6",
        "RuleAction" : "allow",
        "Egress" : "true",
        "CidrBlock" : "0.0.0.0/0",
        "PortRange" : {"From" : "1024", "To" : "65535"}
      }
    },

    "SubnetNetworkAclAssociation" : {
      "Type" : "AWS::EC2::SubnetNetworkAclAssociation",
      "Properties" : {
        "SubnetId" : { "Ref" : "PublicAuthentication" },
        "NetworkAclId" : { "Ref" : "NetworkAcl" }
      }
    },

    
    "OpenVPNSFTP": {
      "Type": "AWS::EC2::Instance",
      "Properties": {
        
      "InstanceType" : {
        	"Ref" : "InstanceType"
         },
         
      "ImageId" : { "Fn::FindInMap" : [ "ImageMap", { "Ref" : "AWS::Region" }, "OpenVpnAMI" ]},
         
       "KeyName" : {
           "Ref" : "CFKeyName"
         },       
      "NetworkInterfaces": [ {
      "AssociatePublicIpAddress": "true",
      "DeviceIndex": "0",
      "GroupSet" : [ {"Ref" : "BasicSecurityGroup"} ],
     
      "SubnetId": { "Ref" : "PublicAuthentication" }
    } ]
    
      }
     
    },
    
    "NagiosOrKabana": {
      
      "Type": "AWS::EC2::Instance",
      
      "Properties": {
        
         "InstanceType" : {
        	"Ref" : "InstanceType"
         },
         
         "ImageId" : { "Fn::FindInMap" : [ "ImageMap", { "Ref" : "AWS::Region" }, "MonitoringAMI" ]},
         
         "KeyName" : {
           "Ref" : "KeyName"
         },         
      "NetworkInterfaces": [ {
      "AssociatePublicIpAddress": "false",
      "DeviceIndex": "0",
      "GroupSet" : [ {"Ref" : "BasicSecurityGroup"} ],
     
      "SubnetId": { "Ref" : "PrivateMonitoring" }
    } ]
        
      }
      
    },
    
    "ApacheNiFi": {
      "Type": "AWS::EC2::Instance",
      
      
      "Properties": {
          "ImageId" : { "Fn::FindInMap" : [ "ImageMap", { "Ref" : "AWS::Region" }, "NiFiAMI" ]},
          "InstanceType" : {
        	"Ref" : "InstanceType"
         },
         "KeyName" : {
           "Ref" : "KeyName"
         },        
       "NetworkInterfaces": [ {
      "AssociatePublicIpAddress": "false",
      "DeviceIndex": "0",
      "GroupSet" : [ {"Ref" : "BasicSecurityGroup"} ],
     
      "SubnetId": { "Ref" : "PrivateDataLanding" }
    } ]
        
      }
     
    },
    
     "Cloudera": {
      "Type": "AWS::EC2::Instance",
      "Properties": {
        
         "ImageId" : { "Fn::FindInMap" : [ "ImageMap", { "Ref" : "AWS::Region" }, "ClouderaAMI" ]},
          "InstanceType" : {
        	"Ref" : "ClouderaInstanceType"
         },
         "KeyName" : {
           "Ref" : "KeyName"
         },         
       "NetworkInterfaces": [ {
      "AssociatePublicIpAddress": "false",
      "DeviceIndex": "0",
     "GroupSet" : [ {"Ref" : "BasicSecurityGroup"} ],
      "SubnetId": { "Ref" : "PrivateDataLake" }
    } ]
        
      }
    
    },
    
    "OpenOrRevoR": {
      "Type": "AWS::EC2::Instance",
      "Properties": {
        
         "ImageId" : { "Fn::FindInMap" : [ "ImageMap", { "Ref" : "AWS::Region" }, "RstatAMI" ]},
          "InstanceType" : {
        	"Ref" : "InstanceType"
         },
         "KeyName" : {
           "Ref" : "KeyName"
         },       
       "NetworkInterfaces": [ {
      "AssociatePublicIpAddress": "false",
      "DeviceIndex": "0",
     "GroupSet" : [ {"Ref" : "BasicSecurityGroup"} ],
      "SubnetId": { "Ref" : "PrivateAnalytics" }
    } ]
       
      }
    
    },
    
    "QlikQlikviewTableau": {
      "Type": "AWS::EC2::Instance",
      "Properties": {
        
         "ImageId" : { "Fn::FindInMap" : [ "ImageMap", { "Ref" : "AWS::Region" }, "VisualAMI" ]},
          "InstanceType" : {
        	"Ref" : "InstanceType"
         },
         "KeyName" : {
           "Ref" : "KeyName"
         },       
       "NetworkInterfaces": [ {
      "AssociatePublicIpAddress": "false",
      "DeviceIndex": "0",
  	  "GroupSet" : [ {"Ref" : "BasicSecurityGroup"} ],
      "SubnetId": { "Ref" : "PrivateVisualization" }
    } ]
        
      }
    
    }
    
  }
}

Go ahead try and run this stack from your eclipse by right clicking on the page, click on “Run on AWS”, then click “Create stack”.

You can then hop on to your console to watch it in action as your VPC, subnets, and EC2 machines are being created.

Some tips before I conclude the article:

• There is no delete stack command in Eclipse, so I deleted mine using AWS CLI with the following command:” C:\Program Files\Amazon\AWSCLI> aws cloudformation delete-stack –stack-name SunTest2 “
• You can also delete the stack from the cloudformation web console. But it’s not recommended that you delete individual stack components manually. It also defeats the purpose of using CF.
• If you need to update anything in your stack, you can do it in Eclipse with the “Update Stack” command.
• Please restrict your Stack names to the following characters: [Az az 0-9], or your stack creation will fail.
• Resource Properties and Parameters are case sensitive, so Default is not the same as default.

That’s it for now, and stay tuned for follow up articles that go on more details about CF stack creation, updating, and troubleshooting.

AWS Cloud Formation (CF) is a service that allows Enterprises to manage their infrastructure as code. The CF templates are easy to read in both JSON, or the recently supported YAML data serialization language.

You can use your favorite JSON editor to create, and edit templates, and then upload them to your AWS CF Service to use in creating a stack.

Vi, Notepad++, Sublime, Atom, are some of the well known JSON editors, but if you need more than that, such as the ability to validate JSON syntax, as well as deploy your stack from the editor, then I would recommend Eclipse Java EE IDE with AWS toolkit.

Please follow these steps to get your IDE installed and configured:

1- Download Eclipse: Eclipse Neon 64 bit download

2- Choose the Eclipse Java EE IDE

3- Create your workspace, this is the directory where your projects are going to be saved.

4- Launch the Eclipse Marketplace, search for AWS, then install the AWS toolkit for Eclipse 2.0

5- Configure connection to your AWS account , by adding your IAM user Access key and secret access key:

You can retrieve the account keys from your IAM console by click in the account name, and click on “create access key” under the “Security Credentials “tab.

6- Your Eclipse for Java with AWS toolkit should be setup, and connected to your AWS account. You should be able to see any EC2 instances, EBS volumes, S3 buckets, and other resources you have configured.

But make sure you have the correct region selected first.

7- Click on Cloud Formation under the Java “src” folder, and you will see stacks that you have created.

Double click on your stack to open it.

If your stack doesn’t open the first time, close eclipse, reopen it, then try to launch your stack again.

If you have worked with Eclipse before, you would know about these quirks, and that every now and then, you will have to change your workspace.

8- Tips from Amazon AWS site:

• Only files that end in .template can be launched from the Eclipse IDE. If your file ends with another extension, such as .json, you will need to rename it first with a .template extension to use this feature.
• Right click on template editor, and click validate
• Your template will be validated for JSON correctness only; it will not be validated for CloudFormation correctness. A stack template validated in this way can still fail to launch or update.

9- If you don’t have any CF stacks created, you can create one by clicking on “New AWS Java Project”. (Click on the Amazon Icon on the tool bar to get the menu)

Choose AWS CF sample

9- Alternatively, you can start your template using CF builder in your AWS CF console, save it as a “.template” file, and drag and drop it under your cloud formation menu in Eclipse.

Once you create your stack with the CF designer in the AWS console, save it to your local storage, and then load it into your eclipse in order to add parameters to your resources, finalize it, and run it.

Eclipse is by no means perfect, since I think that the JSON validation is not good enough, and that it should also validate CF syntax. I am also trying to figure out, curly braces highlighting, so I know where is the closing brace without having to count. I checked the project’s GitHub repository, and someone has already put in a feature request for that.

I think it will just get better from here, I am impressed with what the AWS toolkit has to offer, and looking forward to learning more.

In the next blog, I will write about creating a CF template to deploy a stack from Eclipse.