Saturday, 2 March 2019

How to create a dynamic DNS with AWS Route 53

Have you ever wanted to host some pet project or even a small website on your own domestic network? If so, you must have stumbled across the DNS resolution issue: since we depend on our ISP to get our "real IP address", there's no guarantee that the IP we see today will stay any longer the same.

You could update your domain's DNS records every time you detect the IP has changed, but obviously that's tedious and error prone. That's when dynamic DNS (DDNS) comes in. Services like No-IP,Duck DNS,etc can be helpful.

In this post I'll go through the steps involved when it comes to setting up your own DDNS service when you own a domain that has been registered in AWS Route 53.

Prerequisites

Before running the commands I suggest here, we need a couple of things to set up. Let's start by installing the required packages if you haven't already installed them.

  • sudo apt-get install awscli
  • sudo apt-get install jq

Configure AWS credentials

$ aws configure
AWS Access Key ID [None]: AKIAIOSFODNN7EXAMPLE
AWS Secret Access Key [None]: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
Default region name [None]: eu-west-1
Default output format [None]: json

A hosted zone in AWS Route 53

$ aws route53 list-hosted-zones
{
    "HostedZones": [
        {
            "ResourceRecordSetCount": 5, 
            "CallerReference": " ... ", 
            "Config": {
                "Comment": "HostedZone created by Route53 Registrar", 
                "PrivateZone": false
            }, 
            "Id": "/hostedzone/Z2TLEXAMPLEZONE", 
            "Name": "abelperez.info."
        }
    ]
}

What's the plan?

The scenario I'm covering here is probably one of the most common. I want to create a new subdomain that points to my external IP and update that record as the external IP changes. To achieve that, we'll follow the steps:

  • Find out the external IP.
  • Get the desired hosted zone.
  • Create the A record (or update it if it already exists).
  • Set it up to run regularly (cron job).

Script step by step

Lest's start like any script, the shebang indicator. Then, input variables, in this case we define the domain name and the public record, notice the "." at the end of the public record, this is required by route 53.

#!/bin/bash
DOMAIN_NAME=abelperez.info
PUBLIC_RECORD=rpi.$DOMAIN_NAME.

Find the external IP, there are many ways to obtain this value, but since we're using AWS, let's get it from checkip endpoint. For debugging purposes we echo the IP found.

IP=$(curl -s http://checkip.amazonaws.com)
echo Found IP=$IP

Determine the hosted zone id, this step is optional if you want to use a hard-coded zone id that can be copied from the AWS Route 53 console. In this case I've invoked list-zones-by-name command which gives the hosted zone information for a specific domain, the format is as above in the prerequisites section. To extract the exact id I used a combination of jq and sed commands.

R53_HOSTED_ZONE=`aws route53 list-hosted-zones-by-name \
--dns-name $DOMAIN_NAME \
--query HostedZones \
| jq -r ".[] | select(.Name == \"$DOMAIN_NAME.\").Id" \
| sed 's/\/hostedzone\///'`

Now that we have all the required information, let's prepare the A record JSON input that will be used by aws route53 change-resource-record-sets command. the action "UPSERT" creates or updates the record accordingly, otherwise we'd need to manually check if it exists or not before updating it. Again for debugging purposes, we echo the final JSON.

read -r -d '' R53_ARECORD_JSON << EOM
{
  "Changes": [
    {
      "Action": "UPSERT",
      "ResourceRecordSet": {
        "Name": "$PUBLIC_RECORD",
        "Type": "A",
        "TTL": 300,
        "ResourceRecords": [
          {
            "Value": "$IP"
          }
        ]
      }
    }
  ]
}
EOM

echo About to execute change
echo "$R53_ARECORD_JSON" 

With the input ready, we invoke the Route 53 command to create/update the A record. This command will return immediately but the operation will take a few seconds to complete. If we want to make sure we know when it's completed, then we need to get the change id returned by the command, in this case, it's stored in the variable. This is optional, as well as the next step.

R53_ARECORD_ID=`aws route53 change-resource-record-sets \
--hosted-zone-id $R53_HOSTED_ZONE \
--change-batch "$R53_ARECORD_JSON" \
--query ChangeInfo.Id \
--output text`

echo Waiting for the change to update.

At this point, the request to create/update the A record is in progress, we could finish the script right now. However, I'd like to get a final confirmation that the operation has been completed. To do that, we can use the wait command providing the change id from the previous request.

aws route53 wait resource-record-sets-changed --id $R53_ARECORD_ID

echo Done.

And now it's actually completed. You should save this to a file, maybe name it update-dns.sh. It will need execute permission.

$ chmod u+x ./update-dns.sh

Set up cron job

In this particular instance I want this script to run in one of my raspberry pis, so I proceeded to copy the script file to pi user's home directory (/home/pi/)

pi@raspberrypi:~ $ ls
update-dns.sh

Now we'll set up a user cron job, we do that by running the command crontab -u followed by the user under which the job should run, this job doesn't require any system-wide privilege therefore it can run as the regular user, pi. -e to edit the file.

pi@raspberrypi:~ $ crontab -u pi -e

All we need to do is append the following text to the file content you are prompted with. The first two numbers correspond to minute and hour to run. For testing purposes I set it near the current time at the moment of testing it. The script output is appended/redirected to a text file so we can review afterwards if desired.

23 22 * * * /home/pi/update-dns.sh >> /home/pi/cron-update-dns.txt

See it in action

Once we've saved the crontab file, if it's the time where the cron job is about to start, we can test it by running tail -f command

pi@raspberrypi:~ $ tail -f cron-update-dns.txt

Finally, don't forget to update the port forwarding section on your home router so your open port directs traffic to a specific device, in my case, to that particular raspberry pi.

Sunday, 17 February 2019

How to Install docker on raspbian stretch

As part of one of my recent experiments I wanted to install docker on my raspberry pi. The procedure nowadays seems significantly better than a couple of years ago when I first tried. However, there are always some little details that can be a bit frustrating when following steps.

Although the process to install docker is well described in their website, it's focused on Debian distribution. We all know that Raspbian is based on Debian and therefore most of the instructions for will apply without much trouble.

That said, I will detail the exact steps I followed to get docker installed on my raspberry pi, this procedure was tested on 3B, Zero and Zero W.

First, update the package list, if you haven't done yet after setting up your raspberry pi.

$ sudo apt-get update

Install packages to allow apt to use a repository over HTTPS.

$ sudo apt-get install \
apt-transport-https \
ca-certificates \
curl \
gnupg2 \
software-properties-common

Add Docker’s official GPG key. Note that here the url points to raspbian directory and not to debian as per the original instructions.

curl -fsSL https://download.docker.com/linux/raspbian/gpg | sudo apt-key add -

The next step recommended didn't work for me.

sudo add-apt-repository \
   "deb [arch=armhf] https://download.docker.com/linux/raspbian \
   $(lsb_release -cs) \
   stable"

It failed with an error similar to this:

Traceback (most recent call last):
File "/usr/bin/add-apt-repository", line 95, in 
  sp = SoftwareProperties(options=options)
File "/usr/lib/python3/dist-packages/softwareproperties/SoftwareProperties.py"
, line 109, in __init__
  self.reload_sourceslist()
File "/usr/lib/python3/dist-packages/softwareproperties/SoftwareProperties.py"
, line 599, in reload_sourceslist
  self.distro.get_sources(self.sourceslist)
File "/usr/lib/python3/dist-packages/aptsources/distro.py", line 89, 
in get_sources (self.id, self.codename))
aptsources.distro.NoDistroTemplateException: Error: could not find a 
distribution template for Raspbian/stretch

Instead, I tried this other way, more explicitly setting up the repository into a new docker.list sources file.

$echo "deb [arch=armhf] https://download.docker.com/linux/raspbian $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list

With the new source repository in the list, update the package list again.

$ sudo apt-get update

Now we are in a position to install docker-ce from the repository, in this particular case, I ran into an issue with the latest version, documented in github basically they suggest to go back to 18.06.1 which we can get from the command below as per official docker instructions on how to get a specific version.

$ apt-cache madison docker-ce
docker-ce | 5:18.09.0~3-0~raspbian-stretch | https://download.docker.com/linux/raspbian stretch/stable armhf Packages
docker-ce | 18.06.2~ce~3-0~raspbian | https://download.docker.com/linux/raspbian stretch/stable armhf Packages
docker-ce | 18.06.1~ce~3-0~raspbian | https://download.docker.com/linux/raspbian stretch/stable armhf Packages
docker-ce | 18.06.0~ce~3-0~raspbian | https://download.docker.com/linux/raspbian stretch/stable armhf Packages

As of this writing, the version I tried is 18.06.2~ce~3-0~raspbian, which can be installed using the apt-get command.

$ sudo apt-get install docker-ce=18.06.2~ce~3-0~raspbian containerd.io

Also notice that by using this vesion, there's no need to install docker-ce-cli.

Testing everything is running as expected.

$ sudo docker info
Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 1
Server Version: 18.06.2-ce
 --- more data --- 

Run the hello world. Usually we get the command sudo docker run hello-world but that only works on "normal" architectures, in this case, we're using ARMv6/7, but to make it more compatible with all of them, I ran the image from arm32v5/hello-world.

$ sudo docker run arm32v5/hello-world
Unable to find image 'arm32v5/hello-world:latest' locally
latest: Pulling from arm32v5/hello-world
590e13f69e4a: Pull complete 
Digest: sha256:8a6a26a494c03e91381161abe924a39a2ff72de13189edcc2ed1695e6be12a5f
Status: Downloaded newer image for arm32v5/hello-world:latest

Hello from Docker!
This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:
 1. The Docker client contacted the Docker daemon.
 2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
    (arm32v5)
 3. The Docker daemon created a new container from that image which runs the
    executable that produces the output you are currently reading.
 4. The Docker daemon streamed that output to the Docker client, which sent it
    to your terminal.

To try something more ambitious, you can run an Ubuntu container with:
 $ docker run -it ubuntu bash

Share images, automate workflows, and more with a free Docker ID:
 https://hub.docker.com/

For more examples and ideas, visit:
 https://docs.docker.com/get-started/

And that's all, docker installed up and running on a raspberry pi.

UPDATE:

If you'd like to get rid of the sudo command every time you use docker, then you can run the following command found in the official guide as a post-installation step. You don't need to create the group as it's already created by docker installation. It adds your user to the docker group.

$ sudo usermod -aG docker $USER

You need to log out and back in for this change to take effect. After that you'll be able to run all docker commands as a regular user.

Thursday, 7 February 2019

Querying Aurora serverless database remotely using Lambda - part 3

This post is part of a series

In the previous part, we've set up the Aurora MySql cluster. At this point we can start creating the client code to allow querying.

The Lambda code

In this example I'll be using .NET Core 2.1 as Lambda runtime and C# as programming language. The code is very simple and should be easy to port to your favourite runtime/language.

Lambda Input

The input to my function consists of two main pieces of information: database connection information and the query to execute.

    public class ConnectionInfo
    {
        public string DbUser { get; set; }
        public string DbPassword { get; set; }
        public string DbName { get; set; }
        public string DbHost { get; set; }
        public int DbPort { get; set; }
    }

    public class LambdaInput
    {
        public ConnectionInfo Connection { get; set; }

        public string QueryText { get; set; }
    }

Lambda Code

The function itself returns a List of dictionary where each item of the list represents a "record" from the query result, these are in a key/value form where key is the "field" name and the value is the what comes form the query.

    public List<Dictionary<string, object>> RunQueryHandler(LambdaInput input, ILambdaContext context)
    {
        var cxnString = GetCxnString(input.Connection);
        var query = input.QueryText;

        var result = new List<Dictionary<string, object>>();
        using (var conn = new MySql.Data.MySqlClient.MySqlConnection(cxnString))
        {
            var cmd = GetCommand(conn, query);
            var reader = cmd.ExecuteReader();

            var columns = new List<string>();

            for (int i = 0; i < reader.FieldCount; i++)
            {
                columns.Add(reader.GetName(i));
            }

            while (reader.Read())
            {
                var record = new Dictionary<string, object>();
                foreach (var column in columns)
                {
                    record.Add(column, reader[column]);
                }
                result.Add(record);
            }
        }
        return result;
    }

Support methods

Here is the code of the missing methods: GetCxnString and GetCommand not really complicated.

    private static readonly string cxnStringFormat = "server={0};uid={1};pwd={2};database={3};Connection Timeout=60";

    private string GetCxnString(ConnectionInfo cxn)
    {
        return string.Format(cxnStringFormat, cxn.DbHost, cxn.DbUser, cxn.DbPassword, cxn.DbName);
    }

    private static MySqlCommand GetCommand(MySqlConnection conn, string query)
    {
        conn.Open();
        var cmd = conn.CreateCommand();
        cmd.CommandText = query;
        cmd.CommandType = CommandType.Text;
        return cmd;
    }

Project file

Before compiling and packaging the code we need a project file, assuming you don't have one already, this is how it looks like to be able to run in AWS Lambda environment.

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <TargetFramework>netcoreapp2.1</TargetFramework>
    <GenerateRuntimeConfigurationFiles>true</GenerateRuntimeConfigurationFiles>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Amazon.Lambda.Core" Version="1.0.0" />
    <PackageReference Include="Amazon.Lambda.Serialization.Json" Version="1.3.0" />
    <PackageReference Include="MySql.Data" Version="8.0.13" />
    <PackageReference Include="Newtonsoft.Json" Version="11.0.2" />
  </ItemGroup>

  <ItemGroup>
    <DotNetCliToolReference Include="Amazon.Lambda.Tools" Version="2.2.0" />
  </ItemGroup>


Preparing Lambda package

Assuming you have both the code and csproj file in the current directory, we just run dotnet lambda package command as per below, where -c sets the Configuration to release, -f sets the target framework to netcoreapp2.1 and -o sets the output zip file name.

$ dotnet lambda package -c release -f netcoreapp2.1 -o aurora-lambda.zip
Amazon Lambda Tools for .NET Core applications (2.2.0)
Project Home: https://github.com/aws/aws-extensions-for-dotnet-cli, https://github.com/aws/aws-lambda-dotnet

Executing publish command
Deleted previous publish folder
... invoking 'dotnet publish', working folder '/home/abel/Downloads/aurora_cluster_sample/bin/release/netcoreapp2.1/publish'

( ... ) --- removed code for brevity ---

... zipping:   adding: aurora.lambda.deps.json (deflated 76%)
Created publish archive (/home/abel/Downloads/aurora_cluster_sample/aurora-lambda.zip).
Lambda project successfully packaged: /home/abel/Downloads/aurora_cluster_sample/aurora-lambda.zip

Next, we upload the resulting zip file to an S3 bucket of our choice. In this example I'm using a bucket named abelperez-temp and I'm uploading the zip file to a folder named aurora-lambda so I keep some form of organisation in my file directory.

$ aws s3 cp aurora-lambda.zip s3://abelperez-temp/aurora-lambda/
upload: ./aurora-lambda.zip to s3://abelperez-temp/aurora-lambda/aurora-lambda.zip

Lambda stack

To create the Lambda function, I've put together a CloudFormation template that includes:

  • AWS::EC2::SecurityGroup contains outbound traffic rule to allow port 3306
  • AWS::IAM::Role contains an IAM role to allow the Lambda function to write to CloudWatch Logs and interact with ENIs
  • AWS::Lambda::Function contains the function definition

Here is the full template, the required parameters are VpcId, SubnetIds and LambdaS3Bucket which we should get from previous stacks' outputs. The template outputs the function full name, which we'll need to be able to invoke it later.

Special attention to the Lambda function definition, the property Handler, in .NET runtime is in the form of AssemblyName::Namespace.ClassName::MethodName and the property Code containing the S3 location of the zip file we uploaded earlier.

Description: Template to create a lambda function 

Parameters: 
  LambdaS3Bucket:
    Type: String
  DbClusterPort: 
    Type: Number
    Default: 3306
  VpcId: 
    Type: String
  SubnetIds: 
    Type: CommaDelimitedList

Resources:
  LambdaSg:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: Allow outbound traffic to MySQL host
      VpcId:
        Ref: VpcId
      SecurityGroupEgress:
        - IpProtocol: tcp
          FromPort: !Ref DbClusterPort
          ToPort: !Ref DbClusterPort
          CidrIp: 0.0.0.0/0

  AWSLambdaExecutionRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Principal:
              Service:
                - lambda.amazonaws.com
            Action: sts:AssumeRole
      Path: /
      Policies:
        - PolicyName: PermitLambda
          PolicyDocument:
            Version: 2012-10-17
            Statement:
            - Effect: Allow
              Action:
              - logs:CreateLogGroup
              - logs:CreateLogStream
              - logs:PutLogEvents
              - ec2:CreateNetworkInterface
              - ec2:DescribeNetworkInterfaces
              - ec2:DeleteNetworkInterface
              Resource: 
                - "arn:aws:logs:*:*:*"
                - "*"
  HelloLambda:
    Type: AWS::Lambda::Function
    Properties:
      Handler: aurora.lambda::project.lambda.Function::RunQueryHandler
      Role: !GetAtt AWSLambdaExecutionRole.Arn
      Code:
        S3Bucket: !Ref LambdaS3Bucket
        S3Key: aurora-lambda/aurora-lambda.zip
      Runtime: dotnetcore2.1
      Timeout: 30
      VpcConfig:
        SecurityGroupIds:
          - !Ref LambdaSg
        SubnetIds: !Ref SubnetIds

Outputs:
  LambdaFunction:
    Value: !Ref HelloLambda

To deploy this stack we use the following command where we pass the parameters specific to our VPC (VpcId and SubnetIds) as well as the S3 bucket name.

$ aws cloudformation deploy --stack-name fn-stack \
--template-file aurora_lambda_template.yml \
--parameter-overrides VpcId=vpc-0b442e5d98841996c SubnetIds=subnet-013d0bbb3eca284a2,subnet-00c67cfed3ab0a791 LambdaS3Bucket=abelperez-temp \
--capabilities CAPABILITY_IAM

Waiting for changeset to be created..
Waiting for stack create/update to complete
Successfully created/updated stack - fn-stack

Let's get the outputs as we'll need this information later. We have the Lambda function full name.

$ aws cloudformation describe-stacks --stack-name fn-stack --query Stacks[*].Outputs
[
    [
        {
            "OutputKey": "LambdaFunction",
            "OutputValue": "fn-stack-HelloLambda-C32KDMYICP5W"
        }
    ]
]

Invoking Lambda function

Now that we have deployed the function and we know its full name, we can invoke it by using dotnet lambda invoke-function command. Part of this job is to prepare the payload which is a JSON in put corresponding to the Lambda input defined above.

{
    "Connection": {
        "DbUser": "master", 
        "DbPassword": "Aurora.2019", 
        "DbName": "dbtest", 
        "DbHost": "db-stack-auroramysqlcluster-xxx.rds.amazonaws.com", 
        "DbPort": 3306
    }, 
    "QueryText":"show databases;"
}

Here is the command to invoke the Lambda function, including the payload parameter encoded to escape the quotes and all in a single line. There are better ways to do this, but for the sake of this demonstration, it's good enough.

$ dotnet lambda invoke-function \
--function-name fn-stack-HelloLambda-C32KDMYICP5W \
--payload "{ \"Connection\": {\"DbUser\": \"master\", \"DbPassword\": \"Aurora.2019\", \"DbName\": \"dbtest\", \"DbHost\": \"db-stack-auroramysqlcluster-xxx.rds.amazonaws.com\", \"DbPort\": 3306}, \"QueryText\":\"show databases;\" }" \
--region eu-west-1

Amazon Lambda Tools for .NET Core applications (2.2.0)
Project Home: https://github.com/aws/aws-extensions-for-dotnet-cli, https://github.com/aws/aws-lambda-dotnet

Payload:
[{"Database":"information_schema"},{"Database":"dbtest"},{"Database":"mysql"},{"Database":"performance_schema"}]

Log Tail:
START RequestId: 595944b5-73bb-4536-be92-a42652125ba8 Version: $LATEST
END RequestId: 595944b5-73bb-4536-be92-a42652125ba8
REPORT RequestId: 595944b5-73bb-4536-be92-a42652125ba8  Duration: 11188.62 ms   Billed Duration: 11200 ms       Memory Size: 128 MB     Max Memory Used: 37 MB

Now we can see the output in the Payload section. And that's how we can query remotely any Aurora serverless cluster without having to set up any EC2 instance. This could be extended to handle different SQL operations such as Create, Insert, Delete, etc.

Monday, 4 February 2019

Querying Aurora serverless database remotely using Lambda - part 2

This post is part of a series

In the previous part, we've set up the base layer to deploy our resources. At this point we can create the database cluster.

Aurora DB Cluster

Assuming we have our VPC ready with at least two subnets to comply with high availability best practices, let's create our cluster, I've put together a CloudFormation template that includes:

  • AWS::EC2::SecurityGroup contains inbound traffic rule to allow port 3306
  • AWS::RDS::DBSubnetGroup contains a group of subnets to deploy the cluster
  • AWS::EC2::DBCluster contains all the parameters to create the database cluster

Here is the full template, the only required parameters are VpcId and SubnetIds, but feel free to override any of the database cluster parameters such as database name, user name, password, etc. The template outputs the IDs corresponding to newly created resources such as the database cluster DNS endpoint, port and the security group.

Description: Template to create a serverless aurora mysql cluster

Parameters: 
  DbClusterDatabaseName: 
    Type: String
    Default: dbtest
  DbClusterIdentifier: 
    Type: String
    Default: serverless-mysql-aurora
  DbClusterParameterGroup: 
    Type: String
    Default: default.aurora5.6
  DbClusterMasterUsername: 
    Type: String
    Default: master
  DbClusterMasterPassword: 
    Type: String
    Default: Aurora.2019
  DbClusterPort: 
    Type: Number
    Default: 3306
  VpcId: 
    Type: String
  SubnetIds: 
    Type: CommaDelimitedList

Resources:
  DbClusterSg:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: Allow MySQL port to client host
      VpcId:
        Ref: VpcId
      SecurityGroupIngress:
        - IpProtocol: tcp
          FromPort: !Ref DbClusterPort
          ToPort: !Ref DbClusterPort
          CidrIp: 0.0.0.0/0

  DbSubnetGroup: 
    Type: "AWS::RDS::DBSubnetGroup"
    Properties: 
      DBSubnetGroupDescription: "aurora subnets"
      SubnetIds: !Ref SubnetIds

  AuroraMysqlCluster:
    Type: AWS::RDS::DBCluster
    Properties:
      DatabaseName:
        Ref: DbClusterDatabaseName
      DBClusterParameterGroupName:
        Ref: DbClusterParameterGroup
      DBSubnetGroupName:
        Ref: DbSubnetGroup
      Engine: aurora
      EngineMode: serverless
      MasterUsername:
        Ref: DbClusterMasterUsername
      MasterUserPassword:
        Ref: DbClusterMasterPassword
      ScalingConfiguration:
        AutoPause: true
        MinCapacity: 2
        MaxCapacity: 4
        SecondsUntilAutoPause: 1800
      VpcSecurityGroupIds:
        - !Ref DbClusterSg
        
Outputs:
  DbClusterEndpointAddress:
    Value: !GetAtt AuroraMysqlCluster.Endpoint.Address
  DbClusterEndpointPort:
    Value: !GetAtt AuroraMysqlCluster.Endpoint.Port
  DbClusterSgId:
    Value: !Ref DbClusterSg

To deploy this stack we use the following command where we pass the parameters specific to our VPC (VpcId and SubnetIds).

$ aws cloudformation deploy --stack-name db-stack \
--template-file aurora_cluster_template.yml \
--parameter-overrides VpcId=vpc-0b442e5d98841996c SubnetIds=subnet-013d0bbb3eca284a2,subnet-00c67cfed3ab0a791

Waiting for changeset to be created..
Waiting for stack create/update to complete
Successfully created/updated stack - db-stack

Let's get the outputs as we'll need this information later. We have the cluster endpoint DNS name and the port as per our definition.

$ aws cloudformation describe-stacks --stack-name db-stack --query Stacks[*].Outputs
[
    [
        {
            "OutputKey": "DbClusterEndpointAddress",
            "OutputValue": "db-stack-auroramysqlcluster-1d1udg4ringe4.cluster-cnfxlauucwwi.eu-west-1.rds.amazonaws.com"
        },
        {
            "OutputKey": "DbClusterSgId",
            "OutputValue": "sg-072bbf2078caa0f46"
        },
        {
            "OutputKey": "DbClusterEndpointPort",
            "OutputValue": "3306"
        }
    ]
]

In the next part, we'll create the Lambda function to query this database remotely.

Wednesday, 30 January 2019

Querying Aurora serverless database remotely using Lambda - part 1

This post is part of a series

Why aurora serverless?

For those who like experimenting with new technology, AWS feeds us with a lot of new stuff every year at re:Invent conference, and many more times. In August 2018, Amazon announced its general availability. I was intrigued by this totally new way to doing database, so I started to play with it.

The first road block I found was connectivity from my local environment. I wanted to connect to my new cluster using my traditional MySQL Workbench client. It turned out to be one of the limitations clearly explained by Amazon, and as of today:

You can't give an Aurora Serverless DB cluster a public IP address. You can access an Aurora Serverless DB cluster only from within a virtual private cloud (VPC) based on the Amazon VPC service.

Most common workarounds involve the use of an EC2 instance to either run a MySQL client from there or SSH tunnel and allow connections from outside of the VPC. In both cases we'll be charged for the use of the EC2 instace. At the moment of this writing there is a solution for that, but still in beta: Data API.

With all that said, I decided to explore my own way in the meantime by creating a serverless approach involving Lambda to query my database.

Setting up the base - VPC

First, we need a VPC. For testing only, it doesn't really matter if we just use the default VPC in the current region. But if you'd like to get started with a blue print, there is this public CloudFormation template, it contains a sample VPC with two public subnets and two private subnets as well as all the required resources to guarantee connectivity (IGW, NAT, Route tables, etc.).

I prefer to isolate my experiments from the rest of the resources, that's why I came up with a template containing the bare minimum to get a VPC up and running. It only includes:

  • AWS::EC2::VPC the VPC itself - default CIDR block 10.192.0.0/16
  • AWS::EC2::Subnet private subnet 1 - default CIDR block 10.192.20.0/24
  • AWS::EC2::Subnet private subnet 2 - default CIDR block 10.192.21.0/24

Here is the full template, the only required parameter is EnvironmentName, but feel free to override any of the CIDR blocks. The template outputs the IDs corresponding to newly created resources such as the VPC and the subnets.

Description: >-
  This template deploys a VPC, with two private subnets spread across 
  two Availability Zones. This VPC does not provide any internet 
  connectivity resources such as IGW, NAT Gw, etc.

Parameters:
  EnvironmentName:
    Description: >-
      An environment name that will be prefixed to resource names
    Type: String

  VpcCIDR:
    Description: >-
      Please enter the IP range (CIDR notation) for this VPC
    Type: String
    Default: 10.192.0.0/16

  PrivateSubnet1CIDR:
    Description: >-
      Please enter the IP range (CIDR notation) for the private subnet 
      in the first Availability Zone
    Type: String
    Default: 10.192.20.0/24

  PrivateSubnet2CIDR:
    Description: >-
      Please enter the IP range (CIDR notation) for the private subnet 
      in the second Availability Zone
    Type: String
    Default: 10.192.21.0/24

Resources:
  VPC:
    Type: AWS::EC2::VPC
    Properties:
      CidrBlock: !Ref VpcCIDR
      EnableDnsSupport: true
      EnableDnsHostnames: true
      Tags:
        - Key: Name
          Value: !Ref EnvironmentName

  PrivateSubnet1:
    Type: AWS::EC2::Subnet
    Properties:
      VpcId: !Ref VPC
      AvailabilityZone: !Select [ 0, !GetAZs '' ]
      CidrBlock: !Ref PrivateSubnet1CIDR
      MapPublicIpOnLaunch: false
      Tags:
        - Key: Name
          Value: !Sub ${EnvironmentName} Private Subnet (AZ1)

  PrivateSubnet2:
    Type: AWS::EC2::Subnet
    Properties:
      VpcId: !Ref VPC
      AvailabilityZone: !Select [ 1, !GetAZs '' ]
      CidrBlock: !Ref PrivateSubnet2CIDR
      MapPublicIpOnLaunch: false
      Tags:
        - Key: Name
          Value: !Sub ${EnvironmentName} Private Subnet (AZ2)

Outputs:
  VPC:
    Description: A reference to the created VPC
    Value: !Ref VPC

  PrivateSubnet1:
    Description: A reference to the private subnet in the 1st Availability Zone
    Value: !Ref PrivateSubnet1

  PrivateSubnet2:
    Description: A reference to the private subnet in the 2nd Availability Zone
    Value: !Ref PrivateSubnet2

To create a stack from this template we run the following command (or go to AWS console and upload the template)

$ aws cloudformation deploy --stack-name vpc-stack \
--template-file vpc_template.yml \
--parameter-overrides EnvironmentName=Dev

Waiting for changeset to be created..
Waiting for stack create/update to complete
Successfully created/updated stack - vpc-stack

After successful stack creation, we can get the outputs as we'll need them for the next step.

$ aws cloudformation describe-stacks --stack-name vpc-stack --query Stacks[*].Outputs
[
    [
        {
            "Description": "A reference to the private subnet in the 1st Availability Zone",
            "OutputKey": "PrivateSubnet1",
            "OutputValue": "subnet-013d0bbb3eca284a2"
        },
        {
            "Description": "A reference to the private subnet in the 2nd Availability Zone",
            "OutputKey": "PrivateSubnet2",
            "OutputValue": "subnet-00c67cfed3ab0a791"
        },
        {
            "Description": "A reference to the created VPC",
            "OutputKey": "VPC",
            "OutputValue": "vpc-0b442e5d98841996c"
        }
    ]
]

In the next part, we'll create the Database cluster using these resources as a base layer.

Thursday, 27 September 2018

How to implement a CloudFormation Include custom tag using YamlDotNet

YAML has become a popular format used to describe a wide range of information in a more readable way. Depending on the concrete use case, these files can grow significantly. One example of this is AWS CloudFormation templates which can be written either in JSON or YAML.

Especially when working on Serverless projects, it doesn’t matter if it’s serverless framework, AWS SAM or just pure CloudFormation. The fact is the main file to maintain is (most of the time) a YAML template that grows over time.

I’d like to focus on a more concrete example: a typical serverless microservice architecture consists of a backend database, a function and a HTTP endpoint (i.e DynamoDB, Lambda and API Gateway). It’s also a common practice to use OpenAPI specification aka Swagger to describe the Web API interface.

In this case, when trying to use Lambda integration (either custom or proxy) we can’t use variables or intrinsic functions within the swagger file (we have to hardcode the lambda invocation url including account number and region, apart from the fact that function name might change if using autogenerated by CloudFormation), unless swagger content is inline in the Body property, which makes the file to grow by a great deal.

As of this writing, there are some strategies proposed by AWS to mitigate this problem, I personally find them a bit cumbersome to use on a daily basis:

  • AWS::CloudFormation::Stack It adds complexity to the whole process since we have to pass parameters to nested stacks and retrieve outputs from them in order to access information from both parties. The template for the nested stack must be stored on an Amazon S3 bucket, which adds friction to our development workflow while building.
  • AWS::Include Transform which is a form of macro hosted by AWS CloudFormation, is simpler than nested stacks but currently it still has some limiations:
    • The snippet has to be stored on an Amazon S3 bucket.
    • If the snippets change, your stack doesn't automatically pick up those changes.
    • It does not currently support using shorthand notations for YAML snippets.

Personally, I prefer a solution where at development time I can split the template in logical parts and then, before deploying it to AWS, compose them in one piece. I like the idea of include partials in specific parts of the parent file.

How to implement this include mechanism ?

YAML provides an extension mechanism named tags, where we can associate a particular data type with a tag (it’s basically a prefix added to a value). In YamlDotNet this is implemented by creating a custom type converter and mapping the new tag with the custom type converter.

IncludeTagConverter (custom type converter)

public class IncludeTagConverter: IYamlTypeConverter
{
    public bool Accepts(Type type)
    {
        return typeof(IncludeTag).IsAssignableFrom(type);
    }

    public object ReadYaml(IParser parser, Type type)
    {
        parser.Expect<MappingStart>();
        var key = parser.Expect<Scalar>();
        var val = parser.Expect<Scalar>();
        parser.Expect<MappingEnd>();

        if (key.Value != "File")
        {
            throw new YamlException(key.Start, val.End, "Expected a scalar named 'File'");
        }

        var input = File.ReadAllText(val.Value);
        var data = YamlSerializer.Deserialize(input);
        return data;
    }

    public void WriteYaml(IEmitter emitter, object value, Type type)
    {
    }
}

IncludeTag class

public class IncludeTag
{
    public string File { get; set; }
}

In this case we are indicating that IncludeTagConverter class should be used if the desiralization mechanism needs to deserialize an object of type IncludeTag. At the end of ReadYaml method, we call a helper class that starts deserialization process again with the content of the “included file”.

YamlSerializer helper class

public class YamlSerializer
{
    private const string IncludeTag = "!Include";

    public static object Deserialize(string yaml)
    {
        var reader = new StringReader(yaml);
        var deserializer = new DeserializerBuilder()
            .WithTypeConverter(new IncludeTagConverter())
            .WithTagMapping(IncludeTag, typeof(IncludeTag))
            .Build();
        var data = deserializer.Deserialize(reader);
        return data;
    }

    public static string Serialize(object data)
    {
        var serializer = new SerializerBuilder().Build();
        var yaml = serializer.Serialize(data);
        return yaml;
    }
}

In this helper class, we tell the deserializer that we are using a type converter and that it has to map !Include tags to data type of IncludeTag. This way, when it encounters an !Include in the yaml file, it will use our type converter to deserialize the content instead, which in turn, will read whatever file we put the name in File: key and will trigger the deserialization process again, thus, allowing us to execute this at several levels in a recursive way.

How do we compose a yaml ?

Once we have the whole file yaml object in memory, by triggering the deserialization process on the main yaml file, like this:

var data = YamlSerializer.Deserialize(input);

We only need to call Serialize again, and since we converted all the !Include tags into normal maps, sequence or scalars, there’s nothing extra we need to do to serialize it back using the default implementation.

var output = YamlSerializer.Serialize(data);

The output will be the composed file which can be saved and used after that.

Example:

Main yaml file (cloud-api.yaml)

Description: Template to create a serverless web api 

Resources:
  ApiGatewayRestApi:
    Type: AWS::ApiGateway::RestApi
    Properties:
      Name: Serverless API
      Description: Serverless API - Using CloudFormation and Swagger
      Body: !Include
        File: simple-swagger.yaml

simple-swagger.yaml file

swagger: "2.0"

info:
  version: 1.0.0
  title: Simple API
  description: A simple API to learn how to write OpenAPI Specification

paths:
  /persons: !Include
    File: persons.yaml
  /pets: !Include
    File: pets.yaml

persons.yaml file

get:
  summary: Gets some persons
  description: Returns a list containing all persons.
  responses:
    200:
      description: A list of Person
      schema:
        type: array
        items:
          required:
            - username
          properties:
            firstName:
              type: string
            lastName:
              type: string
            username:
              type: string

pets.yaml file

get:
  summary: Gets some pets
  description: Returns a list containing all pets.
  responses:
    200:
      description: A list of pets
      schema:
        type: array
        items:
          required:
            - petname
          properties:
            petname:
              type: string
            ownerName:
              type: string
            breed:
              type: string

Final result (composed file)

Description: Template to create a serverless web api
Resources:
  ApiGatewayRestApi:
    Type: AWS::ApiGateway::RestApi
    Properties:
      Name: Serverless API
      Description: Serverless API - Using CloudFormation and Swagger
      Body:
        swagger: 2.0
        info:
          version: 1.0.0
          title: Simple API
          description: A simple API to learn how to write OpenAPI Specification
        paths:
          /persons:
            get:
              summary: Gets some persons
              description: Returns a list containing all persons.
              responses:
                200:
                  description: A list of Person
                  schema:
                    type: array
                    items:
                      required:
                      - username
                      properties:
                        firstName:
                          type: string
                        lastName:
                          type: string
                        username:
                          type: string
          /pets:
            get:
              summary: Gets some pets
              description: Returns a list containing all pets.
              responses:
                200:
                  description: A list of pets
                  schema:
                    type: array
                    items:
                      required:
                      - petname
                      properties:
                        petname:
                          type: string
                        ownerName:
                          type: string
                        breed:
                          type: string

Thursday, 26 July 2018

How to push Windows and IIS logs to CloudWatch using unified CloudWatch Agent automatically

CloudWatch is a powerful monitoring and management tool, collects monitoring and operational data in the form of logs, metrics, and events, providing you with a unified view of AWS resources. One of the most common use cases is collecting logs from web applications.

Log files are generated locally in the form of text files and some running process monitor them and then decide where to send them. This is usually performed by the SSM Agent, however, as per AWS documents:

"Important The unified CloudWatch Agent has replaced SSM Agent as the tool for sending log data to Amazon CloudWatch Logs. Support for using SSM Agent to send log data will be deprecated in the near future. We recommend that you begin using the unified CloudWatch Agent for your log collection processes as soon as possible."

Assigning permissions to EC2 instances

EC2 instances need permission to access CloudWatch logs, if your current instances don’t have any roles associated, then create one with the CloudWatchAgentServerPolicy managed policy attached.

If your instances already have a role then you can add the policy to the existing role. In either case, the instance needs to perform operations such as CreateLogGroup, CreateLogStream, PutLogEvents and so on.

Install the CloudWatch Agent

On Windows Server, the installation process consists of three basic steps:

  1. Download the package from https://s3.amazonaws.com/amazoncloudwatch-agent/windows/amd64/latest/AmazonCloudWatchAgent.zip
  2. Unzip to a local folder
  3. Change directory to the folder containing unzipped package and run install.ps1

For more information about how to install the agent, see AWS documents.

Here is a powershell snippet to automate this process.

# Install the CloudWatch Agent
$zipfile = "AmazonCloudWatchAgent.zip"
$tempDir = Join-Path $env:TEMP "AmazonCloudWatchAgent"
Invoke-WebRequest -Uri "https://s3.amazonaws.com/amazoncloudwatch-agent/windows/amd64/latest/AmazonCloudWatchAgent.zip" -OutFile $zipfile
Expand-Archive -Path $zipfile -DestinationPath $tempDir -Force
cd $tempDir
Write-Host "Trying to uninstall any previous version of CloudWatch Agent"
.\uninstall.ps1

Write-Host "install the new version of CloudWatch Agent"
.\install.ps1

Creating configuration file

Before launching the agent, a configuration file is required, this configuration file can seem daunting at first, especially because it’s a different format from one used in SSM Agent. This configuration file contain three sections: agent, metrics and logs.

In this case, we are interested only in section logs which in turn has two main parts: windows_events (system or application events we can find in Windows Event Viewer) and files (any log files including IIS logs).

There are two common parameters required:

  • log_group_name - Used in CloudWatch to identify a log group, it should be something meaningful such as the event type or website name.
  • log_stream_name - Used in CloudWatch to identify a log stream within a log group, typically it’s a reference to the current EC2 instance.

Collecting Windows Events

Here is an example of a Windows Event log

{
    "event_levels": ["ERROR","INFORMATION"],
    "log_group_name": "/eventlog/application",
    "event_format": "text",
    "log_stream_name": "EC2AMAZ-NPQGPRK",
    "event_name": "Application"
}

Key points:

  • event_levels can be one or more of (INFORMATION, WARNING, ERROR, CRITICAL,VERBOSE).
  • event_name is typically one of (System, Security, Application)
  • event_format is text or xml.

Collecting IIS logs

Here is an example of an IIS website logs

{
    "log_group_name": "/iis/website1",
    "timezone": "UTC",
    "timestamp_format": "%Y-%m-%d %H:%M:%S",
    "encoding": "utf-8",
    "log_stream_name": "EC2AMAZ-NPQGPRK",
    "file_path": "C:\\inetpub\\logs\\LogFiles\\W3SVC2\\*.log"
}

Key points:

  • timezone and timestamp_format are optional.
  • encoding defaults to utf-8
  • file_path uses the standard Unix glob matching rules to match files, while all the examples in AWS docs display concrete log files, the example above matches all .log files within IIS logs folder, this is important since IIS create new files based on a rotation and we can’t predict their names.

These sections can be repeated for every website and for every Windows Event we’d like to push logs to CloudWatch. If we have several EC2 instances as web servers, this process can be tedious and error prone, therefore it should be automated. Here is an example of a powershell snippet.

$windowsLogs = @("Application", "System", "Security")
$windowsLoglevel = @("ERROR", "INFORMATION")
$instance = hostname

$iissites = Get-Website | Where-Object {$_.Name -ne "Default Web Site"}

$iislogs = @()
foreach ($site in $iissites) {
    $iislog = @{
        file_path = "$($site.logFile.directory)\w3svc$($site.id)\*.log"
        log_group_name = "/iis/$($site.Name.ToLower())"
        log_stream_name = $instance
        timestamp_format = "%Y-%m-%d %H:%M:%S"
        timezone = "UTC"
        encoding = "utf-8"
    }
    $iislogs += $iislog
}

$winlogs = @()
foreach ($event in $windowsLogs) {
    $winlog = @{
        event_name = $event
        event_levels = $windowsLoglevel
        event_format ="text"
        log_group_name = "/eventlog/$($event.ToLower())"
        log_stream_name = $instance
    }
    $winlogs += $winlog
}

$config = @{
    logs = @{
        logs_collected = @{
            files = @{
                collect_list = $iislogs
            }
            windows_events = @{
                collect_list = $winlogs
            }
        }
        log_stream_name = "generic-logs"
    }
}

# this could be any other location as long as it’s absolute
$configfile = "C:\Users\Administrator\amazon-cloudwatch-agent.json"

$json = $config | ConvertTo-Json -Depth 6 

# Encoding oem is important as the file is required without any BOM 
$json | Out-File -Force -Encoding oem $configfile

For more information on how to create this file, see AWS documents.

Starting the agent

With the configuration file in place, it’s time to start the agent, to do that, change directory to CloudWatch Agent installation path, typically within Program Files\Amazon\AmazonCloudWatchAgent and run the following command line:

.\amazon-cloudwatch-agent-ctl.ps1 -a fetch-config -m ec2 -c file:configuration-file-path -s 

Key points:

  • -a is short for -Action, fetch-config indicates it will reload configuration file.
  • -m is short for -Mode, in this case ec2 as opposed to onPrem.
  • -c is short for -ConfigLocation which is the configuration file previously generated.
  • -s is short for -Start which indicates to start the service after loading configuration.

Here is a powershell snippet covering this part of the process.

cd "${env:ProgramFiles}\Amazon\AmazonCloudWatchAgent"
Write-Host "Starting CloudWatch Agent"
.\amazon-cloudwatch-agent-ctl.ps1 -a fetch-config -m ec2 -c file:$configfile -s

Let’s test it.

Assuming we have 3 websites running in our test EC2 instance, let’s name them.

  • website1 - hostname: web1.local
  • website2 - hostname: web2.local
  • website3 - hostname: web3.local

After some browsing to generate some traffic, let’s inspect CloudWatch.

Some Windows Events also in CloudWatch Logs

Here is the complete powershell script.