Showing posts with label serverless. Show all posts
Showing posts with label serverless. Show all posts

Saturday, 11 April 2020

AWS HttpApi with Cognito as JWT Authorizer

With the recent release of HttpApi from AWS I've been playing with it for a bit and I wanted to see how far I can get it to use authorization without handling any logic in my application.

Creating a base code

Started with a simple base, let's set up the initial scenario which is no authentication at all. The architecture is the typical HttpApi -> Lambda, in this case, the Lambda content is irrelevant and therefore I've just used an inline code to test it's working.

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
  Sample SAM Template using HTTP API and Cognito Authorizer
Resources:

  # Dummy Lambda function 
  HttpApiTestFunction:
    Type: AWS::Serverless::Function
    Properties:
      InlineCode: |
        exports.handler = function(event, context, callback) {
          const response = {
            test: 'Hello HttpApi',
            claims: event.requestContext.authorizer && 
                    event.requestContext.authorizer.jwt.claims
          };
          callback(null, response);
        };
      Handler: index.handler
      Runtime: nodejs12.x
      Timeout: 30
      MemorySize: 256
      Events:
        GetOpen:
          Type: HttpApi
          Properties:
            Path: /test
            Method: GET
            ApiId: !Ref HttpApi
            Auth:
              Authorizer: NONE

  HttpApi:
    Type: AWS::Serverless::HttpApi
    Properties:
      CorsConfiguration: 
        AllowOrigins:
          - "*"

Outputs:
  HttpApiUrl:
    Description: URL of your API endpoint
    Value: !Sub 'https://${HttpApi}.execute-api.${AWS::Region}.${AWS::URLSuffix}/'
  HttpApiId:
    Description: Api id of HttpApi
    Value: !Ref HttpApi

With the outputs of this template we can hit the endpoint and see that in fact, it's accessible. So far, nothing crazy here.

$ curl https://abc1234.execute-api.us-east-1.amazonaws.com/test
{"test":"Hello HttpApi"}

Creating a Cognito UserPool and Client

The claims object is not populated because the request wasn't authenticated since no token was provided, as expected.

Let's create the Cognito UserPool with a very simple configuration assuming lots of default values since they're not relevant for this example.

  ## Add this fragment under Resources:

  # User pool - simple configuration 
  UserPool:
    Type: AWS::Cognito::UserPool
    Properties: 
      AdminCreateUserConfig: 
        AllowAdminCreateUserOnly: false
      AutoVerifiedAttributes: 
        - email
      MfaConfiguration: "OFF"
      Schema: 
        - AttributeDataType: String
          Mutable: true
          Name: name
          Required: true
        - AttributeDataType: String
          Mutable: true
          Name: email
          Required: true
      UsernameAttributes: 
        - email
  
  # User Pool client
  UserPoolClient:
    Type: AWS::Cognito::UserPoolClient
    Properties: 
      ClientName: AspNetAppLambdaClient
      ExplicitAuthFlows: 
        - ALLOW_USER_PASSWORD_AUTH
        - ALLOW_USER_SRP_AUTH
        - ALLOW_REFRESH_TOKEN_AUTH
      GenerateSecret: false
      PreventUserExistenceErrors: ENABLED
      RefreshTokenValidity: 30
      SupportedIdentityProviders: 
        - COGNITO
      UserPoolId: !Ref UserPool

  ## Add this fragment under Outputs:

  UserPoolId:
    Description: UserPool ID
    Value: !Ref UserPool
  UserPoolClientId:
    Description: UserPoolClient ID
    Value: !Ref UserPoolClient

Once we have the cognito UserPool and a client, we are in a position to start putting things together. But before there are a few things to clarify:

  • Username is the email and only two fields are required to create a user: name and email.
  • The client defines both ALLOW_USER_PASSWORD_AUTH and ALLOW_USER_SRP_AUTH Auth flows to be used by different client code.
  • No secret is generated for this client, if you intend to use other flows, you'll need to create other clients accordingly.

Adding authorization information

Next step is to add authorization information to the HttpApi.

  ## Replace the HttpApi resource with this one.

  HttpApi:
    Type: AWS::Serverless::HttpApi
    Properties:
      CorsConfiguration: 
        AllowOrigins:
          - "*"
      Auth:
        Authorizers:
          OpenIdAuthorizer:
            IdentitySource: $request.header.Authorization
            JwtConfiguration:
              audience:
                - !Ref UserPoolClient
              issuer: !Sub https://cognito-idp.${AWS::Region}.amazonaws.com/${UserPool}
        DefaultAuthorizer: OpenIdAuthorizer

We've added authorization information to the HttpApi where JWT issuer is the the Cognito UserPool previously created and the token are intended only for that client.

If we test again, nothing changes because the event associated with the lambda function says explictly "Authorizer: NONE".

To test this, we'll create a new event associated with the same lambda function but this time we'll add some authorization information to it.

        ## Add this fragment at the same level as GetOpen
        ## under Events as part of the function properties

        GetSecure:
          Type: HttpApi
          Properties:
            ApiId: !Ref HttpApi
            Method: GET
            Path: /secure
            Auth:
              Authorizer: OpenIdAuthorizer

If we test the new endpoint /secure, then we'll see the difference.

$ curl -v https://abc1234.execute-api.us-east-1.amazonaws.com/secure

>>>>>> removed for brevity >>>>>>
> GET /secure HTTP/1.1
> Host: abc1234.execute-api.us-east-1.amazonaws.com
> User-Agent: curl/7.52.1
> Accept: */*
> 
* Connection state changed (MAX_CONCURRENT_STREAMS updated)!
< HTTP/2 401 
< date: Sat, 11 Apr 2020 17:19:50 GMT
< content-length: 26
< www-authenticate: Bearer
< apigw-requestid: K1RYliB1IAMESNA=
< 
* Curl_http_done: called premature == 0
* Connection #0 to host abc1234.execute-api.us-east-1.amazonaws.com left intact

{"message":"Unauthorized"}

At this point we have a new endpoint that requires an access token. Now we need a token, but to get a token, we need a user first.

Fortunately Cognito can provide us with all we need in this case. Let's see how.

Creating a Cognito User

Cognito cli provides the commands to sign up and verify user accounts.

$ aws cognito-idp sign-up \
  --client-id asdfsdfgsdfgsdfgfghsdf \
  --username abel@example.com \
  --password Test.1234 \
  --user-attributes Name="email",Value="abel@example.com" Name="name",Value="Abel Perez" \
  --profile default \
  --region us-east-1

{
    "UserConfirmed": false, 
    "UserSub": "aaa30358-3c09-44ad-a2ec-5f7fca7yyy16", 
    "CodeDeliveryDetails": {
        "AttributeName": "email", 
        "Destination": "a***@e***.com", 
        "DeliveryMedium": "EMAIL"
    }
}

After creating the user, it needs to be verified.

$ aws cognito-idp admin-confirm-sign-up \
  --user-pool-id us-east-qewretry \
  --username abel@example.com \
  --profile default \
  --region us-east-1

This commands gives no output, to test we are good to go, let's use the admin-get-user command

$ aws cognito-idp admin-get-user \
  --user-pool-id us-east-qewretry \
  --username abel@example.com \
  --profile default \
  --region us-east-1 \
  --query UserStatus

"CONFIRMED"

We have a confirmed user!

Getting a token for the Cognito User

To obtain an Access Token, we use the Cognito initiate-auth command providing the client, username and password.

$ TOKEN=`aws cognito-idp initiate-auth \
  --client-id asdfsdfgsdfgsdfgfghsdf \
  --auth-flow USER_PASSWORD_AUTH \
  --auth-parameters USERNAME=abel@example.com,PASSWORD=Test.1234 \
  --profile default \
  --region us-east-1 \
  --query AuthenticationResult.AccessToken \
  --output text`

$ echo $TOKEN

With the access token in hand, it's time to test the endpoint with it.

$ curl -H "Authorization:Bearer $TOKEN" https://abc1234.execute-api.us-east-1.amazonaws.com/secure
# some formatting added here
{
    "test": "Hello HttpApi",
    "claims": {
        "auth_time": "1586627310",
        "client_id": "asdfsdfgsdfgsdfgfghsdf",
        "event_id": "94872b9d-e5cc-42f2-8e8f-1f8ad5c6e1fd",
        "exp": "1586630910",
        "iat": "1586627310",
        "iss": "https://cognito-idp.us-east-1.amazonaws.com/us-east-qewretry",
        "jti": "878b2acd-ddbd-4e68-b097-acf834291d09",
        "sub": "cce30358-3c09-44ad-a2ec-5f7fca7dbd16",
        "token_use": "access",
        "username": "cce30358-3c09-44ad-a2ec-5f7fca7dbd16"
    }
}

Voilà! We've accessed the secure endpoint with a valid access token.

What about groups ?

I wanted to know more about possible granular control of the authorization and I went and created two Cognito Groups let's say Group1 and Group2. Then, I added my newly created user to both groups and repeated the experiment.

Once the user was added to the groups, I got a new token and issued the request to the secure endpoint.

$ curl -H "Authorization:Bearer $TOKEN" https://abc1234.execute-api.us-east-1.amazonaws.com/secure
# some formatting added here
{
    "test": "Hello HttpApi",
    "claims": {
        "auth_time": "1586627951",
        "client_id": "2p9k1pfhtsbr17a2fukr5mqiiq",
        "cognito:groups": "[Group2 Group1]",
        "event_id": "c450ae9e-bd4e-4882-b085-5e44f8b4cefd",
        "exp": "1586631551",
        "iat": "1586627951",
        "iss": "https://cognito-idp.us-east-1.amazonaws.com/us-east-qewretry",
        "jti": "51a39fd9-98f9-4359-9214-000ea40b664e",
        "sub": "cce30358-3c09-44ad-a2ec-5f7fca7dbd16",
        "token_use": "access",
        "username": "cce30358-3c09-44ad-a2ec-5f7fca7dbd16"
    }
}

Notice within the claims object, a new one has come up: "cognito:groups" and the value associated with it is "[Group2 Group1]".

Which means that we could potentially check this claim value to make some decisions in our application logic without having to handle all of the authentication inside the application code base.

This opens the possibility for more exploration within the AWS ecosystem. I hope this has been helpful, the full source code can be found at https://github.com/abelperezok/http-api-cognito-jwt-authorizer.

Monday, 23 March 2020

AWS Serverless Web Application Architecture

Recently, I've been exploring ideas about how to put together different AWS services to achieve a totally serverless architecture for web applications. One of the new services is the HTTP API which simplifies the integration with Lambda.

One general principle I want to follow when designing these architectural models is the separation of three main subsystems:

  • Identity service to handle authentication and authorization
  • Static assets traffic being segregated
  • Dynamic page rendering and server side logic
  • Application configuration outside the code

All these component will be publicly accessible via Route 53 DNS record sets pointing to the relevant endpoints.

Other general ideas across all design diagrams below are:

  • Cognito will handle authentication and authorization
  • S3 will store all static assets
  • Lambda will execute server side logic
  • SSM Parameter Store will hold all configuration settings

Architecture variant 1 - HTTP API and CDN publicly exposed

In this first approach, we have the S3 bucket behind CloudFront which is a common pattern when creating CDN-like structures. CloudFront takes care of all the caching behaviours as well as distributing the cached versions all over the Edge locations, so subsequent requests will be dispatched at a reduced latency. Also, CloudFront has only one cache behaviour, which is the default and one origin which is the S3 bucket.

It's also important to notice the CloudFront distribution has an Alternative Domain Name set to the relevant record set e.g. media.example.com and let's not forget about referencing the ACM SSL certificate so we can use the custom url and not the random one from CloudFront.

From the HTTP API perspective, it has only one integration which is a Lambda integration on the $default route, which means all requests coming from the HTTP endpoint will be directed to the Lambda function in question.

Similar to the case of CloudFront, the HTTP API requires a Custom Domain and Certificate to be able to use a custom url as opposed to the random one given by the API service on creation.

Architecture variant 2 - HTTP API behind CloudFront

In this second approach, we still have the S3 bucket behind CloudFront following the same pattern. However we've placed the HTTP API also behind CloudFront.

CloudFront becomes the traffic controller in this case, where several cache behaviours can be defined to make the correct decision where to route the request to.

Both record sets (media and webapp) are pointing to the same CloudFront distribution, it's the application logic's responsibility to request all static assets using the appropriated domain nam.

Since the HTTP API is behind a CF distribution, I'd suggest to set it up as Regional endpoint.

Architecture variant 3 - No CloudFront at all

Continue playing with this idea, what if we don't use CloudFront distribution at all? I gave it a go and it turns out that it's possible to achieve similar results.

We can use two HTTP APIs and set one to forward traffic to S3 for static assets and the other one to Lambda as per the usual pattern, each of those with Custom Domain and that solves the problem.

But I wanted to push it a little bit further, this time I tried with only one HTTP API and setting several routes e.g "/css/*", "/js/*" integrates with S3 and any other integrates with Lambda, it's then, application logic's responsibility to request all static assets using the appropriated url

Conclusion

These are some ideas I've been experimenting with, the choice of including or not a CloudFront distribution is dependent on the concrete use case, whether the source of our requests is local or globally diverse. Also, whether it is more suitable to have static assets under a subdomain or virtual directory under the same host name.

Never underestimate the power and flexibility of an API Gateway, especially the new HTTP API where it can front any number of combination of resources in the back end.

Thursday, 7 February 2019

Querying Aurora serverless database remotely using Lambda - part 3

This post is part of a series

In the previous part, we've set up the Aurora MySql cluster. At this point we can start creating the client code to allow querying.

The Lambda code

In this example I'll be using .NET Core 2.1 as Lambda runtime and C# as programming language. The code is very simple and should be easy to port to your favourite runtime/language.

Lambda Input

The input to my function consists of two main pieces of information: database connection information and the query to execute.

    public class ConnectionInfo
    {
        public string DbUser { get; set; }
        public string DbPassword { get; set; }
        public string DbName { get; set; }
        public string DbHost { get; set; }
        public int DbPort { get; set; }
    }

    public class LambdaInput
    {
        public ConnectionInfo Connection { get; set; }

        public string QueryText { get; set; }
    }

Lambda Code

The function itself returns a List of dictionary where each item of the list represents a "record" from the query result, these are in a key/value form where key is the "field" name and the value is the what comes form the query.

    public List<Dictionary<string, object>> RunQueryHandler(LambdaInput input, ILambdaContext context)
    {
        var cxnString = GetCxnString(input.Connection);
        var query = input.QueryText;

        var result = new List<Dictionary<string, object>>();
        using (var conn = new MySql.Data.MySqlClient.MySqlConnection(cxnString))
        {
            var cmd = GetCommand(conn, query);
            var reader = cmd.ExecuteReader();

            var columns = new List<string>();

            for (int i = 0; i < reader.FieldCount; i++)
            {
                columns.Add(reader.GetName(i));
            }

            while (reader.Read())
            {
                var record = new Dictionary<string, object>();
                foreach (var column in columns)
                {
                    record.Add(column, reader[column]);
                }
                result.Add(record);
            }
        }
        return result;
    }

Support methods

Here is the code of the missing methods: GetCxnString and GetCommand not really complicated.

    private static readonly string cxnStringFormat = "server={0};uid={1};pwd={2};database={3};Connection Timeout=60";

    private string GetCxnString(ConnectionInfo cxn)
    {
        return string.Format(cxnStringFormat, cxn.DbHost, cxn.DbUser, cxn.DbPassword, cxn.DbName);
    }

    private static MySqlCommand GetCommand(MySqlConnection conn, string query)
    {
        conn.Open();
        var cmd = conn.CreateCommand();
        cmd.CommandText = query;
        cmd.CommandType = CommandType.Text;
        return cmd;
    }

Project file

Before compiling and packaging the code we need a project file, assuming you don't have one already, this is how it looks like to be able to run in AWS Lambda environment.

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <TargetFramework>netcoreapp2.1</TargetFramework>
    <GenerateRuntimeConfigurationFiles>true</GenerateRuntimeConfigurationFiles>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Amazon.Lambda.Core" Version="1.0.0" />
    <PackageReference Include="Amazon.Lambda.Serialization.Json" Version="1.3.0" />
    <PackageReference Include="MySql.Data" Version="8.0.13" />
    <PackageReference Include="Newtonsoft.Json" Version="11.0.2" />
  </ItemGroup>

  <ItemGroup>
    <DotNetCliToolReference Include="Amazon.Lambda.Tools" Version="2.2.0" />
  </ItemGroup>


Preparing Lambda package

Assuming you have both the code and csproj file in the current directory, we just run dotnet lambda package command as per below, where -c sets the Configuration to release, -f sets the target framework to netcoreapp2.1 and -o sets the output zip file name.

$ dotnet lambda package -c release -f netcoreapp2.1 -o aurora-lambda.zip
Amazon Lambda Tools for .NET Core applications (2.2.0)
Project Home: https://github.com/aws/aws-extensions-for-dotnet-cli, https://github.com/aws/aws-lambda-dotnet

Executing publish command
Deleted previous publish folder
... invoking 'dotnet publish', working folder '/home/abel/Downloads/aurora_cluster_sample/bin/release/netcoreapp2.1/publish'

( ... ) --- removed code for brevity ---

... zipping:   adding: aurora.lambda.deps.json (deflated 76%)
Created publish archive (/home/abel/Downloads/aurora_cluster_sample/aurora-lambda.zip).
Lambda project successfully packaged: /home/abel/Downloads/aurora_cluster_sample/aurora-lambda.zip

Next, we upload the resulting zip file to an S3 bucket of our choice. In this example I'm using a bucket named abelperez-temp and I'm uploading the zip file to a folder named aurora-lambda so I keep some form of organisation in my file directory.

$ aws s3 cp aurora-lambda.zip s3://abelperez-temp/aurora-lambda/
upload: ./aurora-lambda.zip to s3://abelperez-temp/aurora-lambda/aurora-lambda.zip

Lambda stack

To create the Lambda function, I've put together a CloudFormation template that includes:

  • AWS::EC2::SecurityGroup contains outbound traffic rule to allow port 3306
  • AWS::IAM::Role contains an IAM role to allow the Lambda function to write to CloudWatch Logs and interact with ENIs
  • AWS::Lambda::Function contains the function definition

Here is the full template, the required parameters are VpcId, SubnetIds and LambdaS3Bucket which we should get from previous stacks' outputs. The template outputs the function full name, which we'll need to be able to invoke it later.

Special attention to the Lambda function definition, the property Handler, in .NET runtime is in the form of AssemblyName::Namespace.ClassName::MethodName and the property Code containing the S3 location of the zip file we uploaded earlier.

Description: Template to create a lambda function 

Parameters: 
  LambdaS3Bucket:
    Type: String
  DbClusterPort: 
    Type: Number
    Default: 3306
  VpcId: 
    Type: String
  SubnetIds: 
    Type: CommaDelimitedList

Resources:
  LambdaSg:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: Allow outbound traffic to MySQL host
      VpcId:
        Ref: VpcId
      SecurityGroupEgress:
        - IpProtocol: tcp
          FromPort: !Ref DbClusterPort
          ToPort: !Ref DbClusterPort
          CidrIp: 0.0.0.0/0

  AWSLambdaExecutionRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Principal:
              Service:
                - lambda.amazonaws.com
            Action: sts:AssumeRole
      Path: /
      Policies:
        - PolicyName: PermitLambda
          PolicyDocument:
            Version: 2012-10-17
            Statement:
            - Effect: Allow
              Action:
              - logs:CreateLogGroup
              - logs:CreateLogStream
              - logs:PutLogEvents
              - ec2:CreateNetworkInterface
              - ec2:DescribeNetworkInterfaces
              - ec2:DeleteNetworkInterface
              Resource: 
                - "arn:aws:logs:*:*:*"
                - "*"
  HelloLambda:
    Type: AWS::Lambda::Function
    Properties:
      Handler: aurora.lambda::project.lambda.Function::RunQueryHandler
      Role: !GetAtt AWSLambdaExecutionRole.Arn
      Code:
        S3Bucket: !Ref LambdaS3Bucket
        S3Key: aurora-lambda/aurora-lambda.zip
      Runtime: dotnetcore2.1
      Timeout: 30
      VpcConfig:
        SecurityGroupIds:
          - !Ref LambdaSg
        SubnetIds: !Ref SubnetIds

Outputs:
  LambdaFunction:
    Value: !Ref HelloLambda

To deploy this stack we use the following command where we pass the parameters specific to our VPC (VpcId and SubnetIds) as well as the S3 bucket name.

$ aws cloudformation deploy --stack-name fn-stack \
--template-file aurora_lambda_template.yml \
--parameter-overrides VpcId=vpc-0b442e5d98841996c SubnetIds=subnet-013d0bbb3eca284a2,subnet-00c67cfed3ab0a791 LambdaS3Bucket=abelperez-temp \
--capabilities CAPABILITY_IAM

Waiting for changeset to be created..
Waiting for stack create/update to complete
Successfully created/updated stack - fn-stack

Let's get the outputs as we'll need this information later. We have the Lambda function full name.

$ aws cloudformation describe-stacks --stack-name fn-stack --query Stacks[*].Outputs
[
    [
        {
            "OutputKey": "LambdaFunction",
            "OutputValue": "fn-stack-HelloLambda-C32KDMYICP5W"
        }
    ]
]

Invoking Lambda function

Now that we have deployed the function and we know its full name, we can invoke it by using dotnet lambda invoke-function command. Part of this job is to prepare the payload which is a JSON in put corresponding to the Lambda input defined above.

{
    "Connection": {
        "DbUser": "master", 
        "DbPassword": "Aurora.2019", 
        "DbName": "dbtest", 
        "DbHost": "db-stack-auroramysqlcluster-xxx.rds.amazonaws.com", 
        "DbPort": 3306
    }, 
    "QueryText":"show databases;"
}

Here is the command to invoke the Lambda function, including the payload parameter encoded to escape the quotes and all in a single line. There are better ways to do this, but for the sake of this demonstration, it's good enough.

$ dotnet lambda invoke-function \
--function-name fn-stack-HelloLambda-C32KDMYICP5W \
--payload "{ \"Connection\": {\"DbUser\": \"master\", \"DbPassword\": \"Aurora.2019\", \"DbName\": \"dbtest\", \"DbHost\": \"db-stack-auroramysqlcluster-xxx.rds.amazonaws.com\", \"DbPort\": 3306}, \"QueryText\":\"show databases;\" }" \
--region eu-west-1

Amazon Lambda Tools for .NET Core applications (2.2.0)
Project Home: https://github.com/aws/aws-extensions-for-dotnet-cli, https://github.com/aws/aws-lambda-dotnet

Payload:
[{"Database":"information_schema"},{"Database":"dbtest"},{"Database":"mysql"},{"Database":"performance_schema"}]

Log Tail:
START RequestId: 595944b5-73bb-4536-be92-a42652125ba8 Version: $LATEST
END RequestId: 595944b5-73bb-4536-be92-a42652125ba8
REPORT RequestId: 595944b5-73bb-4536-be92-a42652125ba8  Duration: 11188.62 ms   Billed Duration: 11200 ms       Memory Size: 128 MB     Max Memory Used: 37 MB

Now we can see the output in the Payload section. And that's how we can query remotely any Aurora serverless cluster without having to set up any EC2 instance. This could be extended to handle different SQL operations such as Create, Insert, Delete, etc.

Monday, 4 February 2019

Querying Aurora serverless database remotely using Lambda - part 2

This post is part of a series

In the previous part, we've set up the base layer to deploy our resources. At this point we can create the database cluster.

Aurora DB Cluster

Assuming we have our VPC ready with at least two subnets to comply with high availability best practices, let's create our cluster, I've put together a CloudFormation template that includes:

  • AWS::EC2::SecurityGroup contains inbound traffic rule to allow port 3306
  • AWS::RDS::DBSubnetGroup contains a group of subnets to deploy the cluster
  • AWS::EC2::DBCluster contains all the parameters to create the database cluster

Here is the full template, the only required parameters are VpcId and SubnetIds, but feel free to override any of the database cluster parameters such as database name, user name, password, etc. The template outputs the IDs corresponding to newly created resources such as the database cluster DNS endpoint, port and the security group.

Description: Template to create a serverless aurora mysql cluster

Parameters: 
  DbClusterDatabaseName: 
    Type: String
    Default: dbtest
  DbClusterIdentifier: 
    Type: String
    Default: serverless-mysql-aurora
  DbClusterParameterGroup: 
    Type: String
    Default: default.aurora5.6
  DbClusterMasterUsername: 
    Type: String
    Default: master
  DbClusterMasterPassword: 
    Type: String
    Default: Aurora.2019
  DbClusterPort: 
    Type: Number
    Default: 3306
  VpcId: 
    Type: String
  SubnetIds: 
    Type: CommaDelimitedList

Resources:
  DbClusterSg:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: Allow MySQL port to client host
      VpcId:
        Ref: VpcId
      SecurityGroupIngress:
        - IpProtocol: tcp
          FromPort: !Ref DbClusterPort
          ToPort: !Ref DbClusterPort
          CidrIp: 0.0.0.0/0

  DbSubnetGroup: 
    Type: "AWS::RDS::DBSubnetGroup"
    Properties: 
      DBSubnetGroupDescription: "aurora subnets"
      SubnetIds: !Ref SubnetIds

  AuroraMysqlCluster:
    Type: AWS::RDS::DBCluster
    Properties:
      DatabaseName:
        Ref: DbClusterDatabaseName
      DBClusterParameterGroupName:
        Ref: DbClusterParameterGroup
      DBSubnetGroupName:
        Ref: DbSubnetGroup
      Engine: aurora
      EngineMode: serverless
      MasterUsername:
        Ref: DbClusterMasterUsername
      MasterUserPassword:
        Ref: DbClusterMasterPassword
      ScalingConfiguration:
        AutoPause: true
        MinCapacity: 2
        MaxCapacity: 4
        SecondsUntilAutoPause: 1800
      VpcSecurityGroupIds:
        - !Ref DbClusterSg
        
Outputs:
  DbClusterEndpointAddress:
    Value: !GetAtt AuroraMysqlCluster.Endpoint.Address
  DbClusterEndpointPort:
    Value: !GetAtt AuroraMysqlCluster.Endpoint.Port
  DbClusterSgId:
    Value: !Ref DbClusterSg

To deploy this stack we use the following command where we pass the parameters specific to our VPC (VpcId and SubnetIds).

$ aws cloudformation deploy --stack-name db-stack \
--template-file aurora_cluster_template.yml \
--parameter-overrides VpcId=vpc-0b442e5d98841996c SubnetIds=subnet-013d0bbb3eca284a2,subnet-00c67cfed3ab0a791

Waiting for changeset to be created..
Waiting for stack create/update to complete
Successfully created/updated stack - db-stack

Let's get the outputs as we'll need this information later. We have the cluster endpoint DNS name and the port as per our definition.

$ aws cloudformation describe-stacks --stack-name db-stack --query Stacks[*].Outputs
[
    [
        {
            "OutputKey": "DbClusterEndpointAddress",
            "OutputValue": "db-stack-auroramysqlcluster-1d1udg4ringe4.cluster-cnfxlauucwwi.eu-west-1.rds.amazonaws.com"
        },
        {
            "OutputKey": "DbClusterSgId",
            "OutputValue": "sg-072bbf2078caa0f46"
        },
        {
            "OutputKey": "DbClusterEndpointPort",
            "OutputValue": "3306"
        }
    ]
]

In the next part, we'll create the Lambda function to query this database remotely.

Wednesday, 30 January 2019

Querying Aurora serverless database remotely using Lambda - part 1

This post is part of a series

Why aurora serverless?

For those who like experimenting with new technology, AWS feeds us with a lot of new stuff every year at re:Invent conference, and many more times. In August 2018, Amazon announced its general availability. I was intrigued by this totally new way to doing database, so I started to play with it.

The first road block I found was connectivity from my local environment. I wanted to connect to my new cluster using my traditional MySQL Workbench client. It turned out to be one of the limitations clearly explained by Amazon, and as of today:

You can't give an Aurora Serverless DB cluster a public IP address. You can access an Aurora Serverless DB cluster only from within a virtual private cloud (VPC) based on the Amazon VPC service.

Most common workarounds involve the use of an EC2 instance to either run a MySQL client from there or SSH tunnel and allow connections from outside of the VPC. In both cases we'll be charged for the use of the EC2 instace. At the moment of this writing there is a solution for that, but still in beta: Data API.

With all that said, I decided to explore my own way in the meantime by creating a serverless approach involving Lambda to query my database.

Setting up the base - VPC

First, we need a VPC. For testing only, it doesn't really matter if we just use the default VPC in the current region. But if you'd like to get started with a blue print, there is this public CloudFormation template, it contains a sample VPC with two public subnets and two private subnets as well as all the required resources to guarantee connectivity (IGW, NAT, Route tables, etc.).

I prefer to isolate my experiments from the rest of the resources, that's why I came up with a template containing the bare minimum to get a VPC up and running. It only includes:

  • AWS::EC2::VPC the VPC itself - default CIDR block 10.192.0.0/16
  • AWS::EC2::Subnet private subnet 1 - default CIDR block 10.192.20.0/24
  • AWS::EC2::Subnet private subnet 2 - default CIDR block 10.192.21.0/24

Here is the full template, the only required parameter is EnvironmentName, but feel free to override any of the CIDR blocks. The template outputs the IDs corresponding to newly created resources such as the VPC and the subnets.

Description: >-
  This template deploys a VPC, with two private subnets spread across 
  two Availability Zones. This VPC does not provide any internet 
  connectivity resources such as IGW, NAT Gw, etc.

Parameters:
  EnvironmentName:
    Description: >-
      An environment name that will be prefixed to resource names
    Type: String

  VpcCIDR:
    Description: >-
      Please enter the IP range (CIDR notation) for this VPC
    Type: String
    Default: 10.192.0.0/16

  PrivateSubnet1CIDR:
    Description: >-
      Please enter the IP range (CIDR notation) for the private subnet 
      in the first Availability Zone
    Type: String
    Default: 10.192.20.0/24

  PrivateSubnet2CIDR:
    Description: >-
      Please enter the IP range (CIDR notation) for the private subnet 
      in the second Availability Zone
    Type: String
    Default: 10.192.21.0/24

Resources:
  VPC:
    Type: AWS::EC2::VPC
    Properties:
      CidrBlock: !Ref VpcCIDR
      EnableDnsSupport: true
      EnableDnsHostnames: true
      Tags:
        - Key: Name
          Value: !Ref EnvironmentName

  PrivateSubnet1:
    Type: AWS::EC2::Subnet
    Properties:
      VpcId: !Ref VPC
      AvailabilityZone: !Select [ 0, !GetAZs '' ]
      CidrBlock: !Ref PrivateSubnet1CIDR
      MapPublicIpOnLaunch: false
      Tags:
        - Key: Name
          Value: !Sub ${EnvironmentName} Private Subnet (AZ1)

  PrivateSubnet2:
    Type: AWS::EC2::Subnet
    Properties:
      VpcId: !Ref VPC
      AvailabilityZone: !Select [ 1, !GetAZs '' ]
      CidrBlock: !Ref PrivateSubnet2CIDR
      MapPublicIpOnLaunch: false
      Tags:
        - Key: Name
          Value: !Sub ${EnvironmentName} Private Subnet (AZ2)

Outputs:
  VPC:
    Description: A reference to the created VPC
    Value: !Ref VPC

  PrivateSubnet1:
    Description: A reference to the private subnet in the 1st Availability Zone
    Value: !Ref PrivateSubnet1

  PrivateSubnet2:
    Description: A reference to the private subnet in the 2nd Availability Zone
    Value: !Ref PrivateSubnet2

To create a stack from this template we run the following command (or go to AWS console and upload the template)

$ aws cloudformation deploy --stack-name vpc-stack \
--template-file vpc_template.yml \
--parameter-overrides EnvironmentName=Dev

Waiting for changeset to be created..
Waiting for stack create/update to complete
Successfully created/updated stack - vpc-stack

After successful stack creation, we can get the outputs as we'll need them for the next step.

$ aws cloudformation describe-stacks --stack-name vpc-stack --query Stacks[*].Outputs
[
    [
        {
            "Description": "A reference to the private subnet in the 1st Availability Zone",
            "OutputKey": "PrivateSubnet1",
            "OutputValue": "subnet-013d0bbb3eca284a2"
        },
        {
            "Description": "A reference to the private subnet in the 2nd Availability Zone",
            "OutputKey": "PrivateSubnet2",
            "OutputValue": "subnet-00c67cfed3ab0a791"
        },
        {
            "Description": "A reference to the created VPC",
            "OutputKey": "VPC",
            "OutputValue": "vpc-0b442e5d98841996c"
        }
    ]
]

In the next part, we'll create the Database cluster using these resources as a base layer.

Monday, 30 April 2018

Using Lambda@Edge to reduce infrastructure complexity

In my previous series I went through the process of creating the cloud resources to host a static website as well as the development pipeline to automate the process from push code in source control to deploy on a S3 bucket.

One of the challenges was how to approach the www to non-www redirection, the proposed solution consisted of duplicating the CloudFront distributions and the S3 website buckets in order to get the traffic end to end, the reason why I took this approach was because CloudFront doesn't have (to the best of my knowledge at the time) the ability to issue redirect, instead it just pass traffic to different origins based on configuration.

What is Lamda@Edge ?

Well, I was wrong, there is in fact a way to make CloudFront to issue redirects, it's called Lambda@Edge, a special flavour of Lambda functions that are executed on Edge locations and therefore closer to the end user. It allows a lot more than just issuing HTTP redirects.

In practice this means we can intercept any of the four events that happen when the user request a page to CloudFront and execute our Lambda code.

  • After CloudFront receives a request from a viewer (viewer request)
  • Before CloudFront forwards the request to the origin (origin request)
  • After CloudFront receives the response from the origin (origin response)
  • Before CloudFront forwards the response to the viewer (viewer response)

In this post, we're going to leverage Lambda@Edge to create another variation of the same infrastructure by using hooking our Lambda function to Viewer Request event, it will look like this one when finished.

How does it change from previous approach?

This time we still need two Route 53 record sets, because we're still handling both abelperez.info and www.abelperez.info.

We need only one S3 bucket, since redirection will be issued by Lambda@Edge, so no need for Redirection Bucket resource.

We need only one CloudFront distribution as there a single origin, but this time the CloudFront distribution will have two CNAMEs in order to handle both www and non-www. We'll also link the lambda function with the event as part of default cache behaviour.

Finally we need to create a Lambda function to performs the redirection when necessary.

Creating Lambda@Edge function

Creating a Lambda@Edge function is not too far from creating an ordinary Lambda function, but we need to be aware of some limitations (at least at the moment of writing), they need to be created only in N. Virginia (US-East-1) region and the available runtime is NodeJs 6.10.

Following the steps from AWS CloudFront Developer Guide, you can create your own Lambda@Edge function and connect it to a CloudFront distribution. Here are some of my highlights:

  • Be aware of the required permissions, telling Lambda to create the role is handy.
  • Remove triggers before creating the function as it can take longer to replicate
  • You need to publish a version of the function before associating with any trigger.

The code I used is very simple, it only reads host header from the request and verify if it's equal to 'abelperez.info' to send a custom response with a HTTP 301 redirection to www domain host, in any other case, it just let it pass ignoring the request, therefore CloudFront will proceed with the request life cycle.

exports.handler = function(event, context, callback) {
  const request = event.Records[0].cf.request;
  const headers = request.headers;
  const host = headers.host[0].value;
  if (host !== 'abelperez.info') {
      callback(null, request);
      return;
  }
  const response = {
      status: '301',
      statusDescription: 'Moved Permanently',
      headers: {
          location: [{
              key: 'Location',
              value: 'https://www.abelperez.info',
          }],
      },
  };
  callback(null, response);
};

Adding the triggers

Once we have created and published the Lambda fuction, it's time to add the trigger, in this case it's CloudFront and we need to provide the CloudFront distribution Id and select the event type which is viewer-request as stated above.

From this point, we've just created a Lambda@Edge function!

Let's test it

The easiest way to test what we've done is to issue a couple of curl commands, one requesting www over HTTPS expecting a HTTP 200 with our HTML and another one request to non-www over HTTP expecting a HTTP 302 with the location pointing to www domain. Here is the output.

abel@ABEL-DESKTOP:~$ curl https://www.abelperez.info
<html>
<body>
<h1>Hello from S3 bucket :) </h1>
</body>
</html>
abel@ABEL-DESKTOP:~$ 
abel@ABEL-DESKTOP:~$ curl http://abelperez.info
<html>
<head><title>301 Moved Permanently</title></head>
<body bgcolor="white">
<center><h1>301 Moved Permanently</h1></center>
<hr><center>CloudFront</center>
</body>
</html>

An automated version of this can be found at my github repo

Thursday, 5 April 2018

Completely serverless static website on AWS

Why serverless ? The basic idea behind this is not to worry about the underlying infrastructure, the Cloud provider will expose services through several interfaces where we allocate resources and use them, and more importantly, pay only for what we use.

This approach helps to prove concepts with little or no budget, also allows to scale as the business grows on demand. All that solves the problem of over provisioning and paying for idle boxes.

One of the common scenarios is having a content website, in this case I'll focus on static website. In this series I'll explain step by step how to create all the environment from development to production on AWS.

At the end of this series we'll have created this infrastructure:

Serverless static website - part 1 - In this part you'll learn how to start with AWS and how to use Amazon S3 to host a static website, making it publicly accesible by its public endpoint url.

Serverless static website - part 2 - In this part you'll learn how to get your own domain and put it in use straight away with the static website.

Serverless static website - part 3 - In this part you'll learn how to get around the problem of having www as well as non-www domain, and how get always redirect to the www endpoint.

Serverless static website - part 4 - In this part you'll learn how to create a SSL certificate via Amazon Certificate Manager and verify the domain identity.

Serverless static website - part 5 - In this part you'll learn how to distribute the content throughout the AWS edge locations and handling SSL traffic.

Serverless static website - part 6 - In this part you'll learn how to set up a git repository using CodeCommit repository, so you can store your source files.

Serverless static website - part 7 - In this part you'll learn how to set up a simple build pipeline using a CodeBuild project.

Serverless static website - part 8 - In this part you'll learn how to automate the process of triggering the build start action when some changes are pushed to the git repository.

Serverless static website - part 9 - In this part you'll learn how the whole process can be automated by using CloudFormation templates to provision all the resources previously described manually.

Serverless static website - part 9

One of the greatest benefits of cloud computing is the ability to automate processes and up to this point, we've learned how to set everything up via AWS console web interface.

Why automate?

It is always good to know how to manage stuff via console in case we need to manually modify something, but we should aim to avoid this practice. Instead, limit the use of the console to a bare minimum and the rest of the time, aim for some automated way, this has the following advantages:

  • We can keep source code / templates under source control, allowing to keep track of changes.
  • It can be reused to replicate in case of a new customer / new environment / etc.
  • No need to remember where every single option is located as the UI can change.
  • It can be transferred to another account in a matter of a few minutes.

In AWS world, the automated process is achieved by creating templates in CloudFormation and deploying them to stacks.

I have already created a couple of CloudFormation templates to automate all the process described to this point, they can be found at my GitHub repo.

CloudFormation templates

In order to automate this infrastructure, I've divided resources into two separate templates: one containing the SSL certificate and the other containing all the rest of the resources. The reason why the SSL certificate is in another template is because it needs to be run on N. Virginia region (US-East-1) as explained earlier when we created it manually, it's a CloudFront requirement.

Templates can contain parameters that make them more flexible, in this case, there is a parameter that controls the creation of a redirection bucket, we might have a scenario when we want just a website on a sub domain and we might not want to redirect from the naked domain. These are the parameters:

SSL Certificate template

  • DomainName: The site domain name (naked domain only)

Infrastructure template

  • HostedZone: This is just for reference, it should not be changed
  • SSLCertificate: This should be taken from the output of the SSL certificate template
  • DomainName: Same as above, only naked domain
  • SubDomainName: specific sub domain, typically www
  • IncludeRedirectToSubDomain: Whether it should include a redirection bucket from the naked domain to the subdomain

Creating SSL certificate Stack

First, let's make sure we are in N. Virginia region. Go to CloudFormation console, once there, click Create Stack button. We are presented with Select Template screen, where we'll choose a template from my repository (ssl-certificate.yaml) by selecting Upload a template to Amazon S3 radio button.

Click Next, you'll see the input parameters page including the stack name which I'll name abelperez-info-ssl-stack to give it some meaningful name.

After entering the required information, click Next and Next again, then on Create button. You'll see the Create in progress status in the stack.

At this point, the SSL certificate is being created and will require the identity verification just like when it was created manually, this will block the stack creation until the validation process is finished, so go ahead and check your email and follow the link to validate the identity to proceed with the stack creation.

Once the creation is done you'll see the Create complete status in the stack, on the lower pane, select Outputs, you'll find SSLCertificateArn. Take that value and copy it somewhere temporarily, we'll need it for our next stack.

Creating Infrastructure Stack

Following a similar process, let's create the second stack containing most of the resources to provision. In this case we are not forced to create it in any specific region, we can choose any provided all the services are available, for this example I'll Ireland (EU-West-1). The template can be downloaded from my repository (infrastructure.yaml).

This time, you are presented a different set of parameters, SSL Certificate will be the output of the previous stack as explained above. Domain name will be exactly the same as in the previous stack, give we are using the SSL certificate for this domain. Subdomain will be www and I'll include a redirection as I expect users to be redirected from abelperez.info to www.abelperez.info. I'll name the stack abelperez-info-infra-stack just to make it meaningful.

Since this template will create IAM users, roles and policies, we need to acknowledge this by ticking the box.

Once we hit Create, we can see the Create in progress screen.

This process can take up to 30 minutes, so please be patient, this takes so long time to create the stack because we are provisioning CloudFront distributions and they can take some time to propagate.

Once the stack is created, we can take note of a couple of values from the output: CodeCommit repository url (either SSH or HTTPS) and the Static bucket name.

Manual steps to finish the set up.

With all resource automatically provisioned by the templates, we are in a position where the only thing we need is to link our local SSH key with the IAM user. To do that, let's do exactly what we did when it was set up manually in part 6.

In my case, I chose to use SSH key, so I went to IAM console, found the user, under Security Credentials, I uploaded my SSH public key.

We also need to update the buildspec.yml to run our build, can be downloaded from the above linked GitHub repository. The placeholder <INSERT-YOUR-BUCKET-NAME-HERE> must be replaced with the actual bucket name, in this instance the bucket name generated is abelperez-info-infra-stack-staticsitebucket-1ur0k115f2757 and my buildspec.yml looks like:

version: 0.2

phases:
  build:
    commands:
      - mkdir dist
      - cp *.html dist/

  post_build:
    commands:
      - aws s3 sync ./dist 
        s3://abelperez-info-infra-stack-staticsitebucket-1ur0k115f2757/ 
        --delete --acl=public-read

Let's test it!

Our test consist of cloning the git repository from CodeCommit, add two files: index.html and buildspec.yml. Then we'll perform a git push and we expect it will trigger the build executing the s3 sync command and copying the index.html to our destination bucket which will be behind CloudFront and CNamed by Route 53. In the end, we should be able to just browse www.abelperez.info and get whatever the result is in index.html just uploaded.

Just a note, if you get a HTTP 403 instead of the expected HTML, then just wait for a few minutes, CloudFront/Route 53 might not be fully propagated just yet.

abel@ABEL-DESKTOP:~$ git clone ssh://git-codecommit.eu-west-1.amazonaws.com/v1/repos/www.abelperez.info-web
Cloning into 'www.abelperez.info-web'...
warning: You appear to have cloned an empty repository.
abel@ABEL-DESKTOP:~$ cd www.abelperez.info-web/
abel@ABEL-DESKTOP:~/www.abelperez.info-web$ git add buildspec.yml index.html 
abel@ABEL-DESKTOP:~/www.abelperez.info-web$ git commit -m "initial commit"
[master (root-commit) dc40888] initial commit
 Committer: Abel Perez Martinez 
Your name and email address were configured automatically based
on your username and hostname. Please check that they are accurate.
You can suppress this message by setting them explicitly. Run the
following command and follow the instructions in your editor to edit
your configuration file:

    git config --global --edit

After doing this, you may fix the identity used for this commit with:

    git commit --amend --reset-author

 2 files changed, 16 insertions(+)
 create mode 100644 buildspec.yml
 create mode 100644 index.html
abel@ABEL-DESKTOP:~/www.abelperez.info-web$ git push
Counting objects: 4, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (4/4), done.
Writing objects: 100% (4/4), 480 bytes | 0 bytes/s, done.
Total 4 (delta 0), reused 0 (delta 0)
To ssh://git-codecommit.eu-west-1.amazonaws.com/v1/repos/www.abelperez.info-web
 * [new branch]      master -> master
abel@ABEL-DESKTOP:~/www.abelperez.info-web$ curl https://www.abelperez.info
<html>
<body>
<h1>Hello from S3 bucket :) </h1>
</body>
</html>

Saturday, 24 February 2018

Serverless static website - part 8

All the way to this point, we have created the resources that allow us to store our source code and deploy it to S3 bucket. However, this build/deploy process has to be triggered manually by going to the console, finding the CodeBuild project and clicking Start build button.

This doesn't sound good enough for a complete development cycle. We need something that links changes in CodeCommit repository and CodeBuild project together so the build process is triggered automatically when changes are pushed to git. That magic link is called CloudWatch

Amazon CloudWatch is a monitoring service for AWS cloud resources and applications you run on AWS. Amongst other things it allows to create event rules that watch service events and trigger actions on target services. In this case we are going to watch changes in CodeCommit repository and link that to the Start build action on CodeBuild.

Creating the Rule

First, we need to go to CloudWatch console, to get there, select Services from the top menu, then under Management Tools, select CloudWatch. Once there, select Rules from the left hand side menu, under Events category.

Configure the Event source

We'll see two panes, the left hand side is for the event source, the right one is to choose the target of our rule.

On the left side, select Event Pattern radio button option (default one), next from the drop down menu, select Events by Service. For Service Name, find the option corresponding to CodeCommit and Event Type CodeCommit Repository State Change.

Since we are only interested in the changes of the repository we previously created, select Specific resource(s) by ARN radio button option. Then, enter the ARN of the corresponding CodeCommit repository, which can be found by going to Settings in CodeCommit console. It should be something like arn:aws:codecommit:eu-west-1:111111111111:abelperez.info-web.

Configure the Target

On the right side, click Add Target, then from the drop down menu, select CodeBuild project. A new form is displayed where we need to enter the project ARN, this is a bit tricky because for some reason, CodeBuild console doesn't show this information. Head to ARN namespaces documentation page and find CodeBuild, the format will be something like arn:aws:codebuild:us-east-1:111111111111:project/abelperez.info-builder.

Next step is to define the role, which in this case, we'll leave as it is proposed to Create a new role for this specific resource, AWS will figure out what permissions based on the type of resource we set as target, this might not be entirely accurate in the case of Lambda as target, we'd need to edit the role and add some policy for specific access.

We skipped Configure input on purpose as no input is required to trigger the build, so no need to change that setting.

Finish the process

Click on Configure details to go to step 2 where the process will finish.

Enter name and optionally a description, leave State as Enabled so it will be active from the moment we create it. Click on Create rule finish the process.

Let's test it

How can we know if the rule is working as expected? Since we connected changes on CodeCommit repository with starting a build on CodeBuild, let's do some changes to our repository, for example, adding a new file index3.html

abel@ABEL-DESKTOP:~/Downloads/abelperez.info-web$ git add index3.html 
abel@ABEL-DESKTOP:~/Downloads/abelperez.info-web$ git commit -m "3rd file"
[master e953674] 3rd file
 1 file changed, 1 insertion(+)
 create mode 100644 index3.html
abel@ABEL-DESKTOP:~/Downloads/abelperez.info-web$ git push
Counting objects: 3, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (2/2), done.
Writing objects: 100% (3/3), 328 bytes | 0 bytes/s, done.
Total 3 (delta 1), reused 0 (delta 0)
To ssh://git-codecommit.eu-west-1.amazonaws.com/v1/repos/abelperez.info-web
   27967c1..e953674  master -> master

Now let's check CodeBuild console to see the build being triggered by the rule we've created

As expected, the build ran without any human intervention, which was the goal of the CloudWatch Event rule. Since the build succeeded, we can also check the existence of the new file in the bucket.

abel@ABEL-DESKTOP:~/Downloads$ aws s3 ls s3://www.abelperez.info
2018-02-21 23:19:53         18 index.html
2018-02-21 23:19:53         49 index2.html
2018-02-21 23:19:53         49 index3.html

Tuesday, 20 February 2018

Serverless static website - part 7

Now that we have our source code versioned and safely stored, we need a way to make it go to our S3 bucket, otherwise it won't be visible to the public. There are, as usual, many ways to address this scenario, but as a developer, I can't live without CI/CD integration for all my processes. In this case, we have a very simple process, and we can combine the building and deploying steps in one go. To do that, once again, AWS offers some useful services, I'll be using CodeBuild for this task.

Creating the access policy

Just like anything in AWS, a CodeBuild project needs access to other AWS services and resources. The way this is done is by applying service roles with policies. As per AWS docs, a CodeBuild project needs the following permissions:

  • logs:CreateLogGroup
  • logs:CreateLogStream
  • logs:PutLogEvents
  • codecommit:GitPull
  • s3:GetObject
  • s3:GetObjectVersion
  • s3:PutObject

The difference is that the proposed policy in the docs does not restrict access to any resource, it suggests to change the * to something more specific. The following is the resulting policy restricting access to logs in the log group created by the CodeBuild project. Also I've added full access to the bucket, since we'll be copying files to that bucket. In summary, the policy below grants the following permissions (account id is masked with 111111111111):

  • Access to create new log groups in CloudWatch
  • Access to create new log streams within the log group created by the build project
  • Access to create new log events within the log stream created by the build project
  • Access to pull data through Git to CodeCommit repository
  • Full Access to our S3 bucket where the files will go
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "logs:CreateLogGroup",
                "s3:ListObjects"
            ],
            "Resource": "*",
            "Effect": "Allow",
            "Sid": "AccessLogGroupsS3ListObjects"
        },
        {
            "Action": [
                "codecommit:GitPull",
                "logs:CreateLogStream",
                "logs:PutLogEvents"
            ],
            "Resource": [
                "arn:aws:logs:eu-west-1:111111111111:log-group:/aws/codebuild/
                 abelperez-stack-builder",
                "arn:aws:logs:eu-west-1:111111111111:log-group:/aws/codebuild/
                 abelperez-stack-builder:*",
                "arn:aws:codecommit:eu-west-1:111111111111:abelperez.info-web"
            ],
            "Effect": "Allow",
            "Sid": "AccessGitLogs"
        },
        {
            "Action": "s3:*",
            "Resource": [
                "arn:aws:s3:::www.abelperez.info",
                "arn:aws:s3:::www.abelperez.info/*"
            ],
            "Effect": "Allow",
            "Sid": "AccessS3staticBucket"
        }
    ]
}

Creating the service role

Go to IAM console, select Roles from the left hand side menu, click on Create Role. On the Select type of trusted entity screen, select AWS service EC2, Lambda and others and from the options, CodeBuild

Click Next: Permissions, at this point, we can create a policy with the JSON above or continue to Review and then add the policy later. To create the policy, click Create Policy button, select JSON tab, paste the JSON above, Review and Create Policy, then back to role permission screen, refresh and select the policy. Give it a name, something like CodeBuildRole or anything meaningful and click on Create Role to finish the process.

Creating the CodeBuild project

With all permissions set, let's start with the actual CodeBuild project, where we'll specify the source code origin, build environment, artifacts and role. On CodeBuild console, if it's the first project, click Get Started, otherwise select Build projects from the left hand side menu, and click on Create Project

Enter the following values:

  • Project name* a meaningful name i.e "abelperez-stack-builder"
  • Source provider* select "AWS CodeCommit"
  • Repository* select the repository previously created
  • Environment image* select "Use an image managed by AWS CodeBuild"
  • Operating system* select "Ubuntu"
  • Runtime* select "Node.js"
  • Runtime version* select "aws/codebuild/nodejs:6.3.1"
  • Buildspec name leave default "buildspec.yml"
  • Artifacts: Type* select "No artifacts"
  • Role name* select the previously created role

After finishing all the input, review and save the project.

Creating the build spec file

When creating the CodeBuild project, one of the input parameters is the Buildspec file name, which refers to the description of our build process. In this case, it's very simple process:

  • BUILD: Create a directory dist to copy all output files
  • BUILD: Copy all .html file to dist
  • POST_BUILD: Synchronise all content of dist folder with www.abelperez.info S3 bucket

The complete buildspec.yml containing the above steps:

version: 0.2

phases:
  build:
    commands:
      - mkdir dist
      - cp *.html dist/

  post_build:
    commands:
      - aws s3 sync ./dist s3://www.abelperez.info/ --delete --acl=public-read

The last step is executed on post build phase, which is at the end of the all build steps when everything is ready to package / deploy or in this case ship to our S3 bucket. This is done by using the sync subcommand of s3 service from AWS Command line interface. It basically takes an origin and a destination and make them to be in sync.

--delete instructs to remove files in the destination that are not in the source, because by default they are not deleted.

--acl=public-read instructs to edit bucket objects' ACL, in this case read access to everyone.

At this point, we can remove the bucket policy previously created if we wish so, since every single file will be publicly accessible, that would allow to have other folders in the same bucket and not being accessible. It will depend on the use case for the website.

Let's test it

It's time to verify if everything is in the right place. We have the build project with some basic commands to run and assuming the above code is saved in a file named buildspec.yml at root level of the code repository, the next step is to commit and push that file to CodeCommit.

Let's go back to CodeBuild console, select the build project we've just created and click Start build, then choose the right branch (master in my case) and once again Start build.

Tuesday, 6 February 2018

Serverless static website - part 6

Up to this point we've created a simple HTML page and set up all the plumbing to make it visible to the world in a distributed and secure way. Now, if we want to update the content, we can either go to S3 console and upload manually the files or from the AWS CLI synchronise a local folder with a bucket. None of those ways seem to scale in the long run. As a developer I like to keep things under control, even more, under source control.

It can be done by using any popular cloud hosted version control systems, such as github, but in this example I'll use the one provided by AWS, it's called CodeCommit and it's a Git repository compatible with all current git tools.

Creating the repository

In CodeCommit console, which can be accessed by selecting Services, then under Developer Tools, select CodeCommit. Once there, click on Create repository button, or Get started if you haven't created any repository before.

Next, give it a name and a description and you'll see something like this:

Creating the user

Now, we need a user for ourselves or if someone else is going to contribute to that repository. There are many ways to address this part, I'll go for creating a user with a specific policy to grant access to that particular CodeCommit repository

In IAM (Identity and Access Management) console, select Users from left hand side menu, then click Add user. Next, give it a name and Programmatic access, this user won't access the console so it doesn't need login and password, not even access keys.

Click Next, Skip permissions for now, we'll deal with that in the next section. Review and create the user.

Creating the policy

Back to IAM console dashboard, select Policies from the left menu, then click on Create policy, select JSON tab and copy the following:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AllowGitCommandlineOperations",
            "Effect": "Allow",
            "Action": [
                "codecommit:GitPull",
                "codecommit:GitPush"
            ],
            "Resource": [
                "arn:aws:codecommit:::abelperez.info-web"
            ]
        }
    ]
}

Details about creating policies is beyond the scope of this post, but essentially this policy grants pull and push operations on abelperez.info-web repository. Click Review Policy and give it a meaningful name such as CodeCommit-MyRepoUser-Policy or something that states what permissions are granted, just to keep everything organised. Create the policy and you'll be able to to see it if filtering by the name.

Assigning the policy to the user

Back to IAM console dashboard, select Users, select the user we've created before, on tab Permissions, click Add Permissions

Once there, select Attach existing policies directly from the top three buttons. Then, filter by Customer managed policies to make it easier to find our policy (the one created before). Select the policy, Review and Add Permissions.

When it's done, we can see the permission added to the user, like below.

Granting access via SSH key

Once we've assigned the policy, the user should have permission to use git pull / push operations, now we need to link that IAM user with a git user (so to speak). To do that, CodeCommit provides two options: via HTTPS and SSH. The simpler way is using SSH, it only requires to upload the SSH public key to IAM user settings, then add some host/key/id information to SSH configuration file and that's it.

In IAM console, select the previously created user, then on Security Credentials tab, scroll down all the way to SSH keys for AWS CodeCommit, then click on Upload SSH public key button, paste your SSH public key. If you don't have one created yet, have a look at here and for more info, here. Click Upload SSH public key button and we are good to go.

If you are interested in accessing via HTTPS, then have a look at this documentation page on AWS website.

Back to CodeCommit console, select the repository previously created, then on the right hand side, click Connect button. AWS will show a panel with instructions, depending on your operating system, but it's essentially like this:

The placeholder Your-IAM-SSH-Key-ID-Here refers to the ID auto generated by IAM when you upload your SSH key and it has the format like APKAIWBASSHIDEXAMPLE

Let's test it

The following command sequence has been executed after setting up all the previous steps on AWS console. Started by cloning an empty repo, then created a new file. Added and committed that file to the repository and finally pushed that commit to the remote repository

abel@ABEL-DESKTOP:~/Downloads$ git clone ssh://git-codecommit.eu-west-1
.amazonaws.com/v1/repos/abelperez.info-web
Cloning into 'abelperez.info-web'...
warning: You appear to have cloned an empty repository.
abel@ABEL-DESKTOP:~/Downloads$ cd abelperez.info-web/
abel@ABEL-DESKTOP:~/Downloads/abelperez.info-web$ ls
abel@ABEL-DESKTOP:~/Downloads/abelperez.info-web$ echo "<h1>New file</h1>" > index.html
abel@ABEL-DESKTOP:~/Downloads/abelperez.info-web$ cat index.html 
<h1>New file</h1>
abel@ABEL-DESKTOP:~/Downloads/abelperez.info-web$ git add index.html 
abel@ABEL-DESKTOP:~/Downloads/abelperez.info-web$ git commit -m "first file"
[master (root-commit) c81ad93] first file
 Committer: Abel Perez Martinez <abel@ABEL-DESKTOP>
Your name and email address were configured automatically based
on your username and hostname. Please check that they are accurate.
You can suppress this message by setting them explicitly. Run the
following command and follow the instructions in your editor to edit
your configuration file:

    git config --global --edit

After doing this, you may fix the identity used for this commit with:

    git commit --amend --reset-author

 1 file changed, 1 insertion(+)
 create mode 100644 index.html
abel@ABEL-DESKTOP:~/Downloads/abelperez.info-web$ git status
On branch master
Your branch is based on 'origin/master', but the upstream is gone.
  (use "git branch --unset-upstream" to fixup)
nothing to commit, working tree clean
abel@ABEL-DESKTOP:~/Downloads/abelperez.info-web$ git push
Counting objects: 3, done.
Writing objects: 100% (3/3), 239 bytes | 0 bytes/s, done.
Total 3 (delta 0), reused 0 (delta 0)
To ssh://git-codecommit.eu-west-1.amazonaws.com/v1/repos/abelperez.info-web
 * [new branch]      master -> master

After this, as expected, the new file is now visible from CodeCommit console.

Friday, 2 February 2018

Serverless static website - part 5

Once we have created the certificate, we need a bridge between a secure endpoint and the current buckets. One of the most common ways to address this is by using CloudFront content distribution service. By using this, we'll make our website to be distributed to some (depending on the pricing) of the Amazon AWS edge locations, which means that if we have some international audience for our website, it will be served to the client by the closest edge location.

Creating WWW Distribution

First, on AWS Console select CloudFront from the services list, under Networking & Content Delivery category. Once there, click on Create Distribution button, in this case, we are interested in a Web Distribution, so under Web, select Get Started. In the create distribution form, there are some key values that are required to enter in order to make this configuration to work. The rest can stay with default value

  • Origin Domain Name: www.abelperez.info.s3-website-eu-west-1.amazonaws.com (which is the endpoint for the bucket)
  • Viewer Protocol Policy: HTTP and HTTPS (this way if anyone tries to visit through HTTP, it will be redirected to HTTPS endpoint)
  • Price Class: Depending on your needs, in my case, US, Canada en Europe is enough
  • Alternate Domain Names(CNAMEs): www.abelperez.info
  • SSL Certificate: Custom SSL Certificate (example.com): and select your certificate from the dropdown menu
  • Default Root Object: index.html

Note: If you can't see your certificate in the dropdown, make sure you've created / imported in the region N. Virginia.

Once entered all this information, click Create Distribution

Creating non-WWW Distribution

Let's repeat the process to create another distribution, this time some values will be different, as we'll point to the non-www bucket and domain name.

  • Origin Domain Name: abelperez.info.s3-website-eu-west-1.amazonaws.com (which is the endpoint for the bucket)
  • Viewer Protocol Policy: HTTP and HTTPS (this way if anyone tries to visit through HTTP, it will be redirected to HTTPS endpoint)
  • Price Class: Depending on your needs, in my case, US, Canada en Europe is enough
  • Alternate Domain Names(CNAMEs): abelperez.info
  • SSL Certificate: Custom SSL Certificate (example.com): and select your certificate from the dropdown menu (the same as above)
  • Default Root Object: leave empty, since this distribution won't serve any file.

Once entered all this information, click Create Distribution, the creation process will take about 30 minutes, so be patient. When they're done, the status changes to Deployed and they can start receiving traffic. You'll see something like this:

Updating DNS records

Now that we have created the distributions, it's time to update how Route 53 is going to resolve DNS requests to CloudFront distributions instead of the S3 bucket previously configured. To do that, on Route 53 console, select the Hosted Zone, then select one record set, and updated the Alias Target to the corresponding CloudFront distribution Domain Name. Then repeat the process for the other record set.

The following diagram illustrates the interaction between the AWS services we've used up this point. The browser first queries DNS for the given domain name, Route 53, will resolve to the appropriate CloudFront distribution, which can serve the content if it has already cached, otherwise it will request from the bucket and then serve the content. If the request is over HTTP to CloudFront, it will issue a HTTP 301 redirection to the HTTPS endpoint. If the request is over HTTPS to CloudFront, it will use the assigned certificate, which in this case is the same for both www and non-www endpoints.


     < HTTP 301 - Redirecto to HTTPS
 +----------------------------+
 |                            |                                         
 |       GET HTTP >     +-----+------+             +----------------+
 |     abelperez.info   | CloudFront |  GET HTTPS  |   S3 Bucket    |
 |           +--------> | (non-www)  |-----------> | abelperez.info +--+
 |           |          |            |             |                |  |
 |           |          +------+-----+             +----------------+  |
 |     +-----+-----+            \                                      |
 |     |           |             V                                     |
 | +-> | Route 53  |       (SSL certficate)                            |
 | |   |           |             A                                     |
 | |   +-----+-----+            /                                      |
 | |         |          +------+-----+          +--------------------+ |
 | |         |          | CloudFront |GET HTTPS |     S3 Bucket      | |
 | |         +--------> |   (www)    |--------> | www.abelperez.info | |
 | |   GET HTTP >       |            |          |                    | |
 | | www.abelperez.info +-----+------+          +----------+---------+ |
 | |                          |                            |           |
 V |                          |                            |           |
+--+--------+ <---------------+                            |           |
|           |      < HTTP 301 - Redirecto to HTTPS         |           |
|           | <--------------------------------------------+           |
|  Browser  |      < HTTP 200 - OK                                     |
|           | <--------------------------------------------------------+
+-----------+      < HTTP 301 - Redirect to www

One more step

There is a bucket that its sole purpose is to redirect requests from non-www to www domain name, this bucket was set to use HTTP protocol, in this case, we are going to update it to HTTPS, so we can save one redirection step in the process.

Let's test it

Once again, we'll use curl to test all the scenarios, in this case we have four scenarios to test (HTTP/HTTPS and www/non-www)

(1) HTTPS on www.abelperez.info - Expected 200

abel@ABEL-DESKTOP:~$ curl -L -I https://www.abelperez.info
HTTP/2 200 
content-type: text/html
content-length: 79
date: Fri, 02 Feb 2018 00:57:25 GMT
last-modified: Mon, 22 Jan 2018 18:58:47 GMT
etag: "87fa2caa5dc0f75975554d6291b2da71"
server: AmazonS3
x-cache: Miss from cloudfront
via: 1.1 19d823478cf075f6fae7a5cb1336751a.cloudfront.net (CloudFront)
x-amz-cf-id: 7np_vqutTogm9pKceNZ82Zim61Eb0E0D9fJBkFaqNHUz3LF63fEh2w==

(2) HTTPS on abelperez.info - Expected 301 to https://www.abelperez.info

abel@ABEL-DESKTOP:~$ curl -L -I https://abelperez.info
HTTP/2 301 
content-length: 0
location: https://www.abelperez.info/
date: Fri, 02 Feb 2018 01:00:11 GMT
server: AmazonS3
x-cache: Miss from cloudfront
via: 1.1 75235d68607fb64805e0649c6268c52b.cloudfront.net (CloudFront)
x-amz-cf-id: 6WSECgHhkvCqZLW7kInopHnovCPcKU56oNQCZiCv7gaQLv2wSu-Vcw==

HTTP/2 200 
content-type: text/html
content-length: 79
date: Fri, 02 Feb 2018 00:57:25 GMT
last-modified: Mon, 22 Jan 2018 18:58:47 GMT
etag: "87fa2caa5dc0f75975554d6291b2da71"
server: AmazonS3
x-cache: RefreshHit from cloudfront
via: 1.1 7158f458652a2c59cfcb688d5dc80347.cloudfront.net (CloudFront)
x-amz-cf-id: _U7qobfP61P2aYyOakzzfwWjkKYrBeKObtWziPv7NVb5M3yPMlsbrQ==

(3) HTTP on www.abelperez.info - Expected 301 to https://www.abelperez.info

abel@ABEL-DESKTOP:~$ curl -L -I http://www.abelperez.info
HTTP/1.1 301 Moved Permanently
Server: CloudFront
Date: Fri, 02 Feb 2018 01:00:32 GMT
Content-Type: text/html
Content-Length: 183
Connection: keep-alive
Location: https://www.abelperez.info/
X-Cache: Redirect from cloudfront
Via: 1.1 2c7c2f0c6eb6b2586e9f36a7740aa616.cloudfront.net (CloudFront)
X-Amz-Cf-Id: qVYxI7z1DSVpzGrIfGWtHI8dZ1Ywx6dPUf4qGmtXbxl71IvC5R6P6Q==

HTTP/2 200 
content-type: text/html
content-length: 79
date: Fri, 02 Feb 2018 00:57:25 GMT
last-modified: Mon, 22 Jan 2018 18:58:47 GMT
etag: "87fa2caa5dc0f75975554d6291b2da71"
server: AmazonS3
x-cache: RefreshHit from cloudfront
via: 1.1 6b11bd43fbd97ec7bb8917017ae0f954.cloudfront.net (CloudFront)
x-amz-cf-id: w1YRlI4QR5W_bxXVXftmGioMCWoeCpwcCqlj0ucPlizOZVev22RU6g==

(4) HTTP on abelperez.info - Expected 301 to https://abelperez.info which in turn will be another 301 to https://www.abelperez.info

abel@ABEL-DESKTOP:~$ curl -L -I http://abelperez.info
HTTP/1.1 301 Moved Permanently
Server: CloudFront
Date: Fri, 02 Feb 2018 01:01:00 GMT
Content-Type: text/html
Content-Length: 183
Connection: keep-alive
Location: https://abelperez.info/
X-Cache: Redirect from cloudfront
Via: 1.1 60d859e64626d7b8d0cc73d27d6f8134.cloudfront.net (CloudFront)
X-Amz-Cf-Id: eiJCl56CO6aUNA3xRnbf8J_liGfY3oI5jdLdhRRW4LoNFbCMunYyPg==

HTTP/2 301 
content-length: 0
location: https://www.abelperez.info/
date: Fri, 02 Feb 2018 01:01:01 GMT
server: AmazonS3
x-cache: Miss from cloudfront
via: 1.1 f030bd6bd539e06a932b0638e025c51d.cloudfront.net (CloudFront)
x-amz-cf-id: 7KfYWyxPhIXXJjybnISt25apbbHUKx74r9TUI9Kguhn2iQATZELfHg==

HTTP/2 200 
content-type: text/html
content-length: 79
date: Fri, 02 Feb 2018 00:57:25 GMT
last-modified: Mon, 22 Jan 2018 18:58:47 GMT
etag: "87fa2caa5dc0f75975554d6291b2da71"
server: AmazonS3
x-cache: RefreshHit from cloudfront
via: 1.1 3eebab739de5f3b3016088352ebea37f.cloudfront.net (CloudFront)
x-amz-cf-id: R8kB6ndn1K8YOiF6J2deG0QkHh-3QD65q0hfV5vdXm5-_1sNNlc3Ng==