Serverless static website - part 9

One of the greatest benefits of cloud computing is the ability to automate processes and up to this point, we've learned how to set everything up via AWS console web interface.

Why automate?

It is always good to know how to manage stuff via console in case we need to manually modify something, but we should aim to avoid this practice. Instead, limit the use of the console to a bare minimum and the rest of the time, aim for some automated way, this has the following advantages:

• We can keep source code / templates under source control, allowing to keep track of changes.
• It can be reused to replicate in case of a new customer / new environment / etc.
• No need to remember where every single option is located as the UI can change.
• It can be transferred to another account in a matter of a few minutes.

In AWS world, the automated process is achieved by creating templates in CloudFormation and deploying them to stacks.

I have already created a couple of CloudFormation templates to automate all the process described to this point, they can be found at my GitHub repo.

CloudFormation templates

In order to automate this infrastructure, I've divided resources into two separate templates: one containing the SSL certificate and the other containing all the rest of the resources. The reason why the SSL certificate is in another template is because it needs to be run on N. Virginia region (US-East-1) as explained earlier when we created it manually, it's a CloudFront requirement.

Templates can contain parameters that make them more flexible, in this case, there is a parameter that controls the creation of a redirection bucket, we might have a scenario when we want just a website on a sub domain and we might not want to redirect from the naked domain. These are the parameters:

SSL Certificate template

• DomainName: The site domain name (naked domain only)

Infrastructure template

• HostedZone: This is just for reference, it should not be changed
• SSLCertificate: This should be taken from the output of the SSL certificate template
• DomainName: Same as above, only naked domain
• SubDomainName: specific sub domain, typically www
• IncludeRedirectToSubDomain: Whether it should include a redirection bucket from the naked domain to the subdomain

Creating SSL certificate Stack

First, let's make sure we are in N. Virginia region. Go to CloudFormation console, once there, click Create Stack button. We are presented with Select Template screen, where we'll choose a template from my repository (ssl-certificate.yaml) by selecting Upload a template to Amazon S3 radio button.

Click Next, you'll see the input parameters page including the stack name which I'll name abelperez-info-ssl-stack to give it some meaningful name.

After entering the required information, click Next and Next again, then on Create button. You'll see the Create in progress status in the stack.

At this point, the SSL certificate is being created and will require the identity verification just like when it was created manually, this will block the stack creation until the validation process is finished, so go ahead and check your email and follow the link to validate the identity to proceed with the stack creation.

Once the creation is done you'll see the Create complete status in the stack, on the lower pane, select Outputs, you'll find SSLCertificateArn. Take that value and copy it somewhere temporarily, we'll need it for our next stack.

Creating Infrastructure Stack

Following a similar process, let's create the second stack containing most of the resources to provision. In this case we are not forced to create it in any specific region, we can choose any provided all the services are available, for this example I'll Ireland (EU-West-1). The template can be downloaded from my repository (infrastructure.yaml).

This time, you are presented a different set of parameters, SSL Certificate will be the output of the previous stack as explained above. Domain name will be exactly the same as in the previous stack, give we are using the SSL certificate for this domain. Subdomain will be www and I'll include a redirection as I expect users to be redirected from abelperez.info to www.abelperez.info. I'll name the stack abelperez-info-infra-stack just to make it meaningful.

Since this template will create IAM users, roles and policies, we need to acknowledge this by ticking the box.

Once we hit Create, we can see the Create in progress screen.

This process can take up to 30 minutes, so please be patient, this takes so long time to create the stack because we are provisioning CloudFront distributions and they can take some time to propagate.

Once the stack is created, we can take note of a couple of values from the output: CodeCommit repository url (either SSH or HTTPS) and the Static bucket name.

Manual steps to finish the set up.

With all resource automatically provisioned by the templates, we are in a position where the only thing we need is to link our local SSH key with the IAM user. To do that, let's do exactly what we did when it was set up manually in part 6.

In my case, I chose to use SSH key, so I went to IAM console, found the user, under Security Credentials, I uploaded my SSH public key.

We also need to update the buildspec.yml to run our build, can be downloaded from the above linked GitHub repository. The placeholder <INSERT-YOUR-BUCKET-NAME-HERE> must be replaced with the actual bucket name, in this instance the bucket name generated is abelperez-info-infra-stack-staticsitebucket-1ur0k115f2757 and my buildspec.yml looks like:

version: 0.2

phases:
  build:
    commands:
      - mkdir dist
      - cp *.html dist/

  post_build:
    commands:
      - aws s3 sync ./dist 
        s3://abelperez-info-infra-stack-staticsitebucket-1ur0k115f2757/ 
        --delete --acl=public-read

Let's test it!

Our test consist of cloning the git repository from CodeCommit, add two files: index.html and buildspec.yml. Then we'll perform a git push and we expect it will trigger the build executing the s3 sync command and copying the index.html to our destination bucket which will be behind CloudFront and CNamed by Route 53. In the end, we should be able to just browse www.abelperez.info and get whatever the result is in index.html just uploaded.

Just a note, if you get a HTTP 403 instead of the expected HTML, then just wait for a few minutes, CloudFront/Route 53 might not be fully propagated just yet.

abel@ABEL-DESKTOP:~$ git clone ssh://git-codecommit.eu-west-1.amazonaws.com/v1/repos/www.abelperez.info-web
Cloning into 'www.abelperez.info-web'...
warning: You appear to have cloned an empty repository.
abel@ABEL-DESKTOP:~$ cd www.abelperez.info-web/
abel@ABEL-DESKTOP:~/www.abelperez.info-web$ git add buildspec.yml index.html 
abel@ABEL-DESKTOP:~/www.abelperez.info-web$ git commit -m "initial commit"
[master (root-commit) dc40888] initial commit
 Committer: Abel Perez Martinez 
Your name and email address were configured automatically based
on your username and hostname. Please check that they are accurate.
You can suppress this message by setting them explicitly. Run the
following command and follow the instructions in your editor to edit
your configuration file:

    git config --global --edit

After doing this, you may fix the identity used for this commit with:

    git commit --amend --reset-author

 2 files changed, 16 insertions(+)
 create mode 100644 buildspec.yml
 create mode 100644 index.html
abel@ABEL-DESKTOP:~/www.abelperez.info-web$ git push
Counting objects: 4, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (4/4), done.
Writing objects: 100% (4/4), 480 bytes | 0 bytes/s, done.
Total 4 (delta 0), reused 0 (delta 0)
To ssh://git-codecommit.eu-west-1.amazonaws.com/v1/repos/www.abelperez.info-web
 * [new branch]      master -> master
abel@ABEL-DESKTOP:~/www.abelperez.info-web$ curl https://www.abelperez.info
<html>
<body>
<h1>Hello from S3 bucket :) </h1>
</body>
</html>

Serverless static website - part 7

Now that we have our source code versioned and safely stored, we need a way to make it go to our S3 bucket, otherwise it won't be visible to the public. There are, as usual, many ways to address this scenario, but as a developer, I can't live without CI/CD integration for all my processes. In this case, we have a very simple process, and we can combine the building and deploying steps in one go. To do that, once again, AWS offers some useful services, I'll be using CodeBuild for this task.

Creating the access policy

Just like anything in AWS, a CodeBuild project needs access to other AWS services and resources. The way this is done is by applying service roles with policies. As per AWS docs, a CodeBuild project needs the following permissions:

• logs:CreateLogGroup
• logs:CreateLogStream
• logs:PutLogEvents
• codecommit:GitPull
• s3:GetObject
• s3:GetObjectVersion
• s3:PutObject

The difference is that the proposed policy in the docs does not restrict access to any resource, it suggests to change the * to something more specific. The following is the resulting policy restricting access to logs in the log group created by the CodeBuild project. Also I've added full access to the bucket, since we'll be copying files to that bucket. In summary, the policy below grants the following permissions (account id is masked with 111111111111):

• Access to create new log groups in CloudWatch
• Access to create new log streams within the log group created by the build project
• Access to create new log events within the log stream created by the build project
• Access to pull data through Git to CodeCommit repository
• Full Access to our S3 bucket where the files will go

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "logs:CreateLogGroup",
                "s3:ListObjects"
            ],
            "Resource": "*",
            "Effect": "Allow",
            "Sid": "AccessLogGroupsS3ListObjects"
        },
        {
            "Action": [
                "codecommit:GitPull",
                "logs:CreateLogStream",
                "logs:PutLogEvents"
            ],
            "Resource": [
                "arn:aws:logs:eu-west-1:111111111111:log-group:/aws/codebuild/
                 abelperez-stack-builder",
                "arn:aws:logs:eu-west-1:111111111111:log-group:/aws/codebuild/
                 abelperez-stack-builder:*",
                "arn:aws:codecommit:eu-west-1:111111111111:abelperez.info-web"
            ],
            "Effect": "Allow",
            "Sid": "AccessGitLogs"
        },
        {
            "Action": "s3:*",
            "Resource": [
                "arn:aws:s3:::www.abelperez.info",
                "arn:aws:s3:::www.abelperez.info/*"
            ],
            "Effect": "Allow",
            "Sid": "AccessS3staticBucket"
        }
    ]
}

Creating the service role

Go to IAM console, select Roles from the left hand side menu, click on Create Role. On the Select type of trusted entity screen, select AWS service EC2, Lambda and others and from the options, CodeBuild

Click Next: Permissions, at this point, we can create a policy with the JSON above or continue to Review and then add the policy later. To create the policy, click Create Policy button, select JSON tab, paste the JSON above, Review and Create Policy, then back to role permission screen, refresh and select the policy. Give it a name, something like CodeBuildRole or anything meaningful and click on Create Role to finish the process.

Creating the CodeBuild project

With all permissions set, let's start with the actual CodeBuild project, where we'll specify the source code origin, build environment, artifacts and role. On CodeBuild console, if it's the first project, click Get Started, otherwise select Build projects from the left hand side menu, and click on Create Project

Enter the following values:

• Project name* a meaningful name i.e "abelperez-stack-builder"
• Source provider* select "AWS CodeCommit"
• Repository* select the repository previously created
• Environment image* select "Use an image managed by AWS CodeBuild"
• Operating system* select "Ubuntu"
• Runtime* select "Node.js"
• Runtime version* select "aws/codebuild/nodejs:6.3.1"
• Buildspec name leave default "buildspec.yml"
• Artifacts: Type* select "No artifacts"
• Role name* select the previously created role

After finishing all the input, review and save the project.

Creating the build spec file

When creating the CodeBuild project, one of the input parameters is the Buildspec file name, which refers to the description of our build process. In this case, it's very simple process:

• BUILD: Create a directory dist to copy all output files
• BUILD: Copy all .html file to dist
• POST_BUILD: Synchronise all content of dist folder with www.abelperez.info S3 bucket

The complete buildspec.yml containing the above steps:

version: 0.2

phases:
  build:
    commands:
      - mkdir dist
      - cp *.html dist/

  post_build:
    commands:
      - aws s3 sync ./dist s3://www.abelperez.info/ --delete --acl=public-read

The last step is executed on post build phase, which is at the end of the all build steps when everything is ready to package / deploy or in this case ship to our S3 bucket. This is done by using the sync subcommand of s3 service from AWS Command line interface. It basically takes an origin and a destination and make them to be in sync.

--delete instructs to remove files in the destination that are not in the source, because by default they are not deleted.

--acl=public-read instructs to edit bucket objects' ACL, in this case read access to everyone.

At this point, we can remove the bucket policy previously created if we wish so, since every single file will be publicly accessible, that would allow to have other folders in the same bucket and not being accessible. It will depend on the use case for the website.

Let's test it

It's time to verify if everything is in the right place. We have the build project with some basic commands to run and assuming the above code is saved in a file named buildspec.yml at root level of the code repository, the next step is to commit and push that file to CodeCommit.

Let's go back to CodeBuild console, select the build project we've just created and click Start build, then choose the right branch (master in my case) and once again Start build.

Serverless static website - part 6

Up to this point we've created a simple HTML page and set up all the plumbing to make it visible to the world in a distributed and secure way. Now, if we want to update the content, we can either go to S3 console and upload manually the files or from the AWS CLI synchronise a local folder with a bucket. None of those ways seem to scale in the long run. As a developer I like to keep things under control, even more, under source control.

It can be done by using any popular cloud hosted version control systems, such as github, but in this example I'll use the one provided by AWS, it's called CodeCommit and it's a Git repository compatible with all current git tools.

Creating the repository

In CodeCommit console, which can be accessed by selecting Services, then under Developer Tools, select CodeCommit. Once there, click on Create repository button, or Get started if you haven't created any repository before.

Next, give it a name and a description and you'll see something like this:

Creating the user

Now, we need a user for ourselves or if someone else is going to contribute to that repository. There are many ways to address this part, I'll go for creating a user with a specific policy to grant access to that particular CodeCommit repository

In IAM (Identity and Access Management) console, select Users from left hand side menu, then click Add user. Next, give it a name and Programmatic access, this user won't access the console so it doesn't need login and password, not even access keys.

Click Next, Skip permissions for now, we'll deal with that in the next section. Review and create the user.

Creating the policy

Back to IAM console dashboard, select Policies from the left menu, then click on Create policy, select JSON tab and copy the following:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AllowGitCommandlineOperations",
            "Effect": "Allow",
            "Action": [
                "codecommit:GitPull",
                "codecommit:GitPush"
            ],
            "Resource": [
                "arn:aws:codecommit:::abelperez.info-web"
            ]
        }
    ]
}

Details about creating policies is beyond the scope of this post, but essentially this policy grants pull and push operations on abelperez.info-web repository. Click Review Policy and give it a meaningful name such as CodeCommit-MyRepoUser-Policy or something that states what permissions are granted, just to keep everything organised. Create the policy and you'll be able to to see it if filtering by the name.

Assigning the policy to the user

Back to IAM console dashboard, select Users, select the user we've created before, on tab Permissions, click Add Permissions

Once there, select Attach existing policies directly from the top three buttons. Then, filter by Customer managed policies to make it easier to find our policy (the one created before). Select the policy, Review and Add Permissions.

When it's done, we can see the permission added to the user, like below.

Granting access via SSH key

Once we've assigned the policy, the user should have permission to use git pull / push operations, now we need to link that IAM user with a git user (so to speak). To do that, CodeCommit provides two options: via HTTPS and SSH. The simpler way is using SSH, it only requires to upload the SSH public key to IAM user settings, then add some host/key/id information to SSH configuration file and that's it.

In IAM console, select the previously created user, then on Security Credentials tab, scroll down all the way to SSH keys for AWS CodeCommit, then click on Upload SSH public key button, paste your SSH public key. If you don't have one created yet, have a look at here and for more info, here. Click Upload SSH public key button and we are good to go.

If you are interested in accessing via HTTPS, then have a look at this documentation page on AWS website.

Back to CodeCommit console, select the repository previously created, then on the right hand side, click Connect button. AWS will show a panel with instructions, depending on your operating system, but it's essentially like this:

The placeholder Your-IAM-SSH-Key-ID-Here refers to the ID auto generated by IAM when you upload your SSH key and it has the format like APKAIWBASSHIDEXAMPLE

Let's test it

The following command sequence has been executed after setting up all the previous steps on AWS console. Started by cloning an empty repo, then created a new file. Added and committed that file to the repository and finally pushed that commit to the remote repository

abel@ABEL-DESKTOP:~/Downloads$ git clone ssh://git-codecommit.eu-west-1
.amazonaws.com/v1/repos/abelperez.info-web
Cloning into 'abelperez.info-web'...
warning: You appear to have cloned an empty repository.
abel@ABEL-DESKTOP:~/Downloads$ cd abelperez.info-web/
abel@ABEL-DESKTOP:~/Downloads/abelperez.info-web$ ls
abel@ABEL-DESKTOP:~/Downloads/abelperez.info-web$ echo "<h1>New file</h1>" > index.html
abel@ABEL-DESKTOP:~/Downloads/abelperez.info-web$ cat index.html 
<h1>New file</h1>
abel@ABEL-DESKTOP:~/Downloads/abelperez.info-web$ git add index.html 
abel@ABEL-DESKTOP:~/Downloads/abelperez.info-web$ git commit -m "first file"
[master (root-commit) c81ad93] first file
 Committer: Abel Perez Martinez <abel@ABEL-DESKTOP>
Your name and email address were configured automatically based
on your username and hostname. Please check that they are accurate.
You can suppress this message by setting them explicitly. Run the
following command and follow the instructions in your editor to edit
your configuration file:

    git config --global --edit

After doing this, you may fix the identity used for this commit with:

    git commit --amend --reset-author

 1 file changed, 1 insertion(+)
 create mode 100644 index.html
abel@ABEL-DESKTOP:~/Downloads/abelperez.info-web$ git status
On branch master
Your branch is based on 'origin/master', but the upstream is gone.
  (use "git branch --unset-upstream" to fixup)
nothing to commit, working tree clean
abel@ABEL-DESKTOP:~/Downloads/abelperez.info-web$ git push
Counting objects: 3, done.
Writing objects: 100% (3/3), 239 bytes | 0 bytes/s, done.
Total 3 (delta 0), reused 0 (delta 0)
To ssh://git-codecommit.eu-west-1.amazonaws.com/v1/repos/abelperez.info-web
 * [new branch]      master -> master

After this, as expected, the new file is now visible from CodeCommit console.

Mercurial hooks to enforce commit rules

If you've been in the software industry for a while, there's something that's very clear, no software can be done on your own for too long. Working as a team requires some form organisation, some rules, some standards so everyone is on the same page. The more standardised, the easier to automate processes. Let's start on simple day to day tasks such as committing code to source control.

In the team where I'm working, we set a couple of rules for all to follow.

Commit messages normalized

Since we are supposed to associate every commit with a corresponding ticket on our trello board, that is a fundamental part of the commit message. The format we set is as follows:

[Committer/Committer] - ticket ### - commit message

Where:

Committer is the initials of the dev(s) involved in that particular commit

Ticket ### is the trello card number associated with the task

Commit message is a meaningful message that summarises what has changed

spaces and dashes are mandatory

Limiting the maximum number of committed files

A common rule of thumb is that we should limit the number of modified files to a maximum so we can trace and understand what has changed in order to review when we need to find some specific change. However, it's not set in stone what that maximum is, so we opted for a manageable amount of files, initially 10 but it turns out that it works better with 20 and it's still a manageable amount.

How do we enforce this?

After some research, I put my bet on mercurial hooks, since we use mercurial as source control and bitbucket as code repository. Mercurial hooks allows the possibility of executing custom code when some event happens, for example before the commit transaction is finished. At this point, we can know all about the changes that are about to be committed. The main point of interest is the number of files in the current changeset and the description aka commit message.

Before considering a commit as valid, the previously mentioned rules are verified and if one of them is invalid, the commit is aborted.

What if we are merging ?

There is a scenario that needs to be treated differently, it's when a commit is done after the result of a merge. Even if every single commit is limited to a maximum number of modified files, when merging with a different branch such as stable cut, inevitably there will be a lot more files than the maximum and that's acceptable, so in this case, we ignore the rule of the number of files and the commit message should be different i.e Merge with stable.

Putting it all together

Installing the hook.

Inside .hg folder, there is a file named hgrc, it defines mercurial configuration for the current repository. Add the following lines.

[hooks]
pretxncommit = python:hooks/commit-rules.py:checkCommit

Here we assign the python function checkCommit to pretxtcommit event hook, and such function is located inside hooks/commit-rules.py file relative to the repository root. I was tempted to keep this file inside .hg but this is not possible.

Let's inspect what's inside this commit-rules.py

import re

maximum_files = 20

def checkCommit(ui, repo, **kwargs):
 
 # Check the commit message for compliance with our team rules, if not compliant
 # Then abort the commit transaction displaying useful information to the user
 hg_commit_message = repo['tip'].description()
 is_incorrect_message = checkMessage(ui, hg_commit_message)
 if is_incorrect_message:
  return True

 # Check if the current revision has more than one parent
 # if more than one, the we are on a merge => do not check the number of files
 revision_parents = len(repo['tip'].parents())
 if revision_parents > 1:
  return False
 
 # Check the number of files on current revision, if they're more than the maximum 
 # Then abort the commit transaction displaying useful information to the user
 repo_files_count = len(repo['tip'].files()) 
 is_incorrect_commit_files = checkFiles(ui, repo_files_count)
 if is_incorrect_commit_files:
  return True
  
 return False

def checkMessage(ui, message): 
 
 # Regular expression to check the commit message structure - for a single commit
 # Valid examples:
 #
 # [DG/APM] - ticket 123 - Two committers text for a valid commit
 # [APM] - ticket 123 - Single committer text for a valid commit 
 #
 commit_re = re.compile(r'(\[[A-Za-z\/]+\])\s\-\sticket\s(\d+)\s\-\s(.*)')
 commit_check = commit_re.match(message)
 
 # regular expression to check the commit message structure - for a merge commit
 # Valid examples:
 #
 # Merge with stable
 # Merge with default
 #
 merge_re = re.compile(r'Merge with [A-Za-z0-9- ]+')
 merge_check = merge_re.match(message)
 
 if not commit_check and not merge_check:
  ui.warn('Commit message does not comply with our team rules\n')
  ui.warn('\n')
  ui.warn('Use format "[Committer/Commiter] - ticket ### - commit message"\n')
  ui.warn('\n')
  ui.warn('* Committer initials should be separated by /\n')
  ui.warn('* Ticket ### should match Trello ticket number\n')
  ui.warn('\n')
  ui.warn('For merge, use format "Merge with branch-name"\n')
  ui.warn('\n')
  
  # return true if is invalid 
  return True
 
 return False

def checkFiles(ui, files):
 if (files > maximum_files):
  ui.warn("Commit with too many files (" + str(files) + ") - Maximum files (" + str(maximum_files) +")\n")
  # return true if is invalid 
  return True
 
 return False

Since this code has to run on each dev computer and does not propagate for security reasons according to mercurial documentation, we made a way to "install" it when the build operation is run in Visual Studio by creating a pre build event that runs a powershell script.

Here is the powershell script that installs the hook when it's not already there.

Param(
    [Parameter(Mandatory=$True)]
    [string]$hgrcFile
)

if (-not (Test-Path $hgrcFile) ) {
    exit
}

#$hgrcFile = ".\.hg\hgrc"
$hgrc = Get-Content $hgrcFile -ErrorAction Stop

$lineHook = "[hooks]"
$linePreTxnCommit = "pretxncommit = python:hooks/commit-rules.py:checkCommit"
$containsHooks = $hgrc.Contains($lineHook)
$containsPreTxtCommit = $hgrc.Contains($linePreTxnCommit)

if (-not ($containsHooks -and $containsPreTxtCommit)) 
{
    Write-Host "hook not installed. Installing ..."

    $hgrc += ""
    $hgrc += $lineHook
    $hgrc += $linePreTxnCommit
    
    $hgrc | Out-File $hgrcFile -Encoding utf8

    Write-Host "Done"
}

And here is the pre build event fragment from the main .csproj file in our solution

<PropertyGroup>
    <PreBuildEvent>
        powershell -ExecutionPolicy ByPass -File "$(SolutionDir)install-hook.ps1" -hgrcFile "$(SolutionDir).hg\hgrc"
    </PreBuildEvent>
</PropertyGroup>

For more reference about mercurial hooks, check Mercurial hooks examples