Serverless static website - part 5

Once we have created the certificate, we need a bridge between a secure endpoint and the current buckets. One of the most common ways to address this is by using CloudFront content distribution service. By using this, we'll make our website to be distributed to some (depending on the pricing) of the Amazon AWS edge locations, which means that if we have some international audience for our website, it will be served to the client by the closest edge location.

Creating WWW Distribution

First, on AWS Console select CloudFront from the services list, under Networking & Content Delivery category. Once there, click on Create Distribution button, in this case, we are interested in a Web Distribution, so under Web, select Get Started. In the create distribution form, there are some key values that are required to enter in order to make this configuration to work. The rest can stay with default value

• Origin Domain Name: www.abelperez.info.s3-website-eu-west-1.amazonaws.com (which is the endpoint for the bucket)
• Viewer Protocol Policy: HTTP and HTTPS (this way if anyone tries to visit through HTTP, it will be redirected to HTTPS endpoint)
• Price Class: Depending on your needs, in my case, US, Canada en Europe is enough
• Alternate Domain Names(CNAMEs): www.abelperez.info
• SSL Certificate: Custom SSL Certificate (example.com): and select your certificate from the dropdown menu
• Default Root Object: index.html

Note: If you can't see your certificate in the dropdown, make sure you've created / imported in the region N. Virginia.

Once entered all this information, click Create Distribution

Creating non-WWW Distribution

Let's repeat the process to create another distribution, this time some values will be different, as we'll point to the non-www bucket and domain name.

• Origin Domain Name: abelperez.info.s3-website-eu-west-1.amazonaws.com (which is the endpoint for the bucket)
• Viewer Protocol Policy: HTTP and HTTPS (this way if anyone tries to visit through HTTP, it will be redirected to HTTPS endpoint)
• Price Class: Depending on your needs, in my case, US, Canada en Europe is enough
• Alternate Domain Names(CNAMEs): abelperez.info
• SSL Certificate: Custom SSL Certificate (example.com): and select your certificate from the dropdown menu (the same as above)
• Default Root Object: leave empty, since this distribution won't serve any file.

Once entered all this information, click Create Distribution, the creation process will take about 30 minutes, so be patient. When they're done, the status changes to Deployed and they can start receiving traffic. You'll see something like this:

Updating DNS records

Now that we have created the distributions, it's time to update how Route 53 is going to resolve DNS requests to CloudFront distributions instead of the S3 bucket previously configured. To do that, on Route 53 console, select the Hosted Zone, then select one record set, and updated the Alias Target to the corresponding CloudFront distribution Domain Name. Then repeat the process for the other record set.

The following diagram illustrates the interaction between the AWS services we've used up this point. The browser first queries DNS for the given domain name, Route 53, will resolve to the appropriate CloudFront distribution, which can serve the content if it has already cached, otherwise it will request from the bucket and then serve the content. If the request is over HTTP to CloudFront, it will issue a HTTP 301 redirection to the HTTPS endpoint. If the request is over HTTPS to CloudFront, it will use the assigned certificate, which in this case is the same for both www and non-www endpoints.

     < HTTP 301 - Redirecto to HTTPS
 +----------------------------+
 |                            |                                         
 |       GET HTTP >     +-----+------+             +----------------+
 |     abelperez.info   | CloudFront |  GET HTTPS  |   S3 Bucket    |
 |           +--------> | (non-www)  |-----------> | abelperez.info +--+
 |           |          |            |             |                |  |
 |           |          +------+-----+             +----------------+  |
 |     +-----+-----+            \                                      |
 |     |           |             V                                     |
 | +-> | Route 53  |       (SSL certficate)                            |
 | |   |           |             A                                     |
 | |   +-----+-----+            /                                      |
 | |         |          +------+-----+          +--------------------+ |
 | |         |          | CloudFront |GET HTTPS |     S3 Bucket      | |
 | |         +--------> |   (www)    |--------> | www.abelperez.info | |
 | |   GET HTTP >       |            |          |                    | |
 | | www.abelperez.info +-----+------+          +----------+---------+ |
 | |                          |                            |           |
 V |                          |                            |           |
+--+--------+ <---------------+                            |           |
|           |      < HTTP 301 - Redirecto to HTTPS         |           |
|           | <--------------------------------------------+           |
|  Browser  |      < HTTP 200 - OK                                     |
|           | <--------------------------------------------------------+
+-----------+      < HTTP 301 - Redirect to www

One more step

There is a bucket that its sole purpose is to redirect requests from non-www to www domain name, this bucket was set to use HTTP protocol, in this case, we are going to update it to HTTPS, so we can save one redirection step in the process.

Let's test it

Once again, we'll use curl to test all the scenarios, in this case we have four scenarios to test (HTTP/HTTPS and www/non-www)

(1) HTTPS on www.abelperez.info - Expected 200

abel@ABEL-DESKTOP:~$ curl -L -I https://www.abelperez.info
HTTP/2 200 
content-type: text/html
content-length: 79
date: Fri, 02 Feb 2018 00:57:25 GMT
last-modified: Mon, 22 Jan 2018 18:58:47 GMT
etag: "87fa2caa5dc0f75975554d6291b2da71"
server: AmazonS3
x-cache: Miss from cloudfront
via: 1.1 19d823478cf075f6fae7a5cb1336751a.cloudfront.net (CloudFront)
x-amz-cf-id: 7np_vqutTogm9pKceNZ82Zim61Eb0E0D9fJBkFaqNHUz3LF63fEh2w==

(2) HTTPS on abelperez.info - Expected 301 to https://www.abelperez.info

abel@ABEL-DESKTOP:~$ curl -L -I https://abelperez.info
HTTP/2 301 
content-length: 0
location: https://www.abelperez.info/
date: Fri, 02 Feb 2018 01:00:11 GMT
server: AmazonS3
x-cache: Miss from cloudfront
via: 1.1 75235d68607fb64805e0649c6268c52b.cloudfront.net (CloudFront)
x-amz-cf-id: 6WSECgHhkvCqZLW7kInopHnovCPcKU56oNQCZiCv7gaQLv2wSu-Vcw==

HTTP/2 200 
content-type: text/html
content-length: 79
date: Fri, 02 Feb 2018 00:57:25 GMT
last-modified: Mon, 22 Jan 2018 18:58:47 GMT
etag: "87fa2caa5dc0f75975554d6291b2da71"
server: AmazonS3
x-cache: RefreshHit from cloudfront
via: 1.1 7158f458652a2c59cfcb688d5dc80347.cloudfront.net (CloudFront)
x-amz-cf-id: _U7qobfP61P2aYyOakzzfwWjkKYrBeKObtWziPv7NVb5M3yPMlsbrQ==

(3) HTTP on www.abelperez.info - Expected 301 to https://www.abelperez.info

abel@ABEL-DESKTOP:~$ curl -L -I http://www.abelperez.info
HTTP/1.1 301 Moved Permanently
Server: CloudFront
Date: Fri, 02 Feb 2018 01:00:32 GMT
Content-Type: text/html
Content-Length: 183
Connection: keep-alive
Location: https://www.abelperez.info/
X-Cache: Redirect from cloudfront
Via: 1.1 2c7c2f0c6eb6b2586e9f36a7740aa616.cloudfront.net (CloudFront)
X-Amz-Cf-Id: qVYxI7z1DSVpzGrIfGWtHI8dZ1Ywx6dPUf4qGmtXbxl71IvC5R6P6Q==

HTTP/2 200 
content-type: text/html
content-length: 79
date: Fri, 02 Feb 2018 00:57:25 GMT
last-modified: Mon, 22 Jan 2018 18:58:47 GMT
etag: "87fa2caa5dc0f75975554d6291b2da71"
server: AmazonS3
x-cache: RefreshHit from cloudfront
via: 1.1 6b11bd43fbd97ec7bb8917017ae0f954.cloudfront.net (CloudFront)
x-amz-cf-id: w1YRlI4QR5W_bxXVXftmGioMCWoeCpwcCqlj0ucPlizOZVev22RU6g==

(4) HTTP on abelperez.info - Expected 301 to https://abelperez.info which in turn will be another 301 to https://www.abelperez.info

abel@ABEL-DESKTOP:~$ curl -L -I http://abelperez.info
HTTP/1.1 301 Moved Permanently
Server: CloudFront
Date: Fri, 02 Feb 2018 01:01:00 GMT
Content-Type: text/html
Content-Length: 183
Connection: keep-alive
Location: https://abelperez.info/
X-Cache: Redirect from cloudfront
Via: 1.1 60d859e64626d7b8d0cc73d27d6f8134.cloudfront.net (CloudFront)
X-Amz-Cf-Id: eiJCl56CO6aUNA3xRnbf8J_liGfY3oI5jdLdhRRW4LoNFbCMunYyPg==

HTTP/2 301 
content-length: 0
location: https://www.abelperez.info/
date: Fri, 02 Feb 2018 01:01:01 GMT
server: AmazonS3
x-cache: Miss from cloudfront
via: 1.1 f030bd6bd539e06a932b0638e025c51d.cloudfront.net (CloudFront)
x-amz-cf-id: 7KfYWyxPhIXXJjybnISt25apbbHUKx74r9TUI9Kguhn2iQATZELfHg==

HTTP/2 200 
content-type: text/html
content-length: 79
date: Fri, 02 Feb 2018 00:57:25 GMT
last-modified: Mon, 22 Jan 2018 18:58:47 GMT
etag: "87fa2caa5dc0f75975554d6291b2da71"
server: AmazonS3
x-cache: RefreshHit from cloudfront
via: 1.1 3eebab739de5f3b3016088352ebea37f.cloudfront.net (CloudFront)
x-amz-cf-id: R8kB6ndn1K8YOiF6J2deG0QkHh-3QD65q0hfV5vdXm5-_1sNNlc3Ng==

Serverless static website - part 4

Security is an aspect that should never be left behind, even if we are only displaying content to general public. As per Google's strategy to favour secure traffic, they've started to penalise websites that are only on HTTP, we are going to follow this idea and add HTTPS and SSL certificate to this website.

Requesting a certificate

On AWS console, we have a Certificate Manager under Security, Identity & Compliance category. Once there, we have two choices: Import a certificate or Request a certificate. We'll use the request option. It's important for this to work with further steps to make sure we are in N. Virginia region before requesting a certificate. Click on Request a certificate button

In this example, I'm interested in creating a certificate that will validate the main domain abelperez.info, the subdomain www.abelperez.info and any subdomain I create in the future, that is done by using the wildcard *.abelperez.info.

Next step is validate the identity, basically we need to prove that we in fact own the domain we are creating the certificate for. In this case I chose Email validation. The request in progress should look like this.

Verifying domain ownership

At this point, we should receive an email (or a few of them) asking to validate our email address by following a link.

The verification page looks like this one, just click I approve.

It's important to note that in this case, because I'm requesting a certificate for *.abelperez.info and abelperez.info, I'll receive two requests and I have to verify both of them, if that's your case, make sure you've verified all the requests, otherwise the operation will be incomplete, and the certificate won't be issued until all validation are completed. Once it's verified, the certificate should look like this.

Now that we have valid certificate for our domain, it can be attached to other entities like CloundFront CDN distribution, Elastic Load balancer, etc.

Serverless static website - part 3

To www or not to www ? well, that's not the real question. Apparently, nowadays this topic has changed a little bit from the early times of the World Wide Web. In this article, there are some interesting pieces of information for those interested in that topic.

The bottom line is whatever your choice is, there has to be consistency, in this example, the canonical name will be www.abelperez.info, but I also want that if anyone browses just abelperez.info, they have to be redirected to the www version. How do we do that on AWS ?

Creating the redirect bucket

As discussed earlier, an S3 bucket can be configured as a website, but it will only listen on a single hostname (www.abelperez.info), therefore if we want to listen on another hostname (abelperez.info), we need another S3 bucket, again matching the bucket name with the hostname.

On AWS S3 console, create a new bucket named (your domain name without www). This time, instead of selecting Use this bucket to host a website, let's select Redirect requests where:

• Target bucket or domain: is the www hosthame
• Protocol: will be http for now

The configuration should look like this

Creating the Record set

Now, we need to route DNS requests to the new website previously created. To do that, let's go to Route 53 console, select your hosted zone, then click on Create Record Set button. Specify the following values:

• Name: leave in blank - it will be non www DNS entry
• Type: A
• Alias: Yes
• Alias Target: You should see the bucket name in the list

What have we done so far ?

We have created two DNS entries pointing to two different S3 buckets (effectively two web servers). One web server (www) will serve the content as explained earlier. The other web server (non www) will issue 301 redirect code to the www version. This way, the browser will request again, but now with the www version which will get served by the www web server, delivering the content as expected.

The following diagram illustrates the workflow

                                                 +----------------+
                GET abelperez.info               |   S3 Bucket    |
                +------------------------------> | abelperez.info +----+
                |                                |                |    |
          +-----+-----+                          +----------------+    |
          |           |                                                |
      +-> | Route 53  |                                                |
      |   |           |                                                |
      |   +-----+-----+                      +--------------------+    |
      |         |                            |     S3 Bucket      |    |
      |         +--------------------------> | www.abelperez.info |    |
      |         GET www.abelperez.info       |                    |    |
      |                                      +----------+---------+    |
+-----+-----+                                           |              |
|           | <-----------------------------------------+              |
|  Browser  |        HTTP 200 - OK                                     |
|           | <--------------------------------------------------------+
+-----------+        HTTP 301 - Permanent Redirect to www

Let's test it

To test this behaviour, one easy way is using the universal tool CURL, we'll use two switches:

• -L Follows the redirects
• -I Fetches headers only

For more information about CURL, see the manpage.

There are two scenarios to test: first, when we try to hit non www host. We can see the first request gets HTTP 301 code and the second gets HTTP 200.

abel@ABEL-DESKTOP:~$ curl -L -I http://abelperez.info
HTTP/1.1 301 Moved Permanently
x-amz-id-2: pLVO9p67k51FJpZCSbF2LxJyrB8w9WyEkgNXHF0Zq8twe3Dw1ud3OiIHRzN0y5B4wDvwngLGEBg=
x-amz-request-id: AD13DD8436422AAC
Date: Sat, 27 Jan 2018 17:50:19 GMT
Location: http://www.abelperez.info/
Content-Length: 0
Server: AmazonS3

HTTP/1.1 200 OK
x-amz-id-2: s/6E2lV7nYtfBq96Qftwip7lzMvIkOMuIq0jbwCisYU0V7ujMRisPuqPsNt2vMuBWFIYuwkqLFs=
x-amz-request-id: 1C258F07FD183836
Date: Sat, 27 Jan 2018 17:50:19 GMT
Last-Modified: Wed, 08 Nov 2017 09:10:40 GMT
ETag: "a339a5d4a0ad6bb215a1cef5221b0f6a"
Content-Type: text/html
Content-Length: 85
Server: AmazonS3

The second scenario is trying to go directly to www host, and the request gets HTTP 200 straight away.

abel@ABEL-DESKTOP:~$ curl -L -I http://www.abelperez.info
HTTP/1.1 200 OK
x-amz-id-2: apCkPouaYzoy5gjemxU+BjDLbQxxE46EUhDXBHirq6PK0OZbubP2BVhWllxlSV99zg5UB3tGbd8=
x-amz-request-id: D928A1DF3B3EB0DE
Date: Sat, 27 Jan 2018 18:18:19 GMT
Last-Modified: Wed, 08 Nov 2017 09:10:40 GMT
ETag: "a339a5d4a0ad6bb215a1cef5221b0f6a"
Content-Type: text/html
Content-Length: 85
Server: AmazonS3

Serverless static website - part 2

In the previous part I explained how to create a s3 bucket and make it behave like a web server, then we put some files on it and were able to publicly browse using the HTTP endpoint provided by s3. However, this is not the most appealing url to associate with our website, we most likely want to own a domain name.

Buying a domain name

If we want a domain, there are a lot of places where you can buy one, they are usually cheap, unless we go crazy with the name. Here is the list of ICANN-Accredited Registrars where you can choose your favourite. There is one particular domain registrar that might be convenient, if you search amazon in that list, it will come up, yes, Amazon also sell domain names. In this example I purchased mine through them, just to keep everything in the same place, you are free to choose your domain registrar, it doesn't make a big difference when setting everything up.

Route 53 is the service that manages DNS on AWS. To register a domain, once in Route 53 console, go to Domains / Registered domains. Then Choose a domain name with your favourite TLD and follow the process, it's that simple.

When you register a new domain, it can take some times a couple of days to complete the process, this is indicated in the console by listing the new domain under Pending requests, once the process is complete it will appear under Registered domain. Your console should look like this by then. More information about registering a domain with Route 53, see here.

Creating a Public Hosted Zone

Once you have domain name of yours, whether registered with Route 53 or with another registrar, it's the time to create a public hosted zone, which is basically a container for all DNS records associated with your domain and subdomains. To do that, go to Route 53 console, choose Hosted Zones, hit the Create Hosted Zone button, then provide the domain name, and optionally a comment, also make sure Public Hosted Zone is selected. When it's done, it should look like this.

If you chose to buy your domain with a different registrar, then you'll have to update your name servers with the new ones created by the Hosted Zone. See AWS docs on how to do that, here.

Creating a Record Set

In AWS world, a Record Set is similar to standard DNS record, but with some extensions. In this particular case, we'll use Alias. To create a record, select the Hosted Zone and click Create Record Set button. Specify the following values:

• Name: www
• Type: A
• Alias: Yes
• Alias Target: You should see the bucket name in the list

It's important to note that the name of the bucket where we are hosting our files must match the record name. So in this case, my record is www.abelperez.info, that's exactly the name of the bucket, otherwise Route 53 won't be able to make the association between those two.

Click on Create button, and that's it. We've just linked the domain name to a serverless web server for static files.

Let's test it

To test all this, it's very simple, let's browse to http://www.abelperez.info (your domain of course) and see what happens

Your browser content should display this

For more information about Route 53 Alias records values, see here.

Serverless static website - part 1

Cloud computing has become nowadays one of the most popular practices, but it doesn't have to be limited only to a reduced group of specialists. In fact, I'll show you how to take advantage of this new thing called "serverless" on AWS to host a static website

The idea behind this is not to worry about the underlying infrastructure, AWS will handle that. In this case, we'll start by using one of the oldest services they provide, Amazon S3.

Accessing Amazon AWS

If you are new to Amazon AWS, you should start by creating a new account and use the free 1 year trial, which allows a lot of room for playing with it.

In this example I show a real domain I already own and a simple HTML file (very simple)

Creating the S3 bucket

Once you've logged in AWS console, choose Services from the top left menu, then S3 in Storage category or just browse to https://s3.console.aws.amazon.com/s3/

Click on Create Bucket and follow the wizard. It's important to note that if you're trying to use your own domain name within AWS (using Route 53) the name of your bucket has to match the domain name. In this case I'm using www.abelperez.info, therefore, that's my bucket name. Also, if you plan to use a CDN (using CloundFront distribution), this is not relevant, however I encourage this practice as it will organise better the buckets for their purpose. For more information about how to create a bucket, see here

Your S3 console should look similar to this

Converting the S3 bucket in a website

Next step, on the same console page, select the bucket and click on Properties, then find Static website hosting. In the expanded form, choose the option Use this bucket to host a website and specify index and error documents accordingly, typically the same as proposed by the watermark. Save and take not that the format of your HTTP endpoint (public url) is like http://www.abelperez.info.s3-website-eu-west-1.amazonaws.com

It follows the convention of http://<your-bucket-name>.s3-website-<region-where-it-was-created>.amazonaws.com

It should look like this

Granting public read access to all files

Granting public read to all objects is usually deemed as a bad practice, but in this particular case (and for now), we need a way to make public all files we upload to the bucket. S3 Buckets allow a very granular permission system on all objects inside a bucket, that means every time we upload a file, we are responsible for choosing the right permissions. However, this process can be tedious and prone to error (it's easy to forget)

Let's add a Bucket Policy that will entitle all clients request to read the objects inside our bucket. To do that, select the bucket in the console and click on Permissions tab, then click on Bucket Policy and add the following text, just replacing www.abelperez.info with your bucket name

{
    "Version": "2012-10-17",
    "Id": "PublicReadAccess",
    "Statement": [
        {
            "Sid": "GrantPublcReadAccess",
            "Effect": "Allow",
            "Principal": "*",
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::www.abelperez.info/*"
        }
    ]
}

Creating AWS policies is out of the scope of this post, but basically it allows the GetObject action to all objects inside the bucket named www.abelperez.info

Finally, upload some files

Just for demonstration purposes, I've uploaded a single HTML file, at the moment of this writing is located at https://www.abelperez.info/index.html. You can use any HTML, CSS, Javascript, images files you like.

On way of uploading files to a S3 bucket is by using the console, like we've done all the other operations, select the bucket, and using Upload and Create folder buttons you can recreate your website directory tree. There are better ways to upload files, for example, using the command line or the Rest API.

Your bucket content should look like this

Let's test it

To test all this, it's actually very simple, let's take the HTTP endpoint from earlier and paste in the browser http://www.abelperez.info.s3-website-eu-west-1.amazonaws.com

Your browser content should display this

For full AWS documentation about this topic, see here

Mercurial hooks to enforce commit rules

If you've been in the software industry for a while, there's something that's very clear, no software can be done on your own for too long. Working as a team requires some form organisation, some rules, some standards so everyone is on the same page. The more standardised, the easier to automate processes. Let's start on simple day to day tasks such as committing code to source control.

In the team where I'm working, we set a couple of rules for all to follow.

Commit messages normalized

Since we are supposed to associate every commit with a corresponding ticket on our trello board, that is a fundamental part of the commit message. The format we set is as follows:

[Committer/Committer] - ticket ### - commit message

Where:

Committer is the initials of the dev(s) involved in that particular commit

Ticket ### is the trello card number associated with the task

Commit message is a meaningful message that summarises what has changed

spaces and dashes are mandatory

Limiting the maximum number of committed files

A common rule of thumb is that we should limit the number of modified files to a maximum so we can trace and understand what has changed in order to review when we need to find some specific change. However, it's not set in stone what that maximum is, so we opted for a manageable amount of files, initially 10 but it turns out that it works better with 20 and it's still a manageable amount.

How do we enforce this?

After some research, I put my bet on mercurial hooks, since we use mercurial as source control and bitbucket as code repository. Mercurial hooks allows the possibility of executing custom code when some event happens, for example before the commit transaction is finished. At this point, we can know all about the changes that are about to be committed. The main point of interest is the number of files in the current changeset and the description aka commit message.

Before considering a commit as valid, the previously mentioned rules are verified and if one of them is invalid, the commit is aborted.

What if we are merging ?

There is a scenario that needs to be treated differently, it's when a commit is done after the result of a merge. Even if every single commit is limited to a maximum number of modified files, when merging with a different branch such as stable cut, inevitably there will be a lot more files than the maximum and that's acceptable, so in this case, we ignore the rule of the number of files and the commit message should be different i.e Merge with stable.

Putting it all together

Installing the hook.

Inside .hg folder, there is a file named hgrc, it defines mercurial configuration for the current repository. Add the following lines.

[hooks]
pretxncommit = python:hooks/commit-rules.py:checkCommit

Here we assign the python function checkCommit to pretxtcommit event hook, and such function is located inside hooks/commit-rules.py file relative to the repository root. I was tempted to keep this file inside .hg but this is not possible.

Let's inspect what's inside this commit-rules.py

import re

maximum_files = 20

def checkCommit(ui, repo, **kwargs):
 
 # Check the commit message for compliance with our team rules, if not compliant
 # Then abort the commit transaction displaying useful information to the user
 hg_commit_message = repo['tip'].description()
 is_incorrect_message = checkMessage(ui, hg_commit_message)
 if is_incorrect_message:
  return True

 # Check if the current revision has more than one parent
 # if more than one, the we are on a merge => do not check the number of files
 revision_parents = len(repo['tip'].parents())
 if revision_parents > 1:
  return False
 
 # Check the number of files on current revision, if they're more than the maximum 
 # Then abort the commit transaction displaying useful information to the user
 repo_files_count = len(repo['tip'].files()) 
 is_incorrect_commit_files = checkFiles(ui, repo_files_count)
 if is_incorrect_commit_files:
  return True
  
 return False

def checkMessage(ui, message): 
 
 # Regular expression to check the commit message structure - for a single commit
 # Valid examples:
 #
 # [DG/APM] - ticket 123 - Two committers text for a valid commit
 # [APM] - ticket 123 - Single committer text for a valid commit 
 #
 commit_re = re.compile(r'(\[[A-Za-z\/]+\])\s\-\sticket\s(\d+)\s\-\s(.*)')
 commit_check = commit_re.match(message)
 
 # regular expression to check the commit message structure - for a merge commit
 # Valid examples:
 #
 # Merge with stable
 # Merge with default
 #
 merge_re = re.compile(r'Merge with [A-Za-z0-9- ]+')
 merge_check = merge_re.match(message)
 
 if not commit_check and not merge_check:
  ui.warn('Commit message does not comply with our team rules\n')
  ui.warn('\n')
  ui.warn('Use format "[Committer/Commiter] - ticket ### - commit message"\n')
  ui.warn('\n')
  ui.warn('* Committer initials should be separated by /\n')
  ui.warn('* Ticket ### should match Trello ticket number\n')
  ui.warn('\n')
  ui.warn('For merge, use format "Merge with branch-name"\n')
  ui.warn('\n')
  
  # return true if is invalid 
  return True
 
 return False

def checkFiles(ui, files):
 if (files > maximum_files):
  ui.warn("Commit with too many files (" + str(files) + ") - Maximum files (" + str(maximum_files) +")\n")
  # return true if is invalid 
  return True
 
 return False

Since this code has to run on each dev computer and does not propagate for security reasons according to mercurial documentation, we made a way to "install" it when the build operation is run in Visual Studio by creating a pre build event that runs a powershell script.

Here is the powershell script that installs the hook when it's not already there.

Param(
    [Parameter(Mandatory=$True)]
    [string]$hgrcFile
)

if (-not (Test-Path $hgrcFile) ) {
    exit
}

#$hgrcFile = ".\.hg\hgrc"
$hgrc = Get-Content $hgrcFile -ErrorAction Stop

$lineHook = "[hooks]"
$linePreTxnCommit = "pretxncommit = python:hooks/commit-rules.py:checkCommit"
$containsHooks = $hgrc.Contains($lineHook)
$containsPreTxtCommit = $hgrc.Contains($linePreTxnCommit)

if (-not ($containsHooks -and $containsPreTxtCommit)) 
{
    Write-Host "hook not installed. Installing ..."

    $hgrc += ""
    $hgrc += $lineHook
    $hgrc += $linePreTxnCommit
    
    $hgrc | Out-File $hgrcFile -Encoding utf8

    Write-Host "Done"
}

And here is the pre build event fragment from the main .csproj file in our solution

<PropertyGroup>
    <PreBuildEvent>
        powershell -ExecutionPolicy ByPass -File "$(SolutionDir)install-hook.ps1" -hgrcFile "$(SolutionDir).hg\hgrc"
    </PreBuildEvent>
</PropertyGroup>

For more reference about mercurial hooks, check Mercurial hooks examples

A bug hidden for 4 years !

I have to admit I've abandoned my file uploader for some time, and I've been revisiting the code recently as part of some experiments with .NET Core 2.

After some work done to make it work with the new framework, to my surprise, it just looked to work perfectly until I tried to validate one of the basic implicit requirements: the uploaded copy must be exactly the same as the original file. It turned out that even if the files had the same exact size, to the byte, they weren't identical, how was that possible ?

Symptom:

After uploading a file larger than the packet size (default to 4 MB), the resulting copy was not identical to the original.

Simple test:

Upload a zip file larger than 4 MB, try to open it. It was corrupt.

The most uprising fact was that if I ran the .NET 4 version it still worked as expected ! Next, I migrated the solution to .NET 4.7 and still worked as expected, apparently it was only affecting to .NET Core version. So, under the circumstance, I tried to compare request / response side by side working version vs bugged version.

A few things looked a bit suspicious:

• some extra headers sent on .NET working version
• Transfer-Encoding: chunked from .NET Core bugged version

None of those were related. Back to square one.

I tried reducing the packet size to 1024 bytes and upload a text file so I could see how the content differed but no luck, it was indeed the same (verified comparing documents with Diff tool) Well now deep debugging showed that the buffer wasn't being written completely to output file, that would explain the corrupted resulting file but not the size difference. Let's see the code.

Original version

var buffer = new byte[contentLength];
var bytesread = inputStream.Read(buffer, 0, buffer.Length);
if (bytesread > 0)
{
    using (var fs = File.Open(actualfile, FileMode.Append, FileAccess.Write))
    {
        fs.Write(buffer, 0, buffer.Length);
    }
    //json.Data = new {guid, packet};
    return new UploadPacketResult(guid, packet);
}

Spot something weird ?

It turns out that Stream.Read can perform a blocking operation that reads up to the length specified but not necessarily the whole buffer, now it's obvious! but how it didn't fail before !! I don't have an answer for that :)

Well, here is the fixed version

var buffer = new byte[contentLength];
using (var fs = File.Open(actualfile, FileMode.Append, FileAccess.Write))
{
    int read = 0;
    while ((read = inputStream.Read(buffer, 0, buffer.Length)) > 0)
    {
        fs.Write(buffer, 0, read);
    }
}
//json.Data = new {guid, packet};
return new UploadPacketResult(guid, packet);

And that's the way it was supposed to be done in the first place, but there was no test case that proved otherwise (until now, 4 years later)