Wednesday, 12 July 2023

My journey through BLFS - part 2

Continuing from where I left in part 1 with a functional X Windows System, it's time to aim for something closer to a usable system.

16 May 2022 - GTK+ 2

I cloned the VM to have a checkpoint to restart from in case anything went horribly wrong. At this point there are many possible paths but I decided to go for LXDE due to its small memory footprint and dependencies compared to other more mainstream desktop environments.

Building GTK+ 2

GTK+ is the centerpiece for all graphics used in LXDE. I started with version 2. As explained in the guide, some libraries will need to be rebuilt with extra support previously disabled. On that note I had to rebuild cairo, pango, harfbuzz due to the later addition of gobject-introspection.

GTK+ 2 up and running from the demo app.

Quick gotcha: ISO Codes source code was not available from the proposed location, instead I got it from http://ftp.debian.org/debian/pool/main/i/iso-codes/iso-codes_4.9.0.orig.tar.xz not a big issue but it required a bit of googling around.

I wanted to see if it was possible to build LXDE with GTK+ version 3, so I built GTK+ 3, here is how it looks from the demo app.

17 May 2022

LSB Tools link is broken it gives 404, instead of releases, I got it from Github repositor tags page https://github.com/djlucas/LSB-Tools/archive/refs/tags/v0.9.tar.gz

Building apps with GTK+ 3 support.

Building LXSession

./configure \
  --prefix=/usr \
  --enable-gtk3 \
  --enable-buildin-clipboard \
  --enable-buildin-polkit

Building PCManFM

./configure \
  --prefix=/usr \
  --sysconfdir=/etc \
  --with-gtk=3

Building libindicator-12.10.1

It has a bug which is allegedly fixed, however, I could find the issue in the source code I downloaded from https://launchpad.net/libindicator/+download as per this comment. https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=806470.

CFLAGS=-Wno-error ./configure \
  --prefix=/usr \
  --with-gtk=3 \
  --enable-deprecations=yes

18 May 2022

I tried many different ways to get LXPanel built with GTK+ 3 to work. Long story short, I couldn’t.

Building LXPanel (not possible to install with GTK 3)

./configure \
  --prefix=/usr \
  --with-plugins=-netstat

It required wireless-tools-devel.

Building libwnck, download source from: https://download.gnome.org/sources/libwnck/3.24/libwnck-3.24.1.tar.xz

./configure \
  --prefix=/usr \
  --disable-static

Building keybinder, download source from: https://github.com/kupferlauncher/keybinder/releases/download/keybinder-3.0-v0.3.2/keybinder-3.0-0.3.2.tar.gz

./configure --prefix=/usr

End of LXPanel, at this point all dependencies were met and LXPanel is installed.

Building LXAppearance

./configure \
  --prefix=/usr \
  --sysconfdir=/etc \
  --enable-dbus \
  --enable-gtk3

9 June 2022

I gave up on LXPanel with GTK+ 3, the main reason being that even if it installed correctly in the system, it constantly crashed randomly and this seems to be a known bug. I built libfm-gtk as well so LXPanel has a GTK+ 2 dependency built before and ./configure is happy.

Unfortunately (or fortunately) I was very busy during the following months and I couldn't spend time on this. Finally I got some spare time and I went back to it.

3rd March 2023

Installed Lightdm and lightdm-gtk-greeter.

At this point, a new problem appeared, the VM integration didn't work until I log in which is when SPICE agent runs. I added a note for future research.

I need to find out how to run spice-vdagent with lightdm or the greeter. Potentially exploring more here: https://wiki.ubuntu.com/LightDM.

Copying the desktop entry to the relevant location works to autostart the SPICE agent when you log in to the session.

sudo cp \
  /sources/spice-vdagent-0.21.0/data/spice-vdagent.desktop \
  /etc/xdg/autostart/

I installed Mousepad without any trouble.

Important note: compiling Firefox requires a lot a lot of RAM, no less than 8GB!

It also requires send-notify program to complete installation

Gtkmm - it wouldn’t compile unless disabling building documentation in all subprojects:

meson --prefix=/usr --buildtype=release .. \
  -Dmm-common:use-network=true \
  -Dbuild-documentation=false \
  -Datkmm-1.6:build-documentation=false \
  -Dcairomm-1.0:build-documentation=false \
  -Dglibmm-2.4:build-documentation=false \
  -Dlibsigcplusplus-2.0:build-documentation=false \
  -Dpangomm-1.4:build-documentation=false 

Sunday, 8 May 2022

My journey through BLFS - part 1

Having successfully finished LFS, I felt inspired to continue the journey, since all I had was a very minimal system. As I used version LFS 11.0-systemd, it only made sense to continue with the same version BLFS 11.0-systemd

15 Feb 2022 - SSH & curl

The top priority for me was to be able to connect remotely, primarily motivated by the fact that I was typing all commands from the guide since I left the chroot. OpenSSH to the rescue.

Another priority was the ability to download packages to continue building the system with new packages as I followed along, but there was no curl or wget. I could have used some bash magic or python minimal script, however, the main issue was the lack of support to download files via HTTPS. This is where make-ca, pk11-kit, GnuTLS and a few others came handy.

During all that time, I was making use of virtio 9p to share a folder with my host, this would simulate copying files manually before being able to use the network to download the rest.

6 April 2022 - LLVM compiled - required a lot of RAM

At some point I went down the rabbit hole, probably my mistake being too ambitious, building packages that I didn’t need so early in the process, but hey, that’s the learning process. A particular package that took me too deep in the dependency hell was texlive-20210325 as I was also building the documentation along with the binaries, to such extent that I decided to leave it and use the binary option (install-tl-unx) which is also huge in size.

One issue I encountered while building some of those dependencies of dependencies, I was building LLVM when all of the sudden the compiling process was killed, not enough memory. I set up 4GB swap space and tried again, to my surprise, it got killed again, so I ended up creating a swap file of 8 GB to complete the compilation process.

Other interesting milestones were:

Linux PAM - it allowed a more streamlined integration with many programs. In my particular use case, I wanted to control how I was using the “su” command without having to type root password every time and without using sudo. It also made me rebuild the whole systemd again to include PAM support.

Polkit - a somehow ubiquitous dependency for GUI apps and because it has quite a few dependencies that take some time to build like js engine, some of the next packages benefited from not having to build them as dependencies.

Having built all this, it was time to start preparing one of the most daunting parts of the whole BLFS, the X Window System !

25 April 2022 - X Window System up and running

This part is another one, where you can see yourself building and installing tons of packages and not seeing anything encouraging. It’s not until that very last moment of truth when you run startx that you see the result of all that hard work.

To my surprise, it worked the first time ! Well, not without some issues, but I have to say, the default values for xorg configuration are pretty good. Without proper graphic card drivers and barely any configuration in place, it managed to launch the X server with twm and some insanely big terminal emulators.

It took me some time to get my head around this poor man’s window manager, especially because for some reason the mouse movement was a bit awkward when the VM’s window had a different aspect ratio from the guest resolution.

Some issues I noted to fix immediately after this successful milestone:

  • Mouse cursor trapped in VM’s window
  • Clipboard integration host-guest
  • Mouse movement impacted by VM’s window size

26 April 2022 - Qemu agent integration

All the issues above mentioned are related to the same root cause, not having the "guest drivers" running in the VM. In this case, I needed both the qemu-agent and spice-vdagent running as well as the actual kernel drivers.

In order to get spice channel working, I had to add virtio vsockets in the kernel, so that the devices /dev/virtio-ports/com.redhat.spice.0 and /dev/virtio-ports/org.qemu.guest_agent.0 were available after setting them up in VM’s hardware configuration. Together with some udev rules provided by the agent, it was all I needed to get the daemons running.

Something I found out, which is not covered in the guide (again, because it was specific to my VM scenario) when building spice-vdagent, I had to add a few extra parameters to make it work:

CFLAGS=-Wno-error ./configure \
  --prefix=/usr \
  --with-init-script=systemd \
  --with-session-info=systemd

For some reason it wasn’t recognising that it’s being compiled in a running systemd environment, so I provided the flags to force using it, also the warning being treated as errors.

The second part of the integration, as per documentation is a per X-session process spice-vdagent, which I had to somehow manually invoke, and when I did, magic happened ! The mouse cursor was free as a bird to move all over my screen. Next, figure out how to adjust the resolution and bundle everything together to run automatically when all X starts up.

This is the tweaked version of my ".xinitrc" which worked well enough to get by without having any desktop environment to handle all these tasks.

xrandr --output Virtual-0 --mode 1366x768
spice-vdagent
xrdb -load $HOME/.Xresources
xsetroot -solid gray &
xclock -g 50x50-0+0 -bw 0 &
xload -g 50x50-50+0 -bw 0 &
xterm -g 80x24+0+0 &
xterm -g 80x24+700+320 &
twm

To wrap this part up, it’s been an incredible learning experience, this time focusing on building packages that are closer to what it’s being used on a daily basis in any Linux system. This is the base to add a desktop environment which will be my next task, at this point I’m leaning towards LXDE, but I’ll keep a backup copy at this stage in case I’d like to go for a different one later on.

The journey continues.

Wednesday, 4 May 2022

My journey through LFS

Building a whole Linux system completely from the ground up is something I wanted to do for some time and finally I had all I needed to take the plunge. In this post I’m sharing the highlights of this journey. Also, this is something I’m doing in my spare time, hence the time gaps between milestones.

1st Jan 2022 - Started reading the guide

I decided to go for the LFS-11.0-systemd released in September 2021, the latest stable at the time and I'm comfortable enough with systemd.

Before even starting with LFS I needed a host system, I gave it a try to the latest Arch iso at the time, which was archlinux-2022.01.01-x86_64.iso. Also, as my first attempt in such a venture, I went for a virtual machine using KVM/QEMU with its own challenges which are not fully covered in the guide.

25 Jan 2022 - Created VM with vanilla Arch as host system

Some initial decisions about my target system:

  • Virtual machine KVM/QEMU
  • 4 CPU (compiling can be CPU intensive)
  • 2 GB RAM (aim for a small system)
  • 60 GB Disk storage (that was too optimistic)
  • UEFI boot (mimic modern systems)
  • Network - bridged (ease of connection via SSH)
  • 1 partition for everything (simplicity)
  • No swap partition, swap file when needed

I learned that a plain vanilla Arch Linux base system is good enough for building a full LFS with no extra packages. Every requirement was met out of the box, that’s a good start !

Once I had my VM up and running with all the base configuration, including my SSH key in place so I can connect to it via SSH so it makes my life easier by enabling me to copy and paste, I made a clone of it just in case I had to go back and start again.

The first few days required a lot of patience and will power, since you’re only compiling and compiling and not seeing much in use. I had the impression sometimes that some of the commands to follow were somehow hacking my way to just get it compiled despite the environment conditions.

It’s only once you enter the chroot that it starts to feel like you’re getting somewhere.

10 Feb 2022 - chroot getting ready

Working inside the chroot can give you some frustration since it’s such a minimal system, the LFS developers made good effort to provide enough tooling to get the job done, but anything outside that, you’ll be faced with the dreaded "-bash: xxxx: command not found"

This was the first time I installed neofetch in the pseudo new system, mostly since I was intrigued by which logo it would display and how it’d find system information when there is barely any system at all.

Because I used UEFI in my target system, I had to jump ahead to BLFS for boot loader installation and configuration, that break in the flow can be a little disrupting and confusing, but nothing major.

14 Feb 2022 - First successful boot

There is always a moment that makes or breaks the installation, in this case is when configuring the Linux kernel. As almost expected after such a long build up preparing and configuring all the packages, I was greeted by a big kernel panic when I booted the system for the first time outside of the chroot.

It couldn’t find /dev/vda3 from the kernel command line argument root=/dev/vda3 which happens to be the root partition in the virtual hard drive. Some symbols were not selected (or indicated in the guide, since it’s specific to my setup using a VM) such as CONFIG_VIRTIO_BLK, more information on that can be found at linux-kvm website.

After adding the missing symbols and rebuilding the kernel a couple of times, following the classical trial and error method, I could see a normal boot screen all the way to the login prompt, which I have to say was lightning fast !

At this point, the system had been built but this was not the end of the journey, there was some extra networking configuration required, again, due to use of a VM which required virtio network as well as 9p so I had a mechanism to share files between the host and the guest LFS.

One more thing before calling it a milestone complete, I got counted and this is my information:

You have successfully registered!
ID: 29352
Name: Abel Perez Martinez
First LFS Version: 11.0-systemd

Date: 14 Feb 2022

My key takeaways from this process:

  • Arch Linux is minimal but powerful
  • Be patient, compiling takes time
  • Compiling software takes a lot of disk space
  • Use -j4 whenever possible
  • Make a backup of your VM after a significant milestone
  • This is only the beginning, it’s a minimal system

Sunday, 17 October 2021

How to fix “failed to parse field [xxxxxx] in document with id [yyyyy]. Preview of field's value: ‘zzzzzzz’”

I came across this issue while chasing the infamous “[ warn] [engine] failed to flush chunk” in fluent-bit connected to Elasticsearch. For some context, I'm using Amazon EKS to run my workloads and I use fluent-bit to parse the logs and push them to Elasticsearch so I can query them later on using Kibana.

The first step in this investigation was to set "Trace_Error On" in [OUTPUT] section as part of the fluent-bit configuration, a config map in this instance.

What is the problem ?

[2021/10/07 11:03:51] [ warn] [engine] failed to flush chunk '1-1633604630.
197484999.flb', retry in 11 seconds: task_id=29, input=tail.0 > output=es.0 
(out_id=0)
[2021/10/07 11:03:52] [error] [output:es:es.0] error: Output

{
    "create": {
        "_index": "fluent-bit-000135",
        "_type": "_doc",
        "_id": "qQZsWnwBkg-cCPSefqtj",
        "status": 400,
        "error": {
            "type": "mapper_parsing_exception",
            "reason": "failed to parse field [log_processed.Properties.
StatusCode] of type [long] in document with id 'qQZsWnwBkg-cCPSefqtj'. 
Preview of field's value: 'Unauthorized'",
            "caused_by": {
                "type": "illegal_argument_exception",
                "reason": "For input string: \"Unauthorized\""
            }
        }
    }
},

... more logs, removed for brevity ...

This essentially means that the field "log_processed.Properties.StatusCode" was initially mapped (automatically in my case) as "long" and therefore it won't allow the current value "Unauthorized" due to it not being able to parse that as a long value. It's probably my mistake initially not having mapped explictily to a "text" value instead, I could've avoided this situation. But I didn't and here we are.

What is the solution ?

Basically, you need to let Elasticsearch know about the data types in your indexes. A common pattern when working with Elasticearch and Kibana is to creata a State management policy to automatically rolls over your data to a new index and eventually delete old data to avoid disk space to run too low. This requires to have index templates in place so Elasticsearch knows how to create new indexes when the time comes to roll over.

I already use a policy that deletes indices older than 14 days and rolls over after either 1 day or 1 GB of size.

At this point, I'm not terribly bothered about existing data, my priority is to keep new data in good shape. Old data should be reindexed if you want to rescue it and make it available to search on.

What needs to be done is to add a mapping to the index template (assuming you have already one) that contains the property name and type explictily declared.

How to do it ?

Luckily, Kibana gives us a good developer console, that we can run the commands against Elasticsearch endpoints.

Using Kibana Dev Tools Console which you can find under
"https://<your-endpoint>/_plugin/kibana/app/dev_tools#/console"

1 - Update the template

Under "mappings/properties", add a new entry with the property name and type, in this case "log_processed.Properties.StatusCode" of type "text". This will not impact any existing data.

PUT /_index_template/ism_retention_14d
{
  "index_patterns": ["fluent-bit-*"],
  "template": {
    "mappings": {
      "properties": {
        "@timestamp": {
          "type": "date_nanos"                 
        },
        "log_processed.Properties.StatusCode": {
          "type": "text"                 
        }
      }
    },
    "settings": {
      "index.opendistro.index_state_management.rollover_alias":"fluent-bit",
      "sort.field": [ "@timestamp"],          
      "sort.order": [ "desc"]
    }
  }
}

2- Verify it's updated

This is only to double check that the change has been applied with the correct values for new indexes to be created matching the pattern (fluent-bit-*) in this case.

GET /_index_template/ism_retention_14d
{
  "index_templates" : [
    {
      "name" : "ism_retention_14d",
      "index_template" : {
        "index_patterns" : [
          "fluent-bit-*"
        ],
        "template" : {
          "settings" : {
            "index" : {
              "opendistro" : {
                "index_state_management" : {
                  "rollover_alias" : "fluent-bit"
                }
              },
              "sort" : {
                "field" : [
                  "@timestamp"
                ],
                "order" : [
                  "desc"
                ]
              }
            }
          },
          "mappings" : {
            "properties" : {
              "@timestamp" : {
                "type" : "date_nanos"
              },
              "log_processed.Properties.StatusCode" : {
                "type" : "text"
              }
            }
          }
        },
        "composed_of" : [ ]
      }
    }
  ]
}

3- Create new index (+1 the current sequence)

Find out which is the latest sequence number in this index parttern (fluent-bit-000135 in this case), add 1 and create a new index with that name (fluent-bit-000136 in this case).

PUT /fluent-bit-000136
{
  "acknowledged" : true,
  "shards_acknowledged" : true,
  "index" : "fluent-bit-000136"
}

4- Verify the mapping is actually in place

Once the index has been created, it should have inherited that mappings from the template updated above.

GET /fluent-bit-000136/_mapping/field/log_processed.Properties.StatusCode
{
  "fluent-bit-000136" : {
    "mappings" : {
      "log_processed.Properties.StatusCode" : {
        "full_name" : "log_processed.Properties.StatusCode",
        "mapping" : {
          "StatusCode" : {
            "type" : "text"
          }
        }
      }
    }
  }
}

5- Update the alias with "is_write_index"

The new index is now ready to start receiving data. To accomplish that, set the "is_write_index" property to true in the new index and to false in the current target index.

POST /_aliases
{
  "actions" : [
    { "add" : { 
      "index" : "fluent-bit-000135", 
      "alias" : "fluent-bit", 
      "is_write_index": false 
      }
    },
    { "add" : {
      "index" : "fluent-bit-000136", 
      "alias" : "fluent-bit", 
      "is_write_index": true 
      }
    }
  ]
}

6- Verify the aliases are correct

Verify the "is_write_index" property is set appropriately to each index from the previous step.

GET fluent-bit-000135/_alias
{
  "fluent-bit-000135" : {
    "aliases" : {
      "fluent-bit" : {
        "is_write_index" : false
      }
    }
  }
}

GET fluent-bit-000136/_alias
{
  "fluent-bit-000136" : {
    "aliases" : {
      "fluent-bit" : {
        "is_write_index" : true
      }
    }
  }
}

Conclusion

Following these steps will help you to stop those errors that prevent the logs from being added to Elasticsearch and therefore missing from your queries and potentially from alerts based on that information.

Having state management policies is crucial to achieve a sensible retention policy as well as avoiding creating too large indexes. It also eases the process of updating the template from time to time to keep up with your applications' needs.

Saturday, 11 April 2020

AWS HttpApi with Cognito as JWT Authorizer

With the recent release of HttpApi from AWS I've been playing with it for a bit and I wanted to see how far I can get it to use authorization without handling any logic in my application.

Creating a base code

Started with a simple base, let's set up the initial scenario which is no authentication at all. The architecture is the typical HttpApi -> Lambda, in this case, the Lambda content is irrelevant and therefore I've just used an inline code to test it's working.

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
  Sample SAM Template using HTTP API and Cognito Authorizer
Resources:

  # Dummy Lambda function 
  HttpApiTestFunction:
    Type: AWS::Serverless::Function
    Properties:
      InlineCode: |
        exports.handler = function(event, context, callback) {
          const response = {
            test: 'Hello HttpApi',
            claims: event.requestContext.authorizer && 
                    event.requestContext.authorizer.jwt.claims
          };
          callback(null, response);
        };
      Handler: index.handler
      Runtime: nodejs12.x
      Timeout: 30
      MemorySize: 256
      Events:
        GetOpen:
          Type: HttpApi
          Properties:
            Path: /test
            Method: GET
            ApiId: !Ref HttpApi
            Auth:
              Authorizer: NONE

  HttpApi:
    Type: AWS::Serverless::HttpApi
    Properties:
      CorsConfiguration: 
        AllowOrigins:
          - "*"

Outputs:
  HttpApiUrl:
    Description: URL of your API endpoint
    Value: !Sub 'https://${HttpApi}.execute-api.${AWS::Region}.${AWS::URLSuffix}/'
  HttpApiId:
    Description: Api id of HttpApi
    Value: !Ref HttpApi

With the outputs of this template we can hit the endpoint and see that in fact, it's accessible. So far, nothing crazy here.

$ curl https://abc1234.execute-api.us-east-1.amazonaws.com/test
{"test":"Hello HttpApi"}

Creating a Cognito UserPool and Client

The claims object is not populated because the request wasn't authenticated since no token was provided, as expected.

Let's create the Cognito UserPool with a very simple configuration assuming lots of default values since they're not relevant for this example.

  ## Add this fragment under Resources:

  # User pool - simple configuration 
  UserPool:
    Type: AWS::Cognito::UserPool
    Properties: 
      AdminCreateUserConfig: 
        AllowAdminCreateUserOnly: false
      AutoVerifiedAttributes: 
        - email
      MfaConfiguration: "OFF"
      Schema: 
        - AttributeDataType: String
          Mutable: true
          Name: name
          Required: true
        - AttributeDataType: String
          Mutable: true
          Name: email
          Required: true
      UsernameAttributes: 
        - email
  
  # User Pool client
  UserPoolClient:
    Type: AWS::Cognito::UserPoolClient
    Properties: 
      ClientName: AspNetAppLambdaClient
      ExplicitAuthFlows: 
        - ALLOW_USER_PASSWORD_AUTH
        - ALLOW_USER_SRP_AUTH
        - ALLOW_REFRESH_TOKEN_AUTH
      GenerateSecret: false
      PreventUserExistenceErrors: ENABLED
      RefreshTokenValidity: 30
      SupportedIdentityProviders: 
        - COGNITO
      UserPoolId: !Ref UserPool

  ## Add this fragment under Outputs:

  UserPoolId:
    Description: UserPool ID
    Value: !Ref UserPool
  UserPoolClientId:
    Description: UserPoolClient ID
    Value: !Ref UserPoolClient

Once we have the cognito UserPool and a client, we are in a position to start putting things together. But before there are a few things to clarify:

  • Username is the email and only two fields are required to create a user: name and email.
  • The client defines both ALLOW_USER_PASSWORD_AUTH and ALLOW_USER_SRP_AUTH Auth flows to be used by different client code.
  • No secret is generated for this client, if you intend to use other flows, you'll need to create other clients accordingly.

Adding authorization information

Next step is to add authorization information to the HttpApi.

  ## Replace the HttpApi resource with this one.

  HttpApi:
    Type: AWS::Serverless::HttpApi
    Properties:
      CorsConfiguration: 
        AllowOrigins:
          - "*"
      Auth:
        Authorizers:
          OpenIdAuthorizer:
            IdentitySource: $request.header.Authorization
            JwtConfiguration:
              audience:
                - !Ref UserPoolClient
              issuer: !Sub https://cognito-idp.${AWS::Region}.amazonaws.com/${UserPool}
        DefaultAuthorizer: OpenIdAuthorizer

We've added authorization information to the HttpApi where JWT issuer is the the Cognito UserPool previously created and the token are intended only for that client.

If we test again, nothing changes because the event associated with the lambda function says explictly "Authorizer: NONE".

To test this, we'll create a new event associated with the same lambda function but this time we'll add some authorization information to it.

        ## Add this fragment at the same level as GetOpen
        ## under Events as part of the function properties

        GetSecure:
          Type: HttpApi
          Properties:
            ApiId: !Ref HttpApi
            Method: GET
            Path: /secure
            Auth:
              Authorizer: OpenIdAuthorizer

If we test the new endpoint /secure, then we'll see the difference.

$ curl -v https://abc1234.execute-api.us-east-1.amazonaws.com/secure

>>>>>> removed for brevity >>>>>>
> GET /secure HTTP/1.1
> Host: abc1234.execute-api.us-east-1.amazonaws.com
> User-Agent: curl/7.52.1
> Accept: */*
> 
* Connection state changed (MAX_CONCURRENT_STREAMS updated)!
< HTTP/2 401 
< date: Sat, 11 Apr 2020 17:19:50 GMT
< content-length: 26
< www-authenticate: Bearer
< apigw-requestid: K1RYliB1IAMESNA=
< 
* Curl_http_done: called premature == 0
* Connection #0 to host abc1234.execute-api.us-east-1.amazonaws.com left intact

{"message":"Unauthorized"}

At this point we have a new endpoint that requires an access token. Now we need a token, but to get a token, we need a user first.

Fortunately Cognito can provide us with all we need in this case. Let's see how.

Creating a Cognito User

Cognito cli provides the commands to sign up and verify user accounts.

$ aws cognito-idp sign-up \
  --client-id asdfsdfgsdfgsdfgfghsdf \
  --username abel@example.com \
  --password Test.1234 \
  --user-attributes Name="email",Value="abel@example.com" Name="name",Value="Abel Perez" \
  --profile default \
  --region us-east-1

{
    "UserConfirmed": false, 
    "UserSub": "aaa30358-3c09-44ad-a2ec-5f7fca7yyy16", 
    "CodeDeliveryDetails": {
        "AttributeName": "email", 
        "Destination": "a***@e***.com", 
        "DeliveryMedium": "EMAIL"
    }
}

After creating the user, it needs to be verified.

$ aws cognito-idp admin-confirm-sign-up \
  --user-pool-id us-east-qewretry \
  --username abel@example.com \
  --profile default \
  --region us-east-1

This commands gives no output, to test we are good to go, let's use the admin-get-user command

$ aws cognito-idp admin-get-user \
  --user-pool-id us-east-qewretry \
  --username abel@example.com \
  --profile default \
  --region us-east-1 \
  --query UserStatus

"CONFIRMED"

We have a confirmed user!

Getting a token for the Cognito User

To obtain an Access Token, we use the Cognito initiate-auth command providing the client, username and password.

$ TOKEN=`aws cognito-idp initiate-auth \
  --client-id asdfsdfgsdfgsdfgfghsdf \
  --auth-flow USER_PASSWORD_AUTH \
  --auth-parameters USERNAME=abel@example.com,PASSWORD=Test.1234 \
  --profile default \
  --region us-east-1 \
  --query AuthenticationResult.AccessToken \
  --output text`

$ echo $TOKEN

With the access token in hand, it's time to test the endpoint with it.

$ curl -H "Authorization:Bearer $TOKEN" https://abc1234.execute-api.us-east-1.amazonaws.com/secure
# some formatting added here
{
    "test": "Hello HttpApi",
    "claims": {
        "auth_time": "1586627310",
        "client_id": "asdfsdfgsdfgsdfgfghsdf",
        "event_id": "94872b9d-e5cc-42f2-8e8f-1f8ad5c6e1fd",
        "exp": "1586630910",
        "iat": "1586627310",
        "iss": "https://cognito-idp.us-east-1.amazonaws.com/us-east-qewretry",
        "jti": "878b2acd-ddbd-4e68-b097-acf834291d09",
        "sub": "cce30358-3c09-44ad-a2ec-5f7fca7dbd16",
        "token_use": "access",
        "username": "cce30358-3c09-44ad-a2ec-5f7fca7dbd16"
    }
}

Voilà! We've accessed the secure endpoint with a valid access token.

What about groups ?

I wanted to know more about possible granular control of the authorization and I went and created two Cognito Groups let's say Group1 and Group2. Then, I added my newly created user to both groups and repeated the experiment.

Once the user was added to the groups, I got a new token and issued the request to the secure endpoint.

$ curl -H "Authorization:Bearer $TOKEN" https://abc1234.execute-api.us-east-1.amazonaws.com/secure
# some formatting added here
{
    "test": "Hello HttpApi",
    "claims": {
        "auth_time": "1586627951",
        "client_id": "2p9k1pfhtsbr17a2fukr5mqiiq",
        "cognito:groups": "[Group2 Group1]",
        "event_id": "c450ae9e-bd4e-4882-b085-5e44f8b4cefd",
        "exp": "1586631551",
        "iat": "1586627951",
        "iss": "https://cognito-idp.us-east-1.amazonaws.com/us-east-qewretry",
        "jti": "51a39fd9-98f9-4359-9214-000ea40b664e",
        "sub": "cce30358-3c09-44ad-a2ec-5f7fca7dbd16",
        "token_use": "access",
        "username": "cce30358-3c09-44ad-a2ec-5f7fca7dbd16"
    }
}

Notice within the claims object, a new one has come up: "cognito:groups" and the value associated with it is "[Group2 Group1]".

Which means that we could potentially check this claim value to make some decisions in our application logic without having to handle all of the authentication inside the application code base.

This opens the possibility for more exploration within the AWS ecosystem. I hope this has been helpful, the full source code can be found at https://github.com/abelperezok/http-api-cognito-jwt-authorizer.

Monday, 23 March 2020

AWS Serverless Web Application Architecture

Recently, I've been exploring ideas about how to put together different AWS services to achieve a totally serverless architecture for web applications. One of the new services is the HTTP API which simplifies the integration with Lambda.

One general principle I want to follow when designing these architectural models is the separation of three main subsystems:

  • Identity service to handle authentication and authorization
  • Static assets traffic being segregated
  • Dynamic page rendering and server side logic
  • Application configuration outside the code

All these component will be publicly accessible via Route 53 DNS record sets pointing to the relevant endpoints.

Other general ideas across all design diagrams below are:

  • Cognito will handle authentication and authorization
  • S3 will store all static assets
  • Lambda will execute server side logic
  • SSM Parameter Store will hold all configuration settings

Architecture variant 1 - HTTP API and CDN publicly exposed

In this first approach, we have the S3 bucket behind CloudFront which is a common pattern when creating CDN-like structures. CloudFront takes care of all the caching behaviours as well as distributing the cached versions all over the Edge locations, so subsequent requests will be dispatched at a reduced latency. Also, CloudFront has only one cache behaviour, which is the default and one origin which is the S3 bucket.

It's also important to notice the CloudFront distribution has an Alternative Domain Name set to the relevant record set e.g. media.example.com and let's not forget about referencing the ACM SSL certificate so we can use the custom url and not the random one from CloudFront.

From the HTTP API perspective, it has only one integration which is a Lambda integration on the $default route, which means all requests coming from the HTTP endpoint will be directed to the Lambda function in question.

Similar to the case of CloudFront, the HTTP API requires a Custom Domain and Certificate to be able to use a custom url as opposed to the random one given by the API service on creation.

Architecture variant 2 - HTTP API behind CloudFront

In this second approach, we still have the S3 bucket behind CloudFront following the same pattern. However we've placed the HTTP API also behind CloudFront.

CloudFront becomes the traffic controller in this case, where several cache behaviours can be defined to make the correct decision where to route the request to.

Both record sets (media and webapp) are pointing to the same CloudFront distribution, it's the application logic's responsibility to request all static assets using the appropriated domain nam.

Since the HTTP API is behind a CF distribution, I'd suggest to set it up as Regional endpoint.

Architecture variant 3 - No CloudFront at all

Continue playing with this idea, what if we don't use CloudFront distribution at all? I gave it a go and it turns out that it's possible to achieve similar results.

We can use two HTTP APIs and set one to forward traffic to S3 for static assets and the other one to Lambda as per the usual pattern, each of those with Custom Domain and that solves the problem.

But I wanted to push it a little bit further, this time I tried with only one HTTP API and setting several routes e.g "/css/*", "/js/*" integrates with S3 and any other integrates with Lambda, it's then, application logic's responsibility to request all static assets using the appropriated url

Conclusion

These are some ideas I've been experimenting with, the choice of including or not a CloudFront distribution is dependent on the concrete use case, whether the source of our requests is local or globally diverse. Also, whether it is more suitable to have static assets under a subdomain or virtual directory under the same host name.

Never underestimate the power and flexibility of an API Gateway, especially the new HTTP API where it can front any number of combination of resources in the back end.

Saturday, 13 July 2019

Run ASP.NET Core 2.2 on a Raspberry Pi Zero

Raspberry PI Zero (and Zero W) is a cool and cheap piece of technology that can run software anywhere. Being a primarily .NET developer, I wanted to give it a try running the current version of ASP.NET Core, which is as of now 2.2.

However, the first problem is that even though .NET Core supports ARM CPUs, it does not support ARM32v6, only v7 and above. After digging a bit, I found out that mono does support that CPU and on top of that, it’s binary compatible with .NET Framework 4.7.

In this post, I’ll summarise several hours of trial and error to get it working. If developing on Windows, targeting both netcoreapp2.2 and net472 is easier since chances are we’ll have all installed already. On Linux, however, it’s not that easy and it’s when Mono comes in to help, we need the reference assemblies to build the net472 version

Let’s check the tools we’ll use:

$ dotnet --version
2.2.202
$ docker --version
Docker version 18.09.0, build 4d60db4

Create a new dotnet core MVC web application to get the starting template.

$ dotnet new mvc -o dotnet.mvc

Update the target framework to TargetFrameworks if we want to actually target both. We could target only net472 if desired. Also, since Microsoft.AspNetCore.App and Microsoft.AspNetCore.Razor.Design metapackages won’t be available on .NET Framework, instead reference directly the Nuget packages we’ll use, here is an example of the ones the default template uses.

<Project Sdk="Microsoft.NET.Sdk.Web">

  <PropertyGroup>
    <TargetFrameworks>net472;netcoreapp2.2</TargetFrameworks>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Microsoft.AspNetCore" Version="2.2.0" />
    <PackageReference Include="Microsoft.AspNetCore.Hosting.WindowsServices" Version="2.2.0" />
    <PackageReference Include="Microsoft.AspNetCore.Mvc" Version="2.2.0" />
    <PackageReference Include="Microsoft.AspNetCore.StaticFiles" Version="2.2.0" />
    <PackageReference Include="Microsoft.AspNetCore.HttpsPolicy" Version="2.2.0" />
    <PackageReference Include="Microsoft.AspNetCore.CookiePolicy" Version="2.2.0" />
    <PackageReference Include="Microsoft.Extensions.Logging.Debug" Version="2.2.0" />
    <PackageReference Include="Microsoft.Extensions.Logging.EventLog" Version="2.2.0" />
    <PackageReference Include="Microsoft.Extensions.Options" Version="2.2.0" />
  </ItemGroup>

</Project>

When targeting net472, we need to use mono’s reference assemblies since they’re not part of dotnet core, to do that, we set the environment variable FrameworkPathOverride to the appropriated path, typically something like /usr/lib/mono/4.7.2-api/.

export FrameworkPathOverride=/usr/lib/mono/4.7.2-api/

Then proceed as usual with dotnet cli

  • dotnet restore
  • dotnet build
  • dotnet run

To make a more consistent build across platforms. We’ll create a Dockerfile, this will also enable us to build locally without having to install mono on our local development environments. This is a multi-stage docker file that builds the application and creates the runtime image.

More information on how to create the docker file for dotnet core applications. https://docs.docker.com/engine/examples/dotnetcore/

FROM pomma89/dotnet-mono:dotnet-2-mono-5-sdk AS build-env
WORKDIR /app

ENV FrameworkPathOverride /usr/lib/mono/4.7.2-api/

# Copy csproj and restore as distinct layers
COPY *.csproj ./
RUN dotnet restore

# Copy everything else and build
COPY . ./
RUN dotnet publish -c Release -o out -f net472

# Build runtime image for ARM v5 using mono
FROM arm32v5/mono:5.20
WORKDIR /app
COPY --from=build-env /app/out .
EXPOSE 5000
ENV ASPNETCORE_URLS http://*:5000
ENTRYPOINT [ "mono", "dotnet.mvc.exe" ]

Since we are building with mono, the resulting executable is in this case dotnet.mvc.exe file. The environment variable ASPNETCORE_URLS is required in order to be able to listen on any network interface, by default it will only listen on localhost which is in fact, the container itself and not our host. Combined with EXPOSE 5000 it’s possible to access the application from outside the container.

The secret ingredient to make it work on a raspberry pi zero (ARM32v6) is the line “FROM arm32v5/mono:5.20” which takes a base docker image using a compatible CPU architecture.

To run this application locally we use the traditional dotnet run but since we’ve specified two target frameworks, the parameter -f is required to specify which one we want to use. In this case, chances are we’re using netcoreapp2.2 locally and net472 to build and run on the raspberry pi.

$ dotnet run -f netcoreapp2.2

Once we’re happy the application works as expected locally, we could build the docker image locally to test the whole docker building process is working before deploying to the device. To test this way, we have to modify the second FROM line and remove the “arm32v5/” from the image name, taking the PC version of mono instead.

The following sequence shows the docker commands.

Build the image locally

$ docker build -t test .

Either run in interactive mode -i -t or run in detached mode -d

$ docker run --rm -it -p 3000:5000 test
$ docker run --rm -d -p 3000:5000 test

Once it’s running, we can just browse to http://localhost:3000/ and verify it’s up and running.

Now we can go back and modify the second FROM line by putting “arm32v5/” where it was.

# build the image and tag it to my docker hub registry 
$ docker build -t abelperezok/mono-mvc:arm32v5-test .
# don’t forget to log in to docker hub
$ docker login
# push this image to docker hub registry 
$ docker push abelperezok/mono-mvc:arm32v5-test

Once it’s uploaded to the registry, we can connect to the raspberry pi and run the follow docker run command.

# pull and run the container from this image 
$ docker run --rm -d -p 3000:5000 abelperezok/mono-mvc:arm32v5-test

Helpful links

https://www.c-sharpcorner.com/article/running-asp-net-core-2-0-via-mono/

https://stackoverflow.com/questions/44770702/build-nuget-package-on-linux-that-targets-net-framework

https://www.mono-project.com/download/stable/#download-lin-debian

https://andrewlock.net/building-net-framework-asp-net-core-apps-on-linux-using-mono-and-the-net-cli/

https://hub.docker.com/r/pomma89/dotnet-mono/

Saturday, 2 March 2019

How to create a dynamic DNS with AWS Route 53

Have you ever wanted to host some pet project or even a small website on your own domestic network? If so, you must have stumbled across the DNS resolution issue: since we depend on our ISP to get our "real IP address", there's no guarantee that the IP we see today will stay any longer the same.

You could update your domain's DNS records every time you detect the IP has changed, but obviously that's tedious and error prone. That's when dynamic DNS (DDNS) comes in. Services like No-IP,Duck DNS,etc can be helpful.

In this post I'll go through the steps involved when it comes to setting up your own DDNS service when you own a domain that has been registered in AWS Route 53.

Prerequisites

Before running the commands I suggest here, we need a couple of things to set up. Let's start by installing the required packages if you haven't already installed them.

  • sudo apt-get install awscli
  • sudo apt-get install jq

Configure AWS credentials

$ aws configure
AWS Access Key ID [None]: AKIAIOSFODNN7EXAMPLE
AWS Secret Access Key [None]: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
Default region name [None]: eu-west-1
Default output format [None]: json

A hosted zone in AWS Route 53

$ aws route53 list-hosted-zones
{
    "HostedZones": [
        {
            "ResourceRecordSetCount": 5, 
            "CallerReference": " ... ", 
            "Config": {
                "Comment": "HostedZone created by Route53 Registrar", 
                "PrivateZone": false
            }, 
            "Id": "/hostedzone/Z2TLEXAMPLEZONE", 
            "Name": "abelperez.info."
        }
    ]
}

What's the plan?

The scenario I'm covering here is probably one of the most common. I want to create a new subdomain that points to my external IP and update that record as the external IP changes. To achieve that, we'll follow the steps:

  • Find out the external IP.
  • Get the desired hosted zone.
  • Create the A record (or update it if it already exists).
  • Set it up to run regularly (cron job).

Script step by step

Lest's start like any script, the shebang indicator. Then, input variables, in this case we define the domain name and the public record, notice the "." at the end of the public record, this is required by route 53.

#!/bin/bash
DOMAIN_NAME=abelperez.info
PUBLIC_RECORD=rpi.$DOMAIN_NAME.

Find the external IP, there are many ways to obtain this value, but since we're using AWS, let's get it from checkip endpoint. For debugging purposes we echo the IP found.

IP=$(curl -s http://checkip.amazonaws.com)
echo Found IP=$IP

Determine the hosted zone id, this step is optional if you want to use a hard-coded zone id that can be copied from the AWS Route 53 console. In this case I've invoked list-zones-by-name command which gives the hosted zone information for a specific domain, the format is as above in the prerequisites section. To extract the exact id I used a combination of jq and sed commands.

R53_HOSTED_ZONE=`aws route53 list-hosted-zones-by-name \
--dns-name $DOMAIN_NAME \
--query HostedZones \
| jq -r ".[] | select(.Name == \"$DOMAIN_NAME.\").Id" \
| sed 's/\/hostedzone\///'`

Now that we have all the required information, let's prepare the A record JSON input that will be used by aws route53 change-resource-record-sets command. the action "UPSERT" creates or updates the record accordingly, otherwise we'd need to manually check if it exists or not before updating it. Again for debugging purposes, we echo the final JSON.

read -r -d '' R53_ARECORD_JSON << EOM
{
  "Changes": [
    {
      "Action": "UPSERT",
      "ResourceRecordSet": {
        "Name": "$PUBLIC_RECORD",
        "Type": "A",
        "TTL": 300,
        "ResourceRecords": [
          {
            "Value": "$IP"
          }
        ]
      }
    }
  ]
}
EOM

echo About to execute change
echo "$R53_ARECORD_JSON" 

With the input ready, we invoke the Route 53 command to create/update the A record. This command will return immediately but the operation will take a few seconds to complete. If we want to make sure we know when it's completed, then we need to get the change id returned by the command, in this case, it's stored in the variable. This is optional, as well as the next step.

R53_ARECORD_ID=`aws route53 change-resource-record-sets \
--hosted-zone-id $R53_HOSTED_ZONE \
--change-batch "$R53_ARECORD_JSON" \
--query ChangeInfo.Id \
--output text`

echo Waiting for the change to update.

At this point, the request to create/update the A record is in progress, we could finish the script right now. However, I'd like to get a final confirmation that the operation has been completed. To do that, we can use the wait command providing the change id from the previous request.

aws route53 wait resource-record-sets-changed --id $R53_ARECORD_ID

echo Done.

And now it's actually completed. You should save this to a file, maybe name it update-dns.sh. It will need execute permission.

$ chmod u+x ./update-dns.sh

Set up cron job

In this particular instance I want this script to run in one of my raspberry pis, so I proceeded to copy the script file to pi user's home directory (/home/pi/)

pi@raspberrypi:~ $ ls
update-dns.sh

Now we'll set up a user cron job, we do that by running the command crontab -u followed by the user under which the job should run, this job doesn't require any system-wide privilege therefore it can run as the regular user, pi. -e to edit the file.

pi@raspberrypi:~ $ crontab -u pi -e

All we need to do is append the following text to the file content you are prompted with. The first two numbers correspond to minute and hour to run. For testing purposes I set it near the current time at the moment of testing it. The script output is appended/redirected to a text file so we can review afterwards if desired.

23 22 * * * /home/pi/update-dns.sh >> /home/pi/cron-update-dns.txt

See it in action

Once we've saved the crontab file, if it's the time where the cron job is about to start, we can test it by running tail -f command

pi@raspberrypi:~ $ tail -f cron-update-dns.txt

Finally, don't forget to update the port forwarding section on your home router so your open port directs traffic to a specific device, in my case, to that particular raspberry pi.

Sunday, 17 February 2019

How to Install docker on raspbian stretch

As part of one of my recent experiments I wanted to install docker on my raspberry pi. The procedure nowadays seems significantly better than a couple of years ago when I first tried. However, there are always some little details that can be a bit frustrating when following steps.

Although the process to install docker is well described in their website, it's focused on Debian distribution. We all know that Raspbian is based on Debian and therefore most of the instructions for will apply without much trouble.

That said, I will detail the exact steps I followed to get docker installed on my raspberry pi, this procedure was tested on 3B, Zero and Zero W.

First, update the package list, if you haven't done yet after setting up your raspberry pi.

$ sudo apt-get update

Install packages to allow apt to use a repository over HTTPS.

$ sudo apt-get install \
apt-transport-https \
ca-certificates \
curl \
gnupg2 \
software-properties-common

Add Docker’s official GPG key. Note that here the url points to raspbian directory and not to debian as per the original instructions.

curl -fsSL https://download.docker.com/linux/raspbian/gpg | sudo apt-key add -

The next step recommended didn't work for me.

sudo add-apt-repository \
   "deb [arch=armhf] https://download.docker.com/linux/raspbian \
   $(lsb_release -cs) \
   stable"

It failed with an error similar to this:

Traceback (most recent call last):
File "/usr/bin/add-apt-repository", line 95, in 
  sp = SoftwareProperties(options=options)
File "/usr/lib/python3/dist-packages/softwareproperties/SoftwareProperties.py"
, line 109, in __init__
  self.reload_sourceslist()
File "/usr/lib/python3/dist-packages/softwareproperties/SoftwareProperties.py"
, line 599, in reload_sourceslist
  self.distro.get_sources(self.sourceslist)
File "/usr/lib/python3/dist-packages/aptsources/distro.py", line 89, 
in get_sources (self.id, self.codename))
aptsources.distro.NoDistroTemplateException: Error: could not find a 
distribution template for Raspbian/stretch

Instead, I tried this other way, more explicitly setting up the repository into a new docker.list sources file.

$echo "deb [arch=armhf] https://download.docker.com/linux/raspbian $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list

With the new source repository in the list, update the package list again.

$ sudo apt-get update

Now we are in a position to install docker-ce from the repository, in this particular case, I ran into an issue with the latest version, documented in github basically they suggest to go back to 18.06.1 which we can get from the command below as per official docker instructions on how to get a specific version.

$ apt-cache madison docker-ce
docker-ce | 5:18.09.0~3-0~raspbian-stretch | https://download.docker.com/linux/raspbian stretch/stable armhf Packages
docker-ce | 18.06.2~ce~3-0~raspbian | https://download.docker.com/linux/raspbian stretch/stable armhf Packages
docker-ce | 18.06.1~ce~3-0~raspbian | https://download.docker.com/linux/raspbian stretch/stable armhf Packages
docker-ce | 18.06.0~ce~3-0~raspbian | https://download.docker.com/linux/raspbian stretch/stable armhf Packages

As of this writing, the version I tried is 18.06.2~ce~3-0~raspbian, which can be installed using the apt-get command.

$ sudo apt-get install docker-ce=18.06.2~ce~3-0~raspbian containerd.io

Also notice that by using this vesion, there's no need to install docker-ce-cli.

Testing everything is running as expected.

$ sudo docker info
Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 1
Server Version: 18.06.2-ce
 --- more data --- 

Run the hello world. Usually we get the command sudo docker run hello-world but that only works on "normal" architectures, in this case, we're using ARMv6/7, but to make it more compatible with all of them, I ran the image from arm32v5/hello-world.

$ sudo docker run arm32v5/hello-world
Unable to find image 'arm32v5/hello-world:latest' locally
latest: Pulling from arm32v5/hello-world
590e13f69e4a: Pull complete 
Digest: sha256:8a6a26a494c03e91381161abe924a39a2ff72de13189edcc2ed1695e6be12a5f
Status: Downloaded newer image for arm32v5/hello-world:latest

Hello from Docker!
This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:
 1. The Docker client contacted the Docker daemon.
 2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
    (arm32v5)
 3. The Docker daemon created a new container from that image which runs the
    executable that produces the output you are currently reading.
 4. The Docker daemon streamed that output to the Docker client, which sent it
    to your terminal.

To try something more ambitious, you can run an Ubuntu container with:
 $ docker run -it ubuntu bash

Share images, automate workflows, and more with a free Docker ID:
 https://hub.docker.com/

For more examples and ideas, visit:
 https://docs.docker.com/get-started/

And that's all, docker installed up and running on a raspberry pi.

UPDATE:

If you'd like to get rid of the sudo command every time you use docker, then you can run the following command found in the official guide as a post-installation step. You don't need to create the group as it's already created by docker installation. It adds your user to the docker group.

$ sudo usermod -aG docker $USER

You need to log out and back in for this change to take effect. After that you'll be able to run all docker commands as a regular user.

Thursday, 7 February 2019

Querying Aurora serverless database remotely using Lambda - part 3

This post is part of a series

In the previous part, we've set up the Aurora MySql cluster. At this point we can start creating the client code to allow querying.

The Lambda code

In this example I'll be using .NET Core 2.1 as Lambda runtime and C# as programming language. The code is very simple and should be easy to port to your favourite runtime/language.

Lambda Input

The input to my function consists of two main pieces of information: database connection information and the query to execute.

    public class ConnectionInfo
    {
        public string DbUser { get; set; }
        public string DbPassword { get; set; }
        public string DbName { get; set; }
        public string DbHost { get; set; }
        public int DbPort { get; set; }
    }

    public class LambdaInput
    {
        public ConnectionInfo Connection { get; set; }

        public string QueryText { get; set; }
    }

Lambda Code

The function itself returns a List of dictionary where each item of the list represents a "record" from the query result, these are in a key/value form where key is the "field" name and the value is the what comes form the query.

    public List<Dictionary<string, object>> RunQueryHandler(LambdaInput input, ILambdaContext context)
    {
        var cxnString = GetCxnString(input.Connection);
        var query = input.QueryText;

        var result = new List<Dictionary<string, object>>();
        using (var conn = new MySql.Data.MySqlClient.MySqlConnection(cxnString))
        {
            var cmd = GetCommand(conn, query);
            var reader = cmd.ExecuteReader();

            var columns = new List<string>();

            for (int i = 0; i < reader.FieldCount; i++)
            {
                columns.Add(reader.GetName(i));
            }

            while (reader.Read())
            {
                var record = new Dictionary<string, object>();
                foreach (var column in columns)
                {
                    record.Add(column, reader[column]);
                }
                result.Add(record);
            }
        }
        return result;
    }

Support methods

Here is the code of the missing methods: GetCxnString and GetCommand not really complicated.

    private static readonly string cxnStringFormat = "server={0};uid={1};pwd={2};database={3};Connection Timeout=60";

    private string GetCxnString(ConnectionInfo cxn)
    {
        return string.Format(cxnStringFormat, cxn.DbHost, cxn.DbUser, cxn.DbPassword, cxn.DbName);
    }

    private static MySqlCommand GetCommand(MySqlConnection conn, string query)
    {
        conn.Open();
        var cmd = conn.CreateCommand();
        cmd.CommandText = query;
        cmd.CommandType = CommandType.Text;
        return cmd;
    }

Project file

Before compiling and packaging the code we need a project file, assuming you don't have one already, this is how it looks like to be able to run in AWS Lambda environment.

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <TargetFramework>netcoreapp2.1</TargetFramework>
    <GenerateRuntimeConfigurationFiles>true</GenerateRuntimeConfigurationFiles>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Amazon.Lambda.Core" Version="1.0.0" />
    <PackageReference Include="Amazon.Lambda.Serialization.Json" Version="1.3.0" />
    <PackageReference Include="MySql.Data" Version="8.0.13" />
    <PackageReference Include="Newtonsoft.Json" Version="11.0.2" />
  </ItemGroup>

  <ItemGroup>
    <DotNetCliToolReference Include="Amazon.Lambda.Tools" Version="2.2.0" />
  </ItemGroup>


Preparing Lambda package

Assuming you have both the code and csproj file in the current directory, we just run dotnet lambda package command as per below, where -c sets the Configuration to release, -f sets the target framework to netcoreapp2.1 and -o sets the output zip file name.

$ dotnet lambda package -c release -f netcoreapp2.1 -o aurora-lambda.zip
Amazon Lambda Tools for .NET Core applications (2.2.0)
Project Home: https://github.com/aws/aws-extensions-for-dotnet-cli, https://github.com/aws/aws-lambda-dotnet

Executing publish command
Deleted previous publish folder
... invoking 'dotnet publish', working folder '/home/abel/Downloads/aurora_cluster_sample/bin/release/netcoreapp2.1/publish'

( ... ) --- removed code for brevity ---

... zipping:   adding: aurora.lambda.deps.json (deflated 76%)
Created publish archive (/home/abel/Downloads/aurora_cluster_sample/aurora-lambda.zip).
Lambda project successfully packaged: /home/abel/Downloads/aurora_cluster_sample/aurora-lambda.zip

Next, we upload the resulting zip file to an S3 bucket of our choice. In this example I'm using a bucket named abelperez-temp and I'm uploading the zip file to a folder named aurora-lambda so I keep some form of organisation in my file directory.

$ aws s3 cp aurora-lambda.zip s3://abelperez-temp/aurora-lambda/
upload: ./aurora-lambda.zip to s3://abelperez-temp/aurora-lambda/aurora-lambda.zip

Lambda stack

To create the Lambda function, I've put together a CloudFormation template that includes:

  • AWS::EC2::SecurityGroup contains outbound traffic rule to allow port 3306
  • AWS::IAM::Role contains an IAM role to allow the Lambda function to write to CloudWatch Logs and interact with ENIs
  • AWS::Lambda::Function contains the function definition

Here is the full template, the required parameters are VpcId, SubnetIds and LambdaS3Bucket which we should get from previous stacks' outputs. The template outputs the function full name, which we'll need to be able to invoke it later.

Special attention to the Lambda function definition, the property Handler, in .NET runtime is in the form of AssemblyName::Namespace.ClassName::MethodName and the property Code containing the S3 location of the zip file we uploaded earlier.

Description: Template to create a lambda function 

Parameters: 
  LambdaS3Bucket:
    Type: String
  DbClusterPort: 
    Type: Number
    Default: 3306
  VpcId: 
    Type: String
  SubnetIds: 
    Type: CommaDelimitedList

Resources:
  LambdaSg:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: Allow outbound traffic to MySQL host
      VpcId:
        Ref: VpcId
      SecurityGroupEgress:
        - IpProtocol: tcp
          FromPort: !Ref DbClusterPort
          ToPort: !Ref DbClusterPort
          CidrIp: 0.0.0.0/0

  AWSLambdaExecutionRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Principal:
              Service:
                - lambda.amazonaws.com
            Action: sts:AssumeRole
      Path: /
      Policies:
        - PolicyName: PermitLambda
          PolicyDocument:
            Version: 2012-10-17
            Statement:
            - Effect: Allow
              Action:
              - logs:CreateLogGroup
              - logs:CreateLogStream
              - logs:PutLogEvents
              - ec2:CreateNetworkInterface
              - ec2:DescribeNetworkInterfaces
              - ec2:DeleteNetworkInterface
              Resource: 
                - "arn:aws:logs:*:*:*"
                - "*"
  HelloLambda:
    Type: AWS::Lambda::Function
    Properties:
      Handler: aurora.lambda::project.lambda.Function::RunQueryHandler
      Role: !GetAtt AWSLambdaExecutionRole.Arn
      Code:
        S3Bucket: !Ref LambdaS3Bucket
        S3Key: aurora-lambda/aurora-lambda.zip
      Runtime: dotnetcore2.1
      Timeout: 30
      VpcConfig:
        SecurityGroupIds:
          - !Ref LambdaSg
        SubnetIds: !Ref SubnetIds

Outputs:
  LambdaFunction:
    Value: !Ref HelloLambda

To deploy this stack we use the following command where we pass the parameters specific to our VPC (VpcId and SubnetIds) as well as the S3 bucket name.

$ aws cloudformation deploy --stack-name fn-stack \
--template-file aurora_lambda_template.yml \
--parameter-overrides VpcId=vpc-0b442e5d98841996c SubnetIds=subnet-013d0bbb3eca284a2,subnet-00c67cfed3ab0a791 LambdaS3Bucket=abelperez-temp \
--capabilities CAPABILITY_IAM

Waiting for changeset to be created..
Waiting for stack create/update to complete
Successfully created/updated stack - fn-stack

Let's get the outputs as we'll need this information later. We have the Lambda function full name.

$ aws cloudformation describe-stacks --stack-name fn-stack --query Stacks[*].Outputs
[
    [
        {
            "OutputKey": "LambdaFunction",
            "OutputValue": "fn-stack-HelloLambda-C32KDMYICP5W"
        }
    ]
]

Invoking Lambda function

Now that we have deployed the function and we know its full name, we can invoke it by using dotnet lambda invoke-function command. Part of this job is to prepare the payload which is a JSON in put corresponding to the Lambda input defined above.

{
    "Connection": {
        "DbUser": "master", 
        "DbPassword": "Aurora.2019", 
        "DbName": "dbtest", 
        "DbHost": "db-stack-auroramysqlcluster-xxx.rds.amazonaws.com", 
        "DbPort": 3306
    }, 
    "QueryText":"show databases;"
}

Here is the command to invoke the Lambda function, including the payload parameter encoded to escape the quotes and all in a single line. There are better ways to do this, but for the sake of this demonstration, it's good enough.

$ dotnet lambda invoke-function \
--function-name fn-stack-HelloLambda-C32KDMYICP5W \
--payload "{ \"Connection\": {\"DbUser\": \"master\", \"DbPassword\": \"Aurora.2019\", \"DbName\": \"dbtest\", \"DbHost\": \"db-stack-auroramysqlcluster-xxx.rds.amazonaws.com\", \"DbPort\": 3306}, \"QueryText\":\"show databases;\" }" \
--region eu-west-1

Amazon Lambda Tools for .NET Core applications (2.2.0)
Project Home: https://github.com/aws/aws-extensions-for-dotnet-cli, https://github.com/aws/aws-lambda-dotnet

Payload:
[{"Database":"information_schema"},{"Database":"dbtest"},{"Database":"mysql"},{"Database":"performance_schema"}]

Log Tail:
START RequestId: 595944b5-73bb-4536-be92-a42652125ba8 Version: $LATEST
END RequestId: 595944b5-73bb-4536-be92-a42652125ba8
REPORT RequestId: 595944b5-73bb-4536-be92-a42652125ba8  Duration: 11188.62 ms   Billed Duration: 11200 ms       Memory Size: 128 MB     Max Memory Used: 37 MB

Now we can see the output in the Payload section. And that's how we can query remotely any Aurora serverless cluster without having to set up any EC2 instance. This could be extended to handle different SQL operations such as Create, Insert, Delete, etc.

Monday, 4 February 2019

Querying Aurora serverless database remotely using Lambda - part 2

This post is part of a series

In the previous part, we've set up the base layer to deploy our resources. At this point we can create the database cluster.

Aurora DB Cluster

Assuming we have our VPC ready with at least two subnets to comply with high availability best practices, let's create our cluster, I've put together a CloudFormation template that includes:

  • AWS::EC2::SecurityGroup contains inbound traffic rule to allow port 3306
  • AWS::RDS::DBSubnetGroup contains a group of subnets to deploy the cluster
  • AWS::EC2::DBCluster contains all the parameters to create the database cluster

Here is the full template, the only required parameters are VpcId and SubnetIds, but feel free to override any of the database cluster parameters such as database name, user name, password, etc. The template outputs the IDs corresponding to newly created resources such as the database cluster DNS endpoint, port and the security group.

Description: Template to create a serverless aurora mysql cluster

Parameters: 
  DbClusterDatabaseName: 
    Type: String
    Default: dbtest
  DbClusterIdentifier: 
    Type: String
    Default: serverless-mysql-aurora
  DbClusterParameterGroup: 
    Type: String
    Default: default.aurora5.6
  DbClusterMasterUsername: 
    Type: String
    Default: master
  DbClusterMasterPassword: 
    Type: String
    Default: Aurora.2019
  DbClusterPort: 
    Type: Number
    Default: 3306
  VpcId: 
    Type: String
  SubnetIds: 
    Type: CommaDelimitedList

Resources:
  DbClusterSg:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: Allow MySQL port to client host
      VpcId:
        Ref: VpcId
      SecurityGroupIngress:
        - IpProtocol: tcp
          FromPort: !Ref DbClusterPort
          ToPort: !Ref DbClusterPort
          CidrIp: 0.0.0.0/0

  DbSubnetGroup: 
    Type: "AWS::RDS::DBSubnetGroup"
    Properties: 
      DBSubnetGroupDescription: "aurora subnets"
      SubnetIds: !Ref SubnetIds

  AuroraMysqlCluster:
    Type: AWS::RDS::DBCluster
    Properties:
      DatabaseName:
        Ref: DbClusterDatabaseName
      DBClusterParameterGroupName:
        Ref: DbClusterParameterGroup
      DBSubnetGroupName:
        Ref: DbSubnetGroup
      Engine: aurora
      EngineMode: serverless
      MasterUsername:
        Ref: DbClusterMasterUsername
      MasterUserPassword:
        Ref: DbClusterMasterPassword
      ScalingConfiguration:
        AutoPause: true
        MinCapacity: 2
        MaxCapacity: 4
        SecondsUntilAutoPause: 1800
      VpcSecurityGroupIds:
        - !Ref DbClusterSg
        
Outputs:
  DbClusterEndpointAddress:
    Value: !GetAtt AuroraMysqlCluster.Endpoint.Address
  DbClusterEndpointPort:
    Value: !GetAtt AuroraMysqlCluster.Endpoint.Port
  DbClusterSgId:
    Value: !Ref DbClusterSg

To deploy this stack we use the following command where we pass the parameters specific to our VPC (VpcId and SubnetIds).

$ aws cloudformation deploy --stack-name db-stack \
--template-file aurora_cluster_template.yml \
--parameter-overrides VpcId=vpc-0b442e5d98841996c SubnetIds=subnet-013d0bbb3eca284a2,subnet-00c67cfed3ab0a791

Waiting for changeset to be created..
Waiting for stack create/update to complete
Successfully created/updated stack - db-stack

Let's get the outputs as we'll need this information later. We have the cluster endpoint DNS name and the port as per our definition.

$ aws cloudformation describe-stacks --stack-name db-stack --query Stacks[*].Outputs
[
    [
        {
            "OutputKey": "DbClusterEndpointAddress",
            "OutputValue": "db-stack-auroramysqlcluster-1d1udg4ringe4.cluster-cnfxlauucwwi.eu-west-1.rds.amazonaws.com"
        },
        {
            "OutputKey": "DbClusterSgId",
            "OutputValue": "sg-072bbf2078caa0f46"
        },
        {
            "OutputKey": "DbClusterEndpointPort",
            "OutputValue": "3306"
        }
    ]
]

In the next part, we'll create the Lambda function to query this database remotely.

Wednesday, 30 January 2019

Querying Aurora serverless database remotely using Lambda - part 1

This post is part of a series

Why aurora serverless?

For those who like experimenting with new technology, AWS feeds us with a lot of new stuff every year at re:Invent conference, and many more times. In August 2018, Amazon announced its general availability. I was intrigued by this totally new way to doing database, so I started to play with it.

The first road block I found was connectivity from my local environment. I wanted to connect to my new cluster using my traditional MySQL Workbench client. It turned out to be one of the limitations clearly explained by Amazon, and as of today:

You can't give an Aurora Serverless DB cluster a public IP address. You can access an Aurora Serverless DB cluster only from within a virtual private cloud (VPC) based on the Amazon VPC service.

Most common workarounds involve the use of an EC2 instance to either run a MySQL client from there or SSH tunnel and allow connections from outside of the VPC. In both cases we'll be charged for the use of the EC2 instace. At the moment of this writing there is a solution for that, but still in beta: Data API.

With all that said, I decided to explore my own way in the meantime by creating a serverless approach involving Lambda to query my database.

Setting up the base - VPC

First, we need a VPC. For testing only, it doesn't really matter if we just use the default VPC in the current region. But if you'd like to get started with a blue print, there is this public CloudFormation template, it contains a sample VPC with two public subnets and two private subnets as well as all the required resources to guarantee connectivity (IGW, NAT, Route tables, etc.).

I prefer to isolate my experiments from the rest of the resources, that's why I came up with a template containing the bare minimum to get a VPC up and running. It only includes:

  • AWS::EC2::VPC the VPC itself - default CIDR block 10.192.0.0/16
  • AWS::EC2::Subnet private subnet 1 - default CIDR block 10.192.20.0/24
  • AWS::EC2::Subnet private subnet 2 - default CIDR block 10.192.21.0/24

Here is the full template, the only required parameter is EnvironmentName, but feel free to override any of the CIDR blocks. The template outputs the IDs corresponding to newly created resources such as the VPC and the subnets.

Description: >-
  This template deploys a VPC, with two private subnets spread across 
  two Availability Zones. This VPC does not provide any internet 
  connectivity resources such as IGW, NAT Gw, etc.

Parameters:
  EnvironmentName:
    Description: >-
      An environment name that will be prefixed to resource names
    Type: String

  VpcCIDR:
    Description: >-
      Please enter the IP range (CIDR notation) for this VPC
    Type: String
    Default: 10.192.0.0/16

  PrivateSubnet1CIDR:
    Description: >-
      Please enter the IP range (CIDR notation) for the private subnet 
      in the first Availability Zone
    Type: String
    Default: 10.192.20.0/24

  PrivateSubnet2CIDR:
    Description: >-
      Please enter the IP range (CIDR notation) for the private subnet 
      in the second Availability Zone
    Type: String
    Default: 10.192.21.0/24

Resources:
  VPC:
    Type: AWS::EC2::VPC
    Properties:
      CidrBlock: !Ref VpcCIDR
      EnableDnsSupport: true
      EnableDnsHostnames: true
      Tags:
        - Key: Name
          Value: !Ref EnvironmentName

  PrivateSubnet1:
    Type: AWS::EC2::Subnet
    Properties:
      VpcId: !Ref VPC
      AvailabilityZone: !Select [ 0, !GetAZs '' ]
      CidrBlock: !Ref PrivateSubnet1CIDR
      MapPublicIpOnLaunch: false
      Tags:
        - Key: Name
          Value: !Sub ${EnvironmentName} Private Subnet (AZ1)

  PrivateSubnet2:
    Type: AWS::EC2::Subnet
    Properties:
      VpcId: !Ref VPC
      AvailabilityZone: !Select [ 1, !GetAZs '' ]
      CidrBlock: !Ref PrivateSubnet2CIDR
      MapPublicIpOnLaunch: false
      Tags:
        - Key: Name
          Value: !Sub ${EnvironmentName} Private Subnet (AZ2)

Outputs:
  VPC:
    Description: A reference to the created VPC
    Value: !Ref VPC

  PrivateSubnet1:
    Description: A reference to the private subnet in the 1st Availability Zone
    Value: !Ref PrivateSubnet1

  PrivateSubnet2:
    Description: A reference to the private subnet in the 2nd Availability Zone
    Value: !Ref PrivateSubnet2

To create a stack from this template we run the following command (or go to AWS console and upload the template)

$ aws cloudformation deploy --stack-name vpc-stack \
--template-file vpc_template.yml \
--parameter-overrides EnvironmentName=Dev

Waiting for changeset to be created..
Waiting for stack create/update to complete
Successfully created/updated stack - vpc-stack

After successful stack creation, we can get the outputs as we'll need them for the next step.

$ aws cloudformation describe-stacks --stack-name vpc-stack --query Stacks[*].Outputs
[
    [
        {
            "Description": "A reference to the private subnet in the 1st Availability Zone",
            "OutputKey": "PrivateSubnet1",
            "OutputValue": "subnet-013d0bbb3eca284a2"
        },
        {
            "Description": "A reference to the private subnet in the 2nd Availability Zone",
            "OutputKey": "PrivateSubnet2",
            "OutputValue": "subnet-00c67cfed3ab0a791"
        },
        {
            "Description": "A reference to the created VPC",
            "OutputKey": "VPC",
            "OutputValue": "vpc-0b442e5d98841996c"
        }
    ]
]

In the next part, we'll create the Database cluster using these resources as a base layer.

Thursday, 27 September 2018

How to implement a CloudFormation Include custom tag using YamlDotNet

YAML has become a popular format used to describe a wide range of information in a more readable way. Depending on the concrete use case, these files can grow significantly. One example of this is AWS CloudFormation templates which can be written either in JSON or YAML.

Especially when working on Serverless projects, it doesn’t matter if it’s serverless framework, AWS SAM or just pure CloudFormation. The fact is the main file to maintain is (most of the time) a YAML template that grows over time.

I’d like to focus on a more concrete example: a typical serverless microservice architecture consists of a backend database, a function and a HTTP endpoint (i.e DynamoDB, Lambda and API Gateway). It’s also a common practice to use OpenAPI specification aka Swagger to describe the Web API interface.

In this case, when trying to use Lambda integration (either custom or proxy) we can’t use variables or intrinsic functions within the swagger file (we have to hardcode the lambda invocation url including account number and region, apart from the fact that function name might change if using autogenerated by CloudFormation), unless swagger content is inline in the Body property, which makes the file to grow by a great deal.

As of this writing, there are some strategies proposed by AWS to mitigate this problem, I personally find them a bit cumbersome to use on a daily basis:

  • AWS::CloudFormation::Stack It adds complexity to the whole process since we have to pass parameters to nested stacks and retrieve outputs from them in order to access information from both parties. The template for the nested stack must be stored on an Amazon S3 bucket, which adds friction to our development workflow while building.
  • AWS::Include Transform which is a form of macro hosted by AWS CloudFormation, is simpler than nested stacks but currently it still has some limiations:
    • The snippet has to be stored on an Amazon S3 bucket.
    • If the snippets change, your stack doesn't automatically pick up those changes.
    • It does not currently support using shorthand notations for YAML snippets.

Personally, I prefer a solution where at development time I can split the template in logical parts and then, before deploying it to AWS, compose them in one piece. I like the idea of include partials in specific parts of the parent file.

How to implement this include mechanism ?

YAML provides an extension mechanism named tags, where we can associate a particular data type with a tag (it’s basically a prefix added to a value). In YamlDotNet this is implemented by creating a custom type converter and mapping the new tag with the custom type converter.

IncludeTagConverter (custom type converter)

public class IncludeTagConverter: IYamlTypeConverter
{
    public bool Accepts(Type type)
    {
        return typeof(IncludeTag).IsAssignableFrom(type);
    }

    public object ReadYaml(IParser parser, Type type)
    {
        parser.Expect<MappingStart>();
        var key = parser.Expect<Scalar>();
        var val = parser.Expect<Scalar>();
        parser.Expect<MappingEnd>();

        if (key.Value != "File")
        {
            throw new YamlException(key.Start, val.End, "Expected a scalar named 'File'");
        }

        var input = File.ReadAllText(val.Value);
        var data = YamlSerializer.Deserialize(input);
        return data;
    }

    public void WriteYaml(IEmitter emitter, object value, Type type)
    {
    }
}

IncludeTag class

public class IncludeTag
{
    public string File { get; set; }
}

In this case we are indicating that IncludeTagConverter class should be used if the desiralization mechanism needs to deserialize an object of type IncludeTag. At the end of ReadYaml method, we call a helper class that starts deserialization process again with the content of the “included file”.

YamlSerializer helper class

public class YamlSerializer
{
    private const string IncludeTag = "!Include";

    public static object Deserialize(string yaml)
    {
        var reader = new StringReader(yaml);
        var deserializer = new DeserializerBuilder()
            .WithTypeConverter(new IncludeTagConverter())
            .WithTagMapping(IncludeTag, typeof(IncludeTag))
            .Build();
        var data = deserializer.Deserialize(reader);
        return data;
    }

    public static string Serialize(object data)
    {
        var serializer = new SerializerBuilder().Build();
        var yaml = serializer.Serialize(data);
        return yaml;
    }
}

In this helper class, we tell the deserializer that we are using a type converter and that it has to map !Include tags to data type of IncludeTag. This way, when it encounters an !Include in the yaml file, it will use our type converter to deserialize the content instead, which in turn, will read whatever file we put the name in File: key and will trigger the deserialization process again, thus, allowing us to execute this at several levels in a recursive way.

How do we compose a yaml ?

Once we have the whole file yaml object in memory, by triggering the deserialization process on the main yaml file, like this:

var data = YamlSerializer.Deserialize(input);

We only need to call Serialize again, and since we converted all the !Include tags into normal maps, sequence or scalars, there’s nothing extra we need to do to serialize it back using the default implementation.

var output = YamlSerializer.Serialize(data);

The output will be the composed file which can be saved and used after that.

Example:

Main yaml file (cloud-api.yaml)

Description: Template to create a serverless web api 

Resources:
  ApiGatewayRestApi:
    Type: AWS::ApiGateway::RestApi
    Properties:
      Name: Serverless API
      Description: Serverless API - Using CloudFormation and Swagger
      Body: !Include
        File: simple-swagger.yaml

simple-swagger.yaml file

swagger: "2.0"

info:
  version: 1.0.0
  title: Simple API
  description: A simple API to learn how to write OpenAPI Specification

paths:
  /persons: !Include
    File: persons.yaml
  /pets: !Include
    File: pets.yaml

persons.yaml file

get:
  summary: Gets some persons
  description: Returns a list containing all persons.
  responses:
    200:
      description: A list of Person
      schema:
        type: array
        items:
          required:
            - username
          properties:
            firstName:
              type: string
            lastName:
              type: string
            username:
              type: string

pets.yaml file

get:
  summary: Gets some pets
  description: Returns a list containing all pets.
  responses:
    200:
      description: A list of pets
      schema:
        type: array
        items:
          required:
            - petname
          properties:
            petname:
              type: string
            ownerName:
              type: string
            breed:
              type: string

Final result (composed file)

Description: Template to create a serverless web api
Resources:
  ApiGatewayRestApi:
    Type: AWS::ApiGateway::RestApi
    Properties:
      Name: Serverless API
      Description: Serverless API - Using CloudFormation and Swagger
      Body:
        swagger: 2.0
        info:
          version: 1.0.0
          title: Simple API
          description: A simple API to learn how to write OpenAPI Specification
        paths:
          /persons:
            get:
              summary: Gets some persons
              description: Returns a list containing all persons.
              responses:
                200:
                  description: A list of Person
                  schema:
                    type: array
                    items:
                      required:
                      - username
                      properties:
                        firstName:
                          type: string
                        lastName:
                          type: string
                        username:
                          type: string
          /pets:
            get:
              summary: Gets some pets
              description: Returns a list containing all pets.
              responses:
                200:
                  description: A list of pets
                  schema:
                    type: array
                    items:
                      required:
                      - petname
                      properties:
                        petname:
                          type: string
                        ownerName:
                          type: string
                        breed:
                          type: string