Adding a CI/CD Pipeline

To reduce the chance of a human error affecting the a production site (especially in a team environment) it is good practice to implement a CI/CD pipeline. This post covers the implementation of a CI/CD pipeline for the AlphaGeek site. In future I will provide a more generic post about CI/CD pipelines.

Pre-requisites

This article will be based on the blog described in Hexo, AWS and Serverless Framework, Securing A Test Environment Using AWS WAF, Securing S3 Bucket from Direct Access and Implementing Unit Testing; some of the concepts can be applied to other uses, but this post will focus on the specific AlphaGeek use case.

Select a CI/CD Pipeline Provider

There are a large number of CI/CD Pipeline providers. Some of them offer self-hosted solutions, others charge for even the most basic account, some are free for open-source projects, and some provide limited accounts for free.

For my purposes I selected SemaphoreCI as it integrates with GitHub, is free (with significant usage limitations) and appears to provide a high level of configuration.

Design Your Pipeline

As the implementation of a CI/CD pipeline is to improve the reliability of my blog I added a number of new packages to my blog's requirements as part of the CI/CD pipeline.

If any task in the pipeline fails all subsequent actions should not occur. The final pipeline design will operate as follows:

Pre-commit Git Hook

A pre-commit git hook will prevent any commits of invalid code.

  • Run unit tests;
  • Run markdown linting;
  • Clean the site build files;
  • Rebuild the site;
  • Apply versioning to the asset files;
  • Scan the generated files for broken links;
  • Scan the generated files for non-W3C compliant HTML;
  • Scan the generated files for accessibility standards compliance but ignore the results.

CI/CD Pipeline

The following CI/CD pipeline will be triggered when a commit to the develop branch is pushed to GitHub.

  • Re-run the pre-commit tasks in the CI/CD environment;
  • Deploy content changes to the development site;
  • Scan the deployed files for broken links.

The following CI/CD pipeline will be triggered when a commit to the master branch is pushed to GitHub.

  • Re-run the pre-commit tasks in the CI/CD environment;
  • Deploy content changes to the production site;
  • Scan the deployed files for broken links.

After all tests have passed an option to deploy infrastructure and code changes will be available though the chosen CI/CD SaaS provider.

Pipeline Implementation

Sign-up to SemaphoreCI

The first task was to sign-up to SemaphoreCI. This was as simple as clicking on the large Sign up with GitHub button and selecting the repository I wanted to integrate.

Set-up the New Test Dependencies

To implement the tests several new dependencies have been added to the blog.

markdownlint-cli

To ensure the markdown for all the posts is formatted consistently markdownlint-cli was installed.

Install markdownlint-cli
1
npm i --save-dev markdownlint-cli

A custom configuration was created for this dependency so it operates how I want it to. This configuration enforces the top level heading as level 2 (level 1 headings are used automatically for the post title in my chosen template); the maximum line length was disabled because it is not compatible with Hexo's default Markdwon interpreter; and I removed the ? character from the heading validation.

Create a .mdconfig file in the root of your project
1
2
3
4
5
6
7
8
9
{
"MD002": {
"level": 2
},
"MD013": false,
"MD026": {
"punctuation": ".,;:!"
}
}

node-w3c-validator

To ensure there is no invalid HTML on the site I implemented node-w3c-validator. This script has a dependency on Java, so that may need to be installed as well.

Install node-w3c-validator
1
npm i --save-dev node-w3c-validator

pa11y-ci

Pa11y-CI is a wrapper for Pa11y to make it easier to integrate in a CI/CD pipeline.

Pa11y scans an HTML file for any accessibility issues. As my blog has a large number of existing issues I have configured the scripts that run this to allow it to fail.

Install pa11y-ci
1
npm i --save-dev pa11y-ci

Add New Scripts to NPM

To simplify running the existing NPM commands and the new dependencies I added a number of elements to the scripts section of the package.json file, modified some of the definitions and re-ordered them to make more sense to me.

<!-- markdownlint-disable MD013 -->

New elements added to the scripts section of package.json
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
"scripts": {
"jest": "jest",
"mdlint-drafts": "markdownlint --config .mdconfig ./source/_drafts",
"mdlint": "markdownlint --config .mdconfig ./source/_posts",
"precheck": "npm run jest && npm run mdlint",
"clean": "hexo clean",
"build": "hexo generate",
"cleanbuild": "npm run clean && npm run build",
"linkcheck": "blcl --filter-level 3 --get --recursive --exclude /atom.xml --exclude /favicon.png --exclude http://2019-01-28-securing-s3.demo.alphageek.com.au.s3-website-us-east-1.amazonaws.com --exclude http://localhost:4000 --exclude http://dev./%3Cyour_domain%3E/ public",
"htmlcheck": "node-w3c-validator -v -s -i public/",
"a11ycheck": "pa11y-ci public/*.html public/*/*.html public/*/*/*.html public/*/*/*/*.html public/*/*/*/*/*.html || true",
"validate": "npm run linkcheck && npm run htmlcheck && npm run a11ycheck",
"buildtest-local": "npm run precheck && npm run cleanbuild && npm run validate",
"precommit": "npm run buildtest-local"
},

<!-- markdownlint-enable MD013 -->

Default theme issues

If you are running the default Hexo theme (landscape), it is not W3C compliant, so you will need to change the htmlcheck script in your package.json to be node-w3c-validator -v -s -i public/ || true. This will display the output when you run the precommit hook, but will not enforce the HTML validation.

Implement the Git Pre-Commit Hook

Implementing a Git pre-commit hook is simply a matter of creating a file at .git/pre-comit and populating it with a valid shell script that completes with an error code of 0.

Create a git pre-commit hook at .git/pre-commit
1
2
3
#!/bin/sh

npm run precommit

Define the Tasks to Run on SemaphoreCI

The SemaphoreCI configuration is done in multiple phases to ensure it works and so a broken build isn't accidentally deployed to production.

In the configuration file, tasks will be run sequentially, and if multiple jobs are defined within a task they can be run in parallel.

SemaphoreCI configuration located at .semaphore/semaphore.yml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
version: v1.0
name: Hexo Serverless Build Pipeline
agent:
machine:
type: e1-standard-2
os_image: ubuntu1804
blocks:
# Prepare the build environment
- name: Prepare
task:
jobs:
# Make sure we've got the right Java version configured
- name: Set Java Version
commands:
- change-java-version 8
# Run NPM install, using Semaphore's cache where possible
- name: NPM Install
commands:
# Update NPM because it's so old
- cache restore npm
- npm i -g npm
- cache store npm .nvm/versions/node/v8.11.3/lib/node_modules/npm
- checkout
# Reuse dependencies from cache and avoid installing them from scratch:
- cache restore node-modules-$(checksum package-lock.json)
- npm ci
- cache store node-modules-$(checksum package-lock.json) node_modules
# Run the validation routines that don't require a build
- name: Validate
task:
prologue:
commands:
- checkout
- cache restore npm
- cache restore node-modules-$(checksum package-lock.json)
jobs:
# Run the jest test suite
- name: Run Jest Tests
commands:
- npm run jest
# Run the Markdown linter
- name: MD Lint
commands:
- npm run mdlint
# Build the deployment files
- name: Build
task:
prologue:
commands:
- checkout
- cache restore npm
- cache restore node-modules-$(checksum package-lock.json)
jobs:
# Ensure we have a clean build directory, generate the files and add asset versioning
- name: Build Site
commands:
- npm run clean
- npm run build
- cache store public-$(find source -type f -exec cat {} + | checksum) public
# Run tests on the deployment files
- name: Test Locally
task:
prologue:
commands:
- checkout
- cache restore npm
- cache restore node-modules-$(checksum package-lock.json)
- cache restore public-$(find source -type f -exec cat {} + | checksum)
jobs:
# Check that all links are valid
- name: Test link validity
commands:
- npm run linkcheck
# Check that the HTML is valid
- name: Test W3C compatability
commands:
- npm run htmlcheck
# Check if we meet a11y standards
- name: Test Accessibility
commands:
- npm run a11ycheck

Commit and Test

As a pre-commit hook has been added your next commit may be rejected until all existing errors have been fixed. In my case I had to fix errors in all of the markdown files for each of my previous posts.

At this point the CI/CD pipeline should successfully build and test the site.

Add UAT and Production Scripts to NPM

To enable the automated deployment of content changes, and to provide functionality to deploy infrastructure and code changes, some additional scripts need to be defined in the package.json file. These scripts will allow for deployment and testing of both the UAT and Production environments.

<!-- markdownlint-disable MD013 -->

New script definitions for package.json
1
2
3
4
5
6
7
8
9
10
11
12
"deploy-uat-infra": "npx serverless deploy -s dev",
"deploy-uat-site": "npx serverless s3deploy -s dev -v",
"deploy-uat": "npm run deploy-uat-infra && npm run deploy-uat-site",
"linkcheck-uat": "npx blc --filter-level 3 --get --recursive --exclude /atom.xml --exclude /favicon.png --exclude http://2019-01-28-securing-s3.demo.alphageek.com.au.s3-website-us-east-1.amazonaws.com --exclude http://localhost:4000 --exclude http://dev./%3Cyour_domain%3E/ --user-agent '<PASSWORD_DEFINED_IN_SERVERLESS_CONFIGURATION> Tester' http://dev.alphageek.com.au",
"test-uat": "npm run deploy-uat && npm run linkcheck-uat",
"buildtest-uat": "npm run buildtest-local && test-uat",
"deploy-prod-infra": "npx serverless deploy -s prod",
"deploy-prod-site": "npx serverless s3deploy -s prod -v",
"deploy-prod": "npm run deploy-prod-infra && npm run deploy-prod-site",
"linkcheck-prod": "npx blc --filter-level 3 --get --recursive --exclude /atom.xml --exclude /favicon.png --exclude http://2019-01-28-securing-s3.demo.alphageek.com.au.s3-website-us-east-1.amazonaws.com --exclude http://localhost:4000 --exclude http://dev./%3Cyour_domain%3E/ http://alphageek.com.au",
"test-prod": "npm run deploy-prod && npm run linkcheck-prod",
"buildtest-prod": "npm run buildtest-uat && test-prod"

<!-- markdownlint-enable MD013 -->

Remove Deployment Functionality from Serverless Framework

As the deployment process has been migrated to NPM commands the serverless.yml configuration file needs to the build and deployment functionality removed. Delete the following lines from the file:

Delete these lines from serverless.yml
1
2
3
4
5
6
7
8
9
scripts:
hooks:
# Run these commands when creating the deployment artifacts
package:createDeploymentArtifacts: >
hexo clean &&
hexo generate
# Run these commands after infrastructure changes have been completed
deploy:finalize: >
sls s3deploy -s ${self:custom.stage}

Install SemaphoreCI CLI

To enable deployment to AWS SemaphoreCI will need to have access to our AWS credentials. Credentials and other secrets should never be stored in a code repository, so we will require a method to securely save the credentials on Semaphore. This can be done using the SemaphoreCI command line utility. You will also need to know your SemaphoreCI organization name and the SemaphoreCI API token (which can be found at the SemaphoreCI Account Page).

Install and configure sem - the SemaphoreCI CLI
1
2
curl https://storage.googleapis.com/sem-cli-releases/get.sh | bash
sem connect <ORGANIZATION>.semaphoreci.com <API_TOKEN>

Provide AWS Credentials to SemaphoreCI

To provide credentials and secrets to SemaphoreCI a file needs to be created. So this doesn't accidentally get committed to the code repository we begin by adding to the .gitignore file.

Add to the .gitignore file
1
2
# Semaphore secret files
.semaphore/secrets/*

It's now safe to create a file with your AWS credentials in it. Create a new file at .semaphore/secrets/aws.yml with the following content (updated with your AWS details).

AWS credentials file at .semaphore/secrets/aws.yml
1
2
3
4
5
6
7
8
9
10
apiVersion: v1beta
kind: Secret
metadata:
name: <YOUR_SITE_NAME>-aws
data:
env_vars:
- name: AWS_ACCESS_KEY_ID
value: "<YOUR_ACCESS_KEY_ID>"
- name: AWS_SECRET_ACCESS_KEY
value: "<YOUR_SECRET_ACCESS_KEY>"

Then this file needs to be imported to SemaphoreCI.

Import the secrets file to SemaphoreCI
1
sem create -f .semaphore/secrets/aws.yml

The value of these secrets will be embedded in the deployment configuration in the next section.

Create UAT and Production Deployment Configuration

Everything is now ready to create the SemaphoreCI deployment scripts.

Start by creating a file to deploy the content changes to S3 for UAT at .semaphore/uat-content.yml.

UAT Content deployment configuration in .semaphore/uat-content.yml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
version: v1.0
name: AlphaGeek UAT Content Deployment Pipeline
agent:
machine:
type: e1-standard-2
os_image: ubuntu1804
blocks:
# Use Serverless to deploy to UAT
- name: Publish to UAT
task:
# Import the secret environment variables
secrets:
- name: alphageek-aws
prologue:
commands:
- checkout
- cache restore npm
- cache restore node-modules-$(checksum package-lock.json)
- cache restore public-$(find source -type f -exec cat {} + | checksum)
jobs:
- name: Deploy Content
commands:
- npm run deploy-uat-site
# Run tests on the UAT site
- name: Test UAT
task:
prologue:
commands:
- checkout
- cache restore npm
- cache restore node-modules-$(checksum package-lock.json)
jobs:
# Check that all links are valid
- name: Test link validity
commands:
- npm run linkcheck-uat

Create a similar file for deploying content changes to S3 for Production at .semaphore/prod-content.yml.

Production content deployment configuration in .semaphore/prod-content.yml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
version: v1.0
name: AlphaGeek Production Content Deployment Pipeline
agent:
machine:
type: e1-standard-2
os_image: ubuntu1804
blocks:
# Use Serverless to deploy to Production
- name: Publish to Prod
task:
# Import the secret environment variables
secrets:
- name: alphageek-aws
prologue:
commands:
- checkout
- cache restore npm
- cache restore node-modules-$(checksum package-lock.json)
- cache restore public-$(find source -type f -exec cat {} + | checksum)
jobs:
- name: Deploy Content
commands:
- npm run deploy-prod-site
# Run tests on the Production site
- name: Test Production
task:
prologue:
commands:
- checkout
- cache restore npm
- cache restore node-modules-$(checksum package-lock.json)
jobs:
# Check that all links are valid
- name: Test link validity
commands:
- npm run linkcheck-prod

Now we create files for both UAT and Production infrastructure and Lambda function changes at .semaphore/uat-infra.yml and .semaphore/prod-infra.yml respectively.

UAT Infrastructure deployment configuration at .semaphore/uat-infra.yml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
version: v1.0
name: AlphaGeek UAT Infrastructure Deployment Pipeline
agent:
machine:
type: e1-standard-2
os_image: ubuntu1804
blocks:
# Use serverless to deploy to UAT
- name: Publish to UAT
task:
# Import the secret environment variables
secrets:
- name: alphageek-aws
prologue:
commands:
- checkout
- cache restore npm
- cache restore node-modules-$(checksum package-lock.json)
jobs:
- name: Deploy Infrastructure
commands:
- npm run deploy-uat-infra
- cache store serverless-$SEMAPHORE_GIT_BRANCH .serverless
# Run tests on the UAT site
- name: Test UAT
task:
prologue:
commands:
- checkout
- cache restore npm
- cache restore node-modules-$(checksum package-lock.json)
jobs:
# Check that all links are valid
- name: Test link validity
commands:
- npm run linkcheck-uat

Production Infrastructure deployment configuration at .semaphore/prod-infra.yml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
version: v1.0
name: AlphaGeek Production Infrastructure Deployment Pipeline
agent:
machine:
type: e1-standard-2
os_image: ubuntu1804
blocks:
# Use serverless to deploy to Production
- name: Publish to Production
task:
# Import the secret environment variables
secrets:
- name: alphageek-aws
prologue:
commands:
- checkout
- cache restore npm
- cache restore node-modules-$(checksum package-lock.json)
jobs:
- name: Deploy Infrastructure
commands:
- npm run deploy-prod-infra
- cache store serverless-$SEMAPHORE_GIT_BRANCH .serverless
# Run tests on the UAT site
- name: Test Production
task:
prologue:
commands:
- checkout
- cache restore npm
- cache restore node-modules-$(checksum package-lock.json)
jobs:
# Check that all links are valid
- name: Test link validity
commands:
- npm run linkcheck-prod

Add UAT and Production Deployment Configuration to Semaphore

Now the deployment processes have been defined we need to add triggeres for them to the primary SemaphoreCI configuration file.

Add to the end of .semaphore/semaphore.yml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
promotions:
- name: Deploy Content to UAT
pipeline_file: uat-content.yml
auto_promote_on:
- result: passed
branch:
- ^develop$
- name: Deploy Infra to UAT
pipeline_file: uat-infra.yml
- name: Deploy Content to Production
pipeline_file: prod-content.yml
auto_promote_on:
- result: passed
branch:
- ^master$
- name: Deploy Infra to Production
pipeline_file: prod-infra.yml

Add New WAF Rules and Implement

Because the link checking tool we're using doesn't support a custom authentication header we need to enable another method to gain access. For this we will use a custom user-agent string. This is defined in config/resources.yml.

Add to config/resources.yml at line 156
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
CustomUserAgentHeader:
Type: AWS::WAF::ByteMatchSet
Properties:
ByteMatchTuples:
-
FieldToMatch:
Type: HEADER
Data: User-Agent
TargetString:
Fn::Join:
- " "
- - ${self:custom.security.passwords.development}
- "Tester"
TextTransformation: NONE
PositionalConstraint: EXACTLY
Name:
Fn::Join:
- "_"
- - ${self:custom.domain.domainname}
- UserAgent
- Header
CustomUserAgentHeaderRule:
Type: AWS::WAF::Rule
Properties:
Name:
Fn::Join:
- "_"
- - ${self:custom.domain.domainname}
- UserAgent
- Header
- Rule
MetricName:
Fn::Join:
- ""
- - ${self:custom.domain.domain}
- ${self:custom.stage}
- UserrAgent
- Header
- Rule
Predicates:
-
DataId:
Ref: CustomUserAgentHeader
Negated: false
Type: ByteMatch

Then add a reference to it in config/other.yml.

Add to the end of config/other.yml
1
2
3
4
5
6
-
Action:
Type: ALLOW
Priority: 2
RuleId:
Ref: CustomUserAgentHeaderRule

Deploy New Infrastructure

We need to manually deploy the new infrastructure before we can push the code to GitHub as some of the functionality in the CI/CD pipeline will fail with the current configuration.

Deploy the new configuration
1
npm run deploy-uat-infra

How to Publish New Content

Publishing new content is now as easy as committing the changes and pushing the develop branch to GitHub and waiting for the deployment to the development site to complete. Once you've completed any user acceptance testing (UAT) you can merge the develop branch into master and push that to GitHub. Once all testing has completed the content will automatically be published to your production site.

How to Deploy New Infrastructure and Lambda Functions

Deploying new infrastructure is similar to the process for new content, but once the tests and build have completed on SemaphoreCI you will need to press a button to deploy. Just follow these simple steps:

  • Login to SemaphoreCI
  • Locate the build you wish to deploy the infrastructure from
  • Open the build
  • Click the Promote button under the deployment you wish to run

The Final Configuration Files

[serverless.yml] []view raw
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
# The name of your project
service: <project>

# Plugins for additional Serverless functionality
plugins:
- serverless-s3-deploy
- serverless-plugin-scripts

# Configuration for AWS
provider:
name: aws
runtime: nodejs8.10
profile: serverless
# Some future functionality requires us to use us-east-1 at this time
region: us-east-1

# This enables us to use the default stage definition, but override it from the command line
stage: ${opt:stage, self:provider.stage}
# This enables us to prepend the stage name for non-production environments
domain:
fulldomain:
prod: ${self:custom.domain.domain}
other: ${self:custom.stage}.${self:custom.domain.domain}
# This value has been customised so I can maintain multiple demonstration sites
domain: ${self:custom.postname}.${self:custom.domain.zonename}
domainname: ${self:custom.domain.fulldomain.${self:custom.stage}, self:custom.domain.fulldomain.other}
# DNS Zone name (this is only required so I can maintain multiple demonstration sites)
zonename: alphageek.com.au
cacheControlMaxAgeHTMLByStage:
# HTML Cache time for production environment
prod: 3600
# HTML Cache time for other environments
other: 0
cacheControlMaxAgeHTML: ${self:custom.domain.cacheControlMaxAgeHTMLByStage.${self:custom.stage}, self:custom.domain.cacheControlMaxAgeHTMLByStage.other}
sslCertificateARN: arn:aws:acm:us-east-1:165657443288:certificate/61d202ea-12f2-4282-b602-9c3b83183c7a
assets:
targets:
# Configuration for HTML files (overriding the default cache control age)
- bucket:
Ref: WebsiteS3Bucket
files:
- source: ./public/
headers:
CacheControl: max-age=${self:custom.domain.cacheControlMaxAgeHTML}
empty: true
globs:
- '**/*.html'
# Configuration for all assets
- bucket:
Ref: WebsiteS3Bucket
files:
- source: ./public/
empty: true
globs:
- '**/*.js'
- '**/*.css'
- '**/*.jpg'
- '**/*.png'
- '**/*.gif'
# AWS Region to S3 website hostname mapping
s3DNSName:
us-east-2: s3-website.us-east-2.amazonaws.com
us-east-1: s3-website-us-east-1.amazonaws.com
us-west-1: s3-website-us-west-1.amazonaws.com
us-west-2: s3-website-us-west-2.amazonaws.com
ap-south-1: s3-website.ap-south-1.amazonaws.com
ap-northeast-3: s3-website.ap-northeast-3.amazonaws.com
ap-northeast-2: s3-website.ap-northeast-2.amazonaws.com
ap-southeast-1: s3-website-ap-southeast-1.amazonaws.com
ap-southeast-2: s3-website-ap-southeast-2.amazonaws.com
ap-northeast-1: s3-website-ap-northeast-1.amazonaws.com
ca-central-1: s3-website.ca-central-1.amazonaws.com
eu-central-1: s3-website.eu-central-1.amazonaws.com
eu-west-1: s3-website-eu-west-1.amazonaws.com
eu-west-2: s3-website.eu-west-2.amazonaws.com
eu-west-3: s3-website.eu-west-3.amazonaws.com
eu-north-1: s3-website.eu-north-1.amazonaws.com
sa-east-1: s3-website-sa-east-1.amazonaws.com
# Determine what resources file to include based on the current stage
customConfigFile: ${self:custom.customConfigFiles.${self:custom.stage}, self:custom.customConfigFiles.other}
customConfigFiles:
prod: prod
other: other

# Define the resources we will need to host the site
resources:
# Include the resources file
- ${file(config/resources.yml)}
# Include the outputs file
- ${file(config/outputs.yml)}
# Include a custom configuration file based on the environment
- ${file(config/resources/environment/${self:custom.customConfigFile}.yml)}

[package.json] []view raw
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
{
"name": "hexo-site",
"version": "0.0.0",
"private": true,
"hexo": {
"version": "3.8.0"
},
"devDependencies": {
"@silvermine/serverless-plugin-cloudfront-lambda-edge": "^2.1.1",
"broken-link-checker-local": "^0.2.0",
"hexo": "^3.7.0",
"hexo-generator-archive": "^0.1.5",
"hexo-generator-category": "^0.1.3",
"hexo-generator-index": "^0.2.1",
"hexo-generator-tag": "^0.2.0",
"hexo-renderer-ejs": "^0.3.1",
"hexo-renderer-marked": "^0.3.2",
"hexo-renderer-stylus": "^0.3.3",
"hexo-server": "^0.3.1",
"jest": "^24.0.0",
"serverless-lambda-version": "^0.1.2",
"serverless-plugin-scripts": "^1.0.2",
"serverless-s3-deploy": "^0.8.0"
},
"scripts": {
"jest": "jest",
"mdlint-drafts": "markdownlint --config .mdconfig ./source/_drafts",
"mdlint": "markdownlint --config .mdconfig ./source/_posts",
"precheck": "npm run jest && npm run mdlint",
"clean": "hexo clean",
"build": "hexo generate",
"cleanbuild": "npm run clean && npm run build",
"linkcheck": "blcl --filter-level 3 --get --recursive --exclude /atom.xml --exclude /favicon.png --exclude http://2019-01-28-securing-s3.demo.alphageek.com.au.s3-website-us-east-1.amazonaws.com --exclude http://localhost:4000 --exclude http://dev./%3Cyour_domain%3E/ public",
"htmlcheck": "node-w3c-validator -v -s -i public/",
"a11ycheck": "pa11y-ci public/*.html public/*/*.html public/*/*/*.html public/*/*/*/*.html public/*/*/*/*/*.html || true",
"validate": "npm run linkcheck && npm run htmlcheck && npm run a11ycheck",
"buildtest-local": "npm run precheck && npm run cleanbuild && npm run validate",
"precommit": "npm run buildtest-local",
"deploy-uat-infra": "npx serverless deploy -s dev",
"deploy-uat-site": "npx serverless s3deploy -s dev -v",
"deploy-uat": "npm run deploy-uat-infra && npm run deploy-uat-site",
"linkcheck-uat": "npx blc --filter-level 3 --get --recursive --exclude /atom.xml --exclude /favicon.png --exclude http://2019-01-28-securing-s3.demo.alphageek.com.au.s3-website-us-east-1.amazonaws.com --exclude http://localhost:4000 --exclude http://dev./%3Cyour_domain%3E/ --user-agent '<PASSWORD_DEFINED_IN_SERVERLESS_CONFIGURATION> Tester' http://dev.alphageek.com.au",
"test-uat": "npm run deploy-uat && npm run linkcheck-uat",
"buildtest-uat": "npm run buildtest-local && test-uat",
"deploy-prod-infra": "npx serverless deploy -s prod",
"deploy-prod-site": "npx serverless s3deploy -s prod -v",
"deploy-prod": "npm run deploy-prod-infra && npm run deploy-prod-site",
"linkcheck-prod": "npx blc --filter-level 3 --get --recursive --exclude /atom.xml --exclude /favicon.png --exclude http://2019-01-28-securing-s3.demo.alphageek.com.au.s3-website-us-east-1.amazonaws.com --exclude http://localhost:4000 --exclude http://dev./%3Cyour_domain%3E/ http://alphageek.com.au",
"test-prod": "npm run deploy-prod && npm run linkcheck-prod",
"buildtest-prod": "npm run buildtest-uat && test-prod"
},
"jest": {
"testPathIgnorePatterns": [
"/node_modules/",
"/source/code/",
"/public/"
]
}
}

[.mdconfig] []view raw
1
2
3
4
5
6
7
8
9
{
"MD002": {
"level": 2
},
"MD013": false,
"MD026": {
"punctuation": ".,;:!"
}
}

[.semaphore/semaphore.yml] []view raw
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
version: v1.0
name: Hexo Serverless Build Pipeline
agent:
machine:
type: e1-standard-2
os_image: ubuntu1804
blocks:
# Prepare the build environment
- name: Prepare
task:
jobs:
# Make sure we've got the right Java version configured
- name: Set Java Version
commands:
- change-java-version 8
# Run NPM install, using Semaphore's cache where possible
- name: NPM Install
commands:
# Update NPM because it's so old
- cache restore npm
- npm i -g npm
- cache store npm .nvm/versions/node/v8.11.3/lib/node_modules/npm
- checkout
# Reuse dependencies from cache and avoid installing them from scratch:
- cache restore node-modules-$(checksum package-lock.json)
- npm ci
- cache store node-modules-$(checksum package-lock.json) node_modules
# Run the validation routines that don't require a build
- name: Validate
task:
prologue:
commands:
- checkout
- cache restore npm
- cache restore node-modules-$(checksum package-lock.json)
jobs:
# Run the jest test suite
- name: Run Jest Tests
commands:
- npm run jest
# Run the Markdown linter
- name: MD Lint
commands:
- npm run mdlint
# Build the deployment files
- name: Build
task:
prologue:
commands:
- checkout
- cache restore npm
- cache restore node-modules-$(checksum package-lock.json)
jobs:
# Ensure we have a clean build directory, generate the files and add asset versioning
- name: Build Site
commands:
- npm run clean
- npm run build
- cache store public-$(find source -type f -exec cat {} + | checksum) public
# Run tests on the deployment files
- name: Test Locally
task:
prologue:
commands:
- checkout
- cache restore npm
- cache restore node-modules-$(checksum package-lock.json)
- cache restore public-$(find source -type f -exec cat {} + | checksum)
jobs:
# Check that all links are valid
- name: Test link validity
commands:
- npm run linkcheck
# Check that the HTML is valid
- name: Test W3C compatability
commands:
- npm run htmlcheck
# Check if we meet a11y standards
- name: Test Accessibility
commands:
- npm run a11ycheck
promotions:
- name: Deploy Content to UAT
pipeline_file: uat-content.yml
auto_promote_on:
- result: passed
branch:
- develop
- name: Deploy Infra to UAT
pipeline_file: uat-infra.yml
- name: Deploy Content to Production
pipeline_file: prod-content.yml
auto_promote_on:
- result: passed
branch:
- master
- name: Deploy Infra to Production
pipeline_file: prod-infra.yml

[.semaphore/prod-content.yml] []view raw
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
version: v1.0
name: AlphaGeek Production Content Deployment Pipeline
agent:
machine:
type: e1-standard-2
os_image: ubuntu1804
blocks:
# Use Serverless to deploy to Production
- name: Publish to Prod
task:
# Import the secret environment variables
secrets:
- name: alphageek-aws
prologue:
commands:
- checkout
- cache restore npm
- cache restore node-modules-$(checksum package-lock.json)
- cache restore public-$(find source -type f -exec cat {} + | checksum)
jobs:
- name: Deploy Content
commands:
- npm run deploy-prod-site
# Run tests on the Production site
- name: Test Production
task:
prologue:
commands:
- checkout
- cache restore npm
- cache restore node-modules-$(checksum package-lock.json)
jobs:
# Check that all links are valid
- name: Test link validity
commands:
- npm run linkcheck-prod

[.semaphore/prod-infra.yml] []view raw
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
version: v1.0
name: AlphaGeek Production Infrastructure Deployment Pipeline
agent:
machine:
type: e1-standard-2
os_image: ubuntu1804
blocks:
# Use serverless to deploy to Production
- name: Publish to Production
task:
# Import the secret environment variables
secrets:
- name: alphageek-aws
prologue:
commands:
- checkout
- cache restore npm
- cache restore node-modules-$(checksum package-lock.json)
jobs:
- name: Deploy Infrastructure
commands:
- npm run deploy-prod-infra
- cache store serverless-$SEMAPHORE_GIT_BRANCH .serverless
# Run tests on the UAT site
- name: Test Production
task:
prologue:
commands:
- checkout
- cache restore npm
- cache restore node-modules-$(checksum package-lock.json)
jobs:
# Check that all links are valid
- name: Test link validity
commands:
- npm run linkcheck-prod

[.semaphore/uat-content.yml] []view raw
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
version: v1.0
name: AlphaGeek UAT Content Deployment Pipeline
agent:
machine:
type: e1-standard-2
os_image: ubuntu1804
blocks:
# Use Serverless to deploy to UAT
- name: Publish to UAT
task:
# Import the secret environment variables
secrets:
- name: alphageek-aws
prologue:
commands:
- checkout
- cache restore npm
- cache restore node-modules-$(checksum package-lock.json)
- cache restore public-$(find source -type f -exec cat {} + | checksum)
jobs:
- name: Deploy Content
commands:
- npm run deploy-uat-site
# Run tests on the UAT site
- name: Test UAT
task:
prologue:
commands:
- checkout
- cache restore npm
- cache restore node-modules-$(checksum package-lock.json)
jobs:
# Check that all links are valid
- name: Test link validity
commands:
- npm run linkcheck-uat

[.semaphore/uat-infra.yml] []view raw
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
version: v1.0
name: AlphaGeek UAT Infrastructure Deployment Pipeline
agent:
machine:
type: e1-standard-2
os_image: ubuntu1804
blocks:
# Use serverless to deploy to UAT
- name: Publish to UAT
task:
# Import the secret environment variables
secrets:
- name: alphageek-aws
prologue:
commands:
- checkout
- cache restore npm
- cache restore node-modules-$(checksum package-lock.json)
jobs:
- name: Deploy Infrastructure
commands:
- npm run deploy-uat-infra
- cache store serverless-$SEMAPHORE_GIT_BRANCH .serverless
# Run tests on the UAT site
- name: Test UAT
task:
prologue:
commands:
- checkout
- cache restore npm
- cache restore node-modules-$(checksum package-lock.json)
jobs:
# Check that all links are valid
- name: Test link validity
commands:
- npm run linkcheck-uat

[config/resources.yml] []view raw
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
Resources:
# Set-up an S3 bucket to store the site
WebsiteS3Bucket:
Type: AWS::S3::Bucket
Properties:
AccessControl: BucketOwnerFullControl
BucketName: ${self:custom.domain.domainname}
PublicAccessBlockConfiguration:
BlockPublicAcls: true
BlockPublicPolicy: true
IgnorePublicAcls: true
RestrictPublicBuckets: true
# Set-up a policy on the bucket so it can be used as a website
WebsiteBucketPolicy:
Type: AWS::S3::BucketPolicy
Properties:
PolicyDocument:
Id:
Fn::Join:
- ""
- - ${self:service.name}
- BucketPolicy
Statement:
- Sid: CloudFrontForGetBucketObjects
Effect: Allow
Principal:
CanonicalUser:
Fn::GetAtt:
- CloudFrontIdentity
- S3CanonicalUserId
Action: 's3:GetObject'
Resource:
Fn::Join:
- ''
-
- 'arn:aws:s3:::'
- Ref: WebsiteS3Bucket
- /*
Bucket:
Ref: WebsiteS3Bucket
# Configure CloudFront to get all content from S3
WebsiteCloudFrontDistribution:
Type: 'AWS::CloudFront::Distribution'
Properties:
DistributionConfig:
WebACLId:
Ref: CustomAuthorizationHeaderRestriction
Aliases:
- ${self:custom.domain.domainname}
- www.${self:custom.domain.domainname}
CustomErrorResponses:
- ErrorCode: '404'
ResponsePagePath: "/error.html"
ResponseCode: '200'
ErrorCachingMinTTL: '30'
DefaultCacheBehavior:
Compress: true
ForwardedValues:
QueryString: false
Cookies:
Forward: all
SmoothStreaming: false
TargetOriginId: defaultOrigin
ViewerProtocolPolicy: redirect-to-https
DefaultRootObject: index.html
Enabled: true
Origins:
- DomainName:
Fn::GetAtt:
- WebsiteS3Bucket
- DomainName
Id: defaultOrigin
S3OriginConfig:
OriginAccessIdentity:
Fn::Join:
- "/"
- - origin-access-identity
- cloudfront
- Ref: CloudFrontIdentity
PriceClass: PriceClass_All
ViewerCertificate:
AcmCertificateArn: ${self:custom.domain.sslCertificateARN}
SslSupportMethod: sni-only
# DNS Record for the domain
WebsiteDNSRecord:
Type: "AWS::Route53::RecordSet"
Properties:
AliasTarget:
DNSName:
Fn::GetAtt:
- WebsiteCloudFrontDistribution
- DomainName
HostedZoneId: Z2FDTNDATAQYW2
HostedZoneName: ${self:custom.domain.domain}.
Name: ${self:custom.domain.domainname}
Type: 'A'
# DNS Record for www.domain
WebsiteWWWDNSRecord:
Type: "AWS::Route53::RecordSet"
Properties:
AliasTarget:
DNSName:
Fn::GetAtt:
- WebsiteCloudFrontDistribution
- DomainName
HostedZoneId: Z2FDTNDATAQYW2
HostedZoneName: ${self:custom.domain.domain}.
Name: www.${self:custom.domain.domainname}
Type: 'A'
# Predicate to match the authorization header
CustomAuthorizationHeader:
Type: AWS::WAF::ByteMatchSet
Properties:
ByteMatchTuples:
-
FieldToMatch:
Type: HEADER
Data: Authorization
TargetString:
Fn::Join:
- " "
- - Custom
- "<Password>"
TextTransformation: NONE
PositionalConstraint: EXACTLY
Name:
Fn::Join:
- "_"
- - ${self:custom.domain.domainname}
- Authorization
- Header
CustomAuthorizationHeaderRule:
Type: AWS::WAF::Rule
Properties:
Name:
Fn::Join:
- "_"
- - ${self:custom.domain.domainname}
- Authorization
- Header
- Rule
MetricName:
Fn::Join:
- ""
- - ${self:custom.stage}
- ${self:service.name}
- Authorization
- Header
- Rule
Predicates:
-
DataId:
Ref: CustomAuthorizationHeader
Negated: false
Type: ByteMatch
CustomUserAgentHeader:
Type: AWS::WAF::ByteMatchSet
Properties:
ByteMatchTuples:
-
FieldToMatch:
Type: HEADER
Data: User-Agent
TargetString:
Fn::Join:
- " "
- - ${self:custom.security.passwords.development}
- "Tester"
TextTransformation: NONE
PositionalConstraint: EXACTLY
Name:
Fn::Join:
- "_"
- - ${self:custom.domain.domainname}
- UserAgent
- Header
CustomUserAgentHeaderRule:
Type: AWS::WAF::Rule
Properties:
Name:
Fn::Join:
- "_"
- - ${self:custom.domain.domainname}
- UserAgent
- Header
- Rule
MetricName:
Fn::Join:
- ""
- - ${self:custom.domain.domain}
- ${self:custom.stage}
- UserrAgent
- Header
- Rule
Predicates:
-
DataId:
Ref: CustomUserAgentHeader
Negated: false
Type: ByteMatch

[config/other.yml] []view raw
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
Resources:
# Require the custom authorisation header with the correct password in non-production environment
CustomAuthorizationHeaderRestriction:
Type: AWS::WAF::WebACL
Properties:
DefaultAction:
Type: BLOCK
Name:
Fn::Join:
- "_"
- - ${self:custom.domain.domainname}
- Authorization
- Header
- Restriction
MetricName:
Fn::Join:
- ""
- - ${self:custom.stage}
- ${self:service.name}
- Authorization
- Header
- Restriction
Rules:
-
Action: ALLOW
Priority: 1
RuleId:
Ref: CustomAuthorizationHeaderRule
-
Action:
Type: ALLOW
Priority: 2
RuleId:
Ref: CustomUserAgentHeaderRule

[test/functions/urlRewrite.test.js.js] []view raw
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
// Load the file to test
const urlRewriteTest = require('../../functions/urlRewrite');
// Load some data that can be reused for other lambda@Edge functions
const lambdaAtEdgeFixture = require('../fixtures/lambdaAtEdge');

test(
'url-rewrite handler appends index.html to root request',
() => {
const expectedResponse = { uri: lambdaAtEdgeFixture.event.root_object.Records[0].cf.request.uri + 'index.html' };
expect(
urlRewriteTest.handler(
lambdaAtEdgeFixture.event.root_object,
lambdaAtEdgeFixture.context.webClient,
lambdaAtEdgeFixture.callback
)
).toEqual(expect.objectContaining(expectedResponse));
}
);

test(
'url-rewrite handler appends index.html to directory requests',
() => {
const expectedResponse = { uri: lambdaAtEdgeFixture.event.subdirectory.Records[0].cf.request.uri + 'index.html' };
expect(
urlRewriteTest.handler(
lambdaAtEdgeFixture.event.subdirectory,
lambdaAtEdgeFixture.context.webClient,
lambdaAtEdgeFixture.callback
)
).toEqual(expect.objectContaining(expectedResponse));
}
);

test(
'url-rewrite handler does not append index.html to root-directory requests',
() => {
const expectedResponse = { uri: lambdaAtEdgeFixture.event.file.Records[0].cf.request.uri };
expect(
urlRewriteTest.handler(
lambdaAtEdgeFixture.event.file,
lambdaAtEdgeFixture.context.webClient,
lambdaAtEdgeFixture.callback
)
).toEqual(expect.objectContaining(expectedResponse));
}
);

test(
'url-rewrite handler does not append index.html to non-directory requests',
() => {
const expectedResponse = { uri: lambdaAtEdgeFixture.event.file_in_subdirectory.Records[0].cf.request.uri };
expect(
urlRewriteTest.handler(
lambdaAtEdgeFixture.event.file_in_subdirectory,
lambdaAtEdgeFixture.context.webClient,
lambdaAtEdgeFixture.callback
)
).toEqual(expect.objectContaining(expectedResponse));
}
);

[config/outputs.yml] []view raw
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
Outputs:
WebsiteURL:
Value:
Fn::GetAtt:
- WebsiteS3Bucket
- WebsiteURL
Description: URL for my website hosted on S3
S3BucketSecureURL:
Value:
Fn::Join:
- ''
-
- 'https://'
- Fn::GetAtt:
- WebsiteS3Bucket
- DomainName
Description: Secure URL of S3 bucket to hold website content

[config/prod.yml] []view raw
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
Resources:
# Allow the custom authorisation header in the production environment
CustomAuthorizationHeaderRestriction:
Type: AWS::WAF::WebACL
Properties:
DefaultAction:
Type: ALLOW
Name:
Fn::Join:
- "_"
- - ${self:custom.domain.domainname}
- Authorization
- Header
- Restriction
MetricName:
Fn::Join:
- ""
- - ${self:custom.stage}
- ${self:service.name}
- Authorization
- Header
- Restriction
Rules:
-
Action:
Type: ALLOW
Priority: 1
RuleId:
Ref: CustomAuthorizationHeaderRule

Example Site