Faster CloudFormation Delivery

with OpenJS Architect

Brian LeRoux

CTO, Begin.com


  • Super webby open source hacker
  • Member of Apache and OpenJS foundations

Key discoveries

  • Infra as Code (IaC) is a crucial best practice

    As important as testing and revision control

  • CloudFormation is everywhere and awesome

    The de facto standard for IaC on AWS

  • OpenJS Architect is even faster

    One of many excellent ways to generate CFN

This is important because

it makes you faster.

For six years in a row, however, our research has consistently shown that speed and stability are outcomes that enable each other.

2019 Accelerate State of DevOps aka "the DORA report".

Speed + stability = 📈

Not an opinion; this is backed by science!

  • Deployment frequency

    How often the org deploys to production

  • Lead time to production

    Time from commit to running in production

  • Change fail rate + time to restore service

    Converse impact of the above processes

Major 🔑 to speed is determinism

Versions belong in version control: including infra!

Faster lead time to production

More iterations == more chances to get it right.

Speed is crucial for startups

to find product/market fit.

Time is money.

Speed is crucial for bigcos

to remain competetive.

Time is money here, too.

Infrastructure as Code

Software delivery is hard

Proven practices make it easier


  • staged delivery local → staging → production
  • static analysis types, linting, compliance scans, etc
  • testing manual, unit, integration, etc
  • continuous integration trunk based dev
  • continuous delivery gitops → devops

Configuration management

YAML... embedded with Bash

Capistrano, Salt, Ansible, Chef, Puppet, etc, etc

Favor convention over configuration.

How we name stuff and where we put it.

A big speed benefit of frameworks are pre-defined conventions.

Use declarative formats for setup automation, to minimize time and cost...

The 12 Factor App

Infra as Code (IaC)

Treats cloud resources as dependencies, stored in source, and thusly versioned.

This is important because

determinism makes you faster.

Speed & stability


  • staged delivery local → staging → production
  • static analysis types, linting, compliance scans, etc
  • testing manual, unit, integration, etc
  • continuous integration trunk based dev
  • continuous delivery gitops → devops
  • infra as code

A few ways to IaC on AWS


  • .arc
  • sam.json
  • serverless.yml
  • terraform.tf

Shoutout: Stackery.io
Format License Governance/Copyright Deployment method
.arc Apache 2 OpenJS Foundation CloudFormation
sam.json Apache 2 Amazon Web Services, Inc CloudFormation
serverless.yml MIT Serverless, Inc CloudFormation
terraform.tf MPL 2 HashiCorp, Inc SDK
Learn more at opengovernance.dev
  • .arcsam.jsoncloudformation.yaml

SDK Deployment

We migrated to CFN, but the SDK has advantages:

  • Very fast
  • Fully in control of the lifecycle
  • No config files

What's the downside of deployment via SDK?

429 TooManyRequestsException

SDK deploys: fast, but brittle

Brittle feels broken & slow

Owning the SDK lifecycle

means implementing

backoff, retry and failure


Multiply this by every service you use, for every AWS account you have. This takes a lot of code.

Certainly, anyone can implement better backoff, retry and failure than the AWS CloudFormation team. 🙄


Is that a good use of your time?

For us, SDK → CloudFormation was a huge win

This is a 6x improvement. And we added functionality! YMMV.

AWS SDK: still ok!


CloudFormation has gaps in functionality
and even occasional bugs you need to patch.

Sometimes you need fast and dirty

which is totally ok for staging. 🤠

CloudFormation rising

  • Deploys are now (mostly) fast
  • Sevice coverage for most serverless things you want is A+
  • New services coming on board faster than before
  • Standard 'package' format: SAR is the distribution channel

CloudFormation

Things we need AWS to improve


  • Performance: can always be better
  • package/deploy should be SDK not CLI scripts
  • Deploy bucket should be a hidden impl detail (versions are in version control!)
  • Can automate disaster! Better retention policies
  • Incoherent error messages (debugging CFN is not fun)
  • Code completion and tooling

CloudFormation Rough Edges


AWS will only grow faster and stronger but the flip side of that is that it will not grow simpler or less complex.


  • Too verbose: tradeoff of low level configurability
  • Too complex: not going have less capabilities ever

XML, JSON, YAML, and TOML


  • 👎🏾 XML: verbose and hard to read
  • 👎🏾 JSON: all the above and no comments!?
  • 👎🏾 TOML: the most syntax!
  • 🤮 YAML: nesting nightmares & interop is fiction

Terraform clearly agreed!

Essential complexity


Selfish requirements for a new config file format. We wanted something we could READ as well as write.


  • must support comments
  • must support scalar types: string, number, boolean
  • must support complex types: array and map
  • must discourage nesting *
  • must discourage syntax *

* Constraints are good!
.arc
# demo scalar types    
@pragma1
foo
1
false

# compound types
@pragma2
1 2 3
obj
  key value
  k true
JSON result
{
  pragma1: [ "foo", 1, false ],            
  pragma2: [
    [ 1, 2, 3 ],
    { obj: {key: "value", k: true }
  ]
}

✓ must support comments
✓ must support scalar types: string, number, boolean
✓ must support compound types: array and map
✓ must discourage nesting
✓ must discourage syntax

Primitives not frameworks.

One of the most important mechanisms we provided was to offer customers a collection of primitives and tools, where they could pick and choose their preferred way to engage with the AWS cloud…

Werner Vogels, banging techno enthusiast (and CTO of Amazon)

AWS does not have to be complex or scary

Curate a subset of AWS primitives for building serverless apps

Architect Primitives

  • @cdnCloudFront
  • @staticS3
  • @htmlAPI Gateway
  • @eventsSNS
  • @queuesSQS
  • @scheduledCloudWatch Events
  • @tablesDynamoDB
  • @macrosCloudFormation

8 is simpler (and faster) than 300+ services. Also lucky! 😻
# hello world 
@app
myapp

@http
get /
See generated CFN
# timer 
@app
myapp

@scheduled
daily-mantra rate(1 day)
See generated CFN
# CRUDy RESTy
@app
myapp

@http
get /
get /api/cats           # list cats
get /api/cats/:catID    # get a cat
post /api/cats          # create cat
patch /api/cats/:catID  # update cat
delete /api/cats/:catID # y tho
See generated CFN
# graphql app    
@app
myapp

@cdn
@static
@http
post /graphql

@tables
data
  scopeID *String
  dataID **String
See generated CFN

Big code savings, big reduction of config. Less dependencies. Less liability. More speed.

Key discoveries

  • Infra as Code (IaC) is a crucial best practice

    As important as testing and revision control

  • CloudFormation is everywhere and awesome

    The de facto standard for IaC on AWS

  • OpenJS Architect is even faster

    One of many excellent ways to generate CFN

Sign up for Begin.com and star us on Github! github.com/architect/architect