Faster CloudFormation Delivery
with OpenJS Architect
This is important because
it makes you faster.
For six years in a row, however, our research has consistently shown that speed and stability are outcomes that enable each other.
2019 Accelerate State of DevOps aka "the DORA report".
Speed + stability = 📈
Not an opinion; this is backed by science!
-
Deployment frequency
How often the org deploys to production
-
Lead time to production
Time from commit to running in production
-
Change fail rate + time to restore service
Converse impact of the above processes
Major 🔑 to speed is determinism
Versions belong in version control: including infra!
Faster lead time to production
More iterations == more chances to get it right.
Speed is crucial for startups
to find product/market fit.
Time is money.
Speed is crucial for bigcos
to remain competetive.
Time is money here, too.
Infrastructure as Code
Software delivery is hard
Proven practices make it easier
- staged delivery local → staging → production
- static analysis types, linting, compliance scans, etc
- testing manual, unit, integration, etc
- continuous integration trunk based dev
- continuous delivery gitops → devops
Configuration management
YAML... embedded with Bash
Capistrano, Salt, Ansible, Chef, Puppet, etc, etcFavor convention over configuration.
How we name stuff and where we put it.
A big speed benefit of frameworks are pre-defined conventions.
Use declarative formats for setup automation, to minimize time and cost...
Infra as Code (IaC)
Treats cloud resources as dependencies, stored in source, and thusly versioned.
This is important because
determinism makes you faster.
Speed & stability
- staged delivery local → staging → production
- static analysis types, linting, compliance scans, etc
- testing manual, unit, integration, etc
- continuous integration trunk based dev
- continuous delivery gitops → devops
- infra as code
| Format | License | Governance/Copyright | Deployment method |
|---|---|---|---|
.arc |
Apache 2 | OpenJS Foundation | CloudFormation |
sam.json |
Apache 2 | Amazon Web Services, Inc | CloudFormation |
serverless.yml |
MIT | Serverless, Inc | CloudFormation |
terraform.tf |
MPL 2 | HashiCorp, Inc | SDK |
.arc→sam.json→cloudformation.yaml
SDK Deployment
We migrated to CFN, but the SDK has advantages:
- Very fast
- Fully in control of the lifecycle
- No config files
What's the downside of deployment via SDK?
429 TooManyRequestsException
SDK deploys: fast, but brittle
Brittle feels broken & slow
Owning the SDK lifecycle
means implementing
backoff, retry and failure
Multiply this by every service you use, for every AWS account you have. This takes a lot of code.
Certainly, anyone can implement better backoff, retry and failure than the AWS CloudFormation team. 🙄
Is that a good use of your time?
For us, SDK → CloudFormation was a huge win
This is a 6x improvement. And we added functionality! YMMV.
AWS SDK: still ok!
CloudFormation has gaps in functionality
and even occasional bugs you need to patch.
Sometimes you need fast and dirty
which is totally ok for staging. 🤠
CloudFormation rising
- Deploys are now (mostly) fast
- Sevice coverage for most serverless things you want is A+
- New services coming on board faster than before
- Standard 'package' format: SAR is the distribution channel
CloudFormation
Things we need AWS to improve
- Performance: can always be better
package/deployshould be SDK not CLI scripts- Deploy bucket should be a hidden impl detail (versions are in version control!)
- Can automate disaster! Better retention policies
- Incoherent error messages (debugging CFN is not fun)
- Code completion and tooling
CloudFormation Rough Edges
AWS will only grow faster and stronger but the flip side of that is that it will not grow simpler or less complex.
- Too verbose: tradeoff of low level configurability
- Too complex: not going have less capabilities ever
XML, JSON, YAML, and TOML
- 👎🏾 XML: verbose and hard to read
- 👎🏾 JSON: all the above and no comments!?
- 👎🏾 TOML: the most syntax!
- 🤮 YAML: nesting nightmares & interop is fiction
Terraform clearly agreed!
Essential complexity
Selfish requirements for a new config file format. We wanted something we could READ as well as write.
- must support comments
- must support scalar types:
string,number,boolean - must support complex types:
arrayandmap - must discourage nesting *
- must discourage syntax *
* Constraints are good!
.arc
# demo scalar types @pragma1 foo 1 false # compound types @pragma2 1 2 3 obj key value k true
JSON result
{
pragma1: [ "foo", 1, false ],
pragma2: [
[ 1, 2, 3 ],
{ obj: {key: "value", k: true }
]
}
✓ must support comments
✓ must support scalar types: string, number, boolean
✓ must support compound types: array and map
✓ must discourage nesting
✓ must discourage syntax
Primitives not frameworks.
One of the most important mechanisms we provided was to offer customers a collection of primitives and tools, where they could pick and choose their preferred way to engage with the AWS cloud…
Werner Vogels, banging techno enthusiast (and CTO of Amazon)
AWS does not have to be complex or scary
Curate a subset of AWS primitives for building serverless apps
Architect Primitives
@cdn→ CloudFront@static→ S3@html→ API Gateway@events→ SNS@queues→ SQS@scheduled→ CloudWatch Events@tables→ DynamoDB@macros→ CloudFormation
8 is simpler (and faster) than 300+ services. Also lucky! 😻
# hello world @app myapp @http get /See generated CFN
# timer @app myapp @scheduled daily-mantra rate(1 day)See generated CFN
# CRUDy RESTy @app myapp @http get / get /api/cats # list cats get /api/cats/:catID # get a cat post /api/cats # create cat patch /api/cats/:catID # update cat delete /api/cats/:catID # y thoSee generated CFN
# graphql app @app myapp @cdn @static @http post /graphql @tables data scopeID *String dataID **StringSee generated CFN