Off-by-none: Issue #35

April 30, 2019

I want my, I want my, I want my SQL… 🎸

Welcome to Issue #35 of Off-by-none. It’s great to have you here! 👋

Last week we (re)addressed the serverless versus containers debate and looked at some interesting thought pieces on serverless. This week, we’ll try to wean ourselves off of our RDBMS addiction, explore some interesting serverless use cases, and share an overwhelming amount of content from our amazing community.

There was a ridiculous amount of serverless goodness this past week, so buckle up. There is a lot to get to. 😉

When you just need to say ‘No’ to SQL… 🙅‍♂️

I really love RDBMS, but as we migrate applications to the cloud, or better yet, build them as modern applications, the scaling limitations of RDBMS start to become readily apparent. There’s no doubt that certain use cases require the flexibility of relational table structures, and for small workloads, an RDS cluster would be just fine to accomplish our immediate goals. But in the “serverless” world, thinking in third normal form not only introduces scaling issues, but also adds complexity that could increase costs and unnecessary overhead.

This past week, I started working on a new application design that has a relatively complex data model. Like most developers, I often turn to what is familiar, and easier for me to implement. DynamoDB is an incredibly powerful NoSQL database, and when building a table that requires just a few simple access patterns, it is my go-to choice. But when I need to implement something with lots of relationships, I often (foolishly) fall back to the comforts of MySQL. I mean, I can 3NF the hell out of a data model, as most of us data nerds can. But this time I decided to take a stand, and see if I can go full NoSQL on this one.

I started by rewatching Rick Houlihan’s re:Invent 2018 Advanced Design Patterns for DynamoDB, and then tweeted about it. Apparently others agree that it is one of the best DynamoDB talks out there.

After that I spent some time with Forrest Brazeal’s excellent From relational DB to single DynamoDB table: a step-by-step exploration piece, looked at a few examples from Alex DeBrie’s amazing DynamoDB Guide, and then dug in to AWS’s Best Practices for DynamoDB. Where am I with my table design? Maybe about halfway there. But I’m confident that once it clicks, designing NoSQL tables with overloaded indexes, adjacency lists, and hierarchical sort keys, will become second nature. I’ve got more to do, and I know I’ll make some mistakes along the way, but it will be worth it.

I totally get that it’s hard to let go of our old ways of thinking and to leave our comfort zones. It was this mindset that led me to build serverless-mysql to try and jam the metaphorical square peg of RDBMS into the round hole of serverless. I even gave a talk recently on Building Resilient Serverless Systems with Non-Serverless Components that focused heavily on ways to keep using RDBMS. But now I feel like I have to do better. Not just for me, but for the serverless community. I have to start thinking more “serverlessly” when it comes to designing data models, and I need to make NoSQL my default choice, regardless of how complex the requirements.

I encourage you to join me, but I promise not to judge you if you’re still not ready. It’s taken me a long time to get here myself, and I totally understand the “I’ll give up my relational database when you pry it from my cold, dead hands” mentality. Serverless is a big enough paradigm shift in and of itself, so changing the way we think about data is a big ask. But I’m making the leap now, so I’ll be here to help when you’re ready. I know we can do this together.

Serverless News & Product Announcements 📢

Serverless Framework v1.41 – X-Ray for API Gateway, Invoke Local with Docker Improvements & More
You know I have a soft spot for the Serverless Framework, but this release is big! The new local executions with Docker improvements are amazing, especially the environment variable support and Layers integration.

A Cloud Guru raises $33 million for cloud platform classes and labs
If you were hoping that A Cloud Guru would produce some more serverless content, then I think an extra $33 million in the bank might just make your wish come true. Congrats to them. 💰

New course on AWS Step Functions from Yan Cui
Yan Cui has just released a new course that will make you an AWS Step Functions expert. And if you sign up for Thundra, you can get free access to Yan’s Production-Ready Serverless course.

Epsagon has also been busy. They released a new documentation site, introduced the new Timeline View, and added Support for Express applications.

When We Say “Full Observability”, We Really Mean “Full Tracing”
Thundra just launched their new “Full Tracing” feature. Serkan Özal explains what that means for monitoring and observability in your serverless applications.

Serverless automation using PowerShell preview in Azure Functions
The PowerShell community can rejoice.

Serverless Use Cases 🗺

How we keep running by running a slackbot.
The team at Nordcloud Engineering took their running challenge to the extreme and created their own serverless slackbot.

From AWS Lambda to AWS Athena
Dhaval Nagar shows you a simple use case for querying your data from S3 with Athena and AWS Lambda.

Serverless: Can It Simplify Data Science Projects?
There has been some debate (and criticism) over using serverless for deep learning and data science projects. But according to Yaron Haviv, it’s possible with the right underlying architecture (i.e. Nuclio).

Dynamic blocklist with Twilio
Remember when Twilio introduced functions and I said there would be all kinds of use cases for it? Well, Cohan Robinson has a good one for you.

Scheduling automatic deletion of AWS CloudFormation stacks
AWS resources that automatically delete themselves. A very interesting use case for building up and tearing down demo stacks automagically (sorry, I hate that word too).

Build and automate a serverless data lake using an AWS Glue trigger for the Data Catalog and ETL jobs
Besides winning the award for most buzzwordy title, it’s also a really sophisticated way to build out a completely serverless data lake.

When you want to understand how all this serverless stuff actually works… 👩‍🏫

Introduction to Serverless Monitoring
If you want to discover different ways of achieving observability in stateless, distributed, and event-driven architectures, take a look at this white paper from Thundra.

Serverless Architecture – Why and How It’s a Smart Choice?
Not only a great primer (and case) for serverless, but also a good explanation as to why serverless is not PaaS.

The perfect tech stack
Serverless obviously 😃. But seriously, Levi Nunnink walks you through some thoughts on choosing the right technology and how developer familiarity plays a big role.

AWS Lambda: Solving the cold start problems while accessing DB in VPC.
Vishnu Subramanian mulls over the cold start problem, but offers a few suggestions to overcome it. Let’s hope the ENI solution is coming sooner rather than later.

How Secrets Manager Schedules Automatic Rotations
You’re totally rotating your secrets, right? 😬 Well, for those of you actually following best practices, Zac Charles gives us some interesting information about how auto-rotation actually works.

AWS Lambdas Can Have Shared State
Obie Fernandez walks you through the power of state machines and how AWS Step Functions can help you compose multiple functions.

Getting Started with AWS Lambda Layers
Still unsure about Lambda Layers? Matthew Vielkind’s post explains what they are, their benefits, how to create and use them, and provides some best practices and tips as well.

Introduction to Distributed Tracing
The benefits of modern, distributed cloud applications can make it harder to debug and fix software. Nitzan Shapira gives us an overview of distributed tracing, and how that helps solve this problem.

Serverless Stories 📖

Real Time Lambda Cost Analysis Using Honeycomb
Michael Garski from Fender Engineering shows how they used Honeycomb to analyze CloudWatch logs from Lambda to better understand their function utilization.

Where do you keep credentials for your Lambda functions?
Davide de Paolis walks us through his team’s journey to choose the best way to store secrets in their serverless applications.

A journey into serverless, part 2
Laurent Leconte and the Moonshot-Internet team layout all the tools they used to go serverless and why they chose them.

Scaling Skip: Serverless Architectures
Serverless made it possible to scale Skip with just a small team of engineers using the Google Cloud Platform.

Serverless Tutorials 🏗

Using AWS CloudFormation Macros and Custom Resources with the Serverless Framework
Well this is pretty cool. Who would have thought to use CloudFormation Macros in a serverless.yml file? Yi Ai, that’s who.

Deploying AWS Lambda functions using AWS CloudFormation (the portable way)
Lambda requires your S3 bucket to reside in the same AWS Region as the function, and creating a single template limits you to just one region. This excellent post discusses three patterns to address this problem.

Creating a lambda function with terraform
As I was reading this, I said to myself, “Why the heck would Gabriel Garrido not just use a serverless deployment framework instead?” He sets the record straight at the end.

Use NLog in .Net Core and AWS Lambda
Simple tutorial by Ofoedu Frank Ebuka that shows you how to setup and use NLog with .Net Core in Lambda.

Serverless AWS Lambda Dependency Injection & Simple Dependency Injection In AWS Lambda .net core
Gary Woodfine has some tips for you .NET Core fans out there.

Up and running with Aurora Serverless
Manish Pandit gives you an overview of Aurora Serverless, how to get it set up, and how to connect to it. If you want to dive into the pricing model, check out my article.

Serverless Reads 🤓

How to Serverless Locally: The Serverless Dev Workflow Challenge
If you’ve tried developing a serverless application locally, you’ve probably needed to resort to a number of tricks to test with cloudside resources. The team at Stackery just introduced a pretty cool way to do this.

What happens when you stop doing Serverless apps for a time?
Excellent post by Paul Johnston that addresses the fact that many (if not most) developers out there are still building things the “old way” (i.e. not using serverless or modern cloud app techniques). He also presents some great ideas at the end to help address the problem.

A Brief, Incomplete, and Mostly Wrong History of Serverless
Michael Hausenblas’ title sums up this piece nicely, but still worth the quick read.

Fargate Patterns
Srini Karlekar covers the basics of Fargate as well as lays out several patterns that can be used to utilize Fargate as part of your serverless applications. Very cool stuff.

How Far Out is AWS Fargate?
Speaking of Fargate, Michael Lavers from IOPipes argues that perhaps AWS has been pitching it wrong. There are lots of great use cases for Fargate, and in Michael’s opinion (and mine) it’s a better alternative to Kubernetes.

Thoughts from Twitter… 🐦

“One interesting consequence of ‘Serviceful Serverless’ and ‘Code is a Liability’ philosophies is that the worst ‘refactoring’ projects are a couple days of work at most.” ~ Joe Emison
A spirited discussion about development choices. You need to dig a bit as the conversation is buried in replies.

“If you’re part of the broader @kubernetesio ecosystem, comments like these should be taken to heart.” ~ Sandeep Parikh
I really love the thinking behind this thread. Complexity for complexity’s sake, especially when it doesn’t benefit the end user, is a tiresome and wasteful exercise.

When you prefer a multimedia experience… 🎞

Securing serverless applications with the right IAM permissions
Marcia Villalba has another episode of Foobar, this time she show you how to use IAM permission correctly with your serverless applications.

The Serverless Show, Ft. Ran Ribenzaft
Hillel Solow is joined by Ran Ribenzaft, Co-Founder & CTO at Epsagon, to chat about all things serverless observability and security.

Serverless Tools 🛠

Serverless AppSync Plugin: Top 10 New Features
Develop and test your AppSync GraphQL API locally on your workstation.

laconiajs/laconia: Create well-crafted serverless applications, effortlessly
This is a relatively new serverless application framework. It enforces hexagonal architectures, which should help you choose consistent patterns for your serverless functions.

jeremydaly/serverless-cloudside-plugin: Serverless plugin for using cloudside resources during local development
A new plugin I built to help me with my local development issues. Still needs some work, and I always appreciate feedback.

When you realize that AWS has been incredibly productive lately… 🏃‍♂️

Amazon Elasticsearch Service now offers improved performance at lower costs
New instance types offer superior performance at lower costs compared to previous-generation instances. Now if we can just make it serverless.

AWS AppSync Now Supports Tagging GraphQL APIs
You can now assign tags to GraphQL APIs in AWS AppSync.

New Quick Start sets up serverless CI/CD for the enterprise on the AWS Cloud
This is very cool. Setting this whole process up manually can be a pain, so having this Quick Start CloudFormation template will make life a lot easier.

AWS Systems Manager Now Supports Use of Parameter Store at Higher API Throughput
Run up to 1,000 requests per second for applications that require higher concurrent access to a large number of parameters.

Now you can tag Amazon DynamoDB tables when you create them
Yes, you could tag DynamoDB tables, but you used to have to wait until they were created to do it.

AWS Systems Manager Parameter Store Introduces Advanced Parameters
Advanced parameters enable you to create more than 10,000 parameters, use a larger parameter value size (up to 8 KB) and add policies to your parameter.

AWS specifies the IP address ranges for Amazon DynamoDB endpoints
This was a surprisingly common issue, so being able to lockdown IP address ranges on your local firewalls or in your VPC security groups is very helpful.

Query for AWS Regions, Endpoints, and More Using AWS Systems Manager
This is another common issue, just trying to find information about different AWS regions and their service endpoints. Now this info is all available via the AWS Systems Manager API, for free.

Hello, World! Welcome to the AWS Cost Management Blog.
AWS has launched a new blog to help you understand and control your AWS costs. Let’s hope this doesn’t take business away from Corey Quinn. 😉

Upcoming Serverless Events 🗓

There are a lot of upcoming serverless events, webinars, livestreams, and more. If you have an event you’d like me to mention, please email me.

May 1, 2019 – Aqua Serverless Security for AWS Lambda

May 1, 2019 – Stackery Livestream – Cloudside Plus Local Development: What should your serverless development workflow look like?

May 8, 2019 – Stackery Livestream – AWS Serverless Application Patterns with Jeremy Daly (yup, that’s me)

May 14, 2019 – PureSec Webinar – PCI Compliance & Serverless: Everything You Need To Know

May 15, 2019 – Serverless Monitoring & Troubleshooting – Expert Roundtable with Yan Cui, Jeremy Daly and Erez Berkner (yes, me again)

May 21, 2019 – Epsagon Webinar: Modernizing Applications with Serverless on AWS

May 22, 2019 – Build On Serverless, hosted by Heitor Lessa (I’ll be a guest on this episode and on the 29th)

June 4, 2019 – ServerlessDays Tel Aviv.

June 21, 2019 – ServerlessDays Milan. I’m giving one of the keynotes!

June 25-26, 2019 – AWS re:Inforce.

Serverless Star of the Week ⭐️

There is a very long list of people that are doing #ServerlessGood and contributing to the Serverless community. These people deserve recognition for their efforts. So each week, I will mention someone whose recent contribution really stood out to me. I love meeting new people, so if you know someone who deserves recognition, please let me know.

This week’s star is Yi Ai (@yia333). Yi is a Lead Application Developer at Neami National in Melbourne, Australia as well as a self-described “aspiring full-stack developer” and a “serverless enthusiast.” Over the last few months, Yi has published several excellent serverless articles on Medium as well as a number of repositories on GitHub. It’s always great to hear from new voices in the serverless space. Thank you, Yi, for taking the time to share your serverless knowledge and experience with the community! I’m looking forward to future articles and projects. 🙌

Final Thoughts 🤔

Wow, this was a long one. Hopefully you haven’t passed out on your keyboard 🤤.

Every week I’m blown away by the amazing things happening with serverless, and the incredible work that the cloud providers, vendors, and developers are doing. It’s such an honor to be part of this amazing community. And I’m glad that you’re all a part of it with me. 🙌

I hope you enjoyed this issue of Off-by-none. Your feedback and suggestions are always most welcome. They help to make this newsletter better each week. You can reach me via Twitter, LinkedIn, Facebook, or email and let me know your thoughts, criticisms, or how you’d like to contribute to Off-by-none.

If you like this newsletter, and think others would too, please do me the honor of sharing it with friends and coworkers who are interested in serverless. 👍

See you next week,
Jeremy