Darren: 
Welcome to Cloud Coffee Talk, AWS Edition.  These are real world problems, solutions and thoughtful discussions about working in the cloud. This podcast is for cloud professionals at every level of the organization, from the executive team to those with their hands on the keyboard, putting out fires and making the world a better place.

 And just to rehash this is really not meant to be cloud 101. It's meant to be coffee talk. Some people who are really passionate about what they do, talking about the cloud.  

Today's topic is a super exciting one. It's Serverless for Everyone.  The reason why it's super exciting is because we have a rock star with us today. We have the Bono of serverless, although I know he's a drummer, so I think Larry Mullen is more appropriate. But Eric Johnson, a Principal Developer Advocate of serverless at AWS is joining us today. How you doing Eric?

EJ: 
I'm good. I like the U2 reference that was well done.

 Darren: 
Thank you. And we're going to actually call you E.J. For the rest of the show because we also have my co-presenter Erik Deroin joining us as well.

EJ: 
That’s not a problem. Hey Erik. 

Erik: 
It's good to see you.  Good to be here

EJ: 
Thank you. Yeah

 Darren: 
So the three of us actually go way back. We actually worked together at a company - might as well keep it nameless at this point, but we had a great working relationship back then. It's great to see, E.J., what you've done with your career and what you've done with being an evangelist and advocate and really helping people learn in this space because you're so good at it. You do amazing talks.  I think my favorite one was two years ago at ReInvent. I think it was Api Gateway one.

 EJ: 
"I didn't know API Gateway could do that"

 Darren: 
"I didn't know API Gateway could do that" - I watched it again recently and I still learned things and it was over a year old and a great presentation.  So funny. I mean 55 minutes into it, you start talking about your daughter and a cat and I just like that you're entertaining.

 EJ:  
That sounds right.

 Darren: 
One of the most entertaining, ReInvent sessions I've ever been to.

 EJ: 
Thank you, I appreciate that. You always like hearing that back. It's a lot better than "Yeah, I tried to get through it but I couldn't". Okay great, So I should change that. So that's great to hear that. Not just fun but helpful.

 Darren: 
Combining the depth of the content with something that's engaging and really fun to listen to is really what we all want to do - it's how we all want to learn.

 Erik: 
Thank you.

 Darren: 
Talk a little bit about your path and how you ended up as a guru of serverless.

 EJ: 
Oh man. First of all, I don't know if I would say guru of serverless. I sit in a room with some really smart people, it's like, whoa, I'm the dummy in a room and a lot of times, but you know, coming to serverless, I'll tell you, I don't know how far you want me to go back but I'll stay in my development and technical world. I've had many jobs in my life but I've been doing development for a long time, I've worked in different companies as a developer front and backend. I've even done a little DBA which Darren, I know you were a DBA when we met.

 Darren: 
Oh Yes.

 EJ: 
And now you're probably shocked that I would say I've done that, but I have done a little bit and DevOps, you know, you name it. But I remember when Serverless was first announced, I was watching ReInvent and it literally just blew my mind.  The idea that I would not have to manage - and at the time it was just Lambda - API Gateway came shortly after. But this idea of, here's my code now you do the rest was mind blowing to me.

And so I started dabbling and started playing around with it.  And it was pretty raw when it first came out, I was writing my own scripts for deployment. And then as I moved along, I'd always kind of done it on the side. And then I went to work at a company, a partner of AWS as a Solutions Architect. And a lot of what I was doing was infrastructure, helping companies migrate to the cloud. But so many times I would see this architecture that they're doing and saying "that would be so much easier if you went serverless, you know?

But the truth is, as an infrastructure company, well that's not very billable you know, if you're managing infrastructure for somebody, but they were a good company, they're like, you know, if we can help… So a lot of times they'd say, let's get Eric, this might be serverless. So I'd come in and I'd help out and we do some things like that. And I also did a lot of speaking for this company. I was actually hired as the AWS evangelist for this company. It was Rackspace. I don't mind saying the name, I was a Racker. I'm proud to be a Racker.  When it came that this position opened up and I started talking with Chris Munns who is now my boss, it was just a really logical next step for me. I'm so passionate about serverless, I'm so passionate about not maintaining machines and infrastructure and all that kind of stuff. And then I'm also passionate about speaking. My favorite place to be is to be standing on stage teaching something. You know, I have a lot of folks and I'm wandering a little bit Darren, you may need more editing than any of these others have done, I've had people ask me, hey, why don't you be a comedian, why don't you get up...I don't enjoy that. I love making people laugh. But I like teaching. I like when the light bulb goes on and in serverless you see a lot of people going "with the serverless thing. What do you mean? No servers"? And then you explain this event-driven model and this idea of observing rather than, you know, acting and reacting to events. And you see that light bulb go on. It's like, yeah, that's my job. This this is why I do it. So that's kind of my journey to where I'm at now. Again, that's my favorite thing in the world.  It kind of works in virtual. But being on stage and then that hallway track in-between and people go: "I finally got it!" It's like: "All right!"

 Darren: 
That's awesome. And I mean you're obviously so passionate about it. I mean you have the most interesting job in the world given...

 EJ: 
I think so

 Darren: 
...what you get to do for people. That's really cool. There's a question that Erik and I - the other Erik - have gotten in a little debate about which I'd love to hear your input on is: what does serverless mean? What does serverless mean to you?

 EJ: 
Yeah. So and I always start, when I do my serverless 101, I will start with the line, you can put 100 people in the room and ask them what is serverless and you get 103 answers and the reason you get 103 is because somebody will go: "I like that. I'll use that one, too". Honestly - and I'll give you the AWS answer because it kind of lines up with how I think about it and there is a lot of debate but - and Erik I see you smiling at me - I mean this is an AWS answer but it's my answer too.

Erik: 
I think I use the AWS answer which is why I'm smiling.

 EJ: 
Yeah, we really look at four different things. The first is you pay for what you get right and - actually it starts out with no servers to manage. If you're serverless you're not going to manage servers. It's probably the worst kept secret in the world is there are servers in serverless. Yes, you heard it here - but you don't manage them. You don't patch them. You don't upgrade them. You don't have the buildings that do that. 

The second thing is scalability. So you're going to get the scale.  This is what a lot of folks, when we talk about scalability, they talk about the elasticity and growing out: "Oh it's going to scale up, I got rid of the slowness, great". But the cool thing about serverless, is it scales down too - it scales to zero. And so you get this idea of - going to the third thing. You only pay for what you get - for what you use - or get, however you look at it and so that idea and that's the oft missed one. I know it scales up but it scales down so I'm not paying for it. And then the fourth is that security being wrapped around it and the scalability and things like that. And I know there are a lot of debates and I won't go into all of them, but that's how I look at it. And then the other thing is I don't get terribly hung up on it.  Serverless is those things I guess I could continue, but...

 Darren: 
I think there's an important distinction though.  That is the AWS definition of what serverless is, it's services not systems, things you don't manage, all that makes sense. But when you talk about serverless, you do your presentations, you're really talking about a different way about thinking about developing...

 EJ: 
I am

 Darren:  
Different patterns in the way people are actually managing applications. So even though, you can do RDS and you could do Fargate and you could deploy applications there, when it comes to really architecting your applications serverless, the Lambda, the API Gateway...

 EJ: 
A whole different animal.

 Darren: 
Very different. Right?

 EJ: 
Yeah. You know, and that's a funny thing. We've had internal talks as well as external. And, there's this idea that - serverless used to be Lambda. Lambda is serverless, serverless is Lambda, there it is, but Lambda is just one of the many services now. So when we talk - kind of going to what you're talking about - when we talk about serverless, it's really more of stitching together services to make an application. My general statement is: use a managed service. If all else fails, then code.  Code should be the last thing you do, because the services that, you know, API Gateway can talk directly to Step functions. Step functions can talk directly to DynamoDB. And so what had happened is - Ben Kehoe will probably drive to my house and tackle me for this - but we get into this configuration versus code idea and there's a lot of people would say, well, yaml is code and it probably is. I don't really care where you land on that, I use yaml. But the idea is you get more into configuration than you do writing code. What you get out of that is built in - and I could go on forever - but built in retry. Built in error handling, things like that, that I don't have to write code. I've always said the most brittle part of an application is Eric Johnson's code. That's the truth, right? And you've both probably seen me develop at one point or another and, you're like: Yeah, that's really true.

 Erik: 
So, I guess my question then to follow up on that is how do you bridge that gap to a developer? Darren and I talk a lot about the extraneous cognitive load of these things. About teaching people how to think about serverless or think about cloud in general and getting a developer to think more about configuration than about the code that they actually write. What's your pitch on that? To get people start taking that part seriously and find joy in it?

 EJ: 
Yeah, that is a tough question because so many developers are at different stages in their journey. I have a deck that I do called Leveraging Serverless in Full Stack Development. What it does is it walks through what does an application looks like before “serverless”. You can't see, but I'm doing quotes.  I'm doing single quotes in the air but we all know what that means.  We all started with a computer under our desk where we wrote our apps and we somehow got a static ip, and then maybe it ran in your mom's closet. And then we got crazy. We did multi-tier, we broke it out into servers and so on and so forth. And I go all the way up to a full data center with all the servers you've got to manage. And then I just walk through and I start pulling those out and I drop serverless services in there. I don't think this answers your question directly, but this is kind of how I approach it in understanding what do the services do.  What am I already doing and what's the equivalent replacement to that in a serverless world? 

And that's when you again, we go back to this light bulb of: Wait a minute. So you're telling me, instead of running a farm of servers to host my front end. Let's say I'm running a huge app, I can run one s3 bucket? That's what I'm telling you.  You can host it and then maybe you want to run amplify console, so you get the CI/CD out of that, you get some CloudFront, things like that. But the idea is I'm going to go in there and I'm going to show you through example, I'm going to show you how you get rid of this management overhead.

The code - and Darren - you were kind of alluding to that with serverless and we'll probably come back to this in a minute. There is a difference in how you think on the code, too, and how you think about patterns.  But we're not changing the core of your application. You can still do the most popular "Hello World" of serverless, image manipulation, whatever, but you can still do that and your code is still going to work but I don't have to have a server to bring it up. I don't have to have massive hard drives to the side of the machine...

 Darren: 
Maybe you have an issue of transitioning into more of an event driven architecture, the interaction pieces...

 EJ: 
Yeah, it's the wrapper a lot of times.  It's: a lot of my code still works, but it's how I how I approach that. So Erik, I properly ran around that question, but hopefully that helps.

 Erik: 
Yeah, I think it's a good insight and I think it's a topic I think a lot about in, DevOps kind of land. And adopting a DevOps mentality.  I think nothing speaks more to serverless than a DevOps mentality and trying to get developers to pick up on that and advocate and ingrain that into what they do. So I was kind of touching on that and seeing where you kind of think.  That's to me where the answer lies and that's a big ambiguous answer, of course.

 Darren: 
EJ, in your world, how much do you actually interact with teams that are on this journey? Because obviously you do a lot of one-on-one interaction, you influence a lot of people in your presentations. Do you ever get an opportunity to actually work with development teams who are somewhere on their way of their serverless maturation journey?

 EJ: 
I do, and I'm really grateful for that. Most of my role, like you said, is a one-to-one or one to many, you know, the one-to-one is the hallway track or the one to many, however many are in the room, We don't generally sit with customers as a whole, but once in a while I will.  My two areas of expertise, you know - I do all of serverless - we all kind of have our chunks where we go: that's mine, don't touch that, that's mine. Mine are AWS SAM and API Gateway.  Those are kind of where I really do a lot of my work and so every once in a while, you'll get a team that says: Hey, they really like to talk to you specifically. So I get to do this once a while and I've got to sit down with teams - actually just recently - where I got to sit down and walk them through...They were really struggling with the infrastructure as code with the local development, some of those that - and you're probably going to touch on this - but some of those challenges with serverless. Because it's a different paradigm and development. You're developing against infrastructure launched in the cloud, and how do you test against things like that? So, I sat down with them and got to walk them through that.  It was a lot of fun.

 Darren: 
The whole local development environment, I think I had a general question around how do you keep up with the pace of innovation in this space? And of course I think that there's always going to be this friction when things are changing so fast. There's so many different ways and different patterns, and how to approach these problems. The local development environment is definitely one of those areas that still has a way to go. So how do you keep up with this stuff?

 EJ: 
Yeah, there are some sharp edges local development.  How do I keep up? You know, I kind of commit, I'm a SAM guy all the way.  Serverless framework is a phenomenal framework, they do a great job. CDK, obviously, great job, I know SAM.  That's where I live, that's where I eat, that's where I breathe, so that's what I use.  I keep up by doing, I mean we all know that. I do some reading, but until I've broken something, I don't know how it works.  And I break them good. Part of my job, you know, a lot of my job's obviously that outward, you know, talking stuff, but a lot of my job is taking back stuff.  What I think people don't realize is, I sit every week with our SAM team and I say: Hey, here's what folks are saying and they're immediately responsive. It's really kind of cool. They say: Hey, we're going to address that, we've got some things coming down the road.  So it's a two way conversation always between the product team and myself.  Then I am of course representing the developers, so "Developer Advocate" to product team.  

I think I've chased a rabbit here. I maybe lost track of the original question, but that's a really cool thing for me. That personally keeps me up to date because I know the secrets and stuff coming and I get to play with stuff.  Here just recently, I don't know if you saw we announced CDK support for SAM and I was playing with that several weeks beforehand. It's really cool but what they get is, I find, going back to my initial statement - I broke it. This doesn't work, this is broken.  I get to do a lot of non-happy path stuff and pound against that. So that keeps me up to date. Not everybody gets that opportunity because they don't get that early beta, but even stuff coming out, I mean, the first thing I do is I install it, I try it, I hammer against it and then I report bugs and that's pretty much how I stay up to date.

 Darren:
What is the mechanism for learning serverless? Again, it's changing so fast. There's all this content out there. What's working for people?  Or is it just, hey, you have to get in there and just dive in and iterate, make mistakes, fail...

 EJ: 
Both. Yeah, I mean that second part like you said.  I don't think that's isolated to serverless. I think that's any development.  Unless you're hands on, you can't read about development. You can read and get ideas, but you got to do it. You were going to say something?

 Darren: 
Can I just say the api gateway documentation…

 EJ: 
You may not say it...

 Darren: 
...when you download as a pdf, it's 820 pages long.

 Erik: 
Darren has read all of them and memorized them, I guarantee it.

 Darren: 
Just read, not memorized.

 EJ: 
If you read all that before you do anything with API Gateway...just kidding. Yeah, API Gateways in general are a complex thing. Throttling, caching, WAF, you know, all the things we deal with. Well, how do you do all that?  There're different angles, but to go back to your question. Everybody has a different way of learning. Some people, like I said, that light comes on.  Some people walk away and then they'll come back and go: give me some materials. Ben Smith is one of our other DAS.  He's writing us a series called "Getting Started with Serverless". It really takes this approach of: I don't know anything about it. What is it? Where do I dive in? The end of the line is: read the documentation, that's fine learning services, that's fine. But get your hands in and get dirty. The thing I would really caution folks – caution’s not the right word - I would encourage folks about serverless: understand the services.  Because if you go back to my earlier statement and say serverless is a lot of times stitching together services, if all else fails code.  Understanding when do I use SQS versus when do I use EventBridge?  I know we're talking serverless and obviously I'm talking AWS serverless, you know, so, so we're talking these services, but even in the other services - those services are purpose-built to do something and there is overlap, but they do that thing.  So one of the first things I tell folks is, and this is where you read the synopsis, at least, of those and figure out what the different services do. That way you have your full tool belt going on.

 Darren:
I think that's one of the really interesting things about working with these services, is that again, if you take the pace of innovation with trying to implement these services and you just take the age old approach of just trying it out. You can actually get yourself in a lot of trouble.  I can think about a really simple one. You brought up SQS. SQS to Lambda and the fact that it has a batch, - you could do a batch of 1 to 10, How many people know that when they first try it out? So the details really do matter.

 EJ: 
You and I have had a conversation about batch windows.

 Darren: 
We have, Yeah.

 EJ:
I remember you going: “Ohhhhhhh”.  You're right. It starts with the content. When I go back to my earlier days at Rackspace, I was asked:  How do you keep up with it? Because my joke is always a new service comes out at AWS every seven days. Okay, that's not true, but it feels like it because there's constant innovation going on. I mean, it's a crazy place. I started here 2.5 years ago, and when I started, you made the joke, oh, it's the fire hose. You know, you're learning so much, but I'm happy to say that 2.5 years in, that feeling hasn't changed. It's still just like, ahhhh!

 Darren: 
That of course can be very intimidating for...

 EJ: 
It can be.

 Darren: ...And overwhelming. You hear it a lot from people that work at AWS and from those that work with these services. It's just totally overwhelming. Which actually brings me to...

 Erik: 
Can I chime in a couple of things here? I appreciate that because, I still get people being like, did you see that announcement? Like no, there's so many, I get emails every day of something new.  I just want to talk about what I think has been great about the serverless journey for our company and what we went through as we moved parts of ours into serverless.  We started in serverless framework and SAM was announced as we were kind of starting that and we started to evaluate (SAM) and there was some weird funk that we ran into.  
Then about a year later we started to run into some issues with some of the serverless framework plugins and we like serverless framework that's fine.  But we're having some issues with some plugins and we re-evaluated SAM and were, like: this is ready to go. And what was nice is that when you do serverless right, is that when you really drive this sort of architecture, this event-driven architecture, it's actually really easy and really safe to cut those things over and to move over to something new and to adopt a new piece of technology because you've decoupled it because it's isolated because it's really ready to go.

 EJ: 
That's right

Erik:
It is really safe. And that has been a huge benefit and one of the best and most fun parts of serverless.

 Erik: 
I just wanted to point that out for people who are thinking about it because...get my AWS advocate...

 EJ: 
You're hired!

 Erik: 
Yes!  I would accept in a heartbeat.  But yeah, that's how I would pitch that.

 EJ: 
That's great. I appreciate hearing that.  Darren, I know you want to move on but Erik and I are taking over so...I've done a presentation called 12 factor serverless.  If you're developing right, it just kind of works in serverless because it's this decoupled, even if you're developing everything on, on one machine, they should be decoupled, there should be queues, there should be pub-sub, there should be, it's all these things that make our applications degrade gracefully.  Then we can say, okay, well this part is not working this part, still doing its job, but we'll cue that up and when we get that working it's fine. So when you take and kind of start that move to serverless, it kind of enforces those good practices of, you know, solid development and keep it simple, stupid and all those things.

 Darren: 
So now that we have these massively decoupled services all communicating through APIs or whatever else and you have all these integration points and now you now you have another problem.

 EJ: 
This feels very loaded.

 Darren: 
No, but it's the challenge of moving towards microservices and can you talk a little bit about that? Because these are great and you're absolutely right. I mean, your containing your business logic, you're doing more services, writing less code, things are much cleaner and easy to work with. But you have this massive network of services that are having to talk to each other,

 Erik: 
Can I just say: One function, and If blocks and you solve your problems.

 EJ: 
The Mono-Lambda approach. Fat Lambda, Big Lambda, I've heard all, all the terms, so and that's P-H-A-T. So it's cool, you know, so of course that's probably like 1980's. So there you go. I'm a little older than you guys.  Firstly I think is a lot of these problems - when I talk to companies and talking to developers and teams that are dealing with this kind of thing, they probably, let's say pre-migration, they're still building their legacy apps and things like that, but they've broken it out, they're probably already dealing with that in some ways and moving to serverless can kind of highlight that, you know, 40 Lambdas and they're all in the same application or 400 Lambdas you know, Lambda functions. Oh, Chris Munns, he's going to drive here and tackle me. We don't say Lambdas, we say "Lambda functions" and Chris Munns, I swear it'll wake him up at night if I were to say that right now, so don't edit it out because it is fun to poke. I think the thing that we look at - any way you look at it comes down to organization and yeah, it can be complex when you start breaking these out how micro are your microservices?  And there's a lot of debate, talk, opinions, whatever on:  Do we do a mono Lambda that routes to other Lambda functions?  There’s the: We have one Lambda function that hosts an Express app and then we do all the routing in there, I think folks are doing that with Flask in some different ways.  

I think while that works and it can be effective, I think you're kind of losing some of the serverless of serverless. Not serverless necessarily, but the micro service aspect, in you’re maintaining lines and lines and lines of code in a single Lambda function.

Where it makes more sense and when I'm sitting with companies and talking about organization, I really encourage when you look at an application, it usually has logical breaks, logical domains? Here’s my user management, here's my reporting, here's my order, whatever. Now order may have 30 Lambda functions to deal with all the different things that are going on. Users may have 30 and you know, let's say all of them just for easy math and I can count past two, they all have 30?  So if there's five services, that's 150 Lambdas..Lambda functions. What happens is we look at that, and all those are in the same SAM template or CDK or serverless, and that's unmanageable. The reality is if I take that and I break it down and I say, ok, my user management is just going to be an application in itself. It's a micro app with microservices. I may have just coined that phrase, I don't know, but it's a micro app with microservices. And a whole team dedicated to that boom, boom boom..or one person, I don't know. 

But that's that application. And then to combine these, there's several different ways you can do it. You know, you can do one APIgateway pointing to the different ones using base path mapping, So each one can have their own API gateway and then an umbrella API pointing to that.  Or I could use CloudFront or I could use - I mean there's different ways.  But the idea is we isolate those apps into the smallest logical independent micro application and then it becomes manageable and that's when this whole idea of an API - as a user I need to talk to the reporting. So that's what I'm going to call across that API.  I know their endpoint, I'm going to do that and I'm going to get the data. So we set those contracts and so I can do whatever I want behind that contract, but I set that contract for communication and they know that contract and it makes that communication very easy. So that's really what I encourage, even if you're not serverless, even if you're building an application in a data center the old way - I said it - and those should be broken down into these manageable chunks.

 Erik: 
Go read domain driven design if you haven't, and adopt those practices.

 EJ: 
There you go.

 Darren: 
It's nothing new. And I think that Goldilocks approach in looking at the domains, that specific business needs makes a lot of sense.

 EJ: 
It absolutely applies to serverless as well.

 Darren: Absolutely. I think when, because again, there's so much learning happening in the space, I think that - and there's so much creativity and so many different ways that you could design these, that I think things maybe get a little too micro serviced and I do think that spending some time really strategically focused on what's important to the business, making sure the teams are working in that space...

 EJ: I'm just chuckling because I've built both. I've built the mono-Lambda, built the, if it's this, then route it through this and I've built the 40 Lambdas each with one line of code, It's like, oh my gosh, that's ridiculous. So there's a balance, there is definitely a balance, but I really believe in a single responsibility model for Lambda functions, I do one thing, I do it really well and that's all I do.

 Darren: 
I do think that there is no surprise that in the last,  seems like in the last two years, the whole telemetry space is going bazookas and you know - whatever I just said -  it's really becoming much more important I think as micro services become more of a thing.  How do we keep track of this? How do we trace requests through? Talk a little bit about the telemetry space.

 EJ: 
Yeah. So with distributed applications and what I just described was a super distributed - microapps with microservices, you got to see the whole story and you've got to be able to say, okay, this event came in here. Where did it go next? What did it do? What is the latency, these things like that? And it is critical.  Where we used to be able to get, you know, we had one, function class, whatever handling all that and it could just log out everything and say what it is. That's not the case anymore. An event may hit five different services coming in.  So you need that story.  There're multiple ways of doing it.  Our native version is XRay and when you come in, when you hit an API gateway, it's tagged with an ID and then every single service will report based on that. Super handy.  The cool thing about that is, this interesting thing is with Lambda functions, or Lambda service itself, there's two things you get, you get the service:  Did the service respond properly and then you get the Lambda function:  Did the Lambda function respond properly? So you're able to look at that and go okay:  I got a 400 back from the Lambda service that meant something was wrong with the Lambda service itself. That's generally that not going to be the case. It's generally going to be Eric Johnson's code as we've already discussed. You get to see that and look at that latency and see and DynamoDB the same way.  What is that query looking like?  There's a lot of story that can be told. One of the first things we tell folks is in every service that has tracing. Turn on tracing when I do a template, when I do a SAM template Lambda tracing on, API tracing on,  Step Function tracing on, whatever.

 Darren: 
Even in production, no concerns about latency, performance issues?

 EJ: 
No, I think the story - and I don't have a number on latency so I can't, I can't tell you exactly on that - but I think the story I get is worth it.  Now granted if I do see something, maybe I'll turn it off, but I don't have a number of latency that is added from tracing, but I'm pretty sure it's very, very little.

 Erik: 
I would just add being able to trace through a service oriented architecture is critical in a production environment. I'll actually say that if you don't turn that on, you're making a mistake.

 EJ:  
We talk a lot about observability and monitoring and this idea of, you know, monitoring is, you know, watching for what you expect, observability is finding out what you weren't expecting.  Getting this idea of - and that's where tracing comes in really handy. Oh my gosh, I had no idea because a lot of times our applications are boolean - it either works or doesn't - and if it's working we're happy, if it doesn't, we're not happy, but really it should be:  Is it working as efficiently, both speed and cost,  as it could be? And that's where a lot of that story comes in and that's where it's worth running it. Even in production.

 Darren: 
What a great segway to talk about the serverless bill. I'd be curious:  People's reactions when they actually move over to serverless -  Erik and I talked about this in a previous episode where it's like a rounding error on your bill.

 EJ: 
Yeah.  I can only talk about some of the public things out there. One of the things, well, there's two things I'll bring up.  ACloudGuru is pretty open about their bill and how they run serverless and shocking amount of savings they had. Also, I don't know if you saw about that ReInvent, we changed our billing model for Lambda functions. It's no longer...

 Darren: 
It's single millisecond...

 EJ:  
It's now single millisecond billing. And there's more to that than a lot of people get because it used to be some of my Lambdas are under a second, doesn't matter. But now it's like, well, hang on, now if I can optimize that Lambda function to run at 2 or 10 milliseconds, I'm saving 90% there. It's like what? And so yeah, that was really well, people were pretty excited about that. I'll let you ask your next question before I journey off on this.

 Erik: 
I just wanted to say our company is literally pennies on the dollar is what our cost savings ended up being. It was, it was bananas. Absolutely bananas.

 Darren: 
Pretty spectacular.

 EJ:
Yeah, So I'm going to use that as a segway into - and this is something I talk about a lot. Something I'm very passionate about is, and this really does relate to what you're talking about in in a weird way, I'm really passionate about asynchronous processing versus synchronous processing. And the reason for that is serverless, like anything else, if done wrong can be quite expensive, you know what I mean? Just having a Lambda function, run - it's just running all the time, you know, that's not what Lambda functions were made to do. They're made to invoke, do their job and get out of the way. There's a reason we limit that at 15 minutes. It used to be five minutes now it's 15 minutes because that idea of running that process - and I'll have companies to come to me as say: We got this process that's running over 15 minutes and I really challenge them to say, Okay, can it be broken down in steps and we run in step functions, can it be kicked off asynchronously and checked on later? What we'll find a lot is where companies are doing blocking synchronous calls and running that Lambda function for minutes at a time. They could have triggered a request and then gotten signal back later that it's done. And that Lambda only ran for 10 milliseconds. There's a lot of money to be saved by planning and architecting properly, and this goes back to that, I hate to sound like a - broken horse, kick a dead horse, broken record  - uh, I hate to sound like a broken horse, but it's all about...

 Darren: 
That would sound awful.

 EJ: 
Yeah, it's all about understanding and using the service properly. The right service for the right thing. I do a presentation that's called: Thinking Asynchronously and I built this whole translation application.  You submit a text or phrase and then you would also submit an array of languages that you want to translate to. And it would go to API Gateway and hit a Lambda function. The Lambda function would then call Translate.  For each of those records would translate this to english, translate this to spanish, translate this to whatever. So that was 9, 10, 12 calls, depending on how many cultures you call. It would get them all back, it would then send each one of those individually to Polly to convert that from text to audio. And it would take the buffer back. And when the buffer was complete it would then write it to an S3 bucket. And then when that was complete that would save all the data to Dynamo. So that's a very synchronous, long running application. Right? So that whole time. your Lambda is just churning along and you're paying a lot of money. 

But if I turn that around and I said, Okay, same process, I send it into the Lambda function and the Lambda function breaks down and it kicks each one of those translation requests into an event bus. So now I've got 10 events that were just spread out, fanned out on Amazon EventBridge. Then each one of those kicked off a separate Lambda function that did the translation real quick and wrote it back to DynamoDB, or put it right back on the event bus. So now I've got translation requests on the event bus, and now I've got audio requests in the event bus, then another Lambda kicked off and took those audio requests and handed it to Polly and said, when you're done, drop it in this S3 bucket, I'm not going to wait around. When that S3 bucket received that file, it kicked off another Lambda function that wrote it all to DynamoDB. And so what's happening is you're seeing a Lambda, you're seeing 1..2..3..4 I think five Lambda functions invoked for at worst a second versus one running for 50 seconds and you've cut your billing way down by planning appropriately.

 Darren: 
Order of magnitude savings when you move towards this asynchronous...

 EJ: 
Exactly

 Darren: 
...This really comes back to the earlier conversation and you said, I think thinking "asynchronously", but thinking "serverlessly"

 EJ: 
Yeah, thinking serverlessly, yeah.

 Darren: 
...coming back to that:  To me,  going back to the beginning of this conversation around what is serverless mean? I think to me there really is - that's a really important aspect of this, is thinking about things in a serverless, event driven, asynchronous fashion.  And that's where the game really changes.

 Erik: 
And I guarantee that pattern is repeated all over because as you're describing that, I was like, oh that's literally what we did with an application that does something very different, but it's the exact same workflow that we went through.

 EJ: 
Well and if you take that pattern you think about it, okay, let's say now, I'm getting translations and I'm getting audio recordings, but let's say, I want to do Comprehend on it to make sure it's a happy text. Not a sad text, right? So I could go into my original Lambda function and call Comprehend and kind of add that to my code and intermix it or I just throw another Lambda function on that whose sole job is to do Comprehend and have it make a rule in EventBridge that triggers that function. I can have all these different services running from the same events trucking down the Event bridge. That's a classic use of EventBridge anc understanding how a bus works.

 Darren: 
How are you feeling about HTTP API now?  Finally made it to GA at this ReInvent.

 EJ: 
Yeah, I love HTTP API. I think the simplicity of it, it makes it very easy to use. I know you and I have gone back and forth a couple of times on it. One of the things we were looking to do, parity, we're not there. That's just honest, we're not there. So that’s going outside a little bit of what we were doing. 

I guess the way I look at it is HTTP API is a great starting point. Look, I'm just bringing up applications - I don't need all that other stuff. I don't need, you know, WAF attachment. I don't need API keys, things like that, so I can use HTTP API. So that gives me - obviously it's quite a bit of savings. So that gives me an easy way to ramp that up. But I think sometimes they understand:  REST API is going away. No, it's not. REST is around. It's got a lot more features to it and we're continually adding to it and building onto it. Both flavors are out there.

 Darren: 
Good to know. Yeah, I definitely spend time trying to understand those differences. The feature differences because that is an important aspect because I do like the simplicity of it and the configuration. However, if it doesn't support a feature I need…

 EJ: 
Yeah you need what you need. Right? My advice to most folks is start there and if it doesn't work, you know evaluate features, then move on to REST. The REST API is a good product. Well they're both good products but it's super powerful.  There's just a ton you can do with it.

 Darren: 
So, when I was planning this interview with you - and you really brought to light when you started talking about what you do - you really do have a cool job. I was really wondering like how you decide what you do every day?  Because it just seems like so cool. Do you wake up and like:  Who do I want to talk to, or am I going to make another cool slide deck or another cool demo and submit something else to my GitHub?  What does it look like for you?

 EJ: 
A day in the life. I get up, my wife has a list next to the bed. I do what she says. I'm just kidding.  A day in the life. You know, every day looks differently. On a side note, really funny if you ask my kids to tell you what I do:  "He just yells at the screen a lot or he travels and has parties",  because it seems like every time they call me to talk to me like, oh yeah, daddy's in an after party until, whatever Is that all you do?

No, I don't. But anyway, that's not what you asked. 

There's, some core things that I do.  Number one is I learn, right? I spend a good part of the days learning and I do it like we talked earlier by doing, I will build SAM templates. I will try out applications. I try to figure out the hard, the not so common things of our service that customers are asking about and things like that. So, I learn those so I can then go out and teach them.

 As far as what we decide to talk about:  Obviously we're, you know, we're always watching what's new, what's coming.  One of the things - and I'll just say this and it's one of those from the inside. I can tell you it's absolutely true. AWS is so heavily customer obsessed and driven that we are constantly - I can't tell you how many times during the day someone will call me and say, hey, I need a customer opinion on this. What are you hearing about A or B or X or whatever and so we spend a lot of time bringing that in and they modify that. 

So, what happens is that kind of drives that road map and so I'm evaluating that road map and deciding, okay, what do we need to speak about?  There's this balance of, here's something new coming and I want you to know about, I'm going to present on it and there's a balance of, I really don't think folks are getting A or B and so we spend time trying to read that. So I guess the easiest answer is I do every day what I think is going to help the customer most, I know that sounds cliché, but that's really what it comes down to is:  what's going to get the paper cuts out of the way? What's going to get the obstacles out of way and let people build what they want to build? And so that's, that kind of drives what I do.

 Darren: 
Pretty cool that you get to learn and teach as part of your job,

 EJ: 
I told Chris: You are absolutely - I'm going to die at this job.

 Erik: 
I can't think of somebody who's better suited to that as well.

 EJ: 
Well, thank you. I appreciate that. I appreciate that, you know, my background, we talked about the development, My background, I'm just saying - I was a pastor for many, many years and so this kind of helping people and speaking to people and making people laugh, that is very ingrained in me. So this job is the perfect marriage of tech because I love tech.   In my world where I work, I'm probably one of the lesser technical people. Now, I'm pretty technical. In my house, I'm the most technical, I can run the VCR if you know what that is.  But in the world I sit in, this is a very technical world, but it's a great marriage of that tech and the speaking and helping and stuff. So yeah, I love it.

 Erik: 
I also just love not being the smartest person in the room and just surrounded by people who are smart, just like, oh man, I just like to absorb as much of this as I can with a sponge.

 EJ: 
Yeah, I've always said if you're the smartest person in the room, find a new room because you don't learn.  It's great to teach, but you need to learn. So, you should be in two rooms, be the smartest in one room, be the dumbest in the other.

 Darren: 
We're gonna wrap up with some - just pepper you with some questions and...

  EJ: 
All Right.

 Darren: 
You just answer them however you see fit. You don't have to put context around it.

 EJ: 
All right.

 Darren: 
We want to see the inner workings of your mind.

 EJ: 
You're going to be so disappointed.

 Darren: 
Alright. First: Node or Python?

 EJ: 
Node.

 Darren: 
Why doesn't SAM want to be destroyed?

 EJ: 
Okay. So this is an ongoing. I hope Stefano listens to this. I want SAM destroyed as well. So yes. SAM destroy is something I've been pushing for.

 Darren: 
Okay, when deploying via SAM. Should I use the Stage stage as a drink coaster?

 EJ: 
Uh Say that again.

 Darren: 
When deploying via SAM. Should I use the Stage stage as a drink coaster?

 EJ: 
Yes.

 Darren: 
SSM Parameter store or Secrets Manager?

 EJ: 
Secrets Manager.

 Darren: 
Zip file or container image?

 EJ: 
Zip.  

It depends.  What do you need? You know these are what I use. I'm telling you what I use. But yep. And if you want more context, I'm glad to do that.

 Darren: 
If I use serverless to spin up EC2 instances is it still serverless?

 EJ: 
It is not because there's no such thing as using serverless to spin up EC2. Yeah, that question doesn't make sense.  That's a trick question.

 Darren: 
That's what I was going for.

 Erik: 
I mean, you could include boto3 and you could spin up servers...

 EJ: 
You could.  You absolutely could. I guess so. So if I'm using SAM to build EC2 instances is it still serverless? No, it's not.

 Erik: 
What if I use it to destroy EC2 services, which is a Lambda function I actually have?

 EJ: 
You currently can't do that right now.

 Darren: 
As always, Rube Goldberg would be proud of us.

 Erik:
I think we're doing it through CloudFormation.

 EJ: 
Yes, there you go. There you go.

 Darren: 
A couple more questions:  If a serverless function is deployed but never invoked, does it exist?

 EJ: 
Can you hear it?

 Darren: 
I don't know, can you?

 EJ: 
It does not exist. It does not exist on your bill.  And that's what matters.

 Darren: 
That's an excellent point. Excellent point.  
Vegas or Twitch?

 EJ: 
Vegas twice. Yeah. Yeah, Vegas

 Darren: 
Never thought I'd want to see Vegas as bad as I do this year. 

EJ: 
Yeah, exactly. Every Friday at the end of ReInvent: I'm never coming back to Vegas again. Last year:  Pleeeease! Send me to Vegas!  I'll do anything!

Darren: 
Well, EJ, it was an absolute pleasure to have you come on the show.

EJ: 
Oh, it was a blast.

 Darren: 
I love talking to you. I love listening to you, watching you present. Thank you so much.

 EJ: 
Yeah, I really appreciate y'all having me. This was a lot of fun. I enjoyed both of you. Well, like you said in the beginning, we go way back and really have had a lot of fun working, and Darren you and I driving around California at one point just hanging out, so, yes...great times.

 Darren: 
EJ, Why don't you shout out your contact info?

 EJ: 
Yeah. So, the best way to get me is @edjgeek on twitter.  That is the best way and if you need to go, you know, direct, that's fine. But I really encourage you to follow me on @edjgeek. I talk a lot about serverless, pizza, Dr. Pepper, my family, you get it all.

 Darren:
You also have a lot of great content on GitHub, and Twitch.

 EJ: 
We do, yeah. https://serverlessland.com is kind of our central hub. So https://serverlessland.com is a website that the developer advocacy team maintains. And that is a hub to the Youtube channel. So, we have https://youtube.com/serverlessland.  But https://serverlessland.com is a hub to everything. All our blogs, all our videos, learning paths, everything. So that's the best way to get us.

 Darren: 
Excellent. We'll get that in the release notes, Erik, thank you as well. Any final parting words?

 Erik: 
No, it's just been a real pleasure to be here. I really enjoyed getting to sort of be a little more passive in this one, than in the past and just like learning a bunch and also reinforcing that our company has been doing the right things and we're on the right track and that...

 EJ: 
Good 

Erik:
...we're not alone in this journey. It's nice to get like, you know, the stuff we're making up as you go along was the right answer and we probably could have saved some time having talked to you sooner. 

EJ: 
Well, that's what we try to do.  I love to see companies succeed with this. 

Darren: 
Thanks to our listeners for tuning in. You can find us on twitter at @cloudcoffeetalk. We welcome all your feedback.  Until next time, have fun in the cloud.