A great feature of Lambda is you can trigger it to run in a variety of ways, certainly at least one of these triggers will meet your use case.
It is stupidly cheap.
For Lambda you pay by requests (number of times your function runs) and compute time.
In Free Tier you get one million requests and 400,000 GB-seconds per month, which basically means you can do a ton of stuff for free every month.
Even if you do go over it’s a whopping 20 cents per one million requests after.
By my back of the napkin calculation, you can execute over 20 million Lambda functions for the price of your morning Cold Brew — what a time to be alive!Lambda Data Pipelines — Set It And Forget It?In my last Medium Article we went over setting up a Lambda data pipeline in Python that will scrape Craigslist once a day for data on apartment listings.
In a perfect world we can “set it and forget it” and get data at our regular interval.
Anyone who watched the final season of Game Of Thrones knows we do not live in a perfect world.
If you’ve programmed for more than a hot minute you know bugs happen: things time out, Craigslist could be unavailable, your parser could encounter some godforsaken emoticon character that causes it to crash, etc.
So while we can’t fully set it and forget it, we can set it and only be contacted when something is wrong.
This is done with alarms, but first we need to set up monitoring.
Setting Up MonitoringIf you have created a Lambda function you will notice at the top there are two main panels: Configuration and Monitoring.
Configuration is where you actually make changes to your Lambda function: change the trigger, change the actual code, adjust the time out, etc.
Monitoring is where you check that everything is going swimmingly.
It’s where you can see things like if there was an error, the number of times your Lambda function has run, etc.
However, if you go to the Monitoring tab you will notice a lot of empty charts by default.
That is due to the fact that CloudWatch, which is the tool for monitoring AWS services, does not automatically track your Lambda functions metrics.
Adding CloudWatch Metrics to your Lambda FunctionThe actual issue here is your Lambda function just needs permission to publish to the logs.
Why is this the default?.I have no idea, but let’s change that.
Basically what you need to do is attach a policy that let’s Lambda publish to the logs.
Quick Aside: Permissions in AWS can be its own series of articles, but a good rule of thumb is assume literally nothing has permissions to do anything.
It’s like Congress.
In this example, we have our Lambda function, and it can’t publish to our CloudWatch logs unless we say “Hey Lambda, you can publish to CloudWatch!”.
Lambda is a service and not a human (duh), so the way services get permissions are through roles.
Roles basically say “if a service has this role it can do X, Y and Z”.
The “X”, “Y”, and “Z” are policies.
Policies say you can write to S3, query Redshift or publish to CloudWatch.
A role without policies can’t do anything, a role with policies attached can actually do things for us.
So in summary services have roles, which have policies.
Policies are what actually give permission to do things, so each role has policies that allow those roles to do cool stuff.
Not confusing at all, right?.You will get the hang of it eventually.
Step One: On the Configuration page go down to the role your Lambda function is using and click to view the role.
Your name will vary here from what mine says.
Step Two: Attach a new policy by clicking the “Attach Policy” button.
This will add more permissions to our Lambda function, which will allow it to access CloudWatch metrics.
Step Three: The role you need for CloudWatch metrics is called “AWSLambdaBasicExecutionRole”.
If you search for that role in the search bar, click the check mark to the left of the name (to apply it) and then click “Attach Policy” you will add this policy to your Lambda function.
Step Four: Now you have metrics in your Monitoring tab of your Lambda function!.This will now by default show you important metrics about your Lambda such as Invocations (number of times you function runs), Duration and Error Count.
Huzzah!Setting Up Automatic Emails On ErrorNow we have metrics going to our Monitoring page of our Lambda function.
This is an improvement from having no metrics, but this still leaves us in a situation where our function could fail and we wouldn’t notice, unless we are checking this chart regularly (and that’s annoying).
Ideally we only get a notification when our Lambda function fails and then we can go investigate what happened.
We can again rely on CloudWatch for this.
That last list of steps was so much fun let’s do it again!.Note this looks like a lot of steps but really should take you less than 10 minutes the first time and about 2 minutes each time thereafter.
Step One: Go to CloudWatch.
You can get there from the top Navigation by going Services -> Management & Governance -> CloudWatchStep Two: Click Alarms on the left panel and then click the “Create Alarm” button on the right.
Step Three: Click “Select metric” to choose the metric that will trigger our alarmStep Four: Choose “Lambda”Step Five: Choose “By Function Name”Step Six: Find your Lambda’s function name.
If you only have one Lambda it will be easy, but once you start building out your vast empire of serverless computing dominance this could get quite crowded.
You will see a few entries for each function name as there will be several options under Metric Name.
You can use multiple metrics to trigger an alarm, but what we want is a trigger on error.
Thus the Metric Name you want is Errors (duh).
Select that and then choose “Select metric” in the bottom left.
Step Seven: This step is so intense it gets two pictures!.It is time to configure what will actually trigger our alert.
There are many ways to trigger an alarm and this is just the method I use.
Notice on this screen you have the chart being used to trigger your alarm on the left.
In our case this is our error plot, which is at 0 as we don’t have any errors (huzzah!).
Anyways, there are three things we need to setup for this example: Conditions, Statistic and Period, all of which relate to that chart.
Let’s start with Condition.
This is the condition on the chart that we want to trigger our alarm.
Here we set this to be “Greater Than 0”.
Basically if our chart has a value more than 0 (more than 0 errors) I want to know about it.
Moving onto Statistic: in this case it really doesn’t matter due to our condition.
The average error count will be greater than 0 if we have more than 0 errors, as will the Maximum and Sum, so you can choose any.
This is more important if you are concerned with something like execution time where you may want to know if the average time exceeds some value.
The Period is the frequency with which your alarm is being measured.
Here we choose daily since our Lambda is triggering daily.
This is saying if the maximum value for a days period is greater than 0 then we want to know about it.
This is a good explanation of Period if you are still confused on this.
Combining these features, we have an alarm that is tracking data points daily for if the maximum value on the error chart is greater than 0.
In human speak if there is a data point over 0 for a daily period then ring the alarm bells.
Once this is entered click “Next”.
What is SNS?.The behemoth known as AWS has a lot of cool features, one of those in Simple Notification Service, or SNS.
Basically if you want to contact a human about something that happened on AWS you can use SNS.
Thinking of our current example, we want to email a human if something breaks.
This is one of many uses for SNS.
The way it works is you create “topics” like “My Lambda is aflame!!!” and people can subscribe to that topic.
Thus any time something is published to that topic the subscribers will get a notification.
Think of it as an AWS newsletter.
Step Eight: Time to create an SNS topic.
Basically this is how you can get an email, text or other forms of communication when this alarm goes off.
Give your alarm a name that indicates what this alarm is for.
If you only have one or two alarms it may be obvious what the alarm is for, but as you scale out and have dozens or hundreds of alarms it may not be so clear, so give it a descriptive name.
Then just add your email and click the “Create topic” button.
Note: You will get a confirmation email that you want to subscribe to the SNS topic.
You won’t get an email for an alarm until you confirm that.
This is to prevent evil people like me from signing up poor strangers for my AWS alarms.
Just joking, but you really do need to confirm or you won’t get your alarms and then you just wasted a bunch of time reading this long tutorial for nothing!Step Nine: Add a name and description for your Alarm.
This is included in the body of the email (sample above), so be sure to be descriptive here so you know what the hell your alarm is talking about.
Step Ten: Just confirm everything is set up properly and click “Create Alarm”.
Now you should be all set!Testing Your AlarmThere may be a more sophisticated way to do this, but generally I will go to my Lambda function and tell it to print some undefined variable on the first line.
Make that change, save your Lambda function, and click “Test”.
This will invoke your Lambda and when it fails you should get the alarm (it may take a few minutes).
After you see the alarm is working properly be sure to undo this intended mistake.
If you don’t get the alarm be sure to check that you confirmed the SNS topic subscription and that you set your threshold properly (Step 7).
What does my alarm look like?Assuming my instructions weren’t terrible, and you set up your alarm correctly, you will get an e-mail from AWS anytime an error occurs.
It will look similar to above with the only difference being in the region reported and the name swapped for whatever you called your function.
Inside the email you get some more details like when the error occurred, what Lambda function the error is tied to, and more.
SummaryWe can’t expect our code to always properly execute.
We also can’t be expected to be always checking it, especially if it is running hourly or even every minute.
Lambda powered with CloudWatch alarms makes it easy to know when something goes wrong so we can get in there and debug.
I highly encourage you to take advantage of CloudWatch alarms for maintaining happy and healthy data pipelines.
As always, feel free to connect with me on LinkedIn if you found this article useful or have any questions.
Happy pipelining!.. More details