Build an AWS Serverless application with Python 3.
x, Zappa and DockerPatrick RyanBlockedUnblockFollowFollowingApr 29Zappa.
io is a great Python tooling for creating AWS serverless applications.
From Lambda development, to event handlers, to deploying WSGI applications.
You can deploy a Django or Flask application as a Lambda with an API gateway endpoint and never have to worry about configuring an EC2 instance nor scaling or load balancing.
Is this suitable for every application — well — no.
But for a great number applications it is great.
The tricky part of using Zappa to deploy the application, is when one of the third party libraries that your application depends on has an environment specific implementation.
Some Python libraries have implementations that have ‘c’ language bindings.
Zappa has some third party libraries already compiled and ready to be bundled with your application that will run on AWS Linux, however Zappa does not have all third party libraries.
For any library that you need for which there is no pre-built Zappa AWS version, then you will have to make sure you build the Zappa distribution in a Docker environment that reflects the AWS Linux, Python 3.
6 or 3.
7 environment.
Today I am going to show you how to create an AWS serverless application using Zappa to manage a Lambda function and the event handlers.
One of the event handlers relies on a third party library that must be built on the AWS linux Python 3.
6.
x environment.
This post does assume some familiarity with Zappa so I encourage you to read the Zappa.
io documentation.
It also assumes you have Docker installed for your computer and you have configured the AWS CLI.
My work computer is a Mac, and therefore some of these instructions will need to be modified for a Windows environment.
The Github repo for the code that I am going to use can be found here.
When building applications with Zappa.
io, Zappa will provide the Lambda handler (defaulted to handler.
lambda_handler) and based on the configuration of the application the Zappa supplied Lambda handler knows how to route the function to the appropriate event handler.
In this way the Zappa supplied Lambda handler acts as the controller and delegates the event handling based on the zappa_settings.
json file.
The sample application has 2 CloudWatch events.
One that is triggered every minute and another every 2 minutes.
Each CloudWatch event is handled by a different function.
The CloudWatch event handler that runs every 2 minutes, will put a message into a SQS queue.
There is another event listener on the SQS queue, which is part of the same application, but in a different Python file, that will receive the message as the event and make an SFTP connection to a remote SFTP server and get the directory listings.
The application was put together to demonstrate the capabilities and not to necessarily be useful — it is for illustrative purposes only.
Please note, to run this code, you do have to create the SQS queue before deploying.
Sample Application ArchitectureDockerTo build the serverless application, we have to install Docker.
AWS provides Docker images that encapsulate the AWS Linux environment for many of the languages it supports.
You can find more information here.
As sample docker command to start the AWS Linux Python 3.
6 environment is below.
docker command to run Python 3.
6 environmentThis script uses Python 3.
6 image, but if you prefer to use Python 3.
7, just change the Python version to 3.
7.
Notice we are setting environment variables in the docker shell to hold the AWS credential information.
It does not matter how the environment variables are created in the docker environment.
Any method you like to setup the environment variables in the docker shell will work.
As one example of setting up the environment variables, before running the above docker command in a terminal or shell window, you will need to export the environment variables for AWS region, access key and secret key.
export AWS_SECRET_ACCESS_KEY=<your secret access key here>export AWS_ACCESS_KEY_ID=<your access key here>export AWS_DEFAULT_REGION=<your region here>Once you started the Docker image, you are running in the new AWS Linux environment.
This environment is completely separate from your host computer environment.
The reason we set environment variables for the AWS region, access key, secret key is because the Docker shell does not have access to these values.
GOTCHA: One gotcha that you need to be aware of is that if you are someone that likes to put a virtual environment (i.
e.
venv or a .
venv ) in the same folder as your Python project, I have noticed that the Zappa tooling does not like this.
I recommend for your Zappa projects that you keep your virtual environment outside of the project.
After you set your environment variables, and have run the docker command, your shell will have a ‘bash-4.
2’ prompt.
Typing: python –version will show you 3.
6.
1 for the Python3.
6 environment.
Because this is a separate environment from your host computer, you will have to setup a new virtual python environment.
I recommend the following command while in the Docker shell:python -m venv aws_venvYou will only have to do this once.
When you leave the docker shell, the created directory with the environment is left intact.
Once that is created execute (assumes Mac):source aws_venv/bin/activateand then pip install the dependencies:pip install paramikopip install zappaBy installing these libraries while in the Docker shell, which is an AWS Linux environment, this will make sure the version of the paramiko library is compatible with the Linux environment.
ZappaOnce the Python environment is setup we start to look at Zappa.
{ "comp1": { "project_name": "comp1", "runtime": "python3.
6", "s3_bucket": "my-zappa-deployments", "timeout_seconds": 120, "memory_size": 1024, "use_apigateway": false, "xray_tracing": true, "events": [ { "function": "comp1.
RecurringEvents.
recurring_event_handler", "kwargs": {"key1": "value1", "key2": "value2"}, "expression": "rate(1 minute)" }, { "function": "comp1.
RecurringEvents.
every_2_mins", "kwargs": {"queuename": "zappa-test-queue", "key2": "value2"}, "expression": "rate(2 minutes)" }, { "function": "comp1.
SqsSubscriberComp1.
sqs_listener", "event_source": { "arn": "arn:aws:sqs:us-east-1:888877776666:zappa-test-queue", "batch_size": 1, "enabled": true } }], "delete_s3_zip": false, "log_level": "ERROR", "exclude": [ "comp2" ] }}I am not going to go over the entire configuration but instead just point out some highlights.
First, notice that we actually are not specifying a Lambda.
In this case, we are going to take the Zappa default lambda implementation that knows how to delegate to our event handlers, e.
g.
comp1.
RecurringEvents.
every_2_mins.
The events are defined in the ‘events’ section of the json configuration.
In there we setup 2 CloudWatch event handlers, and 1 SQS event handler.
The functions are just normal Python functions that take an event and context, just like a lambda handler would.
Notice we do not specify the AWS environment to build this to.
By not specifying a ‘profile’ configuration name, Zappa will use the environment variables that we setup when we created our Docker environment.
To deploy this application we execute:zappa deploy comp1Zappa will build a deploy zip file of all of the application code and dependencies from the virtual environment, and setup the AWS components.
When it is done if you go to the AWS console you will see something like:AWS Console View after Zappa DeployTo undeploy the application and the events:zappa undeploy -y comp1If you view the CloudWatch logs you will see the 2 CloudWatch event handlers executingConclusionThis post was meant to show at a high level, how to build an AWS serverless application with Zappa and Docker where the application relies on a third party library that must be compatible with AWS Linux.
The steps are essentially:Start Dockerexport AWS environment variables so they can be used in the Docker shellstart docker image for AWS and Python 3.
6 (or python version of your choice)Create Python virtual environment in your docker shell (Only required once)Activate the new python virtual environmentpip install necessary dependencieszappa deploy and zappa undeploy.