Going Serverless with AWS Lambda Functions — 2Ankit Prakash GuptaBlockedUnblockFollowFollowingJan 13As discussed earlier, Lambda Functions can be written to perform complex operations as well which might require multiple libraries or modules.
For example — Extracting face from images using MXNET, downloading images using links.
For these use cases, we will need multiple libraries packed in the execution environment, as coding it from the scratch will be very complex.
So, to deploy those we will have to provide the required libraries to the lambda instances since, lambda instances only have specific libraries to work with.
One more thing to notice is AWS Lambda Instances are actually Amazon Linux based instances, which are different from Ubuntu environments and might have different codes for different modules.
Due to this reason, a lot of people have faced import issue of PIL library with python while using it in Lambda Functions.
To overcome this issue, I would suggest using docker to build runtime environments for scripts and installing libraries and testing it on the same environment.
I would brief you about the usage of docker in downloading modules for Amazon Linux instances.
Post that, we will see how to form Custom Layers and configure layers for AWS Lambda Functions along with creating a lambda function triggered using Object Creation on S3 Bucket and downloading images using data in the files.
Starting off with using docker to download modules for Amazon Linux Instances.
Docker facilitates containerization or operating-system-level virtualization.
We first need to install docker.
To install docker on Ubuntu (Xenial or Bionic) and to know more about basics of Docker, please refer to Virtualizing Cuda Environment Using Nvidia-Docker and Deploying APIs :After installing docker, we will have to run a docker’s container of AWS Lambda Instance and run its shell.
sudo docker run [ –rm] -it -v <code_dir>:/code lambci/lambda:<runtime> shLet us discuss more about the available Lambda Function Runtime here :lambci/lambda:<runtime> : Image of Lambda Instance, we don’t need to worry about the image being locally available or not, we just need an active internet connection and the docker image will automatically be installed.
We can choose from the following list of the Lambda Instances runtime options.
build-nodejs8.
10build-nodejs4.
3build-nodejs6.
10build-python2.
7build-python3.
6build-python3.
7build-ruby2.
5build-go1.
xbuild-java8build-dotnetcore2.
0build-dotnetcore2.
1For python3.
6 runtime environment , I will run the following command.
sudo docker run — rm -it -v /path/to/code/:/code/ lambci/lambda:build-python3.
6 sh4.
Now after running into the interactive shell, we would use pip3 to download the modules needed.
To download the PIL and requests module we would run the following command in the Lambda’s shell.
pip3 install pillowpip3 install requestsNow, since we have already installed the requirements, we will have to locate the required modules and copy those into the /code/ directory.
By default, most of the downloaded packages are downloaded in /var/lang/lib/python3.
6/site-packages/ directory.
We will copy the PIL and requests module along with their dependencies in the /code/ folder using the following commands which we can also access after closing the docker container using ubuntu from /path/to/code/ directory.
cp -r /var/lang/lib/python3.
6/site-packages/PIL /code/cp -r /var/lang/lib/python3.
6/site-packages/Pillow-5.
3.
0.
dist-info /code/cp -r /var/lang/lib/python3.
6/site-packages/requests /code/cp -r /var/lang/lib/python3.
6/site-packages/urllib3 /code/cp -r /var/lang/lib/python3.
6/site-packages/urllib3–1.
24.
1.
dist-info/ /code/cp -r /var/lang/lib/python3.
6/site-packages/chardet /code/cp -r /var/lang/lib/python3.
6/site-packages/chardet-3.
0.
4.
dist-info/ /code/cp -r /var/lang/lib/python3.
6/site-packages/idna /code/cp -r /var/lang/lib/python3.
6/site-packages/idna-2.
7.
dist-info/ /code/cp -r /var/lang/lib/python3.
6/site-packages/certifi /code/cp -r /var/lang/lib/python3.
6/site-packages/certifi-2018.
11.
29.
dist-info/ /code/Now, after having the required modules in a defined directory, we have correctly downloaded the required libraries.
Now, we will focus on forming Custom Layers on AWS.
After logging into AWS Console and opening the Lambda Dashboard.
Open the Layers Dashboard and click on “Create Layer”.
Subsequently, we will be on the following screen and as you can see, we need to fill in the name of the Layer, description of the layer then select compatible runtimes and upload a zipped file having the modules.
But, the modules or the libraries need to be in a specific structure.
Let us go through that as well.
Layer Creation ScreenTo include libraries in a layer, place them in one of the folders supported by your runtime mentioned below :Node.
js : nodejs/node_modules, nodejs/node8/node_modules (NODE_PATH)Python : python, python/lib/python3.
7/site-packages (site directories)Java — java/lib (classpath)Ruby — ruby/gems/2.
5.
0 (GEM_PATH), ruby/lib (RUBY_LIB)All — bin (PATH), lib (LD_LIBRARY_PATH)We will create a directory in the above mentioned format.
Firstly, we will have to change the read-write permissions of the /path/to/code/ folder using the following command as we were using docker with sudo permissions.
So, either we will have to change the permissions of the folder or switch to a user with sudo (superuser do) permissions.
sudo chmod 777 -R /path/to/code/Post which, we will create a folder inside the /path/to/code/ named python.
mkdir /path/to/code/pythonMove all the folders in the python directory, except the python directory itselfcd /path/to/codemv !(python) -t .
/python/After pushing all these libraries in the above directories, zip the folder and upload it on the AWS Layer Creation Screen.
We can use the following command to do that.
Download zip in Ubuntu.
sudo apt-get install zipRun the following commands to zip the foldercd /path/to/codezip -r ~/lambda-layer_1.
zip pythonNow, Click on the Upload button on the Create Layer Screen, Browse and select the lambda-layer_1.
zip file and then click on Create.
We have created the layer successfully.
We can also create versions of single layer and can also download the package of the layer created.
Now, Carrying forward with creating a Lambda Function which downloads images provided a list of Youtube’s Channel Ids and their profile picture link as a file on S3 and uploads the images in a compressed file on S3.
For that let us create a role for lambda functions, which allows lambda function to create Cloudwatch Logs and gives full access of S3.
Role Creation ScreenOpen IAM service from AWS Console and select Roles.
Click on Create Role and choose Lambda as the service which will use this role and click on Next:Permissions.
Role Review ScreenNow, select the required policies for Full Access to S3 (AmazonS3FullAccess) and Lambda Cloudwatch Logs Creation Policy (AWSLambdaBasicExecutionRole).
And click on Next:Tags and can provide Key-Value Pairs, we will skip tags for now and would click on Next:Review.
Eventually, Providing name and description about the Role and checking the Policies in the Role, we will click on the Create Role Button to complete the Role Creation Process.
Now we will create a Lambda Function, using the above created role and the layer.
After clicking, Create Function button select S3 Trigger and select Object Creation Event and give the Prefix Path at which you are going to upload the input file and click Add.
S3 Trigger ConfigurationNow, we will configure Layer to the Lambda Function, Click on the Layers on the Lambda Designer and then click on Add a layer.
Select the relevant layer and its version and click Add.
Configuring Layer to a Lambda FunctionEventually, change the code in the inline editor.
Use the following code to download the images in a zipped format, change the Timeout and CPU Memory Settings and save the lambda function.
Now, to test the Lambda Function Upload a file on Target S3 Bucket having json dumped in every line.
I have manually copied links of profile pictures of several people and have dumped them in this file.
1546760133{"thumbnail":"https://yt3.
ggpht.
com/a-/AN66SAwK4vgc4xY4WEkqBN-L5PAMxMk78S7N9FjL_Q=s800-mo-c-c0xffffffff-rj-k-no","num":"1"}{"thumbnail":"https://yt3.
ggpht.
com/a-/AN66SAyXDERjWHJoIOMzk3oGIb9AW4epd59EIZoV3w=s800-mo-c-c0xffffffff-rj-k-no","num":"2"}{"thumbnail":"https://yt3.
ggpht.
com/a-/AN66SAxTMaIX7mBvmGmHAdYhZe1C9UuqmH6TWG5nxQ=s800-mo-c-c0xffffffff-rj-k-no","num":"3"}{"thumbnail":"https://yt3.
ggpht.
com/a-/AN66SAwaRnCUkcxfZW1LkRPQNqnrNmaTGWWuSyEvSA=s800-mo-c-c0xffffffff-rj-k-no","num":"4"}{"thumbnail":"https://yt3.
ggpht.
com/a-/AN66SAwgXwU1xHh7-WauvK-4OY7InqChVF-nKSOB0A=s800-mo-c-c0xffffffff-rj-k-no","num":"5"}{"thumbnail":"https://yt3.
ggpht.
com/a-/AN66SAxtsoYVvKUkLEokTKHi9KznmpeXRlWIzXyhBQ=s800-mo-c-c0xffffffff-rj-k-no","num":"6"}Post uploading, we will see a new directory “output_files” created in the lambda_test directory, where the zipped images and error files will be uploaded on S3.
I hope this might have given you a complete overview of how AWS Lambda Function works and how you can easily develop Lambda Functions having different libraries and solve your use cases easily.
I hope you find this article informative and easy to learn, if you have any queries feel free to reach me at info.
ankitp@gmail.
com.