Serverless: Using python libraries in AWS Lambda with layers

AWS Lambda functions using python are very powerful but without the full set of libraries available to python it can be frustrating. You can overcome this frustration and leverage the full set of available libraries with layers in Lambda. Jason DeLano of Bethesda explains the process to import layers into AWS Lambda. Special thanks to my colleague and friend Rory who put me on to using layers with Lambda.

The following assumptions are made for readers of this story.

1. You know how to use the basics of AWS Services (Lambda/IAM)
2. You know the basics of python and writing python functions
3. You know the basics of Docker and how to create a Docker container and have Docker Desktop on your local machine

The goal of the article is to show the reader how to add python libraries into a layer in Lambda so that you can use your desired libraries in the functionality in your code. The reader will come across this article if they are trying to use a python library that is not included in the default AWS Lambda container install for python. The reader probably ran into the following error

“errorMessage”: “Unable to import module ‘lambda_function’: No module named ‘pandas’”,

In this article we will walk you the process of importing the Panda’s python library into a layer using a docker to build your .zip file. From our experience if you do not use docker you can run into version challenges with python running on a mac or a windows machine not being compatible with the linux version Lambda is utilizing. Therefore using a docker image using a linux library compatible with Amazon Linux is the most effective approach to build your python .zip file.

AWS Documentation recommends “You build your layer code into a .zip file archive using the same procedure that you would use for a function deployment package”. AWS Documentation explains how to add layers in the following article.

Creating and sharing Lambda layers

How you got here

You were most likely trying to create a lambda function and reference the pandas or another library. After you tried to import the library in your code you realized that pandas is not included in the base container image that Lamba uses for python

Creating the Lambda function

We will start by creating a new lambda function through the console. In this example we will use Python 3.8. Other versions of python are available at this time and new version will be released in the future so I will add comments at the end on the slight changes for this document using newer versions.

After the function is created and you try to use the pandas library with injecting the following code into the Lambda Code

You will now need to run a test, before you can run the test you will need to create a test event. You can create a quick test event by clicking the Test button. If you don’t have an test event created you will need to configure a test event.

Run the test and you will get an error.

[ERROR] Runtime.ImportModuleError: Unable to import module ‘lambda_function’: No module named ‘pandas’

What do we do now to make it work

In order to build the layer with the pandas library we have found the most effective way is to build a docker container to extract the image so it is consistent across platforms on your client.

Create the Dockerfile to build the desired .zip

Create a new Docker file on your local machine where you have docker installed.

In the same directory where you have the Dockerfile run the following docker commands

You will now have the .zip file needed to create the layer.

Create the layer in AWS Lambda

Launch the AWS Lambda Service and click on the “Layers” action on the menu

Then create a new Layer by clicking the “Create Layer” button

This will allow you to create your new layer. Provide a Name, Description and select your file from your local machine with the pandas.zip. The press the create button.

At this point you will get a confirmation screen with details of the layer you added, copy the ARN in the top right.

Let’s go back to our Lambda function we created earlier and we will click on the Layers section.

After you click on the layers it will scroll you to the bottom of the screen where you can add a layer to the function

You will need to use the ARN which you grabbed from the confirmation page on the layer. You will select the Layer source of “Specify an ARN” and then key in the ARN and then hit the Verify button. After it is verified press the Add button.

If you are very familiar with the AWS Cli and running commands that way versus the console the following article outlines generating layers through the command line tools.

Creating Python AWS Lambda Layers with Docker

After the layer is added you can run the test again and be able to leverage the pandas library. You will see a successful response.