Serverless: Using python libraries in AWS Lamba. Preventing unable to import module ‘lambda_function’

This story takes you through the process of importing and using python libraries into AWS Lamba. Lamba by default includes a set of python libraries but you will often run into deploying python code to Lambda that reference a library that is not there. For example you try to execute your Lambda function and you get the following error message

"errorMessage": "Unable to import module 'lambda_function': No module named 'pymongo'",

This story will take you through how to resolve the error message and proceed with using AWS Lambda functions. I will walk you through creating a Lambda function using a python library that is not in the default python libraries for AWS Lamba. Then I will test and verify the function works properly.

The following assumptions are made for readers of this story.

1. You know how to use the basics of AWS Services (Lamba/S3/IAM)
2. You know the basics of python and writing python functions
3. You know the basics of Docker and how to create a Docker container and have Docker Desktop on your local machine
4. Able to spin up a Docker image with MongoDB or use Mongo Cloud to connect to Mongo instance

I’ll start the story by explaining my goal. In this use case I am looking to upload a file to AWS S3 bucket and then create a record in MongoDB using AWS Lambda function leveraging python. For this use case I will use an existing python library called pymongo. As a personal preference I use Jupyter Notebooks to write and test my python code, but feel free to use your preferred method. After I write and test my function in Jupyter Notebooks, I will build a deployment zip file in Docker, then deploy it to AWS Lambda and and then test the Lambda function.

Step 1: Create the code you would like Lambda to execute.

Note: I have blanked out my mongo password and host name. Please replace with your own username/password and MongoDB hostname. Before you start don’t forget to install the python library you are using with

pip install pymongo

JupyterNotebook to verify code is working

In order to call the my new function from Lambda’s python I will need to define a function called lambda_handler and pass to it an event and context

def lambda_handler(event, context):

In this example I will take the file that is uploaded to S3 bucket and write the name of the file and the bucket it arrived in to a MongoDB database.

First I need to build the lambda_function.py file that I will use. I am saving this file in a directory labeled “code” this is more for organizational purposes when I launch the Docker container script that is dependent on the location. I will combine code snippet for the function I tested in Jupyter with the lambda_handler function. The code I used is below for the lambda_function.py file.

lambda_function.py

After I’m done creating the file I can verify the path and the contents of the file.

Step 2: Create the Dockerfile to build the desired zip file.

The code snippet below is the “Dockerfile” that I will run on my laptop to generate the “zip” file that will be uploaded to AWS Lamba.

I’ll walk you through the Dockerfile code. I start with the “ubutu:20.04” image, I can plug in which ever image I want to start with but, I will need to verify that I’m staring with a base image that can install python3-pip, zip. I then create a function directory and copy the lambda_function into the container. I then install python3-pip, zip and pymongo. Make sure to create your “Dockerfile” in the folder that has the subdirectory “code”.

FROM ubuntu:20.04
RUN mkdir function
RUN cd function
COPY code/lambda_function.py function/lambda_function.py
WORKDIR /function
RUN apt-get update
RUN apt-get install python3-pip -y
RUN apt-get install zip -y
RUN pip install -t . pymongo
RUN zip -r linux-lambda.zip *

Step 3: Execute the Docker build, create, and copy command to create the zip file

First I will build the Docker container from the terminal window on my Mac in the directory I created the “Dockerfile” in.

docker build -t genzipfile .

Then I will create the Docker container

docker create -ti --name tempforzip genzipfile

Then I will copy the file from the Docker container to my local directory

docker cp tempforzip:/function/linux-lambda.zip /Users/jdelano/Documents/linux-lambda.zip

Then Cleanup the Docker container

docker rm -f tempforzip

Step 4: Create and Test my Lambda Function

First I will Login to my AWS Account and create a new Lambda Function. I’m going to use a blueprint for this example so I do not need to play around with IAM roles as it will create them and the S3 Trigger.

AWS Lambda: Create Function

I will name my function and role and then select an S3 bucket to trigger the action off of. If I don’t have an S3 bucket I will need to create one ahead of time. The scroll to the bottom of the screen and press the “Create Function” button

AWS Lambda: Create Function part 2

This will create a sample lamba_function.py but I already have one that I will use in the zip file. On the right side of the screen click the “Upload from” drop down list and select .zip file. I will take the file that I generated in Step 3 and upload it here.

I will now see that you code is successfully deployed with the library folders created. If the zip file is to large I will not be able to see the full folder structure but if it is smaller than 10MB I should see the text and be able to edit my function in-line. Otherwise I will need to edit the code in an editor on my laptop and redo the zip each time I need to change it.

Final Results

If I hit the “Test” button and create a new Test I can verify the function is working. After I verify it through the Lambda function screen I can also test by uploading a file into the S3 bucket that I put the trigger on.

MongoDB Compass: Output Example