How to add dependencies to AWS lambda
AWS Serverless

How to add dependencies to AWS lambda

How to deploy an AWS Lambda with dependencies
Bartosz Mikulski
Bartosz Mikulski – 2 min read

The process of adding dependencies to an AWS Lambda consists of two steps. First, we have to install the dependencies in the source code directory. Later, we have to package our lambda function into a zip file that also contains all of the dependency files.

Table of Contents

  1. Installing dependencies
  2. Packaging the lambda function

As an example, I will install the Python Scrapy dependency because, in addition to pip packages, it also requires some Linux dependencies.

Installing dependencies

There is one little problem with installing Lambda dependencies. Lambdas run on a special Linux distribution made by AWS, and we need the dependencies compiled for that particular distribution.

Because of that, I am going to prepare a Docker image in which I will install all of the required packages and then copy them to the local directory that contains the lambda function.

First, I must create a Dockerfile.

FROM amazonlinux

RUN yum update -y && \
    yum install -y python3 pip3 libxml2 libxslt && \
    pip3 install -t /deps_source lxml scrapy

CMD cp -r /deps_source/* /deps

When I build the Docker image using that file, it will install all of the required packages and store them in a directory inside the image.

docker build - t lambdadeps

The only thing left to do when I run the image is copying the already installed dependencies to a directory that was mounted as a Docker volume.

docker run -v $(pwd)/deps:/deps lambdadeps:latest

That will copy the Python dependencies for Amazon Linux into my local directory.

Packaging the lambda function

Now, I can write a Make script that packages the lambda function and its dependencies. Note that, I have the dependencies in the deps directory, and the lambda function is in the test_lambda.py file.

build:
    docker run -v $(pwd)/deps:/deps lambdadeps:latest
    cd deps && zip -r9 function.zip .
    mv deps/function.zip function.zip
    zip -g function.zip test_lambda.py

You MUST use tabs for indentation in the Makefile. If your IDE replaces tabs with spaces, it will not work! If you copy-paste my code, you have to replace four spaces with a tab.

When the Makefile is ready, I can run the make build command to package the lambda and all its dependencies into a zip file.

After that, I have to deploy the function on AWS. The process depends on the deployment of choice, and it is not in the scope of this article.

Older post

Four books to boost your programmer career

I quit my dream job because of a book

Newer post

10x software architecture: high cohesion

How to achieve high cohesion and a few common problems.

Related Posts

MLOps AWS

How to deploy a Transformer-based model with custom preprocessing code to Sagemaker Endpoints using BentoML

Deploy a machine learning model with custom inference code to a Sagemaker Endpoint using BentoML
Published on: 01 Oct 2021

AWS MLOps

Multimodel deployment in Sagemaker Endpoints

How to deploy multiple models in a single Sagemaker Endpoint?
Published on: 28 May 2021

AWS

How to deploy a REST API AWS Lambda using Chalice and AWS Code Pipeline

How to create a REST API Endpoint using AWS Lambda, Chalice, and AWS Code Pipeline
Published on: 16 Apr 2021

Are you looking for an experienced AI consultant? Do you need assistance with your RAG or Agentic Workflow?
Schedule a call, send me a message on LinkedIn. Schedule a call or send me a message on LinkedIn

>