Accelerate
to market
and go live
in just weeks
Services
Digital & Application Innovation

Rearchitect your legacy systems and build new ​cloud-native applications.

Data & Intelligent Applications

Streamline data workflows and boost your applications with advanced intelligence.

Github Enablement

Optimize your development ​operations with our GitHub Accelerators.

Elevate
your debt
recovery
strategy
Products
CollectionsAI

Streamline the debt recovery process with a chatbot that delivers live calls, texts and emails. Escalate high risk calls to live agents with a co-pilot.

MetricSense

Optimize development with real-time metrics and seamless integrations. Enhance DevEx and drive cost-effectiveness using DORA and SPACE frameworks.

Trusted by leading
enterprise
brands
Case Studies
View All
Learn best practices & techniques from the #dreamteam
Don't just dream of the future... Build It!
Schedule a
call to
learn more

Please choose a time slot below to schedule a call with our experts to discover how we can assist you in reaching your goals!

Ready to get started?
Member Name
Title and role @ architech
Schedule a meeting to learn more...
Accelerate
to market
and go live
in just weeks
Elevate
your debt
recovery strategy
Case Studies
View All
Trusted by
leading enterprise
brands
Learn best
practices &
techniques from
the #dreamteam
Don't just dream
of the future...
Build It!

Please choose a time slot below to schedule a call with our experts to discover how we can assist you in reaching your goals!

Ready to get started?
Schedule a
call to
learn more
Schedule a meeting to learn more

Problem Statement

When using AWS lambda functions with Numpy dependencies, the following challenges are faced.

  • NumPy implementation fails on lambda
  • Python libraries can get huge and will cross the lambda hard limit.
  • NumPy is a sizeable library, and loading it during a cold start can impact function performance.

Lambda Functions

Amazon Web Services (AWS) offers the serverless computing service known as AWS Lambda. You can run code with Lambda without setting up or managing servers. To ensure high availability and dependability, it automatically grows and controls the computer resources required to run your code.

Approach 1

To incorporate Numpy dependencies into the Lambda function, contemplate using Lambda Layers to decrease the size of your deployment package. A Lambda Layer can be crafted with the required dependencies, and subsequently linked to your Lambda function.

Lambda Layers

AWS Lambda Layers are like a box where you can put the extra tools (libraries or dependencies) your Python function needs to work correctly. By using layers, you don’t have to include these tools directly in your function, which can make your function easier to manage and update. This way, if multiple functions need the same tools, they can all use the same layer instead of having their own copy. This is helpful when you want to keep your function lightweight, more organized, and easy to understand, especially when working in a team or on larger projects.

Tools required:

  • Terminal or CLI
  • Docker Desktop

To use Numpy libraries in your Lambda function, follow these steps to create, zip, and upload your Lambda Layers.

  1. Pull the Python 3.10 Docker Image from a Specific URL from AWS ECR using the command.
    “docker pull public.ecr.aws/sam/build-python3.10:1.84.0-20230517004040.”
  2. Run the Docker image and mount it to the current directory using the command.
    “docker run -it -v $(pwd):/var/task public.ecr.aws/sam/build-python3.10:1.84.0-20230517004040“
  3. Install all the required dependencies using the pip python package manager.
    pip install langchain openai numpy -t ./python
  4. Creating a Zip archive named python.zip with all these dependencies.
    zip -r python.zip ./python

Create a lambda layer using this zip file to create a custom lambda layer that can be attached to any lambda function with numpy dependency.

Lambda Function with Numpy dependency layer.
Lambda Function with Numpy dependency layer.

Advantages:

Streamlined Code Sharing:

Lambda Layers allow for the easy sharing of common code and dependencies across multiple Lambda functions, reducing redundancy.

Simplified Updates:

Updates to shared code can be made in a centralized layer, making it straightforward to propagate changes to multiple functions.

Improved Versioning:

Layers support versioning, enabling precise control over dependencies and facilitating backward compatibility.

Limitations:

Size Limitation:

Each layer has a maximum size limit (uncompressed) of 250 MB. This includes all files and libraries within the layer.

Number of Layers:

A Lambda function can reference up to five layers. If you reach this limit, you might need to consolidate layers or include dependencies directly in the deployment package.

Cold Starts:

Layers can contribute to cold start times, especially if large layers are involved. This is because Lambda needs to download and extract the layers before executing the function code.

Immutable Layers:

Once a layer version is created, it is immutable. If you need to make changes, you must create a new version of the layer. This can potentially lead to version management challenges.

To overcome these limitations ECR can be used to store the dockerized containers with all code, its dependencies, and the lambda function.

Approach 2

Alternatively, ECR can be used where the code and its dependencies are stored in a docker container where a docker image is built and stored in an ECR. The lambda function is created using this image with all its dependencies inside the image.

Tools required:

  • AWS CLI version 2
  • Python
  • Docker Desktop

Step 1:

  • Create a python file and name it as app.py
  • Write the Python code required including the def handler.

Step 2:

  • Create a requirements.txt file with all dependencies.

Step 3:

  • Create a Dockerfile.

Step 4:

  • Run the docker file using the “docker build” command.

Step 5:

  • Tag the image using the “docker tag.”
  • Push the image to the ECR using “docker push.”

Step 6:

  • Create a Lambda function using the ECR image with the lambda function and the Python code with all the numpy dependencies.

Using Amazon ECR (Elastic Container Registry) to create Lambda functions offers several advantages, especially when working with containerized applications. By leveraging ECR for your Lambda functions, you benefit from AWS’s managed container registry service, ensuring reliability, security, and ease of use in your serverless container-based applications. The whole series of steps can be automated using GitHub actions.

python code with lambda function definition.

AWS ECR with pushed docker images

AWS Lambda function using ECR image URL

Advantages:

Container Flexibility:

ECR supports containerized applications, providing flexibility in choosing runtimes, dependencies, and overall container environments.

Seamless Integration:

ECR integrates seamlessly with other AWS services, enabling the creation of end-to-end containerized solutions.

Network Efficiency:

Storing images within AWS infrastructure can lead to faster image pull times, reducing network latency.

Limitations:

Initial Setup:

Working with containers requires additional setup compared to Lambda Layers, including building and pushing images to ECR.

Cost Implications:

While ECR offers a cost-effective pricing model, consideration should be given to storage costs and data transfer fees.

Conclusion

In conclusion, the choice between using AWS Lambda Layers and Amazon ECR (Elastic Container Registry) for creating Lambda functions depends on specific use cases, requirements, and preferences. Both approaches offer unique advantages, and the decision should be based on factors such as ease of management, deployment flexibility, and the nature of your application.

Lambda Layers excel in simplicity and shared code scenarios, while ECR offers more flexibility with containerized applications. As you design your serverless architecture, carefully weigh the pros and cons to determine the most suitable approach for your use case.

Balaji Abiraman, DevOps Practice Area Lead
& Joseph Anil Tenneti, DevOps Engineer

You may also like