Serverless Framework: Use Amazon S3 Object Lambda to resize images on the fly (Node.js and Sharp)

Richmond Upon Thames, by Garret Keogh on Unsplash

Goal

In this post, I will show you how to use Amazon S3 Object Lambda to resize images on the fly. The Serverless Framework will be used to define the Infrastructure as Code and to simplify the deployment. Sharp will be used to resize the images. Lambda will be written using the Node.js 14x Lambda runtime

What’s Amazon S3 Object Lambda

One of the most common Lambda patterns is to transform data stored inside Amazon S3. Generally, a lambda function is invoked after a file has been uploaded. Lambda would retrieve that file, apply any needed transformation (e.g. converting type of file) and store the result in S3.

That pattern was working well, however, it would require some work done onto a file despite that being accessed in the future or not.

If you needed to convert a file on the fly you should have created a Lambda function, invoke it via Amazon API GW and wait for the lambda to perform the transformation.

AWS has recently introduced Amazon S3 Object Lambda in a good post by Danilo Poccia. S3 Object Lambda allows creating a Lambda directly connected to the S3 bucket (using S3 Access Points) that is automatically invoked when retrieving the object from S3!

That means that our application needs only to send an S3 Get Object request to retrieve the original or transformed data

Also, a very important peculiarity of using Amazon S3 Object Lambda it’s that the file you want to retrieve doesn’t need to exist on S3! We will make use of this for our scenario

Note: High-level AWS CLI S3 commands (e.g. aws s3 cp) don’t currently support S3 Object Lambda, instead we need to use low-level S3 API commands (e.g. aws s3api get-object)

Common Use Cases

In his post, Danilo highlighted the most common use cases for Amazon S3 Object Lambda:

• Converting across data formats (e.g. XML to JSON)
• Compressing or decompressing files on the fly
• Resizing images on the fly (our use case!)
• Many more (left to the devs’ creativity)

Let’s get started!

Let’s focus on our use case of resizing images on the fly using S3 Object Lambda.

VSCode

Prerequisites

This tutorial takes into consideration a basic familiarity with the Serverless Framework and Node.js. If haven’t ever used it or you just want to refresh your mind, please have a look at my previous tutorial “Serverless Framework: Deploy an HTTP endpoint using NodeJS, Lambda on AWS”

Configuring Serverless

As the first step, we need to configure the infrastructure using the Serverless Framework. In particular, we need to:

• Create the S3 bucket
• Define the resizing Lambda
• Give our Lambda write permission to S3 and permission to use theWriteGetObjectResponse method
• Define the S3 Access Point
• Define the S3 Object Lambda Access Point

The IAM role for the Lambda MUST have the s3-object-lambda:WriteGetObjectResponse permission. However, it doesn’t need read permission as we are going to use a PreSigned URL coming from the context to read the requested image

Resizer Lambda

Secondarily we need to write the Resizer Lambda function

The logic we implement is as following:

• A user requests an image from S3 namedrichmond_600x400.jpg
• S3 automatically invokes the S3 Object Lambda
• Lambda attempts to download that file from S3
• If the file doesn’t exist, the Lambda look for the full-sized image named richmond.jpg and extracts the size from the requested image
• Lambda resizes the image to the size requested 600x400
• Lambda stores the resized image into S3
• Lambda returns the object using the newly introduced method WriteGetObjectResponse

Note: at the time the article has been written, there was a bug (#3675) in the AWS SDK that was causing an error while using the method WriteGetObjectResponse: 405 MethodNotAllowed: The specified method is not allowed against this resource . Because of this bug, the endpoint and service name were not resolved properly. As a temporary workaround we need to set the correct endpoint and prefix. This won’t be required once the bug is closed

Workaround for bug #3675

1/4/2021 Update: The #3675 bug has been fixed by AWS in version 2.876.0 of the aws-sdk-js, therefore the above workaround is no longer needed. I have updated all the scripts and repo to reflect that

Test

To test it we can write a simple python script that uses boto3 to get the image from S3

We can then run it with python3 test/object-lambda.py

Test result

We can check if that worked:

• In S3 a new file called richmond_720x480.jpg has been created

S3 console

• The same file has been downloaded in the local folder namedpictures

Local `pictures` folder

Conclusion

In this tutorial, we’ve learnt:

• What’s Amazon S3 Object Lambda
• Common use cases for S3 Object Lambda
• How to use S3 Object Lambda to resize images on the fly (stored in S3) using the Serverless Framework, Node.js and Sharp

Stay tuned for other tutorials about Serverless Framework!

Here’s the link to the GitHub repo