In this blog post, I will show you how to configure S3 bucket notification, AWS Lambda permission, and the Lambda trigger to receive a notification when a file is added to an S3 bucket. Handling the event in the Lambda function is out of the scope of this article. I will use Terraform to configure the notifications and permissions.
Table of Contents
The first thing we have to do is configuring the bucket notifications. Note that it is impossible to define notifications that have overlapping filter prefixes when the filter suffix is the same!
resource "aws_s3_bucket_notification" "bucket-events" {
bucket = "bucket_name"
queue {
events = ["s3:ObjectCreated:*"]
queue_arn = aws_sqs_queue.queue_name.arn
filter_prefix = "file_key_prefix"
filter_suffix = "file_key_suffix"
}
}
Get Weekly AI Implementation Insights
Join engineering leaders who receive my analysis of common AI production failures and how to prevent them. No fluff, just actionable techniques.
After that, we have to give the bucket_name bucket permission to send events to the queue, and the Lambda function needs permission to read the events:
resource "aws_sqs_queue_policy" "bucket-events-policy" {
queue_url = aws_sqs_queue.queue_name.id
policy = <<EOF
{
"Version": "2012-10-17",
"Id": "${aws_sqs_queue.queue_name.arn}",
"Statement": [
{
"Sid": "First",
"Effect": "Allow",
"Principal": "*",
"Action": "SQS:SendMessage",
"Resource": "${aws_sqs_queue.queue_name.arn}",
"Condition": {
"ArnEquals": {
"aws:SourceArn": "arn:aws:s3:::bucket_name"
}
}
},
{
"Sid": "First",
"Effect": "Allow",
"Principal": "*",
"Action": "SQS:ReceiveMessage",
"Resource": "${aws_sqs_queue.queue_name.arn}",
"Condition": {
"ArnEquals": {
"aws:SourceArn": "arn of the lambda function"
}
}
}
]
}
EOF
}
In the end, we have to add the SQS ARN as the source of the events in the Serverless configuration of the Lambda function:
# Put this in the function part in the Serverless configuration
events:
- sqs: 'SQS ARN'