In this blog post, I will show you how to configure S3 bucket notification, AWS Lambda permission, and the Lambda trigger to receive a notification when a file is added to an S3 bucket. Handling the event in the Lambda function is out of the scope of this article. I will use Terraform to configure the notifications and permissions.
The first thing we have to do is configuring the bucket notifications. Note that it is impossible to define notifications that have overlapping filter prefixes when the filter suffix is the same!
resource "aws_s3_bucket_notification" "bucket-events" {
bucket = "bucket_name"
queue {
events = ["s3:ObjectCreated:*"]
queue_arn = aws_sqs_queue.queue_name.arn
filter_prefix = "file_key_prefix"
filter_suffix = "file_key_suffix"
}
}
Want to build AI systems that actually work?
Download my expert-crafted GenAI Transformation Guide for Data Teams and discover how to properly measure AI performance, set up guardrails, and continuously improve your AI solutions like the pros.
After that, we have to give the bucket_name bucket permission to send events to the queue, and the Lambda function needs permission to read the events:
resource "aws_sqs_queue_policy" "bucket-events-policy" {
queue_url = aws_sqs_queue.queue_name.id
policy = <<EOF
{
"Version": "2012-10-17",
"Id": "${aws_sqs_queue.queue_name.arn}",
"Statement": [
{
"Sid": "First",
"Effect": "Allow",
"Principal": "*",
"Action": "SQS:SendMessage",
"Resource": "${aws_sqs_queue.queue_name.arn}",
"Condition": {
"ArnEquals": {
"aws:SourceArn": "arn:aws:s3:::bucket_name"
}
}
},
{
"Sid": "First",
"Effect": "Allow",
"Principal": "*",
"Action": "SQS:ReceiveMessage",
"Resource": "${aws_sqs_queue.queue_name.arn}",
"Condition": {
"ArnEquals": {
"aws:SourceArn": "arn of the lambda function"
}
}
}
]
}
EOF
}
In the end, we have to add the SQS ARN as the source of the events in the Serverless configuration of the Lambda function:
# Put this in the function part in the Serverless configuration
events:
- sqs: 'SQS ARN'