When we want to remove old files from S3 automatically, we use the lifecycle rules, but I don’t recommend setting them using the AWS Web Interface because, in my opinion, the whole infrastructure should be defined as code.
Thus, it is best to add a Terraform configuration for the bucket we want to clean.
We will need three things:
- the name of the bucket
- the key prefix of files we want to remove
- the number of days after which we want to clean the data
When we have all of that, we can define the lifecycle rule in Terraform:
resource "aws_s3_bucket" "bucket" {
bucket = "bucket_name"
acl = "private"
lifecycle_rule {
id = "remove_old_files"
enabled = true
prefix = "key_prefix/"
expiration {
days = 180
}
}
}
In this example, I configured the expiration rule that removes files older than 180 days from the bucket bucket_name, but it applies only to the files which keys start with the prefix key_prefix/.
Want to build AI systems that actually work?
Download my expert-crafted GenAI Transformation Guide for Data Teams and discover how to properly measure AI performance, set up guardrails, and continuously improve your AI solutions like the pros.