When we want to remove old files from S3 automatically, we use the lifecycle rules, but I don’t recommend setting them using the AWS Web Interface because, in my opinion, the whole infrastructure should be defined as code.

Thus, it is best to add a Terraform configuration for the bucket we want to clean.

We will need three things:

  • the name of the bucket
  • the key prefix of files we want to remove
  • the number of days after which we want to clean the data

When we have all of that, we can define the lifecycle rule in Terraform:

resource "aws_s3_bucket" "bucket" {
    bucket = "bucket_name"
    acl = "private"

    lifecycle_rule {
        id = "remove_old_files"
        enabled = true

        prefix = "key_prefix/"

        expiration {
            days = 180

In this example, I configured the expiration rule that removes files older than 180 days from the bucket bucket_name, but it applies only to the files which keys start with the prefix key_prefix/.

Older post

How to retry a Python function call

How to retry a Python function call in case of an error

Newer post

How to make sure that you did not leave an EMR cluster running

How to get notifications about running EMR cluster