Use LatestOnlyOperator to skip some tasks while running a backfill in Airflow
Airflow

Use LatestOnlyOperator to skip some tasks while running a backfill in Airflow

How to skip some tasks when backfilling a DAG in the past
Bartosz Mikulski
Bartosz Mikulski – 1 min read

Occasionally, when we use Airflow, we have a DAG which always works on the most recent snapshot of data even if we run a backfill for days in the past. It happens when at least one of the tasks downloads a snapshot of the current state from an external API, uploads data into another service (using an API, FTP upload, sending events to a message queue, or anything else), or pushes some information to users.

The last case is the worst. Certainly, we don’t want to send old versions of newsletters to all of the subscribers just because we had to backfill some values in a report.

How do we make sure that such time-sensitive tasks are executed only in the most recent DAG run and don’t start when we run a backfill? We can do that by using the LatestOnlyOperator in Airflow!

It is simple to use it. All we need to do is importing the operator, creating a new instance, and adding it to the DAG:

from airflow.operators.latest_only_operator import LatestOnlyOperator

is_this_latest_dag_run = LatestOnlyOperator(task_id="dont_send_newsletters_during_backfills", dag=dag)

upstream_task >> is_this_latest_dag_run >> send_newsletters >> other_downstream_task

When the operator runs, it checks whether the current time is between the most recent execution time and the next scheduled execution time. If yes, it lets the downstream tasks run. If no, it means that we are running a backfill, and the downstream tasks get skipped.

Older post

Christopher Bergh - How the DataOps principles help data engineers make data pipelines trustworthy

An interview with Christopher Bergh who explains how the DataOps principles help data engineers make data pipelines trustworthy

Newer post

What is the difference between a transformation and an action in Apache Spark?

What is an action in Apache Spark? What do you understand as transformations in Apache Spark?

Related Posts

Airflow Prophet

Anomaly detection in Airflow DAG using Prophet library

How to detect problems in Airflow pipeline using Prophet for time series anomaly detection
Published on: 12 Feb 2021

Airflow

How to restart a stuck Airflow DAG

What to do when an Airflow DAG gets stuck and does not want to run
Published on: 31 Dec 2020

Airflow

Why does the DayOfWeekSensor exist in Airflow?

How to make an Airflow DAG wait until a specified day of the week
Published on: 30 Dec 2020

Are you looking for an experienced AI consultant? Do you need assistance with your RAG or Agentic Workflow?
Schedule a call, send me a message on LinkedIn. Schedule a call or send me a message on LinkedIn

>