How to delay an Airflow DAG until a given hour using the DateTimeSensor
Airflow

How to delay an Airflow DAG until a given hour using the DateTimeSensor

How to use the DateTimeSensor in Airflow
Bartosz Mikulski
Bartosz Mikulski – 1 min read

The DateTimeSensor is another part of the Airflow API that seemingly makes no sense. Why would I want to wait until a specified time? For sure, there is little usage of configuration looking like this:

from airflow.sensors.date_time_sensor import DateTimeSensor

time_sensor = DateTimeSensor(
    task_id='wait_until_new_year',
    target_time='2021-01-01T00:00:00',
)

I could imagine only one situation when it is useful. For example, we may have a DAG that will execute only once when we notify the customers that we have just launched a new product. Such mailing will require a few preparation steps, so we can perform those tasks earlier and then wait until the launch date to send the emails.

However, the DateTimeSensor can be used with templates, and that makes it way more powerful. If I have a DAG that must wait until 10 am to execute some tasks, I can configure the sensor using the execution_date parameter:

time_sensor = DateTimeSensor(
    task_id='are_we_there_yet',
     target_time='{{ execution_date.replace(hour=10) }}', 
)

What is the advantage of this configuration? When I run a backfill for some past date, it will not get stuck at the time sensor (because the date has already passed). If I used TimeDeltaSensor, it would wait the specified number of minutes even during the backfills, which probably makes no sense.

Older post

How to run PySpark code using the Airflow SSHOperator

How to submit a PySpark job using SSHOperator in Airflow

Newer post

Send a Slack message from an Airflow DAG

How to use the SlackAPIPostOperator to send a templated message to a Slack channel

Related Posts

Airflow Prophet

Anomaly detection in Airflow DAG using Prophet library

How to detect problems in Airflow pipeline using Prophet for time series anomaly detection
Published on: 12 Feb 2021

Airflow

How to restart a stuck Airflow DAG

What to do when an Airflow DAG gets stuck and does not want to run
Published on: 31 Dec 2020

Airflow

Why does the DayOfWeekSensor exist in Airflow?

How to make an Airflow DAG wait until a specified day of the week
Published on: 30 Dec 2020

Are you looking for an experienced AI consultant? Do you need assistance with your RAG or Agentic Workflow?
Schedule a call, send me a message on LinkedIn. Schedule a call or send me a message on LinkedIn

>