Dependencies between DAGs: How to wait until another DAG finishes in Airflow?
Airflow Data engineering

Dependencies between DAGs: How to wait until another DAG finishes in Airflow?

How to trigger Airflow DAG when another DAG is completed
Bartosz Mikulski
Bartosz Mikulski – 2 min read

In this article, I am going to show how to set up dependencies between two DAGs. Imagine that I have a DAG that dumps data from production databases and another DAG that aggregates the raw data and pushes the result into a reporting database.

Table of Contents

  1. ExternalTaskSensor
  2. How to use sensors

I want the second DAG to run when the first one finishes, but I don’t want to move its tasks into the first DAG because that would make a mess in the configuration.

Airflow does not allow to set up dependencies between DAGs explicitly, but we can use Sensors to postpone the start of the second DAG until the first one successfully finishes.

ExternalTaskSensor

To configure the sensor, we need the identifier of another DAG (we will wait until that DAG finishes). Additionally, we can also specify the identifier of a task within the DAG (if we want to wait for a single task). If we want to wait for the whole DAG we must set external_task_id = None.

from datetime import datetime
from airflow import DAG
from airflow.operators.dummy_operator import DummyOperator
from airflow.sensors.external_task_sensor import ExternalTaskSensor

dag = DAG('dependency_dag', description='DAG with sensor', schedule_interval='* * * * *',
          start_date=datetime(2019, 7, 10))

sensor = ExternalTaskSensor(task_id='dag_sensor', external_dag_id = 'another_dag_id', external_task_id = None, dag=dag, mode = 'reschedule')

task = DummyOperator(task_id='some_task', retries=1, dag=dag)

task.set_upstream(sensor)

By default, the sensor waits for the successful execution of the dependency on the current day. We can also specify the status and time delta, so it is possible to run a dependency when a DAG fails or run it every hour. For details, look at the allowed_states and execution_delta parameters in the documentation.

How to use sensors

The sensor is just another type of task, so I create a new DAG which begins with a sensor. In the default configuration, the sensor checks the dependency status every minute.

After I configure the sensor, I should specify the rest of the tasks in the DAG. As I wrote in the previous paragraph, we use sensors like regular tasks, so I connect the task with the sensor using the upstream/downstream operator. I do it in the last line:

task.set_upstream(sensor)
Older post

How to run Airflow in Docker (with a persistent database)

How to configure Airflow in a Docker container

Newer post

Bellman equation explained

The fundamental equation of reinforcement learning

Related Posts

Data engineering Project management

Theory of constraints in data engineering

Are you busy, but nothing ever gets done? Perhaps, theory of constraints will help you
Published on: 25 Jun 2021

Airflow Prophet

Anomaly detection in Airflow DAG using Prophet library

How to detect problems in Airflow pipeline using Prophet for time series anomaly detection
Published on: 12 Feb 2021

Data engineering PySpark BDD

Testing data products: BDD for data engineers

How to use BDD to test PySpark code
Published on: 29 Jan 2021

Are you looking for an experienced AI consultant? Do you need assistance with your RAG or Agentic Workflow?
Schedule a call, send me a message on LinkedIn. Schedule a call or send me a message on LinkedIn

>