How to retrieve the statuses of the recent DAG executions from Airflow database
Airflow

How to retrieve the statuses of the recent DAG executions from Airflow database

How to make a dashboard that displays Airflow DAG statuses
Bartosz Mikulski
Bartosz Mikulski – 1 min read

If you want to create a nice dashboard that displays the statuses of the most recent Airflow DAG runs, you will need to retrieve them from the Airflow database:

SELECT
    dag_run.dag_id,
    dag_run.state
FROM dag_run
INNER JOIN (SELECT
                dag_id,
                MAX(execution_date) AS date
            FROM dag_run
            GROUP BY dag_id) mx
    ON
    dag_run.dag_id = mx.dag_id
    AND dag_run.execution_date = mx.date
JOIN dag ON dag.dag_id = dag_run.dag_id AND is_active = 1 AND is_paused = 0

This query retrieves the most recent DAG run and returns its id and state. Additionally, it filters out the paused DAGs, so we don’t pollute the results with things that don’t run in production right now.

Of course, creating a dashboard of Airflow DAG runs takes some time, so you may prefer to use the project we have open-sourced: https://github.com/Wikia/discreETLy. In addition to DAG execution times and statuses, it also displays Athena tables’ last update time and their descriptions.

Older post

How to find and terminate an idle Redshift session

How to find the idle session that is blocking the connection pool in Redshift

Newer post

Christopher Bergh - How the DataOps principles help data engineers make data pipelines trustworthy

An interview with Christopher Bergh who explains how the DataOps principles help data engineers make data pipelines trustworthy

Related Posts

Airflow Prophet

Anomaly detection in Airflow DAG using Prophet library

How to detect problems in Airflow pipeline using Prophet for time series anomaly detection
Published on: 12 Feb 2021

Airflow

How to restart a stuck Airflow DAG

What to do when an Airflow DAG gets stuck and does not want to run
Published on: 31 Dec 2020

Airflow

Why does the DayOfWeekSensor exist in Airflow?

How to make an Airflow DAG wait until a specified day of the week
Published on: 30 Dec 2020

Are you looking for an experienced AI consultant? Do you need assistance with your RAG or Agentic Workflow?
Schedule a call, send me a message on LinkedIn. Schedule a call or send me a message on LinkedIn

>