Doing data quality checks using the SQLCheckOperator

SQLCheckOperator is an Airflow operator that executes a SQL query, expects to receive a single row in the response, and attempts to cast every value in the row to a boolean. It succeeds when all returned values can be cast to true, so the query may return those values:

  • a boolean True

  • a non-zero numeric value (including negative values!)

  • a non-empty string

  • a non-empty list, set, or dictionary

In addition to failing when any of the values is False, the SQLCheckOperator operator also fails when the query returns no rows.

For example, we can use that operator to check the count of values in a table:

from airflow.operators.sql import SQLCheckOperator

operator = SQLCheckOperator(
     sql="SELECT COUNT(*) FROM some_table WHERE some_column='{{ yesterday_ds_nodash }}'"
)
Older post

How to deal with the jinja2 TemplateNotFound error in Airflow

How to fix the TemplateNotFound error while using a custom Airflow operator

Newer post

How to check the next execution date of an Airflow DAG

How to use Airflow CLI to get the next execution date of a DAG

Are you looking for an experienced AI consultant? Do you need assistance with your RAG or Agentic Workflow?
Schedule a call, send me a message on LinkedIn. Schedule a call or send me a message on LinkedIn

>