Airflow

How to deal with the jinja2 TemplateNotFound error in Airflow

How to fix the TemplateNotFound error while using a custom Airflow operator

Bartosz Mikulski 19 Oct 2020 – 2 min read

Occasionally, when we are making a custom Airflow operator, we may end up in a situation when Airflow complains about not existing jinja template. It may happen when we have an Airflow operator, which calls another operator inside its execute function when both operators have a common template field.

In this situation, Airflow may raise an error saying that: jinja2.exception.TemplateNotFound: content_of_the_field. It is strange because if we have not used a jinja template inside the field value, we may expect it to pass the string without using it as a template. Unfortunately, that does not happen.

Dealing with the issue is quite simple. The wrapping Airflow operator should include the field’s content as a jinja constant and pass it into the inner operator. For example, when our code looks like this:

class SomeOperator(BaseOperator):

    template_fields = ('the_field')

    def __init__(self, field, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.field = field
        self.extra_args = args
        self.extra_kwargs = kwargs

    def execute(self, context):
        self.another_operator = SomeOtherOperator(field=self.field, *self.extra_args, **self.extra_kwargs)
        self.another_operator.execute(context)

We can fix the jinja template issue like this:

class SomeOperator(BaseOperator):

    template_fields = ('the_field')

    def __init__(self, field, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.field = "{{ '" + field + "' }}"
        self.extra_args = args
        self.extra_kwargs = kwargs

    ... rest of the code

Older post

How to deal with the jinja2 TemplateNotFound error in Airflow

How to postpone Airflow DAG until files get uploaded into an S3 bucket

Doing data quality checks using the SQLCheckOperator

How to deal with the jinja2 TemplateNotFound error in Airflow

How to postpone Airflow DAG until files get uploaded into an S3 bucket

Doing data quality checks using the SQLCheckOperator

Related Posts

Anomaly detection in Airflow DAG using Prophet library

How to restart a stuck Airflow DAG

Why does the DayOfWeekSensor exist in Airflow?