How to deal with the jinja2 TemplateNotFound error in Airflow

Occasionally, when we are making a custom Airflow operator, we may end up in a situation when Airflow complains about not existing jinja template. It may happen when we have an Airflow operator, which calls another operator inside its execute function when both operators have a common template field.

In this situation, Airflow may raise an error saying that: jinja2.exception.TemplateNotFound: content_of_the_field. It is strange because if we have not used a jinja template inside the field value, we may expect it to pass the string without using it as a template. Unfortunately, that does not happen.

Dealing with the issue is quite simple. The wrapping Airflow operator should include the field’s content as a jinja constant and pass it into the inner operator. For example, when our code looks like this:

class SomeOperator(BaseOperator):

    template_fields = ('the_field')

    def __init__(self, field, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.field = field
        self.extra_args = args
        self.extra_kwargs = kwargs

    def execute(self, context):
        self.another_operator = SomeOtherOperator(field=self.field, *self.extra_args, **self.extra_kwargs)
        self.another_operator.execute(context)

We can fix the jinja template issue like this:

class SomeOperator(BaseOperator):

    template_fields = ('the_field')

    def __init__(self, field, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.field = "{{ '" + field + "' }}"
        self.extra_args = args
        self.extra_kwargs = kwargs

    ... rest of the code
Older post

How to postpone Airflow DAG until files get uploaded into an S3 bucket

How to use Airflow sensors to detect that files have been uploaded into an S3 bucket

Newer post

Doing data quality checks using the SQLCheckOperator

How to use SQLCheckOperator to verify that the database contains an expected number of rows

Are you looking for an experienced AI consultant? Do you need assistance with your RAG or Agentic Workflow?
Schedule a call, send me a message on LinkedIn. Schedule a call or send me a message on LinkedIn

>