What should I do if I wanted to pause an Airflow DAG until one of my articles gets publishes?

Let’s assume that I have already scheduled publication of the text you are reading right now, and I know the article URL. It is going to be https://mikulskibartosz.name/use-httpsensor-to-wait-for-website

Now, I have to define an HTTP connection in my Airflow connections settings. The connection should point to the root address of my website: https://mikulskibartosz.name.

After that, I add a HttpSensor to my DAG. It is going to use the connection defined earlier and wait until the resource is available:

sensor = HttpSensor(
    task_id='http_sensor_check',
    http_conn_id='mikulskibartosz_name',
    endpoint='use-httpsensor-to-wait-for-website',
    poke_interval=60,
    dag=dag)
Stop AI Hallucinations Before They Cost You.
Join engineering leaders getting weekly tactics to prevent failure in customer-facing AI systems. Straight from real production deployments.

The Airflow DAG will wait until the website returns a status code, which indicates success. When you look at this sensor’s code, you will see that it returns False only when the failure was due to 404 Not Found status. In a case of any other failure, it propagates the exception to Airflow.

Stop AI Hallucinations Before They Cost You.
Join engineering leaders getting weekly tactics to prevent failure in customer-facing AI systems. Straight from real production deployments.
Older post

How to add an EMR step in Airflow and wait until it finishes running

How to use AwsHook and EmrStepSensor to add an EMR step and wait until it finishes running

Newer post

How to select a random sample of rows using Athena

How to use a window function to select random rows from Athena

Engineering leaders: Is your AI failing in production? Take the 10-minute assessment
>
×
Stop AI Hallucinations Before They Cost You.
Join engineering leaders getting weekly tactics to prevent failure in customer-facing AI systems. Straight from real production deployments.