Airflow HTTP sensor: wait for an AI video job to finish

An Airflow HTTP sensor (HttpSensor) sends a GET to an endpoint on every poke and succeeds when its response_check callable returns True. To wait for an AI video job, submit the job with HttpOperator, then point an HttpSensor at the job's status URL with a response_check that returns True once the job is done, in reschedule mode and with a timeout of its own.

Airflow facts come from the HTTP provider's operators guide, its HttpSensor, HttpOperator, and connection pages, and Airflow's Sensors, Tasks, and Task SDK references; Sume facts come from Video Generation and Jobs and results. All were read on 2026-09-27. Sume has no Airflow provider: the DAG makes plain HTTPS calls. The polling loop outside Airflow is in How to poll a video generation job status API.

How do I start the job with the HTTP operator?

Create an HTTP connection, sume_api here, with host api.sume.com and schema https; Airflow's HTTP operators default to http. Airflow takes headers as JSON in the connection's Extra, so put Authorization: Bearer <your key> there and keep the key out of DAG code.

HttpOperator sends a POST by default, and its endpoint, data, and headers are templated. Sume's POST /v1/videos answers 202 with id, polling_url, status: "pending", and model. A response_filter that returns id makes the job id the task's result, which the sensor pulls from XCom.

Key the request to the DAG run with Idempotency-Key: video-{{ run_id }}. Airflow says to use run_id rather than the logical date when you need a value unique within a DAG, and Sume answers a replayed key with the original job instead of a second paid one. That covers task retries, and deferrable mode too, where Airflow warns that a triggerer restart can replay a POST.

# Inside your DAG. The sume_api connection holds the host and the key.
import json
from airflow.providers.http.operators.http import HttpOperator
from airflow.providers.http.sensors.http import HttpSensor

def video_done(response):
    status = response.json()["status"]
    if status in ("failed", "cancelled"):
        raise ValueError(f"Sume video job ended as {status}")
    return status == "completed"

submit = HttpOperator(
    task_id="submit_video", http_conn_id="sume_api", endpoint="v1/videos",
    data=json.dumps({"model": "sume/auto", "prompt": "A slow pan across a desk, morning light",
                     "aspect_ratio": "9:16", "duration": 5}),
    headers={"Content-Type": "application/json", "Idempotency-Key": "video-{{ run_id }}"},
    response_filter=lambda response: response.json()["id"],
)
wait = HttpSensor(
    task_id="wait_for_video", http_conn_id="sume_api",
    endpoint="v1/videos/{{ ti.xcom_pull(task_ids='submit_video') }}",
    response_check=video_done, response_error_codes_allowlist=["429"],
    mode="reschedule", poke_interval=90, timeout=20 * 60,
)
submit >> wait

What should response_check return for each job status?

True passes the sensor and False means poke again. For the two failure states, raise: an exception fails the sensor task, and it then follows its retries setting, instead of poking until the timeout.

Why set response_error_codes_allowlist?

By default HttpSensor returns False only on 404. Any other error code raises and fails the sensor itself, with no more poking. Sume polls count against the key's read budget, and a spent budget answers 429, which Sume's docs say to back off from, not to treat as the job's result. With ["429"] in the allowlist, a rate limit costs one poke instead of the task. The list replaces the default ["404"], so a job id Sume doesn't know now fails the sensor instead of poking until the timeout. The read budget is separate from the write budget and forty times its size, so polling won't block your submits.

What timeout and poke interval should the sensor use?

Use mode="reschedule". In the default poke mode a sensor holds a worker slot for its whole run; in reschedule mode it takes one only while checking. Airflow's reference says a reschedule poke interval should be more than one minute, to spare the scheduler, and Sume's docs give 30 seconds as a reasonable polling interval and say video typically takes 30 seconds to several minutes, so 90 seconds suits both.

timeout counts from the first poke, reschedule delays included, and a breach raises AirflowSensorTimeout, which fails the sensor immediately without retrying. Sume's docs call 20 minutes a reasonable client-side deadline for video. A sensor timeout doesn't cancel the job: it keeps running and still bills. Check the same job id again rather than submitting anew; even a rerun of submit_video in the same DAG run sends the same key and gets the original job back.

• HttpSensor doesn't download the video. A completed job lists unsigned_urls, and GET /v1/videos/{id}/content, called with your key, redirects to the file; see How to download a generated video from the API.
• HttpSensor can also run in deferrable mode with deferrable=True.