Data teams on Airflow 3 don't get their pipeline-level failure alerts, because the callback never runs.
Data teams who rely on Apache Airflow to alert them when a pipeline fails
Where it stands
- Sourced (done)
- Kit (done)
- Wanted (next)
- Building
- Solved
How this problem came to be
- Found2 sourcesgithub.com/apache
- Quoted4 claimseach checked against its page
- BriefWrittenand what solved looks like
- Market1 signals1 reactions on the issue
- Kit5 steps21 tools · 6 routes
- People0 want0 live it · 0 would test
The problem, broken down
- Who it hurts
- Teams that upgraded from Airflow 2 to 3 and set on_failure_callback on the whole DAG.
- The moment
- When a pipeline run fails or times out overnight and no one is told.
- What people do today
- Task-level callbacks still fire, so some teams move alerts back to every task, which can miss failures that are not tied to one task.
- Why it's unsolved
- Two pull requests linked to the issue were closed without being merged, and the issue is still marked as needing triage.
- What solved looks like
- A test that reproduces the bug on current main, and a fix so a DAG-level failure callback fires exactly once when a run fails, including on timeout, checked with LocalExecutor and KubernetesExecutor.
Evidence
Every claim is tied to the source it comes from- 1The issue reports that a DAG-level on_failure_callback does not fire in Airflow 3.1.7 with KubernetesExecutor, while task-level callbacks do, and says this worked in Airflow 2.x.
- 2The reporter found ‘callback is empty’ in the scheduler's debug logs for every failed DAG run.
- 3Later comments report the same problem after upgrading to Airflow 3, including on 3.2.2 with LocalExecutor.
- 4One commenter said their team moved failure callbacks to the DAG level after a major outage, to catch every failure case.
Builders still need to know
Written from the cited sources. Anything they don't say is left as a question, not a guess.
Sources
From a public page with a citation.
What the source says, in short
The issue reports that a DAG-level on_failure_callback does not fire in Airflow 3.1.7 with KubernetesExecutor, while task-level callbacks do, and says this worked in Airflow 2.x. The reporter found ‘callback is empty’ in the scheduler's debug logs for every failed DAG run. Later comments report the same problem after upgrading to Airflow 3, including on 3.2.2 with LocalExecutor. One commenter said their team moved failure callbacks to the DAG level after a major outage, to catch every failure case.
Forks 0
Nobody has forked this yet. Be the first.
Market read
How big this is, from the evidence
- GitHub ↗Builders' pull · Medium confidence1 reactions on the issue
How these were measured: Total reactions on the GitHub issue, read from its API.
For whoever takes this up
The builder's path for this problem
Five steps, each with the tools that fit this problem and a word to carry with you. A starter prompt is ready to paste into any assistant.
Open the pathDiscussion 0
No comments yet.