The builder's path
For: Data teams on Airflow 3 don't get their pipeline-level failure alerts, because the callback never runs.
“The true creator is necessity, which is the mother of our invention.”
Routes to a solution
Each route is a different kind of journey, with real people and organisations to contact at each step, starting with those in every country.
Step 1
Build in the open
Publish the code and invite contributors from the community.
Let the people a free tool helps fund its maintainer directly.
Raise and spend money in the open for a community project.
Grants for open source that makes the internet better.
Network of civic tech groups in many countries; find the one near the problem.
Step 2
Fund the maintainers
Sponsors, collectives and open source grants.
Let the people a free tool helps fund its maintainer directly.
Raise and spend money in the open for a community project.
Grants for open source that makes the internet better.
Network of civic tech groups in many countries; find the one near the problem.
Step 3
Get it adopted
Civic tech groups and public bodies take it up.
Y Combinator's free course and co-founder matching.
Raise and spend money in the open for a community project.
Network of civic tech groups in many countries; find the one near the problem.
Volunteer data scientists working with social-sector organisations.
Ideas, sources and tools for this problem, from first interviews to funding.
what DAG and task callbacks are supposed to do
maintainers can say which of the closed approaches was closest
Starter prompt
Built from this problem's brief. Paste it into Claude or any AI assistant.
Help me design and build a first working prototype for this problem in Anywhere. Problem: Data teams on Airflow 3 don't get their pipeline-level failure alerts, because the callback never runs. Who it hurts: Teams that upgraded from Airflow 2 to 3 and set on_failure_callback on the whole DAG. The moment it hurts: When a pipeline run fails or times out overnight and no one is told. What people do today: Task-level callbacks still fire, so some teams move alerts back to every task, which can miss failures that are not tied to one task. Why it is still unsolved: Two pull requests linked to the issue were closed without being merged, and the issue is still marked as needing triage. What solved looks like: A test that reproduces the bug on current main, and a fix so a DAG-level failure callback fires exactly once when a run fails, including on timeout, checked with LocalExecutor and KubernetesExecutor. Where: Anywhere First, list what you would need to learn from the people affected before building, as interview questions. Then propose the simplest version that would help, the screens or messages it needs, and the data it would use. Keep it cheap to run, accessible, and respectful of people's privacy. Flag anything you are unsure about instead of guessing.
Talk to the people who live it before you build.
“You cannot step twice into the same river.”
Get a working version in front of people fast.
Using AI is not cheating. Every tool carries the work of the people who made it, and using it well is how knowledge has always moved forward. The thinking, the care for the people who live the problem, and the judgement are still yours.
“The Analytical Engine weaves algebraical patterns just as the Jacquard-loom weaves flowers and leaves.”
Open data that fits this problem.
“Lead me from the unreal to the real, from darkness to light.”
असतो मा सद्गमय। तमसो मा ज्योतिर्गमय।
asato mā sad gamaya, tamaso mā jyotir gamaya
Tools matched to what this problem needs.
“We are like dwarfs sitting on the shoulders of giants; we see more, and things more distant, not by our own sharp sight but because we are lifted up by their greatness.”
Ship it responsibly, and find help and funding.
“While we teach, we learn.”
Homines dum docent discunt.
Kit written for this problem from its brief; every link was checked. Links go to each provider's own site. Zetema is not paid for any listing.