An Intro to Airflow at January 2022 IndyPy
We kicked off the 2022 season of IndyPy — Indiana’s largest Python meetup founded in 2007 by Six Feet Up CTO and AWS Community Hero, Calvin Hendryx-Parker — with an “Introduction to Airflow.” In this presentation and live demo, Rocco Pascale, a Field Engineer at Astronomer, demonstrates how anyone with Python knowledge can deploy a workflow using Apache Airflow — an open-source platform created to programmatically author, schedule and monitor workflows.
As Rocco explains, by writing a single DAG (Directed Acyclic Graph) file in Python using existing provider packages — Astronomer currently has 82 providers and is regularly expanding its library — users can begin to define complex relationships between data and actions.
Read our live Twitter thread, and watch the video below for a deeper dive into Airflow, DAGS, tasks, operators and the core components that make Airflow run including the webserver, scheduler, metastore, executor, worker and triggerer.
Watch the presentation:
Links and Resources
- The Astronomer Registry includes a full list of providers, sample DAGs and more: https://registry.astronomer.io
- Rocco Pascale’s GitHub: https://github.com/ReadytoRocc
- Airflow Guides: https://www.astronomer.io/guides
- Deferrable Operators: https://www.astronomer.io/guides/deferrable-operators
Thanks for filling out the form! A Six Feet Up representative will be in contact with you soon.