Improving Big Data: A Guide to Enhanced Pipelines
Seeking to improve your big data pipeline? This blog walks you through enhancements made to a client's system using Airflow Datasets, DAG Dependencies, Azure Durable Functions, and edge cases. Learn how we added functionality and flexibility by streamlining data integration, minimizing cost increases, and creating a scalable pipeline development process.
Dave O'Connor |
January 25, 2024
Too Big for DAG Factories?
As your infrastructure scales up, how you go about managing all DAGs in Airflow becomes very important. One method would be to create a “DAG factory,” which can churn out thousands of DAGs dynamically from a single configuration file.
Glenn Franxman |
December 8, 2022
Data Warehousing in Microsoft Azure
Data Warehousing is the process of storing and processing large amounts of data from many disparate sources.
Glenn Franxman |
February 11, 2021
