The pros and cons of Apache Airflow as a workflow management platform for ETL & Data Science and deriving from that the use cases for which Airflow may be a good or a bad choice
Published in · 10 min read · Aug 26, 2020
--
Not so long ago, if you would ask any data engineer or data scientist about what tools do they use for orchestrating and scheduling their data pipelines, the default answer would likely be Apache Airflow. Even though Airflow can solve many current data engineering problems, I would argue that for some ETL & Data Science use cases it may not be the best choice.
In this article, I will discuss the pros and cons that I experienced while working with Airflow in the last two years and derive from that the use cases for which Airflow is still a great choice. I hope that by the end of this article, you will be able to determine whether it fits your ETL & Data Science needs.
Community
Undeniably, Apache Airflow has an amazing community. There is a large number of individuals using Airflow and contributing to this open-source project. If you want to solve a particular data engineering problem, the chances are that somebody in the community has already solved that and shared their solution online or even contributed their implementation to the codebase.
Companies placing strategic bets on Airflow
Many companies decided to invest in Apache Airflow and support its growth, among them:
- Google with its Cloud Composer GCP service,
- Astronomer offering enterprise support in deploying Airflow on Kubernetes,
- Polidea heavily contributing to the codebase with many PMC members
- GoDataDriven offering Apache Airflow training.
The support from those companies ensures that there are people working full-time to further improve the software which guarantees long-term stability, support, and training.
