Learn how to avoid 10 common data engineering pitfalls—like Spark data skew, Airflow retry chaos, schema drift, and more—with practical solutions
Don’t Get Tripped Up! 10 Common Data Engineering Pitfalls Read More »
Learn how LLM + MCP synergy revolutionizes complex tasks. An Apache Airflow 3.0 case study demonstrates auto-updating DAGs and overcoming AI limitations.
Explore how AI in data engineering is shaping the future. This 2025 guide helps new grads build the skills, tools, and mindset to thrive in a cloud-driven, AI-first world.
Data Engineering in 2025: A Practical Guide for New Grads Entering the AI-First Era Read More »
Explore the latest features, UI updates, and key changes in Apache Airflow 3.0. This deep dive covers DAG versioning, event-driven scheduling, Docker setup, and more for data engineers and workflow automation pros.
Unboxing Apache Airflow 3.0: What’s New, What’s Gone, and Why It Matters Read More »
Discover the upcoming features in Apache Airflow 3.0, with insights from the Airflow 3.0 workstream. Get ready for the next big release!
Apache Airflow 3.0 Is Coming Soon: Here’s What You Can Expect Read More »
Integrating Apache Airflow with the AWS ecosystem has become easier than ever with MWAA. To make MWAA work efficiently, I prepared a comprehensive guide using CDK to spin up MWAA and some tips for MWAA specifically to help you understand the deployment for Airflow in AWS.
How to build a web crawler with MWAA (AWS Airflow) with CDK Read More »
The airflow schedule interval could be a challenging concept to comprehend, even for developers work on Airflow for a while find difficult to grasp. A confusing question arises every once a while on StackOverflow is “Why my DAG is not running as expected?”. This problem usually indicates a misunderstanding among the Airflow schedule interval.
Airflow Schedule Interval 101 Read More »
Discover the Essential Reading List for Data Engineers: 10 Classic Books You Can’t Miss. While many free online resources are available, they often lack the depth and context needed to truly master the field. In this article, I will share ten classic books that cover everything from fundamental technical skills like Python and SQL, to more advanced topics like Apache Spark, Apache Flink, Apache Beam, Apache Airflow, Kubernetes, distributed systems, and dimensional modeling.
The Essential Reading List for Data Engineers: 10 Classic Books You Can’t Miss Read More »
Many modern data orchestration projects like Apache Airflow and Luigi are Python-based. Let’s explore the popular data pipeline orchestration options for R.
5 Fantastic Data Pipeline Orchestration Tools For R Read More »
Airflow has been widespread for years. Is Apache Airflow due for a replacement? mage-ai is the new ETL tool for data engineers to check out as a substitution. I have taken a first impression of mage-ai and will share my thoughts.
Is Apache Airflow Due for Replacement? The First Impression Of mage-ai Read More »
