Explore the latest features, UI updates, and key changes in Apache Airflow 3.0. This deep dive covers DAG versioning, event-driven scheduling, Docker setup, and more for data engineers and workflow automation pros.
Unboxing Apache Airflow 3.0: What’s New, What’s Gone, and Why It Matters Read More »
Discover how DuckDB Local UI revolutionises your data exploration experience. After years of using external tools, DuckDB’s native interface provides a seamless, quick, and intuitive way to interact with your data projects
DuckDB Local UI is Awesome! Read More »
DeepSeek SmallPond is here to shake up data engineering. See how this lightweight open-source framework offers a fresh alternative to Apache Spark and Flink for batch and streaming processes.
DeepSeek SmallPond: A Game-Changer for Data Engineers Seeking Lightweight Solutions Read More »
Discover the upcoming features in Apache Airflow 3.0, with insights from the Airflow 3.0 workstream. Get ready for the next big release!
Apache Airflow 3.0 Is Coming Soon: Here’s What You Can Expect Read More »
Integrating Apache Airflow with the AWS ecosystem has become easier than ever with MWAA. To make MWAA work efficiently, I prepared a comprehensive guide using CDK to spin up MWAA and some tips for MWAA specifically to help you understand the deployment for Airflow in AWS.
How to build a web crawler with MWAA (AWS Airflow) with CDK Read More »
If you are reading this blog post, you may have faced the challenge of data validation before, or you might be struggling with it. My goal in this post is to share my experience with data validation
The Foundation of Data Validation Read More »
The airflow schedule interval could be a challenging concept to comprehend, even for developers work on Airflow for a while find difficult to grasp. A confusing question arises every once a while on StackOverflow is “Why my DAG is not running as expected?”. This problem usually indicates a misunderstanding among the Airflow schedule interval.
Airflow Schedule Interval 101 Read More »
Exploratory Data Analysis (EDA) is a methodology in data science as the initial approach to gain insights by visualizing and summarizing data. We will use some exploratory data analysis technics to find the reason behind the bidding war on the housing market.
Bidding War on Housing Market? Let’s Use R For Exploratory Data Analysis Read More »
When it comes to visualizing data with a histogram and dealing with multiple groups, it can be quite challenging. I have recently come across a useful ggplot2 extension called ggridges that has been helpful for my data exploratory tasks.
Visualizing Data with ggridges: Techniques to Eliminate Density Plot Overlaps in ggplot2 Read More »
Slowly Changing Dimension (SCD) is critical to dimensional modeling. We will discuss the eight types of SCDs. By the end, you will clearly understand each type and be able to differentiate between SCDs in dimensional modeling.
