Integrating Apache Airflow with the AWS ecosystem has become easier than ever with MWAA. To make MWAA work efficiently, I prepared a comprehensive guide using CDK to spin up MWAA and some tips for MWAA specifically to help you understand the deployment for Airflow in AWS.
How to build a web crawler with MWAA (AWS Airflow) with CDK Read More »
If you are reading this blog post, you may have faced the challenge of data validation before, or you might be struggling with it. My goal in this post is to share my experience with data validation
The Foundation of Data Validation Read More »
The airflow schedule interval could be a challenging concept to comprehend, even for developers work on Airflow for a while find difficult to grasp. A confusing question arises every once a while on StackOverflow is “Why my DAG is not running as expected?”. This problem usually indicates a misunderstanding among the Airflow schedule interval.
Airflow Schedule Interval 101 Read More »
Exploratory Data Analysis (EDA) is a methodology in data science as the initial approach to gain insights by visualizing and summarizing data. We will use some exploratory data analysis technics to find the reason behind the bidding war on the housing market.
Bidding War on Housing Market? Let’s Use R For Exploratory Data Analysis Read More »
When it comes to visualizing data with a histogram and dealing with multiple groups, it can be quite challenging. I have recently come across a useful ggplot2 extension called ggridges that has been helpful for my data exploratory tasks.
Visualizing Data with ggridges: Techniques to Eliminate Density Plot Overlaps in ggplot2 Read More »
Slowly Changing Dimension (SCD) is critical to dimensional modeling. We will discuss the eight types of SCDs. By the end, you will clearly understand each type and be able to differentiate between SCDs in dimensional modeling.
Sometimes writing SQL can be frustrating, especially when encountering NULL values. This article can help you better understand NULL in SQL
Demystifying Null in SQL: A Comprehensive Guide for Data Professionals Read More »
My main goal is to enable people without programming experience to use the powerful Sankey Diagram by simply uploading the transaction CVS file from the popular site Mint.com.
How I Built a Tool to Visualize Expense In Sankey Diagram Read More »
One key learning I had while chasing the latest tool is: Tools are great, but many data engineering problems cannot be resolved by using the newest tool but by human — Data Engineers. I want to share my thoughts on Why data engineering is about much more than just the tools you use.
Data Engineering: Why It’s About Much More Than Just the Tools You Use Read More »
Let’s bring the data community’s attention to the essential- Building Better Data Warehouses with Dimensional Modeling: A Guide for Data Engineers.
Building Better Data Warehouses with Dimensional Modeling: A Guide for Data Engineers Read More »
