Let’s Orchestrate With Airflow — Step-by-Step Airflow Implementations

Photo by Sachin C Nair: https://www.pexels.com/photo/waterfalls-during-sunset-954929/

You’ll orchestrate the ETL workflow with Airflow in the local environment without downloading ANYTHING.

You’ll use Bash Operator, Python Operator, Task Decorator, and other Airflow functions.

You’ll get the data, load it to PostgreSQL Database and see it with pgAdmin in minutes.

You’ll do everything with the Docker containers.

Easy-to-follow, step-by-step explanations. You’ll love it.

CONTENTS

Introduction

Airflow-Lite Version in Local Environment

DAG — Directed Acyclic Graph

DATA — TLC Trip Record Data

Get the Data

Extract & Load the Data

With the PgAdmin

What if … ?

DAG Runs Monthly

Conclusion

https://airflow.apache.org/

INTRODUCTION

“Airflow is a platform created by the community to programmatically author, schedule, and monitor workflows.” (https://airflow.apache.org/)

Airflow is one of the most known work orchestration tools.

In this tutorial, we will cover Airflow implementations in the local environment.

We will use Airflow Docker Lite Installation.

We will get the data from the source, transform it and load it to the PostgreSQL database and see it with the pgAdmin.

Let’s start.

Image Credit: https://giphy.com/

AIRFLOW-LITE Version in Local Environment

In the previous article, we discussed two different easy installation options for Airflow installation with Docker.

And we saw that we installed Airflow with Docker in minutes.