Saving the Output of Spark ETL Jobs | Towards AI

How You Should Save the Output of your Spark ETL Jobs (If you are not Writing to a Database)

Introduction

In this article, I will share my thoughts on the best way to save the output of Spark ETL jobs so that it is easier to do analytical work later. The code to reproduce the examples can be found here.

Cluster Setup

The cluster I used to run the code in this article is hosted on Databricks with the following configuration:

• Cluster Mode: Standard
• Databricks Runtime Version: 5.5 LTS ML (includes Apache Spark 2.4.3 Scala 2.11)

There are 8 workers and both the workers and driver are m4.xlarge instances (16.0 GB, 4 Cores).

Case Study

Imagine you are in the following scenario:

You just joined an immensely popular online retailer with customers worldwide. This company makes billions of dollars in revenue each year. So you know it generates so much transactional data everyday that it can only be efficiently processed using a distributed computing framework like Apache Spark. You are tasked to build a model to forecast daily turnover and decided to begin by doing some exploratory data analysis on past transactions.

The data engineering team has kindly agreed to give you access to these data in the…