Programming

4 Tips To Write Scalable Apache Spark Code

Introduction

In this article, I will share some tips on how to write scalable Apache Spark code. The examples presented here are actually based on the code I encountered in the real world. So, by sharing these tips, I hope I can help newcomers to write performant Spark code without needlessly increasing their cluster’s resources.

Cluster Setup

The cluster I used to run the code in this article is hosted on Databricks with the following configuration:

• Cluster Mode: Standard
• Databricks Runtime Version: 5.5 LTS ML (includes Apache Spark 2.4.3 Scala 2.11)

There are 8 workers and both the workers and driver are m4.xlarge instances (16.0 GB, 4 Cores).

Background

I recently inherited a notebook meant to track the results of our AB Tests to evaluate the performance of our recommendation engine. Running the notebook from start to finish was excruciatingly slow. This was the cell that took the longest time to execute:

Figure 1: Very bad implementation

The files used to track the data in our AB Test experiments are stored as newline delimited JSON files partitioned into different folders by year, month and day. Each day could have a few hundred…