A Practical Tip When Working With Random Samples On Spark

Introduction

In this article, I will share a crucial tip when using Spark to analyze a random sample of a data frame. The code to reproduce the results can be found here. It’s an HTML version of a Databricks notebook, so all you have to do is download it in its raw form and then display the downloaded file in a web browser.

Problem Statement

Let’s suppose we are working on a large dataset. For the purpose of this article, we will assume that the dataset in question is the flight departure delays dataset, which looks like this:

Since it is a large dataset, we’ve decided to begin our analyses on a subset of it. There are many say we can sample this dataset so let’s just assume that we decided that all the delays for a random sample of origin is appropriate for whatever business question we want our analyses to answer.

Here’s how we can obtain the desired subset using Pandas:

Figure 2: Obtaining the desired subset using Pandas

In Figure 2, we begin by reading the dataset into df . We then build another data frame containing a random sample of origin which we will use to filter df and save…