Differences between RDD, data frames and data sets in Spark

Overview

• Understand the difference between APIs 3 sparks: RDD, data frames and data sets
• We will see how to create RDD, data frames and data sets.

Introduction

Have passed 11 years since Apache Spark began to exist and, impressively and continuously, became the first choice of big data developers. Developers have always loved providing simple and powerful APIs that can perform any type of big data analysis..

Initially, in 2011 came up with the concept of RDD, after in 2013 with Dataframes and later in 2015 with the concept of Datasets. None of them have depreciated, we can still use them all. In this post, we will understand and see the difference between the three.

Table of Contents

1. What are RDD?
2. When to use RDD?
3. What are data frames?
4. What are data sets?
5. RDD vs Dataframes vs Datasets?

What are RDD?

RDDs or resilient distributed data sets are the fundamental data structure of Spark. It is the collection of objects that is capable of storing partitioned data in the multiple nodes of the cluster and also enables them to do the processing in parallel..

It is fault tolerant if you perform multiple transformations on the RDD and then, for any circustance, some node fails. El RDD, then, is able to recover automatically.

There is 3 alternativas para crear un ASD:

1. Parallel an existing data collection
2. Reference to the stored external data file
3. Creating RDD from an existing RDD

When to use RDD?

We can use RDD in the following situations:

1. When we want to do low-level transformations on the dataset. Read more about RDD Transformations: PySpark to perform Transformations
2. Does not automatically infer schema from ingested data, we need to specify the schema of each and every dataset when we create an RDD. Learn how to infer the RDD scheme here: Building Machine Learning Pipelines with PySpark

What are data frames?

It was first introduced in Spark version 1.3 to overcome the limitations of Spark RDD. Spark Dataframes are the distributed collection of data points, but here, data is organized in named columns. Enable developers to debug code throughout runtime, what was not allowed with RDDs.

Data frames can read and write the data in various formats like CSV, JSON, EURO, HDFS and HIVE tables. It is already optimized to process large data sets for most preprocessing tasks, so we don't need to write complex functions on our own.

Uses a catalyst optimizer for optimization purposes. If you want to read more about the catalyst optimizer, I highly recommend that you read this post: Practical tutorial for analyzing data using Spark SQL

Let's see how to create a data frame using PySpark.

What are data sets?

Spark Datasets is an extension of the Data Frames API with the benefits of RDDs and Datasets. It's fast and provides a type-safe interface. Type safety means that the compiler will validate the data types of all columns in the dataset during compilation only and will throw an error if there is any discrepancy in the data types.

RDD users will find it somewhat similar to the code, but it is faster than RDD. Can efficiently process structured and unstructured data.

We still can't create Spark datasets in Python. Dataset API is only enabled in Scala and Java.

RDD vs. Data Frames vs. Data Sets

	RDD	Data frames	Data sets
Data representation	RDD is a distributed collection of data items without any schema.	It is also the distributed collection organized in the named columns.	It is an extension of Dataframes with more features such as type safety and object-oriented interface.
Improvement	No built-in optimization engine for RDD. Developers must write the optimized code themselves.	Uses a catalyst optimizer for optimization.	Also uses a catalyst optimizer for optimization purposes.
Scheme projection	Here, we need to set the schema manually.	It will automatically discover the dataset schema.	In addition, it will automatically find the schema of the dataset through the SQL engine.
Aggregation operation	RDD is slower than data frames and data sets to perform simple operations like grouping data.	Provides a simple API to perform aggregation operations. Performs aggregation faster than RDDs and datasets.	Dataset is faster than RDDs but slightly slower than Dataframes.

Final notes

In this post, we have seen the difference between the three main APIs of Apache Spark. Then, to complete, if you want rich semantics, high-level abstractions, type safety, elija Dataframes o Datasets. If you need more control over the preprocessing part, you can always use the RDDs.

I recommend that you check out these additional resources on Apache Spark to boost your knowledge.:

If you found this post informative, share it with your friends, and also if you want to give any suggestions on what it should cover, feel free to leave them in the notes below.

Related