bytepeaker_logo
Envelope

Spark Streaming | A Beginner's Guide to Spark Streaming

Contents

Overview

  • Understand Spark Streaming and how it works.
  • Learn about Windows on Spark Streaming with an example.

Introduction

According to IBM, the 60% of all sensory information loses value in a few milliseconds if it is not acted upon. Taking into account that the Big Data and analytics market has reached the $ 125 billion and a large part of this will be attributed to IoT in the future, inability to harness information in real time will result in billions of dollars of loss.

Examples of some of these applications include a telecommunications company, which calculates how many of its users have used WhatsApp in the last 30 minutes, a retailer that tracks the number of people who have said positive things about their products today on social media, or a law enforcement agency looking for a suspect using CCTV traffic data.

Esta es la razón principal por la que los sistemas de procesamiento de flujos como Spark Streaming definirán el futuro de la analytics in real time. There is also a growing need to analyze both data at rest and data in motion to power applications., what makes systems like Spark, they can do both, be even more attractive and powerful. It is a system for all Big Data seasons.

You will learn how Spark Streaming not only keeps the familiar Spark API intact, but also, underhood, uses RDD for storage and fault tolerance. This allows Spark professionals to jump into the streaming world right from the start.. With that in mind, let's get right to it.

1flyjc6u-qaq64ydllrzdww-8687944

Introducción a Spark Streaming | de Harshit Agarwal | Half

Table of Contents

  • Apache Spark
  • Ecosistema Apache Spark
  • Spark Streaming: DStreams
  • Spark Streaming: transmission context
  • Example: word count
  • Spark Streaming: Window
  • A window based on: word count
  • a (more efficient) window based: word count
  • Spark Streaming: exit operations

Apache Spark

Apache Spark is a unified computing engine and set of libraries for parallel data processing in computer clusters. At the time of writing this article, Spark is the most developed open source engine for this task, making it a standard tool for any developer or data scientist interested in big data.

Spark supports several widely used programming languages (Python, Java, Scale y R), includes libraries for various tasks ranging from SQL to streaming to machine learning, and runs anywhere, desde una computadora portátil hasta un cluster de miles de servidores. This makes it an easy system to get started and scale up to big data processing or incredibly large scale.. Here are some of the features of Spark:

  • Fast, general-purpose engine for large-scale data processing

  • Spark can efficiently support more types of calculations

  • Can read / write to any Hadoop-compatible system (for instance, HDFS)

  • Speed: in-memory data storage for very fast iterative queries

    • el sistema también es más eficiente que MapReduce para aplicaciones complejas que se ejecutan en el disco

    • until 40 times faster than Hadoop

    • Ingest data from many sources: Kafka, Twitter, HDFS, sockets TCP

    • Results can be sent to file systems, databases, live panels, but not only

7p88f2-2703904

Ecosistema Apache Spark

The following are the components of the Apache Spark ecosystem:

  • Spark Core: basic Spark functionality (task scheduling, memory management, fault recovery, storage systems interaction).

  • Spark SQL: package to work with structured data queried through SQL and HiveQL

  • Spark Streaming: a component that enables the processing of live data streams (for instance, log files, status update messages)

  • MLLib: MLLib is a machine learning library like Mahout. It is built on Spark and has the ability to support many machine learning algorithms.
  • GraphX: For graphs and graphical calculations, Spark has its own graphical calculation engine, called GraphX. It is similar to other widely used graphics processing tools or databases, like Neo4j, Giraffe and many other distributed graphics databases.

teoh7z-1472822

Spark Streaming: abstractions

Spark Streaming has a micro batch architecture as follows:

  • treats the transmission as a series of data batches

  • New batches are created at regular time intervals.

  • the size of the time intervals is called batch break

  • the batch interval is usually between 500 ms and several seconds

opxh3o-6839897

The reduction value of each window is calculated incrementally.

Discretized sequence (DStream)

Discretized current O DStream is the basic abstraction provided by Spark Streaming. Represents a continuous flow of data, either the input data stream received from the source or the processed data stream generated by transforming the input stream. Internally, a DStream is represented by a continuous series of RDDs, which is Spark's abstraction of a distributed and immutable dataset (watch Spark Programming Guide for more details). Each RDD in a DStream contains data of a certain interval.

vqp083-1785373

  • RDD transformations are calculated by the Spark engine

  • DStream operations hide most of these details

  • Any operations applied in a DStream translate into operations in the underlying RDDs.

  • The reduction value of each window is calculated incrementally.

an6nl4-5175935

Spark Streaming: transmission context

It is the main entry point for Spark Streaming functionality. Provides methods used to create DStreams from various input sources. Streaming Spark can be created by providing a Spark Master URL and Application Name, or from an org.apache.spark.SparkConf configuration, o desde un org.apache.spark.SparkContext existente. The associated SparkContext can be accessed using context.sparkContext.

After creating and transforming DStreams, streaming computing can be started and stopped using context.start() Y, respectively. context.awaitTermination() allows current thread to wait for context termination by stop() or for an exception.

To run a SparkStreaming application, we need to define StreamingContext. SparkContext specializes for streaming applications.

The streaming context in Java can be defined as follows:

JavaStreamingContext ssc = new JavaStreamingContext(sparkConf, batchInterval);

where:

  • Maestro is a Spark cluster URL, Mesos o YARN; to run your code in local mode, use “local[K]"Where K> = 2 represents parallelism

  • app name is the name of your application

  • batch interval time interval (in seconds) of each batch

Once built, offer two types of operations:

Some examples are: map (), filter () y reduceByKey ()

    • stateful transformations: uses data from previous batches to calculate the results of the current batch. Include sliding windows, status monitoring over time, etc.

Note that a streaming context can be started only once and must be started after we configure all DStreams and output operations.

Basic data sources

The basic Spark Streaming data sources are listed below:

  • File streams: Used to read data from files on any file system that supports the HDFS API (namely, HDFS, S3, NFS, etc.), you can create a DStream like: 
    ... = streamingContext.fileStream<...>(directory);
  • Streams based on custom receivers: DStreams can be created with data streams received through custom receivers, extending the Receiver class 
    ... = streamingContext.queueStream(queueOfRDDs)
  • RDD queue as flow: To test a Spark Streaming application with test data, you can also create a DStream based on an RDD queue, using 
    ... = streamingContext.queueStream(queueOfRDDs)

Most transformations have the same syntax as applied to RDDs

Transformation

Sense

Map (func)

Returns a new DStream by passing each element of the source DStream through a func function.

flatMap (func)

Similar to the map, but each input element can be assigned to 0 or more output elements.

filter (func)

Return a new DStream by selecting only the records from the source DStream where func returns true.

Union (otherStream)

Returns a new DStream that contains the union of the elements of the source DStream and otherDStream.

join (another flow)

When two pair DStreams are called (K, V) Y (K, W), returns a new DStream of (K, (V, W)) pairs with all element pairs for each key.

Example: word count

SparkConf sparkConf = new SparkConf()
.setMaster("local[2]").setAppName("WordCount");
JavaStreamingContext ssc = ...
JavaReceiverInputDStream<String> lines = ssc.socketTextStream( ... );
JavaDStream<String> words = lines.flatMap(...);
JavaPairdStream<String, Integer> wordCounts = words
                                             .mapToPair(s -> new Tuple2<>(s, 1))
                                             .reduceByKey((i1, i2) -> i1 + i2);

wordCounts.print();

Spark Streaming: Window

The simplest window function is a window, that allows you to create a new DStream, calculado aplicando los parameters de ventana al antiguo DStream. you can use any of the dstream operations in the new stream, so you get all the flexibility you want.

Calculations in windows allow you to apply transformations over a sliding data window. Any window operation needs to specify two parameters:

  • window long
    • The duration of the window in seconds.
  • sliding break
    • The interval at which the window operation is performed in seconds.
    • these parameters must be multiples of the batch interval.

i8chij-9683770

window(windowLength, slideInterval)

Returns a new DStream that is calculated based on batches in window.

...
JavaStreamingContext ssc = ...
JavaReceiverInputDStream<String> lines = ...
JavaDStream<String> linesInWindow =
lines.window(WINDOW_SIZE, SLIDING_INTERVAL);
JavaPairdStream<String, Integer> wordCounts = linesInWindow.flatMap(SPLIT_LINE)
.mapToPair(s -> new Tuple2<>(s, 1))
.reduceByKey((i1, i2) -> i1 + i2);
  • reduceByWindow (func, InvFunc, windowLength, slideInterval)
    • Returns a new sequence of a single item, created by adding elements in the sequence during a sliding interval using func (which must be associative).
    • The reduction value of each window is calculated incrementally.
  • reduceByKeyAndWindow (func, InvFunc, windowLength, slideInterval)
    • When calling in a DStream of (K, V) Pairs, returns a new DStream of (K, V) pairs where the values for each key are added using the given reduction function func over lots in a sliding window.

To perform these transformations, we need to define a directory of checkpoints

Window-based: word count

...
JavaPairdStream<String, Integer> wordCountPairs = ssc.socketTextStream(...)
.flatMap(x -> Arrays.asList(SPACE.split(x)).iterator())
.mapToPair(s -> new Tuple2<>(s, 1));
JavaPairdStream<String, Integer> wordCounts = wordCountPairs
.reduceByKeyAndWindow((i1, i2) -> i1 + i2, WINDOW_SIZE, SLIDING_INTERVAL);
wordCounts.print();
wordCounts.foreachRDD(new SaveAsLocalFile());

a (more efficient) window based: word count

In a more efficient version, the reduction value of each window is calculated incrementally.

note that checkpoints must be enabled to use this operation.

...
ssc.checkpoint(LOCAL_CHECKPOINT_DIR);
...
JavaPairdStream<String, Integer> wordCounts = wordCountPairs.reduceByKeyAndWindow(
(i1, i2) -> i1 + i2,
(i1, i2) -> i1 - i2, WINDOW_SIZE, SLIDING_INTERVAL);

Spark Streaming: exit operations

Las operaciones de salida permiten que los datos de DStream se envíen a sistemas externos como una database o un sistema de archivos

Exit operation

Sense

Print()

Imprime los primeros diez elementos de cada lote de datos en un DStream en el node from the controller that runs the application.

saveAsTextFiles (prefix, [suffix])

Save the content of this DStream as text files. The filename in each batch interval is generated based on the prefix.

saveAsHadoopFiles (prefix, [suffix])

Save the contents of this DStream as Hadoop files.

saveAsObjectFiles (prefix, [suffix])

Save the content of this DStream as SequenceFiles of serialized Java objects.

foreachRDD (func)

Generic output operator that applies a function, func, to each RDD generated from the sequence.

Online References
• Spark Documentation
• Spark Documentation

Conclution

It should be clear that Spark Streaming presents a powerful way of writing streaming applications. Taking a batch job that you are already running and converting it to a streaming job with almost no code changes is simple and extremely useful from an engineering point of view if you need this job to interact closely with the rest of your processing application. data.

I recommend that you consult the following data engineering resources to improve your knowledge:

If you liked the article, leave a comment in the comment section below.

Related Posts:
  1. Streaming estructurado de Spark | Structured streaming with Kafka on Windows
  2. Spark Streaming in real time | Real-time data transmission with Apache Spark
  3. Essential books for beginners on Big Data, Hadoop y Apache Spark
  4. Spark Architecture | Apache Spark architecture for data engineers
Recent posts

Subscribe to our Newsletter

We will not send you SPAM mail. We hate it as much as you.

The data always to the Peak

Twitter Facebook-f Youtube Medium

Gaming

Brands

Business

Languages


Datapeaker